Project

General

Profile

Bug #11398

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

Florence sometimes hides other windows, which breaks tests

Added by bertagaz about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Test suite
Target version:
Start date:
05/05/2016
Due date:
% Done:

100%

Feature Branch:
test/11398-florence-hides-other-windows
Type of work:
Code
Blueprint:
Starter:
No
Affected tool:

Description

Two scenarios are failing from time to time because the Florence keyboard is on top of the window where the test suite expects to see results:

cucumber features/root_access_control.feature:25 # Scenario: If no administrative password is set in Tails Greeter the live user \
  should not be able to get administrative privileges through PolicyKit with the standard passwords.
cucumber features/evince.feature:24 # Scenario: I cannot view a PDF file stored in non-persistent /home/amnesia/.gnupg

In this case, another scenario is also failing:

cucumber features/encryption.feature:32 # Scenario: Symmetric encryption and decryption using OpenPGP Applet

This scenarios are using snapshots, so it might be that this one is taken a bit too soon, and the Florence keyboard hasn't really finished to initialize and is still opened on the desktop.

It happened recently in Jenkins runs test_Tails_ISO_feature-11123-new-mirror-pool #2, test_Tails_ISO_stable #196 and test_Tails_ISO_stable #243

See attached screenshots.


Related issues

Related to Tails - Bug #11464: "all notifications have disappeared" step is fragile when network is unplugged Resolved 05/23/2016
Blocks Tails - Bug #10497: wait_until_tor_is_working helper is fragile Resolved 11/06/2015

Associated revisions

Revision 6e0354cf (diff)
Added by bertagaz about 3 years ago

Mark scenarios fragile because of #11398.

Refs: #11398

Revision 138680c9 (diff)
Added by bertagaz about 3 years ago

Revert "Mark scenarios fragile because of #11398."

This reverts commit 6e0354cf86c84b7a48a13d4ef46fa9493523d27b.

Refs: #11398

Revision c924fb5d (diff)
Added by intrigeri about 3 years ago

Test suite: hide Florence keyboard window when it doesn't vanish by itself.

refs: #11398

Revision 06634d46
Added by anonym about 3 years ago

Merge remote-tracking branches 'origin/test/10376-fix_startup-page_roadmap-test-is-fragile' and 'origin/test/11401-notification-wait-is-fragile' into testing

Fix-committed: #10376, #10497, #11398, #11401, #11464

History

#1 Updated by bertagaz about 3 years ago

  • Feature Branch set to test/11398-florence-hides-other-windows

#2 Updated by intrigeri about 3 years ago

  • Subject changed from Florence sometimes hides other windows to Florence sometimes hides other windows, which breaks tests

I've seen it happen again in the last few days, e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10497-tor-bootstrap-is-fragile/17/. Given that this can basically affect all tests, we can't mark them all as fragile, so IMO this will need to be prioritized quite high, and may block #11355.

This problem might boil down to "hide-on-start=true sometimes fails", i.e. a bug in Florence itself (not found any reported in the Debian or upstream BTS). But I bet this will be hard to debug, and we have other reasons (#8312, that blocks #8309, that is most plausible way of fixing #10576 some day) to get rid of Florence, so IMO a test suite -specific workaround would be good enough, i.e.: if Florence is still visible after a short while, hide it ourselves.

Thoughts?

#3 Updated by bertagaz about 3 years ago

I've seen that too while compiling Jenkins stats. I think I mentioned it in the email I've sent on the @-ci list. I'm wondering if this happen on scenarios that are using snapshots, and when this one are made a bit too fast, while Florence is still visible, then it is interfering.

#4 Updated by intrigeri about 3 years ago

I've seen that too while compiling Jenkins stats.

... and indeed, you created this ticket :)

I'm wondering if this happen on scenarios that are using snapshots, and when this one are made a bit too fast, while Florence is still visible, then it is interfering.

Regardless of when the snapshot is taken and restored, at some point Florence's code to hide itself should run, no? It looks like the logical consequence of what you're suggesting is that this code may not work if run after restoring a snapshot. I don't see why this would be the case, but whatever, I don't mind following your lead :) What kind of solution would you suggest, following up on this tentative explanation?

#5 Updated by intrigeri about 3 years ago

  • Priority changed from Normal to Elevated

FTR, all recent test failures I've seen while running tests on 4 branches intensively were caused either by this problem, or by #11401 => bumping priority.

#6 Updated by intrigeri about 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Target version set to Tails_2.4

Giving it a try.

#7 Updated by intrigeri about 3 years ago

  • % Done changed from 0 to 10

I've pushed something, will merge it into #10497 and subtasks so I see how it works on Jenkins, on the N branches I'm monitoring closely there these days.

I'd welcome an initial very quick code review, to benefit from anonym's expertise, but don't feel pressured: this can totally wait post-2.4~rc1, and perhaps even post-2.4.

#8 Updated by intrigeri about 3 years ago

  • Blocks Bug #10497: wait_until_tor_is_working helper is fragile added

#9 Updated by intrigeri about 3 years ago

  • Related to Bug #11464: "all notifications have disappeared" step is fragile when network is unplugged added

#10 Updated by intrigeri about 3 years ago

  • Assignee changed from intrigeri to anonym
  • % Done changed from 10 to 50
  • QA Check set to Ready for QA

Seems pretty robust now!

#11 Updated by anonym about 3 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

#12 Updated by anonym about 3 years ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF