Project

General

Profile

Bug #10381

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

The "I open the address" steps are fragile

Added by kytv almost 4 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
Start date:
10/15/2015
Due date:
% Done:

100%

Feature Branch:
test/10381-fix-I-open-the-address-test-is-fragile
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

anonym wrote on #10288:

In the error screenshot I can see that the Tor Browser is showing the "The connection has timed out" page. Something must be wrong with our recent improvements in the 'I open the address ...' step since it accepted this page. Possibly the existing condition isn't enough, but we should also check that we do not see this particular error page, or perhaps any error page.


Related issues

Related to Tails - Feature #10379: Check that we do not see any error pages in the "I open the address" step. Rejected 10/15/2015
Related to Tails - Bug #11592: Step "[...] has loaded in the Tor Browser" is fragile Resolved 07/22/2016
Blocked by Tails - Feature #9521: Use the chutney Tor network simulator in our test suite Resolved 04/15/2016
Blocked by Tails - Bug #10497: wait_until_tor_is_working helper is fragile Resolved 11/06/2015
Blocks Tails - Bug #10442: Totem "Watching a WebM video over HTTPS" test never passes on Jenkins Resolved 10/28/2015

Associated revisions

Revision f4304c83 (diff)
Added by anonym over 2 years ago

Test suite: make robust_notification_wait() ensure the applet is closed.

In robust_notification_wait() when we close the notification applet,
other windows may change position, creating a racy situation for any
immediately following action aimed at one such window.

Refs: #10381

Revision d74523ec (diff)
Added by intrigeri over 2 years ago

Unmark as fragile one more test case that was disabled due to "11a7a9a4219" being fragile (refs: #10381).

It was initially disabled in 5d797feab0a790cef4f0b502a724d612dee6462c.

Revision 397f6bd7
Added by anonym over 2 years ago

Merge remote-tracking branch 'origin/test/10381-fix-I-open-the-address-test-is-fragile' into stable

Fix-committed: #10381

History

#1 Updated by kytv almost 4 years ago

  • Blocks Bug #10288: Fix newly identified issues to make our test suite more robust and faster added

#2 Updated by intrigeri almost 4 years ago

  • Blocks deleted (Bug #10288: Fix newly identified issues to make our test suite more robust and faster)

#3 Updated by intrigeri almost 4 years ago

  • Parent task set to #10288

#4 Updated by intrigeri almost 4 years ago

  • Assignee set to anonym

#6 Updated by anonym almost 4 years ago

  • Assignee changed from anonym to kytv
  • Target version changed from Tails_1.7 to Tails_1.8

#7 Updated by kytv almost 4 years ago

  • Feature Branch set to kytv:test/5d797fe-fix-I-open-the-address-test-is-fragile

#8 Updated by kytv almost 4 years ago

  • Feature Branch changed from kytv:test/5d797fe-fix-I-open-the-address-test-is-fragile to kytv:test/10381-fix-I-open-the-address-test-is-fragile

#9 Updated by anonym almost 4 years ago

  • Feature Branch changed from kytv:test/10381-fix-I-open-the-address-test-is-fragile to test/10381-fix-I-open-the-address-test-is-fragile

Branch imported to our Git, scenarios marked as @fragile in devel.

#11 Updated by intrigeri almost 4 years ago

  • Related to Feature #10379: Check that we do not see any error pages in the "I open the address" step. added

#12 Updated by anonym almost 4 years ago

  • Assignee changed from kytv to bertagaz

There might be something smart to do that will fix all of {#10442, #10381, #10376} at the same time, and increase browser page loading throughout the test suite.

#13 Updated by anonym almost 4 years ago

  • Assignee changed from bertagaz to anonym

#14 Updated by intrigeri almost 4 years ago

  • Target version changed from Tails_1.8 to Tails_2.0

(We're going to mark as fragile all tests that depend on Tor to have bootstrapped for the moment => not so urgent.)

#15 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.0 to Tails_2.2

#16 Updated by anonym over 3 years ago

  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.2 to Tails_2.4

#17 Updated by anonym over 3 years ago

  • Priority changed from Elevated to Normal
  • Target version deleted (Tails_2.4)

I'm gonna gamble and focus on Chutney (#9521) hoping it will fix this issue.

#18 Updated by anonym over 3 years ago

  • Blocked by Feature #9521: Use the chutney Tor network simulator in our test suite added

#19 Updated by intrigeri over 3 years ago

  • Assignee changed from anonym to intrigeri
  • Target version set to Tails_2.4

Will give it a try.

#21 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10497: wait_until_tor_is_working helper is fragile added

#22 Updated by intrigeri over 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA

(Or should it be for bertagaz?)

Seems to be fixed by chutney.

#23 Updated by intrigeri over 3 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

Too many false positives, back onto my plate. I've marked more tests as fragile, we'll see.

#24 Updated by intrigeri over 3 years ago

I've just seen it fail again (video attached):

call returned: [0, "Creating logfile at /tmp/dogtail-amnesia/logs/tmp.IAB6SjAMOS_20160524-105252_debug ...\nClicking on [label | Unsafe Browser]\nMouse button 1 click at (309,380)\n", ""]
calling as root: rm -f '/tmp/tmp.IAB6SjAMOS'
call returned: [0, "", ""]
    When I successfully start the Unsafe Browser                                                   # features/step_definitions/browser.rb:18
calling as root: . /usr/local/lib/tails-shell-library/tor-browser.sh && echo ${TBB_INSTALL}/firefox
call returned: [0, "/usr/local/lib/tor-browser/firefox\n", ""]
    And I open Tails homepage in the Unsafe Browser                                                # features/step_definitions/browser.rb:219
      FindFailed: can not find BrowserAddressBar.png on the screen.
      Line ?, in File ? (RuntimeError)
      ./features/step_definitions/browser.rb:89:in `/^I open the address "([^"]*)" in the (.*)$/'
      ./features/step_definitions/browser.rb:220:in `/^I open Tails homepage in the (.+)$/'
      features/tor_enforcement.feature:23:in `And I open Tails homepage in the Unsafe Browser'
    And Tails homepage loads in the Unsafe Browser                                                 # features/step_definitions/browser.rb:228

It seems to me that the click on the new tab button might be lost due to the window moving down when the notification bar closes. If my guess is correct, then think that the solution is easy: in robust_notification_wait, we should wait for the notification bar to actually have disappeared, before returning (and this should be tracked by a different ticket).

#25 Updated by intrigeri over 3 years ago

  • Assignee deleted (intrigeri)
  • Target version deleted (Tails_2.4)
  • % Done changed from 50 to 10

I'm giving up on this one (I was merely looking for very low hanging fruits during this cycle, and I'll let someone else fix the problem, now that it's been diagnosed).

#26 Updated by intrigeri over 3 years ago

  • Blocks Bug #10442: Totem "Watching a WebM video over HTTPS" test never passes on Jenkins added

#27 Updated by bertagaz about 3 years ago

  • Related to Bug #11592: Step "[...] has loaded in the Tor Browser" is fragile added

#28 Updated by intrigeri about 3 years ago

  • Feature Branch changed from test/10381-fix-I-open-the-address-test-is-fragile to wip/test/10381-fix-I-open-the-address-test-is-fragile

#29 Updated by anonym almost 3 years ago

  • Assignee set to anonym
  • Target version set to Tails 2.10

#30 Updated by anonym over 2 years ago

  • % Done changed from 10 to 20
  • Feature Branch changed from wip/test/10381-fix-I-open-the-address-test-is-fragile to test/10381-fix-I-open-the-address-test-is-fragile

intrigeri wrote:

I've just seen it fail again (video attached):

[...]

It seems to me that the click on the new tab button might be lost due to the window moving down when the notification bar closes. If my guess is correct, then think that the solution is easy: in robust_notification_wait, we should wait for the notification bar to actually have disappeared, before returning (and this should be tracked by a different ticket).

Makes a lot of sense. Implemented in the feature branch.

#31 Updated by anonym over 2 years ago

  • Assignee changed from anonym to intrigeri
  • % Done changed from 20 to 50
  • QA Check changed from Dev Needed to Ready for QA

Beyond what you sugested, intrigeri, I also made this step retry not only for the Tor Browser, but for all browsers. If this step is still fragile, it's due to reasons not yet clear to me, so I think this is all I can do for now. We have other similar tickets open (e.g. #11592) which probably captures these other ways we can have failures in this area.

Please review'n'merge into testing and devel!

#32 Updated by anonym over 2 years ago

(Feel free to postpone this ticket -- we do not need it for Tails 2.10)

#33 Updated by intrigeri over 2 years ago

I might handle this tomorrow, we'll see.

#34 Updated by intrigeri over 2 years ago

  • Target version changed from Tails 2.10 to Tails_2.11

#35 Updated by intrigeri over 2 years ago

  • Assignee changed from intrigeri to anonym
  • % Done changed from 50 to 70

Code review passes. Unmarked one more test as fragile, that I think you had forgotten. Works for me locally. So if Jenkins is happy with that change, please merge into stable and devel :)

#36 Updated by anonym over 2 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 70 to 100
  • QA Check changed from Ready for QA to Pass

intrigeri wrote:

So if Jenkins is happy with that change

AFAICT it is...

please merge into stable and devel :)

... so merged!

#37 Updated by intrigeri over 2 years ago

Great!

#38 Updated by anonym over 2 years ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF