Project

General

Profile

Bug #11592

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

Step "[...] has loaded in the Tor Browser" is fragile

Added by bertagaz about 3 years ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Test suite
Target version:
Start date:
07/22/2016
Due date:
% Done:

100%

Feature Branch:
test/11592-load-page-in-torbrowser-is-fragile+force-all-tests
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Browser

Description

We noticed while referencing failures Jenkins that this step sometimes fails (see #11087#note-9). It seems that the Tor Browser has troubles loading the startup page, and that's probably due to the Tor bootstrapping issues that the Chutney integration did not completely solved. A workaround could be to add a retry logic here.

01_22_37_The_Tor_Browser_directory_is_usable.mkv (375 KB) bertagaz, 07/22/2016 05:19 AM

bugfix-10494-retry-htpdate_9.log View (740 KB) bertagaz, 07/22/2016 05:20 AM


Related issues

Related to Tails - Bug #10381: The "I open the address" steps are fragile Resolved 10/15/2015
Related to Tails - Bug #17007: JavaScript sometimes blocked on Tor Browser first start ⇒ "Watching a WebM video over HTTPS" and "Playing an Ogg audio track" scenarios are fragile: blocked by NoScript click-to-play Confirmed
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision be35933d (diff)
Added by bertagaz about 3 years ago

Mark the "[...] has loaded in the Tor Browser" step as fragile.

Refs: #11592

Revision 3b24d296 (diff)
Added by bertagaz about 3 years ago

Revert "Mark the "[...] has loaded in the Tor Browser" step as fragile."

This reverts commit be35933d4d5d27d044ce8aad53f1bb4b31eec01c.

Refs: #11592

Revision c557e759 (diff)
Added by intrigeri about 2 months ago

Test suite: make the "[…] has loaded in the Tor Browser" step more robust (refs: #11592)

Revision b69a4dc6 (diff)
Added by intrigeri about 2 months ago

Test suite: make the "[…] has loaded in the Tor Browser" step more robust (refs: #11592)

For some reason, sometimes the JavaScript that turns "Trying a testing version
of Tails" into "Tails" does not do its job. Waiting more does not help.
This seems to be a minor bug in Tor Browser or our website.

This is the primary cause of fragility of our test suite these days.

Given the benefit of getting more useful feedback from CI greatly outweighs
measuring how often that bug happens, and the scenarios that use this steps are
primarily about something else anyway, let's support both pre-JS and
post-JS titles.

Revision 3e032ce4
Added by segfault about 2 months ago

Merge branch 'test/11592-load-page-in-torbrowser-is-fragile+force-all-tests' into devel (Closes: #11592)

History

#1 Updated by bertagaz about 3 years ago

  • Related to Bug #10381: The "I open the address" steps are fragile added

#2 Updated by bertagaz about 3 years ago

  • Feature Branch set to test/11592-load-page-in-torbrowser-is-fragile

Marking related scenarios as fragile.

#3 Updated by intrigeri about 3 years ago

  • Target version deleted (Tails_2.6)

#4 Updated by intrigeri about 3 years ago

  • Feature Branch changed from test/11592-load-page-in-torbrowser-is-fragile to wip/test/11592-load-page-in-torbrowser-is-fragile

#5 Updated by intrigeri about 3 years ago

  • Affected tool set to Browser

#6 Updated by intrigeri over 2 years ago

  • Priority changed from Normal to Elevated

This affects most Tor Browser scenarios, which means we have to run them locally during the release process. Hence bumping priority.

#7 Updated by intrigeri over 1 year ago

Tor Browser 8 hides the stop/reload button. On the branch for #15023 I'm applying a hack to display it in Tails in order to avoid breaking our test suite. Whenever we work on this ticket, let's take this opportunity to drop the hack and stop relying on the Reload button.

#8 Updated by intrigeri about 2 months ago

While analyzing full test suite failures on the stable branch today, I noticed that this step is the primary reason for failures there, so I looked closer and here's what I found out:

  • I've not seen cases when the startup page did not load at all. That is, presumably, we're not in these situations where some retrying / retry_tor would help.
  • Sometimes the startup page does load but the piece of JS that sets document.title = "Tails" does not run, and then the browser window's title is still "Tails - Trying a testing version of Tails - Tor Browser", which makes the step fail. I guess this can be explained by:
    • a network issue preventing the JS from being loaded → unlikely since all other resources are nicely loaded
    • a bug in Tor Browser
    • the VM being too busy to execute the JS fast enough to satisfy the expectations set by this step
  • The Journal was not saved for any of these failures, which makes it hard to analyze what's going on. My understanding of the code is that it's because the remote shell is not "up", i.e. it's not replying within 3 seconds, which can indicate the VM is too busy or has crashed.
  • In this step, we do try_for(60) { @torbrowser.child?(expected_title, roleName: 'frame') }, which suggests we're inclined to retry the search if it fails. But in the debug log I see that the corresponding dogtail code is run only once, never returns, and then try_for times out after 60 seconds. I suspect that dogtail keeps the VM busy, which would explain the lack of a Journal artifact, but also possibly the fact the JS did not run. I don't know why dogtail is taking so long but I suspect the browser, while loading the page (and then changing its title), is creating/deleting/updating tons of a11y objects, which dogtail might have a hard time keeping track of, therefore causing either huge CPU usage or consuming enough memory that the remote shell and the browser are stuck.

So I'm tempted to add a stupid sleep before this try_for, to give the browser some time to run the JS before we overload the VM with a dogtail search. This might or might not work, but at least it should help validate/invalidate my hunch.

Thoughts?

#9 Updated by intrigeri about 2 months ago

  • Status changed from Confirmed to In Progress

#10 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Confirmed
  • Feature Branch changed from wip/test/11592-load-page-in-torbrowser-is-fragile to test/11592-load-page-in-torbrowser-is-fragile+force-all-tests

intrigeri wrote:

While analyzing full test suite failures on the stable branch today, I noticed that this step is the primary reason for failures there

Same on the devel branch.

So I'm tempted to add a stupid sleep before this try_for, to give the browser some time to run the JS before we overload the VM with a dogtail search. This might or might not work, but at least it should help validate/invalidate my hunch.

Trying this on the branch, we'll see how it goes on Jenkins.

#11 Updated by intrigeri about 2 months ago

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Target version set to Tails_3.16

(All these metadata changes are for evaluating the aforementioned workaround and if it works, get it merged.)

#12 Updated by intrigeri about 2 months ago

#13 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Confirmed
  • Assignee deleted (intrigeri)
  • Target version deleted (Tails_3.16)
  • Feature Branch deleted (test/11592-load-page-in-torbrowser-is-fragile+force-all-tests)

intrigeri wrote:

intrigeri wrote:

So I'm tempted to add a stupid sleep before this try_for, to give the browser some time to run the JS before we overload the VM with a dogtail search. This might or might not work, but at least it should help validate/invalidate my hunch.

Trying this on the branch, we'll see how it goes on Jenkins.

Sleeping 5s does not seem to help at all.

#14 Updated by intrigeri about 2 months ago

  • Status changed from Confirmed to In Progress

#15 Updated by intrigeri about 2 months ago

  • Assignee set to intrigeri
  • Target version set to Tails_4.0
  • Feature Branch set to test/11592-load-page-in-torbrowser-is-fragile+force-all-tests

Giving it another try in the hope this allows us to keep running the full test suite by default on the testing & devel branches.

I've seen this step pass on this branch locally with the two page titles that one can see in practice, which was my goal. Let's see how it goes on Jenkins :)

#16 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)

Same on 5 runs on Jenkins, which is unheard of: recently, almost every test suite run there triggers this bug. So it looks like my workaround works!

#17 Updated by segfault about 2 months ago

LGTM

#18 Updated by segfault about 2 months ago

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

#19 Updated by intrigeri about 1 month ago

  • Related to Bug #17007: JavaScript sometimes blocked on Tor Browser first start ⇒ "Watching a WebM video over HTTPS" and "Playing an Ogg audio track" scenarios are fragile: blocked by NoScript click-to-play added

Also available in: Atom PDF