Project

General

Profile

Bug #17056

Feature #16356: Upgrade to Tor Browser 9.0 (based on Firefox 68)

Make test suite robust with Tor Browser 9.0

Added by intrigeri about 1 month ago. Updated 13 days ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Test suite
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Browser

Description

I see lots of failures which seem to be clear test suite robustness regressions triggered by the upgrade to Tor Browser 9.
This makes it hard to tell what else we need to do on #16356, and whether TB 9 fixes #17007.


Related issues

Related to Tails - Bug #17121: Tor Browser 9 sometimes won't load new URLs Resolved
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision 75524b70 (diff)
Added by intrigeri about 1 month ago

Test suite: don't restart Tor Browser between saving/opening (refs: #17056)

The "I close the Tor Browser" step is now broken when more than one tab is
opened, because Tor Browser 9.0 repaired the behavior controlled by the
browser.tabs.warnOnClose pref. In Tails 3.16 this pref was enabled but it was
kindy buggy: sometimes, no warning was displayed. With Tor Browser 9.0, I see
the "Quit and close tabs?" dialog pop up (as intended) more often, which totally
breaks this close+start sequence of steps, that relied on
browser.tabs.warnOnClose to be broken.

These extra steps were added in commit 47df1d7fbde1a66402781df46d5b6491534f0cba
with the following explanation:

To make this scenario more robust we […] restart the browser between save/open
in order to prevent Dogtail e.g. finding the old page that we just saved
instead of the saved version we just opened.

My understanding is that the only case when Dogtail could possibly find the old
page, after we've opened a new tab where we're going to open the new page, is
when #12191 (Dogtail's showingOnly option is not always honored) happens.
So let's explicitly look for currently displayed frames, so that even when
that bug happens, Dogtail should not find the old page.

Revision d2f19c45 (diff)
Added by intrigeri about 1 month ago

Test suite: increase timeout (refs: #17056)

I really don't understand what makes the "I open the address" step
so fragile on our shared Jenkins while it works just fine locally.
Let's try to wait a bit more.

Revision c3335818 (diff)
Added by intrigeri 29 days ago

Test suite: yet another attempt at making the "I open the address" step more robust (refs: #17056)

This one is mostly hopeless but who knows. I'm curious whether waiting a bit,
before we send Dogtail in berserk mode on the Firefox process, will help Tor
Browser do its job first.

Revision d353d033 (diff)
Added by intrigeri 15 days ago

Test suite: use "Paste & Go" instead of pasting and then pressing Enter (refs: #17056)

On our slower isotesters, pasting the URL does not automatically open the
Visit/Search menu, and the Enter key press is lost. So let's try to skip the
Visit/Search menu, avoid having to coordinate with it before pressing Enter, and
avoid the case when pressing Enter does not work: instead, let's let Tor Browser
itself deal with the whole operation.

Revision 4a0c9814 (diff)
Added by intrigeri 15 days ago

Test suite: use "Paste & Go" instead of pasting and then pressing Enter (refs: #17056)

On our slower isotesters, pasting the URL does not automatically open the
Visit/Search menu, and the Enter key press is lost. So let's try to skip the
Visit/Search menu, avoid having to coordinate with it before pressing Enter, and
avoid the case when pressing Enter does not work: instead, let's let Tor Browser
itself deal with the whole operation.

Revision 42cce01e (diff)
Added by intrigeri 15 days ago

Test suite: use "Paste & Go" instead of pasting and then pressing Enter (refs: #17056)

On our slower isotesters, pasting the URL does not automatically open the
Visit/Search menu, and the Enter key press is lost. So let's try to skip the
Visit/Search menu, avoid having to coordinate with it before pressing Enter, and
avoid the case when pressing Enter does not work: instead, let's let Tor Browser
itself deal with the whole operation.

Revision 62145184 (diff)
Added by intrigeri 15 days ago

Test suite: make usage of "Paste & Go" more robust (refs: #17056).

Revision 68e35ff3 (diff)
Added by intrigeri 15 days ago

Test suite: bypass Tor Browser GUI issues and open URLs via the command line (refs: #17056)

On our slower isotesters, pasting the URL does not automatically open the
Visit/Search menu, and the Enter key press is lost. I've also tried to use
"Paste & Go" and to type the URL instead of pasting in; none of those helped.

So let's have something that works, even if arguably it's not in the spirit of
BDD. OTOH, it exercises opening URLs in the same way as they would if the user
clicked on a link from another application, which is useful too.

Revision 6f715312 (diff)
Added by intrigeri 14 days ago

Test suite: fix opening URLs in the Unsafe Browser (refs: #17056)

Fixup against 68e35ff31517225aa7edb8af77c9526f08508949.

Revision dfeaa6e9 (diff)
Added by anonym 9 days ago

Test suite: revert recent hacks for the "I open ..." step.

It turns out these hacks (introduced for refs: #17056) worked around a
real bug, not a test suite issue, so we are gonna fix the real bug
instead (refs: #17121).

History

#1 Updated by intrigeri about 1 month ago

#2 Updated by intrigeri about 1 month ago

  • Status changed from Confirmed to In Progress

#4 Updated by intrigeri about 1 month ago

  • Blocks Bug #17007: JavaScript sometimes blocked on Tor Browser first start ⇒ "Watching a WebM video over HTTPS" and "Playing an Ogg audio track" scenarios are fragile: blocked by NoScript click-to-play added

#5 Updated by intrigeri about 1 month ago

I had some hope that 465900996ee0a74a57eb891ee553a3c4b79721ef and 5066426db1f6c30bffc5555f60ea25d013023814 would help, but unfortunately they don't: the last test runs on Jenkins show that opening an URL in a new tab ("I open the address") is still extremely fragile.

#6 Updated by intrigeri about 1 month ago

intrigeri wrote:

the last test runs on Jenkins show that opening an URL in a new tab ("I open the address") is still extremely fragile.

… while it always works fine on my local Jenkins. This suggests a race condition that we may win or lose, depending on the particularities of the isotester (most likely: resources, performance, workload).

#7 Updated by intrigeri 25 days ago

  • Assignee deleted (intrigeri)

I won't be able to work on this before Saturday.

#8 Updated by intrigeri 18 days ago

  • Assignee set to intrigeri

#9 Updated by intrigeri 15 days ago

  • Blocks deleted (Bug #17007: JavaScript sometimes blocked on Tor Browser first start ⇒ "Watching a WebM video over HTTPS" and "Playing an Ogg audio track" scenarios are fragile: blocked by NoScript click-to-play)

#10 Updated by intrigeri 15 days ago

the last test runs on Jenkins show that opening an URL in a new tab ("I open the address") is still extremely fragile.

… while it always works fine on my local Jenkins. This suggests a race condition that we may win or lose, depending on the particularities of the isotester (most likely: resources, performance, workload).

My latest batch of test suite runs seem to confirm my initial guess:

  • I never see this problem on my fastest local VMs (workerN.ant01).
  • I have seen this problem a couple times on my slower local VM (isotester1.sib).
  • This problem happens a bunch of times during every test suite run on lizard.

When this problem happens, there's a weird graphical glitch earlier, when we load the /home/testing page: there's some white space between the URL bar and the Tails logo and at the bottom of the page. Maybe that's just the new letterboxing feature and it's fully unrelated.

Next thing I want to try: instead of pasting the URL and then pressing Enter, I'll try opening the address bar's right-click contextual menu and then clicking "Paste & Go".

#11 Updated by intrigeri 15 days ago

  • Feature Branch set to test/17056-tor-browser-9.0+force-all-tests

I'll use this topic branch for experiments, and will feel free to rewrite its history. Once happy I'll merge into the branch for #16356.

#12 Updated by intrigeri 15 days ago

  • Feature Branch deleted (test/17056-tor-browser-9.0+force-all-tests)

Merged something that seems robust enough locally into the branch for #16356. It only fixes the problem for the Tor Browser. OTOH I don't know yet if the Unsafe Browser is affected — we'll see once it works well enough so our test suite can exercise more of it.

#13 Updated by intrigeri 14 days ago

intrigeri wrote:

Merged something that seems robust enough locally into the branch for #16356. It only fixes the problem for the Tor Browser.

Confirmed, this makes the Tor Browser scenarios pass on Jenkins! \o/

OTOH I don't know yet if the Unsafe Browser is affected

Hmm, looks like I broke some Unsafe Browser tests while fixing the Tor Browser ones. Will investigate & fix.

#14 Updated by intrigeri 14 days ago

Hmm, looks like I broke some Unsafe Browser tests while fixing the Tor Browser ones. Will investigate & fix.

6f71531208bac5ab92a9dc83bb3d9c4af4b383d2 might fix that.

#15 Updated by intrigeri 13 days ago

  • Related to Bug #17121: Tor Browser 9 sometimes won't load new URLs added

#16 Updated by intrigeri 13 days ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)

It's now robust, thanks to the workaround for #17121 that I've added (removal of this hack is tracked on that other ticket).

Of course, the Unsafe Browser tests fail, but that's because it was not fully ported to Tor Browser 9 yet, which is tracked on #17055.

Also available in: Atom PDF