Project

General

Profile

Bug #15006

Test suite regression: "I successfully start the Unsafe Browser" fails since Tails 3.3

Added by intrigeri almost 2 years ago. Updated 5 months ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
-
Start date:
11/29/2017
Due date:
% Done:

60%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Unsafe Browser

Description

I guess it's caused by the new pure HTML Unsafe Browser home page. I see that on the stable branch, did not look elsewhere.


Related issues

Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision fe0e759d (diff)
Added by anonym almost 2 years ago

Test suite: bump the Unsafe Browser's start page image.

Will-fix: #15006

History

#1 Updated by intrigeri almost 2 years ago

  • Blocks Feature #13240: Core work 2017Q4: Test suite maintenance added

#2 Updated by intrigeri almost 2 years ago

  • Blocks Feature #14817: Test suite improvements from buster sprint 1 added

#3 Updated by anonym almost 2 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from anonym to intrigeri
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA

Bumped some images in stable:

b749e0441c Test suite: bump TailsHomepage.png.
fe0e759dc0 Test suite: bump the Unsafe Browser's start page image.

#4 Updated by intrigeri almost 2 years ago

  • % Done changed from 50 to 60

LGTM, will now test.

#5 Updated by intrigeri almost 2 years ago

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Ready for QA to Dev Needed

Works in my tests but "the Unsafe Browser works in all supported languages" is still broken if the locales sample includes a language for which we have a translated homepage: we used to look for a picture that didn't vary per-language; since we don't display that picture anymore you simplified things in 38662c3219333d4f2dea5d2936749daa133939f9, which makes sense, but now our test suite is essentially trying to verify that we see the English homepage in all languages, which of fails when the locales sample includes a language for which we have a translated homepage. As per our experiment (try to do such work in a less fine-grained way) I'm not filing another dedicated ticket about it, let's generalize this one instead :)

#6 Updated by intrigeri almost 2 years ago

  • Blocks deleted (Feature #14817: Test suite improvements from buster sprint 1)

#7 Updated by intrigeri almost 2 years ago

Also, it seems your refactoring in 38662c3219333d4f2dea5d2936749daa133939f9 had a nasty side-effect that introduces a regression in "Upgrading Tails with Tails Upgrader through an incremental upgrade": if the Unsafe Browser takes more than 10 seconds to start (which happens in the few recent runs on Jenkins I've seen) then we fail, while previously I think we were waiting up to 6 minutes.

#8 Updated by intrigeri almost 2 years ago

I still a strange failure here on an ISO built from the branch for #14819 at 6c23dc58e241abd46efba7f861baa1b4fdf2e811. I see "Unsafe Browser failed to launch in the following locale(s): en_US.utf8" but the Unsafe Browser did start in English; the test suite got confused, did not even start it in the 2nd language to be tested, and then successfully started it in the 3rd one. I'm attaching the video and here's the debug output:

01:43:11.703914579: calling as root: test -d '/usr/lib/locale/zh_CN.utf8'
01:43:11.824896168: call returned: [0, "", ""]
01:43:11.825769087: calling as root: pidof -x -o '%PPID' gnome-terminal-server
01:43:11.985597426: call returned: [1, "", ""]
01:43:12.024865880: calling as amnesia: xdotool key Super
01:43:12.218533316: call returned: [0, "", ""]
01:43:13.429144086: [log]  TYPE "commandline" 
01:43:13.497129847: [log] ( Ctrl )  TYPE "#ENTER." 
01:43:18.520386107: [log]  TYPE "LANG=en_US.utf8 LC_ALL=en_US.utf8 sudo unsafe-browser#ENTER." 
01:43:20.378738335: [log]  TYPE "#RIGHT.#ENTER." 
01:43:30.932402796: calling as root: pidof -x -o '%PPID' gnome-terminal-server
01:43:31.387661570: call returned: [0, "12272\n", ""]
01:44:05.120614434: Found fuzzy candidate picture for GnomeTerminalWindow.png with similarity 0.4
01:44:05.875618000: [log] CLICK on L(894,130)@S(0)[0,0 1024x768]
01:44:38.765072649: [log]  TYPE "LANG=it_IT.utf8 LC_ALL=it_IT.utf8 sudo unsafe-browser#ENTER." 
01:44:51.774493042: Found fuzzy candidate picture for GnomeQuestionDialogIcon.png with similarity 0.4
01:44:52.014205672: [log]  TYPE "#RIGHT.#ENTER." 
01:44:52.015108795: calling as root: . /usr/local/lib/tails-shell-library/tor-browser.sh && echo ${TBB_INSTALL}/firefox
01:44:52.024615452: [log] ( Ctrl )  TYPE "q" 
01:44:52.304498606: call returned: [0, "/usr/local/lib/tor-browser/firefox\n", ""]
01:44:52.304714124: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:44:52.515435092: call returned: [0, "", ""]
01:44:53.515771341: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:44:53.617212895: call returned: [0, "", ""]
01:44:54.617553746: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:44:54.746847177: call returned: [0, "", ""]
01:44:55.747130393: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:44:55.874164049: call returned: [0, "", ""]
01:44:56.874496324: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:44:57.009500894: call returned: [1, "", ""]
01:44:57.010509348: calling as root: pidof -x -o '%PPID' gnome-terminal-server
01:44:57.149766601: call returned: [0, "12272\n", ""]
01:44:57.995066829: [log] CLICK on L(226,585)@S(0)[0,0 1024x768]
01:45:02.307530592: [log]  TYPE "LANG=ar_EG.utf8 LC_ALL=ar_EG.utf8 sudo unsafe-browser#ENTER." 
01:45:02.390330168: [log]  TYPE "#LEFT.#ENTER." 
01:45:07.597363046: [log] ( Ctrl )  TYPE "q" 
01:45:07.597754992: calling as root: . /usr/local/lib/tails-shell-library/tor-browser.sh && echo ${TBB_INSTALL}/firefox
01:45:07.890353732: call returned: [0, "/usr/local/lib/tor-browser/firefox\n", ""]
01:45:07.890586771: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:45:08.148060863: call returned: [0, "", ""]
01:45:09.148333387: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:45:09.275159227: call returned: [0, "", ""]
01:45:10.275515549: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:45:10.382848078: call returned: [0, "", ""]
01:45:11.383199829: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:45:11.510450811: call returned: [0, "", ""]
01:45:12.510821804: calling as root: test -d '/var/lib/unsafe-browser/chroot'
01:45:12.624946123: call returned: [1, "", ""]
    Then the Unsafe Browser works in all supported languages                       # features/step_definitions/unsafe_browser.rb:31
      Unsafe Browser failed to launch in the following locale(s): en_US.utf8.
      <false> is not true. (Test::Unit::AssertionFailedError)
      features/localization.feature:18:in `Then the Unsafe Browser works in all supported languages'

#9 Updated by intrigeri almost 2 years ago

  • Blocks deleted (Feature #13240: Core work 2017Q4: Test suite maintenance)

#10 Updated by intrigeri almost 2 years ago

#11 Updated by intrigeri almost 2 years ago

I've seen this fail occasionally even with English locales. Attaching the screenshot + Sikuli candidate.

#12 Updated by anonym almost 2 years ago

  • Target version changed from Tails_3.5 to Tails_3.6

#13 Updated by bertagaz over 1 year ago

  • Target version changed from Tails_3.6 to Tails_3.7

#14 Updated by intrigeri over 1 year ago

  • Assignee deleted (anonym)

#15 Updated by intrigeri over 1 year ago

  • Target version deleted (Tails_3.7)

#16 Updated by intrigeri over 1 year ago

intrigeri wrote:

Also, it seems your refactoring in 38662c3219333d4f2dea5d2936749daa133939f9 had a nasty side-effect that introduces a regression in "Upgrading Tails with Tails Upgrader through an incremental upgrade": if the Unsafe Browser takes more than 10 seconds to start (which happens in the few recent runs on Jenkins I've seen) then we fail, while previously I think we were waiting up to 6 minutes.

FTR ae21d2665d4d4da8fabfe2c2676e99d1b2d9d1f5 bumped this timeout to 30 seconds but that's not enough with Firefox 60 on lizard when the box is running tests at full capacity. Bumping to 60 seconds.

#17 Updated by intrigeri 8 months ago

  • Status changed from In Progress to Confirmed

#18 Updated by intrigeri 8 months ago

#19 Updated by intrigeri 8 months ago

#20 Updated by intrigeri 5 months ago

  • Priority changed from Elevated to Normal

(I think this test is fragile but not totally broken on Jenkins.)

Also available in: Atom PDF