Project

General

Profile

Feature #7563

Update the automated test suite for Jessie ISO images

Added by BitingBird about 5 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Test suite
Target version:
Start date:
11/26/2014
Due date:
01/15/2016
% Done:

100%

Feature Branch:
feature/7563-test-suite-on-jessie
Type of work:
Code
Blueprint:
Starter:
No
Affected tool:

Subtasks

Bug #8317: fillram isn't effective for filling the memory in JessieResolved

Bug #8782: Update the "all notifications have disappeared" test suite step for JessieResolved

Bug #9046: Update the encryption test for JessieResolved

Bug #9695: "I start the Tor Browser in offline mode" step starts... Pidgin on JessieResolved

Bug #9692: Test suite's --retry-find option breaks wait_for_gnome_windowRejected

Bug #9704: iptables_parse is buggy for IPv6Resolved

Bug #9705: Update the memory erasure automated tests for JessieResolved

Feature #9706: Jessie: applications menu handling in the test suite is fragileResolved

Bug #9791: Update torified_gnupg feature for JessieResolved

Bug #10325: Update the "Connect to server" tests for JessieRejected

Bug #10336: Update "blocked wireless device" trace for Jessie Rejected

Feature #10402: Make sure the new snapshot system works as intended in JessieResolved

Bug #10403: Screen blanking breaks some tests in JessieResolved

Bug #10467: "I open the address" step is fragile on JessieResolved

Bug #10782: Regression in "I synchronize keys in Seahorse" recovery_procResolved

Bug #10907: usb_install.feature fails when run as part of the entire test suiteResolved

Bug #10926: The I2P AppArmor confinement test case succeeds even when I2P is not confinedResolved


Related issues

Blocked by Tails - Feature #7424: Have Tails based on Jessie building and starting Resolved 07/14/2014
Blocked by Tails - Feature #7882: Fonts look ugly on Jessie Resolved 09/08/2014 10/20/2014
Blocked by Tails - Bug #8256: Running pkexec in Jessie doesn't open the GNOME prompt Resolved 11/13/2014
Blocked by Tails - Bug #8274: liveusb-creator's gdisk scripts returns error in Jessie Resolved 12/18/2014 03/05/2015
Blocked by Tails - Bug #8275: tails-persistence-setup broken in Jessie Resolved 11/18/2014
Blocked by Tails - Bug #8372: Shutdown/reboot broken in Jessie Resolved 12/03/2014
Blocked by Tails - Bug #9053: syslinux/ldlinux.c32 differs from installation source to destination on Jessie Resolved 03/14/2015
Blocked by Tails - Bug #10537: Fix restart-tor and tor-has-bootstrapped semantics vs. test suite on Jessie Resolved 11/12/2015
Blocked by Tails - Bug #10571: The 'application/pgp-keys' MIME type has bad application associations Resolved 11/17/2015
Blocked by Tails - Bug #10539: "Clone and upgrade" on Jessie pretends my Tails was not installed with our Installer Resolved 11/12/2015
Blocked by Tails - Bug #10591: CUPS fails to start with read-only persistent configuration on Jessie Resolved 11/20/2015
Blocked by Tails - Bug #10593: Totem + torsocks can't read videos on Jessie Resolved 11/20/2015
Blocked by Tails - Bug #10660: Tails Installer 4.x crashes when a pristine USB drive is plugged Resolved 11/25/2015
Blocked by Tails - Bug #10696: Tor bridge mode fails on Jessie when time is way off Resolved 11/30/2015
Blocked by Tails - Bug #10732: Fix tor-has-bootstrapped semantics on network reconnect Resolved 12/09/2015

History

#1 Updated by intrigeri about 5 years ago

  • Subject changed from Update the automated test suite to Jessie to Update the automated test suite for Jessie
  • Parent task set to #7423

#2 Updated by intrigeri about 5 years ago

The rationale for doing this before Jessie is frozen is to detect issues ASAP and hopefully fix them in Jessie.

#3 Updated by intrigeri about 5 years ago

  • Blocked by Feature #7424: Have Tails based on Jessie building and starting added

#4 Updated by intrigeri about 5 years ago

  • Tracker changed from Bug to Feature

#5 Updated by anonym almost 5 years ago

#6 Updated by intrigeri almost 5 years ago

  • Due date set to 10/20/2014
  • Priority changed from Normal to High

Raising priority and setting a due date, as the freeze is now very close, and we'll need a few days before the upload deadline to file bugs and help get them fixed. anonym, does that sound realistic, or should we rethink who/when does this?

#7 Updated by intrigeri over 4 years ago

  • Due date changed from 10/20/2014 to 12/01/2014

intrigeri wrote:

Raising priority and setting a due date, as the freeze is now very close, and we'll need a few days before the upload deadline to file bugs and help get them fixed. anonym, does that sound realistic, or should we rethink who/when does this?

This wasn't realistic. Postponing deadline. anonym, do you think this will be enough, or should we rething who does this? (Dropping the ball and asking for help is fine. Silently dropping the ball is less fine :)

#8 Updated by anonym over 4 years ago

  • Blocked by Bug #8256: Running pkexec in Jessie doesn't open the GNOME prompt added

#9 Updated by anonym over 4 years ago

  • % Done changed from 0 to 20
  • Feature Branch set to test/jessie

intrigeri wrote:

This wasn't realistic. Postponing deadline. anonym, do you think this will be enough, or should we rething who does this? (Dropping the ball and asking for help is fine. Silently dropping the ball is less fine :)

Sorry. I've had it on my TODO for quite some time. I've at least started something now, although it's quite a bit more work left.

#10 Updated by anonym over 4 years ago

  • Feature Branch changed from test/jessie to feature/jessie

Working directly in feature/jessie from now on, since the test suite work reveals a lot of regressions that I'm fixing along the way.

At the moment the following features passes:
  • features/torified_browsing.feature
  • features/root_access_control.feature
  • features/dhcp.feature

#11 Updated by intrigeri over 4 years ago

Working directly in feature/jessie from now on, since the test suite work reveals a lot of regressions that I'm fixing along the way.

I think you forgot to push that branch.

#12 Updated by anonym over 4 years ago

  • Blocked by Bug #8274: liveusb-creator's gdisk scripts returns error in Jessie added

#13 Updated by anonym over 4 years ago

  • Blocked by Bug #8275: tails-persistence-setup broken in Jessie added

#14 Updated by anonym over 4 years ago

At the moment (commit 906940a in feature/jessie) the progress is as follows:

Passes completely:
  • features/apt.feature
  • features/build.feature
  • features/dhcp.feature
  • features/firewall_leaks.feature
  • features/i2p.feature
  • features/root_access_control.feature
  • features/time_syncing.feature
  • features/torified_browsing.feature
  • features/torified_gnupg.feature
  • features/unsafe_browser.feature
  • features/untrusted_partitions.feature
Failures because all GNOME 2.x notification area (i.e. "systray") applets are mostly broken in GNOME Shell:
  • features/checks.feature:
    - Scenario: The emergency shutdown applet can shutdown Tails
    - Scenario: The emergency shutdown applet can reboot Tails
  • features/encryption.feature
Failures because USB install (#8274) and persistence (#8275) are broken:
  • features/evince.feature
  • features/pidgin.feature
  • features/totem.feature
  • features/usb_install.feature
Other failures:
  • features/erase_memory.feature: See #7183.
  • features/windows_camouflage.feature: our GNOME flashback theme hasn't been adapted to GNOME Shell.
Ignore:
  • features/truecrypt.feature: we are going to remove truecrypt.

#15 Updated by intrigeri over 4 years ago

At the moment (commit 906940a in feature/jessie) the progress is as follows:

Congrats! We now know what we have to fix, thanks :)

#16 Updated by intrigeri over 4 years ago

  • Category changed from Test suite to 174

Regarding "features/erase_memory.feature: See #7183", #7183 is actually now resolved, so the remaining problem is that the screen is too garbled in the test suite environment, and then we can't find the "Happy dumping!" message, right? Not sure if it's easier to fix in the runtime code or in the test suite itself.

#17 Updated by intrigeri over 4 years ago

anonym wrote:

Failures because all GNOME 2.x notification area (i.e. "systray") applets are mostly broken in GNOME Shell:
  • features/checks.feature:
    - Scenario: The emergency shutdown applet can shutdown Tails
    - Scenario: The emergency shutdown applet can reboot Tails
  • features/encryption.feature

FYI I've added the topIcons extension, so this problem shouldn't be blocking your work anymore :)

#18 Updated by anonym over 4 years ago

  • Category changed from 174 to Test suite

intrigeri wrote:

Regarding "features/erase_memory.feature: See #7183", #7183 is actually now resolved, so the remaining problem is that the screen is too garbled in the test suite environment, and then we can't find the "Happy dumping!" message, right? Not sure if it's easier to fix in the runtime code or in the test suite itself.

Yeah I see that both #7183 and #8098 are resolved, but nothing has changed for me w.r.t. the garbled screen, so I opened #8300 which is awaiting your clarification. And yes, the issue is that we cannot see the "Happy dumping!" message. I'm not sure how we reliably could wait for the memory wiping to be done otherwise.

#19 Updated by anonym over 4 years ago

intrigeri wrote:

anonym wrote:

Failures because all GNOME 2.x notification area (i.e. "systray") applets are mostly broken in GNOME Shell:
  • features/checks.feature:
    - Scenario: The emergency shutdown applet can shutdown Tails
    - Scenario: The emergency shutdown applet can reboot Tails
  • features/encryption.feature

FYI I've added the topIcons extension, so this problem shouldn't be blocking your work anymore :)

Thanks! In commit 4ea49b2 I've fixed encryption.feature.

However, there's still no sign of the emergency shutdown helper applet. Even if we could get it to work with topIcons I do not think it's what we want. I'm opening a new ticket for how to deal with it in Jessie: #8302

#20 Updated by intrigeri over 4 years ago

However, there's still no sign of the emergency shutdown helper applet.

Indeed, since Tails 1.1 it's not hijacking the notification bar anymore.

#21 Updated by anonym over 4 years ago

With the fixes from #8300, #8302 and #8306, I pushed test suite updates so checks.feature passes.

I also did some work for erase_memory.feature, and the two first ("modern computer") scenarios of passes but the rest ("old computer") fails due to low pattern coverage. It becomes something like 55% now (and we end up requiring ~60%) which is getting seriously low, and this is despite there still being ~1300 MiB of free (non-buffer, non-cache) memory. This is getting way too low, so I have opened #8317.

#22 Updated by intrigeri over 4 years ago

  • Blocked by Bug #8317: fillram isn't effective for filling the memory in Jessie added

#23 Updated by intrigeri over 4 years ago

  • Status changed from Confirmed to In Progress

#24 Updated by intrigeri over 4 years ago

  • Due date changed from 12/01/2014 to 02/15/2015
  • % Done changed from 20 to 40

#25 Updated by intrigeri over 4 years ago

  • Blocked by Bug #8372: Shutdown/reboot broken in Jessie added

#27 Updated by BitingBird over 4 years ago

  • Subject changed from Update the automated test suite for Jessie to Update the automated test suite for Jessie ISO images

more precisions ("Jessie ISO images") because I keep confusing with #8165)

#28 Updated by anonym over 4 years ago

  • Target version changed from Tails_2.0 to Tails_1.8

#29 Updated by intrigeri over 4 years ago

Note that I've just merged devel into feature/jessie (post-1.3 release) and there were lots of changes in the test suite, and quite a few merge conflicts (e.g. the Synaptic tests have been rewritten both on feature/jessie and in the devel branch, fortunately more or less converging at least conceptually to similar implementations). So I expect that a lot of new and updated tests will now fail on Jessie ISOs.

#30 Updated by intrigeri over 4 years ago

Note that #8274 has been resolved, which should allow us to adapt more parts of the test suite to Jessie.

#31 Updated by intrigeri over 4 years ago

I think the blockers have been resolved, and work on this can be resumed. And anyway, even without fixing those blockers, I suspect that the test suite of Tails/Jessie ISO images was broken enough by changes we've introduced in the devel branch in the last few months.

#32 Updated by intrigeri over 4 years ago

  • Blocked by Bug #9053: syslinux/ldlinux.c32 differs from installation source to destination on Jessie added

#33 Updated by intrigeri over 4 years ago

I've updated a bunch of things for the usb_install feature, since I wanted to validate the work Alan and I have been doing in the last few days. In particular, it's been ported to UDisks 2, and a few additional low-level tests were added. Now, I gave up after unsuccessfully trying to get rid of FindFailed for: PersistenceWizardWindow.png, that I can't make sense of. So I'm eagerly waiting for this feature to be updated, so that Alan and I know how much work we have left on #8290 and friends, if any :)

#34 Updated by intrigeri over 4 years ago

  • Parent task deleted (#7423)

#35 Updated by intrigeri about 4 years ago

intrigeri wrote:

I've updated a bunch of things for the usb_install feature, since I wanted to validate the work Alan and I have been doing in the last few days. In particular, it's been ported to UDisks 2, and a few additional low-level tests were added. Now, I gave up after unsuccessfully trying to get rid of FindFailed for: PersistenceWizardWindow.png, that I can't make sense of. So I'm eagerly waiting for this feature to be updated, so that Alan and I know how much work we have left on #8290 and friends, if any :)

Fixed in f12419c. This is related to #9692.

#38 Updated by intrigeri almost 4 years ago

We'll have Jessie sprints in November (9-13 and 16-20). It would be awesome if the test suite was a bit more up-to-date by then, and/or if some of you fixed stuff during these sprints.

#39 Updated by intrigeri almost 4 years ago

Given our timeline, it would be good if someone did a full test suite run soon against a Tails/Jessie ISO and added subtasks for broken/missing bits, so that we have an idea of how much work is left to update (and it might be that only doing that finds some bugs in feature/jessie, which I'd love to hear about ASAP :)

#40 Updated by anonym over 3 years ago

intrigeri wrote:

We'll have Jessie sprints in November (9-13 and 16-20). It would be awesome if the test suite was a bit more up-to-date by then, and/or if some of you fixed stuff during these sprints.

Indeed. I plan to have this finished by then. Yay!

Given our timeline, it would be good if someone did a full test suite run soon against a Tails/Jessie ISO and added subtasks for broken/missing bits, so that we have an idea of how much work is left to update (and it might be that only doing that finds some bugs in feature/jessie, which I'd love to hear about ASAP :)

I've done a full run. Massive breakage! :) I've killed ~half of it so far, and I'd rather spend time fixing the issues than redmine wrangling. Sorry!

I've also merged devel in commit b7b2a9a, which had some tricky merge conflicts. It'd be good if someone had a second look.

The rest, b7b2a9a..commit:c63d4b5, is pretty straightforward, except e901c7f (for #9705), although some thought and testing likely will be needed for 51a9e41 (see its commit message). I'll open a ticket for this one, at least.

#41 Updated by intrigeri over 3 years ago

I've also merged devel in commit b7b2a9a, which had some tricky merge conflicts. It'd be good if someone had a second look.

Thanks. Please file a ticket for me to look at it.

#42 Updated by anonym over 3 years ago

anonym wrote:

[...] some thought and testing likely will be needed for 51a9e41 (see its commit message). I'll open a ticket for this one, at least.

Filed as #10395.

intrigeri wrote:

I've also merged devel in commit b7b2a9a, which had some tricky merge conflicts. It'd be good if someone had a second look.

Thanks. Please file a ticket for me to look at it.

Filed as #10402.

#43 Updated by intrigeri over 3 years ago

  • Blocked by deleted (Bug #8317: fillram isn't effective for filling the memory in Jessie)

#44 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10537: Fix restart-tor and tor-has-bootstrapped semantics vs. test suite on Jessie added

#45 Updated by anonym over 3 years ago

  • Blocked by Bug #10571: The 'application/pgp-keys' MIME type has bad application associations added

#46 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10539: "Clone and upgrade" on Jessie pretends my Tails was not installed with our Installer added

#47 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10591: CUPS fails to start with read-only persistent configuration on Jessie added

#48 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10593: Totem + torsocks can't read videos on Jessie added

#49 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10660: Tails Installer 4.x crashes when a pristine USB drive is plugged added

#50 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10696: Tor bridge mode fails on Jessie when time is way off added

#51 Updated by intrigeri over 3 years ago

#52 Updated by intrigeri over 3 years ago

  • Blocked by Bug #9707: Jessie: System sometimes does not poweroff after memory erasure added

#53 Updated by intrigeri over 3 years ago

Anyone looking at this ticket and wondering why it's still open, while all subtasks are resolved: we're tracking progress (and things left to do) in this area on a pad elsewhere.

#54 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10732: Fix tor-has-bootstrapped semantics on network reconnect added

#55 Updated by intrigeri over 3 years ago

  • Target version changed from Tails_1.8 to Tails_2.0

#56 Updated by anonym over 3 years ago

  • Blocked by Bug #10754: Install Electrum 0.2.5.x in Tails/Jessie added

#57 Updated by anonym over 3 years ago

  • Blocked by deleted (Bug #10754: Install Electrum 0.2.5.x in Tails/Jessie)

#58 Updated by anonym over 3 years ago

  • Feature Branch changed from feature/jessie to feature/7563-test-suite-on-jessie

feature/jessie was merged, creating dedicated branch.

#59 Updated by intrigeri over 3 years ago

#60 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.0 to Tails_2.2

#61 Updated by intrigeri over 3 years ago

  • Blocked by deleted (Bug #9707: Jessie: System sometimes does not poweroff after memory erasure)

#62 Updated by intrigeri over 3 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (anonym)
  • Target version changed from Tails_2.2 to Tails_2.0

I think we're done here. Now that 2.0 is out, let's deal with remaining problems (that are only robustness ones) on #10288.

Also available in: Atom PDF