Project

General

Profile

Bug #16281

Update the test suite for buster

Added by CyrilBrulebois 6 months ago. Updated 5 days ago.

Status:
In Progress
Priority:
High
Assignee:
Category:
Test suite
Target version:
Start date:
01/05/2019
Due date:
% Done:

94%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

The test suite won't pass directly for the feature/buster branch, as some reference images will need updating etc. Let's keep track of those updates with this bug report.


Subtasks

Bug #16287: buster vs. ssh.feature: unable to connect to SFTP serverResolved

Bug #16314: IO errors on buster when restoring snapshot from previous run with --keep-snapshotsConfirmed

Bug #16316: Black screen/apparent machine crash in test suite due to snapshot mechanism?Resolved

Bug #16317: persistence.feature fails due to NetworkManager testResolved

Bug #16319: Regression in ssh.feature with busterResolved

Bug #16335: Gobby 0.5→0.6 changes in buster: test suite update neededResolved

Bug #16340: Seahorse/buster: No import button, no more iconsResolved

Bug #16341: Seahorse/buster: No more close buttonResolved

Bug #16616: Re-enable and adjust test for desktop iconsResolved

Bug #16621: Fetching OpenPGP keys using Seahorse fails.Resolved

Bug #16623: Fix tests for mat with mat2Resolved

Bug #16817: Emergency shutdown automated tests on Buster often fail to notice that memory wipe was completedResolvedintrigeri

Bug #16819: Failing scenario: Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when onlineResolvedintrigeri

Bug #16820: UEFI test is broken for Tails based on busterResolvedintrigeri


Related issues

Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed 03/22/2019

Associated revisions

Revision 1bdac672 (diff)
Added by CyrilBrulebois 6 months ago

Test suite: update reference images for Tails Greeter (refs: #16281).

Revision 35178e49 (diff)
Added by CyrilBrulebois 6 months ago

Test suite: update reference image for Gnome Applications menu (refs: #16281).

The background is now blue instead of black by default, but that can
change depending on possible full screen windows. Let's deal with such
cases later, as they are encountered.

Revision 61b1ee84 (diff)
Added by CyrilBrulebois 6 months ago

Test suite: update reference image for Gnome Activities Overview (refs: #16281).

The bottom edge is slightly different but the images are very similar.

Revision aba71b0c (diff)
Added by CyrilBrulebois 6 months ago

Test suite: update reference image for Gnome Activities Overview (refs: #16281).

The bottom edge is slightly different but the images are very similar.

Revision 64a4a6ba (diff)
Added by CyrilBrulebois 6 months ago

Update reference images for persistence-setup wizard (refs: #16281, #16311).

There's a notification pop-up on top of the area covered by the
PersistenceWizardPresets.png reference image. As a consequence cropping
was performed to keep the text part only, trying to be future-proof (in
case the notification pop-up is gotten rid of or its dimensions get
changed).

Revision b8351ada (diff)
Added by CyrilBrulebois 6 months ago

Update more greeter reference images (refs: #16281).

These are needed for root_access_control.feature, only changes are
slight differences in rendering.

Revision c870a2e2 (diff)
Added by CyrilBrulebois 6 months ago

Update reference image for persistence passphrase in the greeter (refs: #16281).

This fixes one failure in with the untrusted_partitions feature.

Revision 3e14345b (diff)
Added by CyrilBrulebois 6 months ago

Update reference image for unblocked partition in the greeter (refs: #16281).

Needed for persistence.feature, now that we have a workaround for snapshot
issues (#16316).

Revision c2954bc0 (diff)
Added by CyrilBrulebois 6 months ago

Update reference image for deletion in persistence-setup (refs: #16281).

Revision bde12eef (diff)
Added by CyrilBrulebois 6 months ago

Update reference images needed for timesync.feature (refs: #16281).

Those were mostly about the tor configuration tool, plus a few specific
settings in the greeter.

Notes:
- Candidate for TailsGreeterSpecificTorConfiguration.png was entirely
wrong.
- TorLauncherConnectingWindow.png had no candidate on the sikuli side,
as the window goes away after a while.

Revision 5e9b0170 (diff)
Added by CyrilBrulebois 6 months ago

Make nautilus vs. ssh work (refs: #16281, #16319).

Revision bbcee2c9 (diff)
Added by CyrilBrulebois 6 months ago

Update reference image for Tails Greeter (refs: #16281).

Needed for shutdown_applet.feature at least.

Revision ba87916f (diff)
Added by CyrilBrulebois 6 months ago

Update reference images for evince feature (refs: #16281).

Note: The checks for apparmor's DENIED lines aren't passing at the
moment (#16321).

Revision 5e939618 (diff)
Added by CyrilBrulebois 6 months ago

Update reference image for “Disable all networking” (refs: #16281).

Without this update, sikuli would find a fuzzy candidate that happens to
be the label describing the various options instead of finding the
actual button. This would lead to a test failure, as eth0 would pop up.

Revision f88918b8 (diff)
Added by CyrilBrulebois 6 months ago

Update reference images for unsafe_browser feature (refs: #16281).

Revision 55da2855 (diff)
Added by CyrilBrulebois 6 months ago

Update reference images for torified_gnupg feature (refs: #16281, #16340).

In the process, switch from a clicking a tool bar button that no longer
exists in buster, and use the context menu to import the selected key
(refs: #16340).

Revision d5b48b58 (diff)
Added by CyrilBrulebois 5 months ago

Update reference images for Pidgin (refs: #16281).

Revision d8d8dd04 (diff)
Added by CyrilBrulebois 3 months ago

Update reference images for Pidgin (refs: #16281).

Revision cfb0c794 (diff)
Added by CyrilBrulebois 3 months ago

Update reference images for Pidgin (refs: #16281).

Revision a20e3731 (diff)
Added by Sandro Knauß 3 months ago

Make "Encryption and verification using GnuPG" succeed (refs: #16281).

History

#1 Updated by CyrilBrulebois 6 months ago

  • Status changed from New to In Progress

#2 Updated by CyrilBrulebois 6 months ago

With these initial commits, at least thunderbird.feature is OK:

5 scenarios (5 passed)
42 steps (42 passed)

#3 Updated by intrigeri 6 months ago

#4 Updated by intrigeri 3 months ago

#5 Updated by intrigeri 3 months ago

#6 Updated by intrigeri 3 months ago

  • Priority changed from Elevated to High

(This is currently the main blocker for us to prioritize coding work on Buster, let's express this via priority.)

#7 Updated by hefee 3 months ago

  • Assignee deleted (CyrilBrulebois)

#8 Updated by anonym about 1 month ago

  • Assignee set to anonym

#9 Updated by intrigeri 7 days ago

After a coordination meeting with anonym & hefee, at this point:

  • We can tell testers that all safety guarantees covered by non-fragile scenarios (that needs to be listed somewhere) are provided by feature/buster. Additional Software may be buggy in some corner cases our test suite exercises, but it shouldn't be dangerous to use as far as we can tell.
  • Wrt. fragile scenarios, they all passed at least once except these ones, that we'll investigate:
    • features/electrum.feature:15 (#16821) → real bug, now fixed
    • features/additional_software_packages.feature:69 (#16819) → test suite bug
    • features/usb_install.feature:87 (#16820) → works on bare metal so it's a test suite bug
    • features/totem.feature:50 → works when tested manually so not blocking the beta; broken on stable too (#10442#note-46)

#10 Updated by intrigeri 6 days ago

I've updated my above comment with info from the pad and tl;dr: all safety guarantees exercised by our test suite are provided by current feature/buster. Every test scenario passed either in the test suite on by testing manually. So yeah, we're good to go and can now publish a beta with security support, that Tails contributors and enthusiasts can use in production!

#11 Updated by intrigeri 6 days ago

Now that we're done with the blockers for beta1 (#16822), I think we should plan the next steps here. I think a XMPP meeting will work best. @hefee, @anonym: anyone wants to coordinate the scheduling process?

#12 Updated by intrigeri 6 days ago

intrigeri wrote:

I think we should plan the next steps here.

A fine goal for such a meeting would be to define what needs to be done so we can call this ticket done and close it: at this point, it's not clear to me what task/goal this ticket is tracking. And then, have a plan to make it happen :)

#13 Updated by intrigeri 5 days ago

Woohoo, in https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_feature-buster/225/, only one ("Browsing the web using the Tor Browser ǂ Watching a WebM video") scenario failed! \o/

Also available in: Atom PDF