Project

General

Profile

Bug #16004

Make our automated test suite take into account USB image updates wrt. supported installation & update methods

Added by u 8 months ago. Updated 1 day ago.

Status:
In Progress
Priority:
Elevated
Category:
Test suite
Target version:
Start date:
09/28/2018
Due date:
% Done:

60%

Estimated time:
24.00 h
QA Check:
Dev Needed
Feature Branch:
test/16004-adapt-usb-scenarios-to-usb-images
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

- .img + .iso as inputs,
- run different tests on them,
- add scenarios for new cases


Related issues

Blocked by Tails - Bug #16419: devel branch FTBFS because time-based snapshots of the "debian" archive are not updated since Jan 28 Resolved 02/05/2019

Associated revisions

Revision c81ef61e (diff)
Added by intrigeri 6 months ago

Document our initial plan to adjust test suite for USB images (refs: #16002, #16003, #16004)

Revision 919d9491 (diff)
Added by CyrilBrulebois 4 months ago

Test suite: make usb-install-tails-greeter use the Tails USB image (refs: #16004).

Let's optimize the test suite by installing the Tails USB image directly,
instead of doing the cloning dance. Remove the parent snapshot accordingly,
as there's no need to be running Tails in the first place anymore.

The Tails USB image could be trusted not to have any persistence
partition but let's keep that check anyway; that's part of a checkpoint.

The (absence of) persistence check used to appear twice; get rid of the
first instance as that one was performed after cloning, with a computer
that was still running. Keep the second one only, which happens after
the just installed system has booted.

Revision dfa58f4d (diff)
Added by CyrilBrulebois 4 months ago

Test suite: remove XXX from scenarios using the updated checkpoint (refs: #16004).

Revision 3a3b130c (diff)
Added by CyrilBrulebois 4 months ago

Test suite: remove XXX from two other scenarios (refs: #16004).

These scenarios are taking the expected shortcut (installing from the Tails
USB image directy), due to the following chain of parent checkpoints:
- usb-install-with-persistence-logged-in
- usb-install-with-persistence-tails-greeter
- usb-install-logged-in
- usb-install-tails-greeter

Revision b6984aec (diff)
Added by CyrilBrulebois 4 months ago

Test suite: make it possible to write an old version of Tails ISO/USB image (refs: #16004).

Revision 90041e5e (diff)
Added by CyrilBrulebois 4 months ago

Test suite: make installing old version of Tails quicker (refs: #16004).

It's way quicker to copy the old version of Tails using its USB image
instead of starting it from DVD and cloning to USB.

Use the implementation of the usb-install-tails-greeter checkpoint, with
a few variations (old Tails USB image, and different device name).

Also, instead of removing the USB device at the end, trigger a proper
shutdown of the computer (the emergency shutdown would be triggered
anyway).

Revision 7f4dac0d (diff)
Added by CyrilBrulebois 4 months ago

Test suite: drop old ISO image scenarios (refs: #16004).

Let's align with the updated documentation: those scenarios aren't
documented any more, drop them entirely.

Revision 73afff29
Added by intrigeri 4 months ago

Merge branch 'test/16004-adapt-usb-scenarios-to-usb-images' into feature/15292-usb-image

This branch might be incomplete (left to check) and it needs a tiny bit of
polishing, but it's definitely an incremental improvement over the current
state of things: this will ensure we have a more relevant test suite
when preparing 3.12~rc1.

refs: #16004

Revision c3d2b7bd (diff)
Added by CyrilBrulebois 4 months ago

Test suite: unplug "old" USB drive (refs: #16004).

Let's restore the previous behaviour, which is unplugging the USB drive
at the end of the scenario. That might not be entirely needed, but
better safe than sorry.

Revision e3c69866 (diff)
Added by CyrilBrulebois 4 months ago

Test suite: document booting and unplugging as being done on purpose (refs: #16004).

Revision 58d83ab8 (diff)
Added by intrigeri about 2 months ago

Test suite: draft Gherkin for installing with GNOME Disks from a USB image (refs: #16004)

History

#1 Updated by u 8 months ago

  • Blocked by Bug #16005: Code review & rubber-duck added

#2 Updated by intrigeri 7 months ago

  • Blocked by deleted (Bug #16005: Code review & rubber-duck)

#3 Updated by CyrilBrulebois 6 months ago

Initial brainstorming:
  • keep the run_test_suite interface as it is, meaning --iso and --old-iso on the .iso files, and no new options for the .img files.
  • check if there's an .img file alongside the .iso one, and use it for tests that are specific to the USB image.

#4 Updated by intrigeri 6 months ago

  • Status changed from Confirmed to In Progress

#5 Updated by intrigeri 6 months ago

#6 Updated by intrigeri 5 months ago

  • Target version changed from Tails_3.11 to Tails_3.12

#7 Updated by u 5 months ago

I'm unsure about the progress of this ticket, if you can, it would be helpful to understand what remains to be done. Thanks!

#8 Updated by intrigeri 5 months ago

  • Feature Branch set to test/16003-growing-system-partition

#9 Updated by u 5 months ago

Hi kibi! i'm still unsure about this one, can you let me know how far you've come and when this would be ready? We need to release a beta that is now late. I'm fine doing these tests later, but I need a clear ETA.

#10 Updated by intrigeri 5 months ago

#11 Updated by u 5 months ago

Hi Cyril, could you try to review this for January 21st latest delay please?
We need this to be done on February 1st and I would like to leave some time for intrigeri in case something needs fixing after the review. Thanks!

#12 Updated by intrigeri 5 months ago

Hi Cyril, could you try to review this for January 21st latest delay please?

To clarify, what's needed here is implementation, not reviewing.
We made an implementation plan during our sprint early December.

#13 Updated by u 5 months ago

intrigeri wrote:

Hi Cyril, could you try to review this for January 21st latest delay please?

To clarify, what's needed here is implementation, not reviewing.
We made an implementation plan during our sprint early December.

Thanks for the clarification. This does not change what I said about the deadline though :)
Do you want to share the implementation plan with me (if it contains any ETAs that is, so that I know what your plans are)?

#14 Updated by intrigeri 5 months ago

Do you want to share the implementation plan with me (if it contains any ETAs that is, so that I know what your plans are)?

It's only a technical plan: clarifying specification of what should be done + ideas/guidance for kibi about how to do it. Nothing like ETAs and such in there. We've pushed it as comments in the affected test suite files on the topic branch.

#15 Updated by CyrilBrulebois 5 months ago

FWIW getting back to the test suite (be it for Buster or for USB image) was done during the FT sprint, and #16003 should be in a rather good shape now. I'll continue with #16004 (this bug) during this week, and send an update on Sunday.

#16 Updated by intrigeri 5 months ago

  • Subject changed from Adjust automated test suite to Make our automated test suite take into account USB image updates wrt. supported installation & update methods

(Slightly more specific ticket title to better express the plan we did in 05f8adba2cee269b03843af1fc8ac89b5d665b3f and c81ef61ea9a852f5d6559cea83cf87443ae4dbbc, which is what this ticket is about.)

#17 Updated by intrigeri 5 months ago

  • Feature Branch deleted (test/16003-growing-system-partition)

#18 Updated by CyrilBrulebois 4 months ago

  • Feature Branch set to test/16004-adapt-usb-scenarios-to-usb-images

I've pushed the test/16004-adapt-usb-scenarios-to-usb-images branch earlier, which addresses a bunch of the XXX's added in the “design doc” commits.

One thing I've left out so far is:

# XXX: rename to tails_installer.feature and move things that don't use Tails
# Installer elsewhere?

which looked a bit cosmetic/clean-up, compared to the commits I pushed, actually changing some codepaths/checkpoint paths.

Regarding the documented use cases, that should be checked by the test suite, this list was gathered:

  • install from Linux: "dd" with GNOME Disks, done; same on Windows/macOS, modulo s/GNOME Disks/Etcher/
  • expert Linux install: dd the .img, done
  • upgrade from "inside" Tails, i.e. without having another up-to-date Tails: from my Tails d/l USB image and install it with GNOME Disks to an intermediary Tails, then boot on that intermediary Tails, and from there upgrade my Tails with Tails Installer
  • Upgrade or install from another up-to-date Tails: restart on the up-to-date Tails, clone with Tails Installer, no ISO/IMG involved

I haven't checked yet what pieces would be missing in the current set of tests.

#19 Updated by intrigeri 4 months ago

Here's a first review, at 7f4dac0d8b85e24a334f5b3e97e9ed95d6e45b5f.

My only comments are about 90041e5ed41377484c688be718fdd4821aba824f:

  • Maybe add a comment to explain why we bother booting the USB stick? (i.e. so that the repartitioning code runs; no other reason I guess?)
  • I'm not sure but I seem to remember that unplugging the USB drive was done on purpose, to get the VM definition back to a clean state avoid tainting the following scenarios. Given what the current next scenario is, it does not matter, but as things are moved / removed, it might matter some day and I don't want to debug weird failures that happen because a scenario is using a different version of Tails from the one I'm expecting. It's possible I'm totally wrong (e.g. if unplugging is a no-op in practice).

That's all for now, great job! Thanks in particular for the great commit messages: this saves me lots of time.

A first job run started on Jenkins, if the disk usage has grown too much, we'll know soon enough. In any case, I'll run the full test suite overnight, measuring the maximum disk space usage: if it has grown, this might indicate regressions (e.g. disks created not as temporarily as they could/should) or be entirely justified. And this full test suite run will also tell us if/how these changes affect other features.

I haven't checked yet what pieces would be missing in the current set of tests.

… and perhaps, what pieces we can drop :)

#20 Updated by intrigeri 4 months ago

A first job run started on Jenkins, if the disk usage has grown too much, we'll know soon enough.

The 2 first (partial) test suite runs on Jenkins went fine: no unexpected breakage.

In any case, I'll run the full test suite overnight, measuring the maximum disk space usage: […]

I forgot to run the measurements but before I started this, my /tmp/TailsToaster had 22G available, i.e. same as our Jenkins isotesters (modulo units discrepancies and rounding), and this full test suite run went fine (no unexpected breakage) so I think we're good on this side.

#21 Updated by intrigeri 4 months ago

  • % Done changed from 0 to 50

Merged into feature/15292-usb-image because this is already better than what we had there.

#22 Updated by intrigeri 4 months ago

  • Target version changed from Tails_3.12 to Tails_3.13

#23 Updated by CyrilBrulebois 4 months ago

From #16004#note-19:

Maybe add a comment to explain why we bother booting the USB stick? (i.e. so that the repartitioning code runs; no other reason I guess?)

Hrm. I guess there aren't that many reasons, except it was done in the usb-install-tails-greeter snapshot I've adapted, which makes it possible to check a few things like the udev-watchdog monitoring.

I'm not sure but I seem to remember that unplugging the USB drive was done on purpose, to get the VM definition back to a clean state avoid tainting the following scenarios. Given what the current next scenario is, it does not matter, but as things are moved / removed, it might matter some day and I don't want to debug weird failures that happen because a scenario is using a different version of Tails from the one I'm expecting. It's possible I'm totally wrong (e.g. if unplugging is a no-op in practice).

I can definitely implement unplugging instead of shutting down, to get unplug + shutdown in a single operation, just to be on the safe side. Checking runtime right now.

#24 Updated by CyrilBrulebois 4 months ago

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check set to Ready for QA

I've pushed two extra commits with a switch back to unplugging, and with added comments at the top of the scenario.

Back to “Ready for QA” as I think I've addressed everything; thanks for the review!

#25 Updated by intrigeri 4 months ago

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check changed from Ready for QA to Dev Needed

This branch FTBFS on Jenkins. I guess you need to merge current devel into it :)

#26 Updated by intrigeri 4 months ago

Hi, I've noticed you were not a watcher for this ticket so you might have missed my previous comment => added you as a watcher + here's a gentle poke to make sure you know this is back on your plate :)

#27 Updated by intrigeri 4 months ago

  • Blocked by Bug #16419: devel branch FTBFS because time-based snapshots of the "debian" archive are not updated since Jan 28 added

#28 Updated by intrigeri 3 months ago

Merged origin/stable into the topic branch, made sure config/base_branch says "stable" (there's a chance we have no major release for a while and I want this to be merged soon).

#29 Updated by intrigeri 3 months ago

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

@CyrilBrulebois incidentally, my last Git mangling fixed the FTBFS => happy to check the test results myself on Jenkins, as an exception :)

#30 Updated by intrigeri 3 months ago

Jenkins is happy, will code-review.

#31 Updated by intrigeri 3 months ago

  • Assignee changed from intrigeri to CyrilBrulebois
  • % Done changed from 50 to 60
  • QA Check changed from Ready for QA to Dev Needed

Code review passes at a40123ad5bc451d59b2e52a035bef56c8f9218cf.

Next step: deal with the leftovers of #16004#note-18.

#32 Updated by u 3 months ago

hi @CyrilBrulebois : we still miss this part on this ticket: deal with the leftovers of #16004#note-18. Can you please schedule this?

#33 Updated by u 3 months ago

  • Parent task changed from #16002 to #15292

#34 Updated by intrigeri 2 months ago

Feel free to skip "rename to tails_installer.feature and move things that don't use Tails Installer elsewhere?", it's not super useful and indeed, mostly cosmetic.

#35 Updated by intrigeri 2 months ago

And regarding the last bit, i.e. "I haven't checked yet what pieces would be missing in the current set of tests": this shouldn't take very long and I'm happy to do it myself, as part of my review / rubber-duck work, if you don't manage to get it done in time for 3.13.

#36 Updated by intrigeri 2 months ago

  • Assignee changed from CyrilBrulebois to intrigeri
  • Target version changed from Tails_3.13 to Tails_3.14
  • QA Check changed from Dev Needed to Info Needed

CyrilBrulebois wrote:

Regarding the documented use cases, that should be checked by the test suite, this list was gathered:

  • install from Linux: "dd" with GNOME Disks, done; same on Windows/macOS, modulo s/GNOME Disks/Etcher/

AFAICT we have no test about installing Tails with GNOME Disks. This is about GNOME Disks outside of Tails anyway and I don't think we're ready to add test cases that boot other operating systems than Tails to our test suite.

  • expert Linux install: dd the .img, done

Indeed, basically all our test cases that boot from USB exercise this.

  • upgrade from "inside" Tails, i.e. without having another up-to-date Tails: from my Tails d/l USB image and install it with GNOME Disks to an intermediary Tails, then boot on that intermediary Tails, and from there upgrade my Tails with Tails Installer

We do exercise the "boot on that intermediary Tails, and from there upgrade my Tails with Tails Installer" part in "Upgrading an old Tails USB installation from another Tails USB drive" but we have no test about "install it with GNOME Disks to an intermediary Tails": the drive that plays the role of the intermediary Tails was prepared in a test suite ad-hoc way, outside of Tails. If we have time left on the budget, let's add a test for this (I'll send you my own clocking data and you can do the maths). Otherwise, forget it.

  • Upgrade or install from another up-to-date Tails: restart on the up-to-date Tails, clone with Tails Installer, no ISO/IMG involved

Cloning with Tails installer: "Installing Tails to a pristine USB drive" exercises the installation; "Re-installing Tails over an existing USB installation with a persistent partition" exercises re-installing; "Upgrading an old Tails USB installation from a Tails DVD" exercises upgrading. So we're good here.

I haven't checked yet what pieces would be missing in the current set of tests.

Done!

#37 Updated by intrigeri 2 months ago

  • Assignee changed from intrigeri to CyrilBrulebois
  • Priority changed from Normal to Elevated
  • Parent task changed from #15292 to #16002

#39 Updated by CyrilBrulebois 2 months ago

  • Private changed from No to Yes

@intrigeri Please get back to me on this topic when it's no longer release days.

#40 Updated by intrigeri 2 months ago

  • Assignee changed from CyrilBrulebois to intrigeri

OK.

#41 Updated by u 2 months ago

  • Private changed from Yes to No

#42 Updated by intrigeri 2 months ago

  • Assignee changed from intrigeri to CyrilBrulebois

@CyrilBrulebois, 3.13 is in the past so I'm sending this back to your plate wrt. my question on #16004#note-38. As discussed elsewhere, no big hurry. But I'd like an answer early April so we can decide what's the next step and eventually wrap this project up :)

#43 Updated by u about 2 months ago

  • Parent task deleted (#16002)

simply unparenting this ticket.

#44 Updated by intrigeri about 2 months ago

  • Parent task set to #16002

u wrote:

simply unparenting this ticket.

@u, FYI this breaks the "estimated time" data of the parent task, which is why I reverted the exact same change 2 weeks ago.

#45 Updated by u about 2 months ago

intrigeri wrote:

u wrote:

simply unparenting this ticket.

@u, FYI this breaks the "estimated time" data of the parent task,

I do apparently not understand why/how this breaks the estimated time" data of the parent task. Each task has their own estimated time. According to #note-38 you seem to consider the estimated time of both tasks together, which is fine by me, but could simply be modified in the "estimated time" field of this ticket task to reflect exactly that thinking. So currently to me it sounds more like a conceptual/UX issue with Redmine. But maybe I need more explanations here.

which is why I reverted the exact same change 2 weeks ago.

I see that you reverted a different change (different parent task) in #note-37, without explaining that you reverted it for the reason you are stating right now.

#47 Updated by CyrilBrulebois about 2 months ago

  • Private changed from No to Yes

[Note deleted, sorry for the noise.]

#48 Updated by CyrilBrulebois about 2 months ago

  • Private changed from Yes to No

#50 Updated by intrigeri about 2 months ago

I do apparently not understand why/how this breaks the estimated time" data of the parent task. Each task has their own estimated time. According to #note-39 you seem to consider the estimated time of both tasks together, which is fine by me, but could simply be modified in the "estimated time" field of this ticket task to reflect exactly that thinking. So currently to me it sounds more like a conceptual/UX issue with Redmine. But maybe I need more explanations here.

@u, you're of course entirely correct that there are several equally valid ways in which we could handle our tickets here :) Now, switching right now to a new organization has a cost for me (I would need to "simply" adjust ticket metadata and then update my habits)… for no perceived value for me. I understand that you did these changes for a reason, presumably to help organize your management work. So I guess we're hitting a typical case of friction here, as in "a change that helps the manager has drawbacks for a worker who does not understand the value of the change and gets defensive". Were we closer to the beginning of this project, it would be interesting to spend more time understanding each other's needs and reach a consensus. Given we're almost done, I'd rather not spend more time on this so I'll reapply your "simply unparenting this ticket" change. We can discuss later and elsewhere how best to handle these things in a way that every affected person understands.

#51 Updated by intrigeri about 2 months ago

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check changed from Info Needed to Dev Needed

#53 Updated by intrigeri about 2 months ago

  • Parent task deleted (#16002)

(As per #16004#note-50, to avoid process getting in the way of doing the work.)

#54 Updated by intrigeri about 2 months ago

  • Assignee changed from intrigeri to CyrilBrulebois

@CyrilBrulebois, I wrote Gherkin for the missing scenario: 58d83ab8b8a2ce154b144a46771f0292a2b3fedc.

Note that we already have code to do "I plug and mount a USB drive containing…"; presumably it's not too hard to extend it to support what we need here. We also have some code to interact with GNOME Disks (features/veracrypt.feature, features/step_definitions/veracrypt.rb).

Happy hacking!

#55 Updated by u about 1 month ago

@CyrilBrulebois is there an ETA for this on your side?

#56 Updated by CyrilBrulebois 1 day ago

  • Target version changed from Tails_3.14 to Tails_3.15

Also available in: Atom PDF