Project

General

Profile

Feature #15319

Feature #15292: Distribute a USB image

Feature #15293: Creating & preparing the disk image

Grow system partition during boot when started from USB

Added by segfault over 1 year ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
10/16/2018
Due date:
% Done:

30%

Estimated time:
4.00 h
Feature Branch:
feature/15292-repartition
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Currently, Tails Installer creates a system partition if size 4 GiB or 8 GiB during installation. It would vastly increase the size of the USB image if we would include such a big partition in it, and unnecessarily so, because the partition is mostly empty. The plan is to instead ship a USB image of minimum size, and increase the size of the system partition during the first boot of the device.


Related issues

Related to Tails - Bug #15681: Check getrandom() status and potential problems Resolved 06/23/2018
Related to Tails - Bug #15987: Check the system partition on every boot and grow it if needed Rejected 09/28/2018
Blocks Tails - Bug #16003: Write automated tests for growing system partition Resolved 09/28/2018

Associated revisions

Revision fedce3f0 (diff)
Added by segfault about 1 year ago

Use set -eu instead of -e in shebang (refs: #15319)

Revision c72bef32 (diff)
Added by segfault about 1 year ago

Use if; then instead of && (refs: #15319)

Revision b2330a96 (diff)
Added by segfault about 1 year ago

Fix mixed spaces and tabs (refs: #15319)

Revision 47f9ca5f (diff)
Added by segfault about 1 year ago

Use fancy sed instead of `grep | sed` (refs: #15319)

Revision 5751abf6 (diff)
Added by segfault about 1 year ago

Quote variables (refs: #15319)

Revision 78e9bcb3 (diff)
Added by segfault about 1 year ago

Use better variable names (refs: #15319)

Revision 3fbc2c50 (diff)
Added by segfault about 1 year ago

Don't use expr (refs: #15319)

Revision b661f52d (diff)
Added by segfault about 1 year ago

Skip partitioning if FSUUID is unset (refs: #15319)

Revision f2626258 (diff)
Added by segfault about 1 year ago

Add comment explaining exit code (refs: #15319)

Revision c92c07bd (diff)
Added by segfault about 1 year ago

Fix partitioning script failing if called without arguments (refs: #15319)

Revision 61baa41c (diff)
Added by segfault about 1 year ago

Include fatresize in initramfs (refs: #15319)

Revision 3726d756 (diff)
Added by segfault about 1 year ago

Grow the filesystem (refs: #15319)

Revision e91b32bf (diff)
Added by segfault 12 months ago

Run partitioning script in init-premount stage (refs: #15319)

... instead of live-realpremount, because the latter is actually
executed after /dev/sda1 is already mounted

Revision 596ed207 (diff)
Added by segfault 12 months ago

Restore boot code in VBR after running fatresize (refs: #15319)

Revision e9a28349 (diff)
Added by segfault 12 months ago

Don't use possibly undefined variable without default value (refs: #15319)

Revision bfa834b9 (diff)
Added by segfault 11 months ago

Don't boot devices smaller than 7200 MiB (refs: #15319)

Revision 2b38ad8c (diff)
Added by segfault 11 months ago

Don't boot devices smaller than 7200 MiB (refs: #15319)

Revision 4365b722 (diff)
Added by intrigeri 11 months ago

Fix rebooting when the boot device is too small (refs: #15319)

It's not clear whether we want to reboot there yet, but in any case,
let's use a syntax that busybox's reboot supports.

Revision 26f771d0 (diff)
Added by intrigeri 11 months ago

When the boot device is too small, wait for user input then shut down the computer (refs: #15319)

Rebooting is not very useful: in many cases, the user will hit exactly
the same problem again. Let's instead shut down the computer so
they have time to unplug the faulty USB stick.

Also, let's wait for user input instead of 10s: the user might be brewing coffee
or something and we're not in any hurry to shut down the system.

Revision 96d2bec6 (diff)
Added by segfault 11 months ago

Use set -eu instead of -e in shebang (refs: #15319)

Revision f396325a (diff)
Added by segfault 11 months ago

Use if; then instead of && (refs: #15319)

Revision bd62d6cf (diff)
Added by segfault 11 months ago

Fix mixed spaces and tabs (refs: #15319)

Revision 3617000f (diff)
Added by segfault 11 months ago

Use fancy sed instead of `grep | sed` (refs: #15319)

Revision 5ea177b8 (diff)
Added by segfault 11 months ago

Quote variables (refs: #15319)

Revision e00e734f (diff)
Added by segfault 11 months ago

Use better variable names (refs: #15319)

Revision 2b43ce24 (diff)
Added by segfault 11 months ago

Don't use expr (refs: #15319)

Revision f8af03f5 (diff)
Added by segfault 11 months ago

Skip partitioning if FSUUID is unset (refs: #15319)

Revision f6c55601 (diff)
Added by segfault 11 months ago

Add comment explaining exit code (refs: #15319)

Revision 83ea07ae (diff)
Added by segfault 11 months ago

Fix partitioning script failing if called without arguments (refs: #15319)

Revision d5341b73 (diff)
Added by segfault 11 months ago

Include fatresize in initramfs (refs: #15319)

Revision f1fb1364 (diff)
Added by segfault 11 months ago

Grow the filesystem (refs: #15319)

Revision 3899598c (diff)
Added by segfault 11 months ago

Run partitioning script in init-premount stage (refs: #15319)

... instead of live-realpremount, because the latter is actually
executed after /dev/sda1 is already mounted

Revision f92311bf (diff)
Added by segfault 11 months ago

Restore boot code in VBR after running fatresize (refs: #15319)

Revision b8965673 (diff)
Added by segfault 11 months ago

Don't use possibly undefined variable without default value (refs: #15319)

Revision 9ff6995e (diff)
Added by segfault 11 months ago

Don't boot devices smaller than 7200 MiB (refs: #15319)

Revision a9c60fdd (diff)
Added by intrigeri 11 months ago

Fix rebooting when the boot device is too small (refs: #15319)

It's not clear whether we want to reboot there yet, but in any case,
let's use a syntax that busybox's reboot supports.

Revision 5a50bc33 (diff)
Added by intrigeri 11 months ago

When the boot device is too small, wait for user input then shut down the computer (refs: #15319)

Rebooting is not very useful: in many cases, the user will hit exactly
the same problem again. Let's instead shut down the computer so
they have time to unplug the faulty USB stick.

Also, let's wait for user input instead of 10s: the user might be brewing coffee
or something and we're not in any hurry to shut down the system.

History

#1 Updated by segfault over 1 year ago

  • QA Check set to Ready for QA

Should be fixed by 689d98876ef00d1c359eb9baf146bf1f38a00a58.

#2 Updated by intrigeri over 1 year ago

  • Related to Bug #15681: Check getrandom() status and potential problems added

#3 Updated by intrigeri over 1 year ago

Recent util-linux has:

commit a9cf659e0508c1f56813a7d74c64f67bbc962538
Author: Carlo Caione <carlo@endlessm.com>
Date:   Mon Mar 19 10:31:07 2018 +0000

    lib/randutils: Do not block on getrandom()

    In Endless we have hit a problem when using 'sfdisk' on the really first
    boot to automatically expand the rootfs partition. On this platform
    'sfdisk' is blocking on getrandom() because not enough random bytes are
    available. This is an ARM platform without a hwrng.

    We fix this passing GRND_NONBLOCK to getrandom(). 'sfdisk' will use the
    best entropy it has available and fallback only as necessary.

    Signed-off-by: Carlo Caione <carlo@endlessm.com>

… which might be relevant.

Context: https://lists.debian.org/msgid-search/20180514003034.GI14763@thunk.org

#4 Updated by segfault over 1 year ago

Thanks. Might be relevant if we hit the same problem. But IIRC we already tested the feature branch and found that it does successfully grow the system partition during first boot without significantly increasing boot time.

#5 Updated by u about 1 year ago

segfault: if this is ready for QA, do you want to assign this to a potential reviewer?

#6 Updated by u about 1 year ago

  • Target version set to Tails_3.10.1

@segfault: please assign to intrigeri if this is ready for QA.

#7 Updated by u about 1 year ago

  • Blocks Bug #15991: Code review & rubber-duck for USB Image added

#8 Updated by u about 1 year ago

  • Estimated time set to 4.00 h

#9 Updated by u about 1 year ago

  • Parent task changed from #15292 to #15293

#10 Updated by u about 1 year ago

  • Blocks Bug #16003: Write automated tests for growing system partition added

#11 Updated by intrigeri about 1 year ago

  • Blocks deleted (Bug #15991: Code review & rubber-duck for USB Image)

#12 Updated by intrigeri about 1 year ago

  • QA Check changed from Ready for QA to Dev Needed

I'll review the branch here although some of these comments probably apply to one of the many sibling tickets. Actually, the ticket setup here seems too fine grained (=> overhead) to me so if you prefer, maybe we can merge all these "what we need to do at early boot" tickets into a single one and use its description to track what's the plan and progress (see e.g. how I'm doing it on #14588).

  • I'm warry of /bin/sh -e: if whatever environment this runs under starts to source this script instead of executing it, then -e will be ignored. set -e would make me feel more comfortable. And while you're at it, go for set -eu :)
  • [ -n "$debug" ] && set -x looks like it will fail under set -e in non-debug mode (but that might be a corner case where it won't actually break, I don't remember). Using an explicit if/then construct would avoid wondering and diving deep into the darkest corners of the dash doc.
  • Have you tested how if [ -b "/dev/disk/by-uuid/${FSUUID}" ] works when booting on DVD?
  • Indentation is inconsistent (mixes tabs & 2-spaces) in the partitioning script.
  • Please add a comment that makes the practical consequences of exit 1 explicit. "Skipping partitioning" suggests the boot process will simply continue but a user-facing message can't convey the amount of technical details I would like to see.
  • | grep ID_PATH= | sed s/ID_PATH=// is not strict enough for such a strict (not anchored regexps) and could be done with sed only; I suggest replacing these with | sed -n '/^ID_PATH=/ s/^ID_PATH=// p'. Same comments for other sed constructs.
  • Please quote variables more consistently: given the nature of this script, robustness even in all unexpected corner cases is primordial and we don't ever want to land into undefined behavior territory.
  • Please give SIZE a more specific name. Also, maybe s/TOTAL_SIZE/DEVICE_SIZE/ (to avoid wondering total size of what?).

Nitpicking:

  • Any reason to use $((…)) in one place and $(expr …) in another?

#13 Updated by intrigeri about 1 year ago

One more thing: please test that the resulting USB stick (after the 1st-boot changes) still boots on non-EFI systems (i.e. the changes made to the GPT were sync'ed to the hybrid/protective/whatever MBR).

#14 Updated by intrigeri about 1 year ago

  • Subject changed from Grow system partition of USB image during boot to Grow system partition during boot when started from USB

#15 Updated by intrigeri about 1 year ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Also, it seems that we need to resize the FAT filesystem as well because while the partition was successfully resized in my tests, the FS is still 2GB:

$ df -h /lib/live/mount/medium/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       2.0G  1.6G  458M  78% /lib/live/mount/medium

#16 Updated by segfault about 1 year ago

  • Target version changed from Tails_3.10.1 to Tails_3.11

This doesn't have to be done in time for 3.10, right? If so, I would like to work on it after 3.10

#17 Updated by intrigeri about 1 year ago

This doesn't have to be done in time for 3.10, right? If so, I would like to work on it after 3.10

Right.

#18 Updated by segfault about 1 year ago

intrigeri wrote:

I'll review the branch here although some of these comments probably apply to one of the many sibling tickets. Actually, the ticket setup here seems too fine grained (=> overhead) to me so if you prefer, maybe we can merge all these "what we need to do at early boot" tickets into a single one and use its description to track what's the plan and progress (see e.g. how I'm doing it on #14588).

  • I'm warry of /bin/sh -e: if whatever environment this runs under starts to source this script instead of executing it, then -e will be ignored. set -e would make me feel more comfortable. And while you're at it, go for set -eu :)

Fixed

  • [ -n "$debug" ] && set -x looks like it will fail under set -e in non-debug mode (but that might be a corner case where it won't actually break, I don't remember). Using an explicit if/then construct would avoid wondering and diving deep into the darkest corners of the dash doc.

It don't think it breaks, because I tested the ISO built from this branch with and without debug flag. I changed it to an if/then construct.

  • Have you tested how if [ -b "/dev/disk/by-uuid/${FSUUID}" ] works when booting on DVD?

I will do that on #16057

  • Indentation is inconsistent (mixes tabs & 2-spaces) in the partitioning script.

Fixed

  • Please add a comment that makes the practical consequences of exit 1 explicit. "Skipping partitioning" suggests the boot process will simply continue but a user-facing message can't convey the amount of technical details I would like to see.

As explained elsewhere (don't know where that was), the status code is irrelevant in the current implementation of live-config. Is that what you want me to write in a comment?

  • | grep ID_PATH= | sed s/ID_PATH=// is not strict enough for such a strict (not anchored regexps) and could be done with sed only; I suggest replacing these with | sed -n '/^ID_PATH=/ s/^ID_PATH=// p'. Same comments for other sed constructs.

Done

  • Please quote variables more consistently: given the nature of this script, robustness even in all unexpected corner cases is primordial and we don't ever want to land into undefined behavior territory.

Done

  • Please give SIZE a more specific name. Also, maybe s/TOTAL_SIZE/DEVICE_SIZE/ (to avoid wondering total size of what?).

Nitpicking:

  • Any reason to use $((…)) in one place and $(expr …) in another?

Fixed

One more thing: please test that the resulting USB stick (after the 1st-boot changes) still boots on non-EFI systems (i.e. the changes made to the GPT were sync'ed to the hybrid/protective/whatever MBR).

Will do that on #16057

Also, it seems that we need to resize the FAT filesystem as well because while the partition was successfully resized in my tests, the FS is still 2GB:

I will look into that later

#19 Updated by segfault about 1 year ago

  • Assignee changed from segfault to intrigeri
  • QA Check changed from Dev Needed to Info Needed

#20 Updated by segfault about 1 year ago

Have you tested how if [ -b "/dev/disk/by-uuid/${FSUUID}" ] works when booting on DVD?

So, $FSUUID is unset when booting on DVD, which means that the script will wait for the timeout (1 minute currently) before skipping the partitioning.

I remember that I knew about this issue back when I last worked on the script (in February), but I can't find any notes / tickets about it and don't remember if I already tried to fix it.

I implemented a fix and will test it after building the ISO.

#21 Updated by intrigeri about 1 year ago

  • Assignee changed from intrigeri to segfault
  • QA Check deleted (Info Needed)

segfault wrote:

intrigeri wrote:

  • Please add a comment that makes the practical consequences of exit 1 explicit. "Skipping partitioning" suggests the boot process will simply continue but a user-facing message can't convey the amount of technical details I would like to see.

As explained elsewhere (don't know where that was), the status code is irrelevant in the current implementation of live-config. Is that what you want me to write in a comment?

Yes, exactly: the fact I did not remember is actually a good example of why a comment would be useful :)

#22 Updated by segfault about 1 year ago

One more thing: please test that the resulting USB stick (after the 1st-boot changes) still boots on non-EFI systems (i.e. the changes made to the GPT were sync'ed to the hybrid/protective/whatever MBR).

Done

#23 Updated by segfault about 1 year ago

intrigeri wrote:

As explained elsewhere (don't know where that was), the status code is irrelevant in the current implementation of live-config. Is that what you want me to write in a comment?

Yes, exactly: the fact I did not remember is actually a good example of why a comment would be useful :)

Done

#24 Updated by segfault about 1 year ago

  • Assignee changed from segfault to intrigeri
  • QA Check set to Info Needed

#25 Updated by segfault about 1 year ago

  • Assignee changed from intrigeri to segfault
  • QA Check deleted (Info Needed)

Also, it seems that we need to resize the FAT filesystem as well because while the partition was successfully resized in my tests, the FS is still 2GB

OK, now I see the problem. I guess we will have to include fatresize in the initramfs then.

#26 Updated by segfault about 1 year ago

I was able to successfully grow the filesystem with fatresize - but then the device doesn't boot anymore, because fatresize overwrites the boot code in the VBR.

The device boots again after restoring the original bootcode (and JMP instruction at the beginning of the VBR). Next step: Implement this in the partitioning script.

Oh, and I already spent a total of 8.00 h on this (including the subticket #16057).

#27 Updated by segfault 12 months ago

  • Assignee changed from segfault to intrigeri
  • QA Check set to Ready for QA

Next step: Implement this in the partitioning script

Done, and seems to work.

#28 Updated by segfault 12 months ago

  • Feature Branch changed from segfault:feature/15292-usb-image to feature/15292-usb-image

#29 Updated by intrigeri 12 months ago

  • Nitpick: if [ -z "${FSUUID}" ] will fail under set -u if $FSUUID is not set (which is different than "set to the empty string"). I think you want if [ -z "${FSUUID:-}" ]. I guess it does not matter much here because live-boot will set FSUUID= but still, it took me some time to understand why it does not break, so when running under set -u, better make it clear. I'm so sorry you have to deal with this shell crap, but that's what early boot is made of :/ Maybe some day we'll write Go or Rust for this sort of things that must run without a big Python/Ruby/Perl interpreter, hey :)
  • IIRC syslinux' ldlinux.sys (generated when we run syslinux) includes hard-coded block numbers, or the MBR hard-codes the block number where it can find ldlinux.sys, or similar. This is "slightly" outside of my comfort zone. Can we assume that fatresize won't mess this up? I know your legacy BIOS tests seem successful, I'm just wondering if we've been lucky so far or if we can really count on it. I guess the safer way to avoid wondering would be to run syslinux again after resizing the FS, but arguably that opens another can of robustness worms.

Once you've fixed the minor shell issue and satisfied my curiosity wrt. ldlinux.sys, feel free to close this ticket as resolved :)

#30 Updated by intrigeri 12 months ago

  • Assignee changed from intrigeri to segfault
  • QA Check changed from Ready for QA to Dev Needed

intrigeri wrote:

Once you've fixed the minor shell issue and satisfied my curiosity wrt. ldlinux.sys,

+ completed the testing on the subtask, of course.

#31 Updated by intrigeri 12 months ago

Actually, I'm not sure we should do nothing at all if the device is smaller than 7200 MiB. Nothing prevents one to install there (only Tails Installer would complain but we're going to deprecate it for initial installation) if they skip the note about this in our doc. And then nothing prevents them from creating a persistent volume. Once that's done, the only way to upgrade is a manual upgrade. But if the size our ISO has grown a little bit in the meantime, that's not gonna work, which is a UX regression compared to the current state of things (I think that Tails Installer refuses to install on a too small device but accepts to upgrade as long as the upgrade fits on the destination device, so that people can keep using and upgrading their existing sticks with a 2.5GB system partition). And given the device is filled by the persistent volume, it's pretty hard to recover from this failure mode.

So I think that we should abort with an error message pointing to the installation doc. Rationale: those sticks are unsupported and we can't provide a good UX on them. Better abort while the user has not invested too much into them, it's still time to reinstall on a bigger device, than let them go on, create a persistent volume and be stuck. If plymouth is already running at this point, fine, we can use it to display the error. Otherwise, we have to display the error on the good ol' text console.

Only newly installed sticks are affected: older ones have a different GUID so we just do nothing (modulo pending discussion on #15987).

If we can easily agree on something like this, let's just implement it. If it's not obvious to you, then let's ask the UX gods™ (aka. sajolida) to tell us what we should do.

Yeah, every plan is doomed to change because it has always missed something :)

#32 Updated by intrigeri 12 months ago

intrigeri wrote:

Only newly installed sticks are affected: older ones have a different GUID so we just do nothing (modulo pending discussion on #15987).

Even if we decide, on #15987, to do what was planned, we can still keep these old and too small sticks running: we error out only if the device is too small AND has the known GUID for initial installations, i.e. it's its first boot. If the device is too small but has another GUID, we exit 0.

#33 Updated by intrigeri 12 months ago

  • Related to Bug #15987: Check the system partition on every boot and grow it if needed added

#34 Updated by u 12 months ago

segfault wrote:

Oh, and I already spent a total of 8.00 h on this (including the subticket #16057).

Thanks! Is this because of a technical difficulty or did we completely underestimate the time for this task?
Maybe we could keep track somewhere on how much this one was overclocked, for future grants? Feel free to send this information to me by mail.

#35 Updated by segfault 12 months ago

We should make sure that both the partition label in the GPT and the label in the FAT are still set to "Tails" after running the partitioning script.

#36 Updated by segfault 11 months ago

  • IIRC syslinux' ldlinux.sys (generated when we run syslinux) includes hard-coded block numbers, or the MBR hard-codes the block number where it can find ldlinux.sys, or similar. This is "slightly" outside of my comfort zone. Can we assume that fatresize won't mess this up?

Yes. The first block number of ldlinux.sys is stored in the code part of the VBR, which we restore after running fatresize. And we can be sure to still find ldlinux.sys at this block number, because fatresize only changes the file allocation table and doesn't move the contents of the filesystem.

#37 Updated by segfault 11 months ago

Nitpick: if [ -z "${FSUUID}" ] will fail under set -u if $FSUUID is not set (which is different than "set to the empty string"). I think you want if [ -z "${FSUUID:-}" ]. I guess it does not matter much here because live-boot will set FSUUID= but still, it took me some time to understand why it does not break, so when running under set -u, better make it clear.

Done in e9a28349a608a6a4ed398f791d4f0765c35eb6dc

#38 Updated by segfault 11 months ago

  • Assignee changed from segfault to sajolida
  • QA Check changed from Dev Needed to Info Needed

intrigeri wrote:

Actually, I'm not sure we should do nothing at all if the device is smaller than 7200 MiB. Nothing prevents one to install there (only Tails Installer would complain but we're going to deprecate it for initial installation) if they skip the note about this in our doc. And then nothing prevents them from creating a persistent volume. Once that's done, the only way to upgrade is a manual upgrade. But if the size our ISO has grown a little bit in the meantime, that's not gonna work, which is a UX regression compared to the current state of things (I think that Tails Installer refuses to install on a too small device but accepts to upgrade as long as the upgrade fits on the destination device, so that people can keep using and upgrading their existing sticks with a 2.5GB system partition). And given the device is filled by the persistent volume, it's pretty hard to recover from this failure mode.

So I think that we should abort with an error message pointing to the installation doc. Rationale: those sticks are unsupported and we can't provide a good UX on them. Better abort while the user has not invested too much into them, it's still time to reinstall on a bigger device, than let them go on, create a persistent volume and be stuck. If plymouth is already running at this point, fine, we can use it to display the error. Otherwise, we have to display the error on the good ol' text console.

Only newly installed sticks are affected: older ones have a different GUID so we just do nothing (modulo pending discussion on #15987).

We discussed this during our last meeting. Currently, we prevent users from installing the USB stick in the first place. With this proposed change, we tell them that they can't use the USB stick after they installed and booted it - not great UX IMO.

I think it would be nicer to still let them use the USB stick, but display a warning that they won't be able to use automatic upgrades. We could display this either in the greeter or as a desktop notification.

We could also prevent creating a persistent partition, so that the user won't have problems migrating to another device.

intrigeri pointed out that quite quickly also manual upgrades won't be possible. I think we could change that by growing the system partition to at least 2 GiB (or 2.5 GiB?), or the whole device, if it is smaller than that.

@sajolida, we would like to have your opinion on this.

#39 Updated by intrigeri 11 months ago

  • IIRC syslinux' ldlinux.sys (generated when we run syslinux) includes hard-coded block numbers, or the MBR hard-codes the block number where it can find ldlinux.sys, or similar. This is "slightly" outside of my comfort zone. Can we assume that fatresize won't mess this up?

Yes. The first block number of ldlinux.sys is stored in the code part of the VBR, which we restore after running fatresize. And we can be sure to still find ldlinux.sys at this block number, because fatresize only changes the file allocation table and doesn't move the contents of the filesystem.

Great news and thanks for checking!

#40 Updated by intrigeri 11 months ago

Done in e9a28349a608a6a4ed398f791d4f0765c35eb6dc

ACK

#41 Updated by sajolida 11 months ago

  • Assignee changed from sajolida to segfault

It's cool that you're giving so much thought on these things! It's a shame that we overlooked them when building the grant proposal :(

To bring more info to the discussion:

  • Nowadays I think it's impossible to buy USB sticks smaller than 7200MiB. So people won't buy a small USB stick just for Tails only to realize that it's too small. They would have to recycle one for it to be too small. So telling them that their USB doesn't work for Tails might not be that bad...
  • After we distribute USB images, the hardest part of the installation process will be to learn how to start from the USB stick (secure boot, boot menu key, etc.). Luckily, once this is learned, doing the same on a different USB stick will be the same. The situation is quite different right now, where installing the USB stick is also a very hard part; but we're getting rid of this.

The 2 solutions that you came up with are good:

  • 1. Is to abort during boot (before reaching the desktop) and tell the user to install on a bigger USB stick. I guess this is relatively cheap to implement. With a bit more complexity, we could display this message between Tails Greeter and the desktop so we can have it translated.
  • 2. Is to let the user use this Tails, with a warning that they won't be able to upgrade and prevent them from creating a persistence.
    • The benefits for the user is that they will feel less immediate frustration and will be able to try out Tails.
    • The downsides for the user is that they will have to install again anyway after the new release is put out. They also won't be able to benefit from the UX goodness of persistence.
    • Solution 2 has immediate gratification but delayed and hidden pain, both in terms of UX and security (for both the lack of upgrades and persistence). Solution 1 is more upfront.
    • Solution 2 also seems more costly to implement as it might require patching Tails Upgrader (what is happening now on small USB sticks right now?) and the persistence setup.

So I think that the UX won't be super great either way and I'm not super convinced by the UX advantages of solution 1 over solution 2. Seeing that we have a very tight budget to work on this project, I would go for solution 1.

Another solution (solution 3) could be to improve solution 1 to propose people to clone to a new USB stick when we detect that the original one is smaller than 7200 MiB. Maybe starting Tails Installer as the only possible thing after Tails Greeter or something, plus a nice message. Note that if we're out of budget and solution 1 is super easy to implement, we could keep this solution 3 for later.

Also, should we add extra warning about the size of the USB stick during the installation steps? Right now we only mention the minimum size in the overview (eg. https://tails.boum.org/install/win/usb-overview/) but it might be worth it to mention it as well in the steps (eg. https://tails.boum.org/install/win/usb/).

#42 Updated by intrigeri 11 months ago

  • Solution 2 also seems more costly to implement as it might require patching Tails Upgrader (what is happening now on small USB sticks right now?)

Tails Upgrader does not care at all about the absolute size of the system partition: it only cares about whether there's enough space left there to download and apply the upgrade. So there's no need to patch it for solution 2, I think: the error message we display already covers that case.

So I think that the UX won't be super great either way and I'm not super convinced by the UX advantages of solution 1 over solution 2. Seeing that we have a very tight budget to work on this project, I would go for solution 1.

+1 (using a pretty big unit)

Another solution (solution 3) could be to improve solution 1 to propose people to clone to a new USB stick when we detect that the original one is smaller than 7200 MiB. Maybe starting Tails Installer as the only possible thing after Tails Greeter or something, plus a nice message. Note that if we're out of budget and solution 1 is super easy to implement, we could keep this solution 3 for later.

For now, to be on the safe side, let's assume we'll be out of budget even if we implement the cheapest possible version of solution 1.

Also, should we add extra warning about the size of the USB stick during the installation steps? Right now we only mention the minimum size in the overview (eg. https://tails.boum.org/install/win/usb-overview/) but it might be worth it to mention it as well in the steps (eg. https://tails.boum.org/install/win/usb/).

Yep. The time when we instruct the candidate user to "Plug the [first/second] USB stick in the computer" seems to be a fine place to remind those who missed this requirement on the overview about it :)
Which made me notice how small, low contrast (light grey on white) this info is there, which is reinforced by the fact it's below the biggest contiguous black-on-white surface on the page (the 2 USB sticks images), that catches the eye and somewhat tends to hide what's around it.

#43 Updated by segfault 11 months ago

  • Assignee changed from segfault to sajolida

I'm not happy about the decision to implement solution 1. IMO, this is a needless restriction, and there are legitimate use cases which don't require being able to upgrade, for example if you just want to try out or demonstrate Tails (think crypto party) or test a USB image in a VM before installing it on a USB stick.

So I tried to implement a possibility to continue the boot after the warning is shown, by dropping into a shell - the default behavior of the panic function in initramfs, which will continue boot if exited. But that's not easy to implement, because the panic function also quits plymouth, which we want to use to display the message to the user.

So this is what I currently implemented:
1. Display "Sorry, this device is too small to run Tails. Please use a device with at least 8 GiB." via plymouth (see screenshot)
2. Wait 10 seconds, then reboot

Instead of rebooting, we could also try sleeping in an endless loop.

#44 Updated by segfault 11 months ago

And FWIW, the implementation of 1 was not particularly easy, because I have to rebuild and boot Tails everytime I want to test a change in the initramfs script (4 times until now) - I think displaying a warning message in the notification area would have been easier.

#45 Updated by segfault 11 months ago

2. Wait 10 seconds, then reboot

The rebooting doesn't work currently, the boot continues after the 10 second wait. I want to know which behavior exactly we want before I spend more time trying to fix this.

#46 Updated by intrigeri 11 months ago

I'm not happy about the decision to implement solution 1. IMO, this is a needless restriction, and there are legitimate use cases which don't require being able to upgrade,

FTR I'm not super happy either and it's always sad to forcibly drop support for use cases (even those that we have explicitly stopped supporting a while ago) that we would like to support. But for me it's the best trade-off and your "needless restriction" phrasing makes it sound like you're missing an important part of the Tails picture, which is expected since you're not exposed to it much personally (yet :)

So let me explain where I'm coming from: we're already struggling with handling all the possible Tails USB sticks that we've let users create; it makes us do more work, and harder work; this impacts help desk and FT people who work with help desk (that is mostly me ATM). So in order to justify making that matrix bigger and this work even harder, the initial development cost is irrelevant: what matters is rather "are the additionally supported use cases worth the running cost of the extra & harder support & maintenance work?". If the answer is "no", then this restriction is not needless, it helps us keep Tails sustainable. So far I've not seen example use cases that come anywhere near convincing me the answer could be "yes" here but I may be missing something.

for example if you just want to try out or demonstrate Tails (think crypto party)

I'm not sure I get what user scenario is affected exactly. Do you mean "I go to a cryptoparty with an empty 4GB USB stick, the people who help me install Tails have no clue about the hardware requirements, so when I start my Tails for the first time I get a nasty error message in plymouth" and "I got to a cryptoparty to demonstrate Tails but I don't remember the hardware requirements so I bring only too small USB sticks"? Or something else?

or test a USB image in a VM before installing it on a USB stick.

I would like to understand this better:

  • Impacted people: who does that and why? Are we talking developers, power users, someone else?
  • Impact on these people: indeed, to do that one will need to dd the .img to a 8GB virtual disk as opposed to passing the .img directly to the virtualization software. Assuming one is not doing it for the first time, in libvirt it's 1 command (dd) vs. 1 command (install the .img with the right permissions to some place where libvirt can read it). So to me it does not make a big difference. But I'd be happy to hear why you feel it's harder.

And FWIW, the implementation of 1 was not particularly easy, because I have to rebuild and boot Tails everytime I want to test a change in the initramfs script (4 times until now)

Ouch. At this point I would probably remaster an existing USB image until I get something that works fine, and only then I would rebuild the whole thing.

I think displaying a warning message in the notification area would have been easier.

But then we would have had to add more code elsewhere specifically to handle this case. More code, more bugs, more maintenance cost, you know the drill :)

#47 Updated by intrigeri 11 months ago

segfault wrote:

I want to know which behavior exactly we want before I spend more time trying to fix this.

Keeping in mind that this is a corner case and we have more important topics to work on, IMO any behavior is OK as long as 1. we display the error message long enough for the user to read it (even if they went to brew coffee or something while Tails was starting); 2. it's hard to circumvent the error and let the boot process continue.

#48 Updated by intrigeri 11 months ago

intrigeri wrote:

segfault wrote:

I want to know which behavior exactly we want before I spend more time trying to fix this.

Keeping in mind that this is a corner case and we have more important topics to work on, IMO any behavior is OK as long as 1. we display the error message long enough for the user to read it (even if they went to brew coffee or something while Tails was starting); 2. it's hard to circumvent the error and let the boot process continue.

I've quickly implemented (on my own time) something that satisfies my own set of requirements and feels good enough to me. Regardless of what we decide, it seems to be an improvement vs. the initial half-working PoC, and unblocks the test suite work that will start tomorrow, but I won't be sad if we drop it. What I've implemented is that if the device is too small, this message is displayed: "Sorry, this device is too small to run Tails. Please use a device with at least 8 GiB. Press ENTER to shut down." and then pressing any key powers off the system. sajolida, good enough? Any improvement suggestion?

#49 Updated by intrigeri 11 months ago

  • Feature Branch changed from feature/15292-usb-image to feature/15292-repartition

#50 Updated by sajolida 11 months ago

  • Assignee changed from sajolida to segfault

Sorry but I'm still not convinced it is worth the extra work and I don't want to dive into this discussion again. I understand that segfault finds it's a needless restriction but my argument was about this extra work being worth it (and not useful in the abstract).

You're already 2 people work on this and I don't think it qualifies as UX core work, so I trust you to come up with a solution that's good enough without me.

And I'm really not against you putting so much care into corner cases.

#51 Updated by intrigeri 11 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (segfault)
  • QA Check deleted (Info Needed)

I think the solution I've implemented is good enough.

Also available in: Atom PDF