Project

General

Profile

Bug #10239

Tails Upgrader sometimes make device unbootable

Added by emmapeel over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
-
Target version:
Start date:
09/23/2015
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Upgrader

Description

It looks like the script that triggers the automatic upgrade is failing to see when there is not enough space available on the Tails partition.

Description of the error:
-------------------------

Instead of receiving the message 'Not enough memory available', the script tries to write the new download, and it ends up with a corrupted boot folder.

What should happen instead:
---------------------------
User should be advised to Update manually instead:

https://tails.boum.org/doc/first_steps/upgrade/#manual


Related issues

Related to Tails - Bug #10235: Document how to fix a Tails device that got broken for too many automatic updates Resolved 09/23/2015
Related to Tails - Bug #10347: Troubleshooting for automatic upgrades that fail to restart Confirmed 10/07/2015
Duplicated by Tails - Bug #10292: Tails 1.6 regression - hangs on Begin: Running /scripts/init-premount Duplicate 09/27/2015

History

#1 Updated by emmapeel over 3 years ago

  • Related to Bug #10235: Document how to fix a Tails device that got broken for too many automatic updates added

#2 Updated by intrigeri over 3 years ago

  • Assignee set to emmapeel
  • Priority changed from Normal to Elevated
  • QA Check changed from Dev Needed to Info Needed

Do you have more info about space available before the upgrade?

#3 Updated by intrigeri over 3 years ago

Actually, I need the same debugging info as https://tails.boum.org/doc/upgrade/error/install/.

#4 Updated by intrigeri over 3 years ago

FTR I have been reported one issue with very similar symptoms after a 1.5->1.5.1 automatic upgrade, and disk space was totally fine. I'm wondering if we've maybe jumped to conclusions a bit too fast regarding the disk space thing. I'm wondering if we might have some missing umount or sync in the reboot process.

#5 Updated by emmapeel over 3 years ago

I am afraid I still haven't got the debugging info.

Users I helped commented on:

Doing many automatic upgrades previously
Fixing the problem with a manual upgrade

I keep asking for the debuging info. I think I will try to recreate this problem today installing Tails 1.2.1 and updating from there. I had one similar report for Tails 1.5.1 but I cannot find it :S

#6 Updated by goupille over 3 years ago

we've got a user that get the exact same error message trying to install Tails 1.6 on a brand new USB stick (16 GB Verbatim Store'n Go V3) with Tails installer (using "clone and install")

#7 Updated by intrigeri over 3 years ago

we've got a user that get the exact same error message trying to install Tails 1.6 on
a brand new USB stick (16 GB Verbatim Store'n Go V3) with Tails installer (using
"clone and install")

This seems to be unrelated. May you please file a dedicated ticket about it?

#8 Updated by goupille over 3 years ago

ok, I'm waiting for more information from the user before opening a new ticket... I figured that, given the doubts about what causes the issue, and the similar error message, it worthed to be said here.

#9 Updated by intrigeri over 3 years ago

I figured that, given the doubts about what causes the issue, and the similar error message, it worthed to be said here.

Sure :)

#10 Updated by intrigeri over 3 years ago

Let's say the problem is "Tails Upgrader sometimes corrupts the device to the point it does not boot anymore". We still don't know whether this bug is caused by lack of storage space. I think the easiest way to prove or disprove that hypothesis would be to install Tails 1.5.1 on a USB stick with Tails Installer, then artificially make it space-constrained by creating a large file in the system partition, and then trying to go through the automatic upgrade and reproduce the problem. One likely will have to test different file sizes and bisect to find a reproducer (if any).

emmapeel, do you feel comfortable trying that? Do you think you can do it by October 15, or to find other people at frontdesk to do so?

If one can't find a good correlation then I think we should look elsewhere. E.g. one could mount stuff with the "sync" option in remount_liveos_rw, and maybe call "sync" manually in the run method in Tails/IUK/Install.pm. If someone finds a reliable way to reproduce the bug, I'm curious to see if any of these hacks help at all.

#11 Updated by intrigeri over 3 years ago

  • Subject changed from Tails 'Automatic upgrade' script fails to realise there is not enough space on device to Tails Upgrader sometimes make device unbootable
  • Status changed from New to Confirmed

(It seems that this has been reported multiple times, seen it on Reddit too, so that's confirmed.)

#12 Updated by intrigeri over 3 years ago

  • Duplicated by Bug #10292: Tails 1.6 regression - hangs on Begin: Running /scripts/init-premount added

#13 Updated by anonym over 3 years ago

I just had a thought: so we require 3x the size of the compressed IUK of free disk space on the boot partition. Hence, if the IUK is compressed so well that it extracts to > 3x the compressed size (also taking the kernel/initrd swap into account), there's an issue. Do we do any other check, later for the actual uncompressed size vs free disk space?

Maybe our UDF:s should record the exact amount of space needed (i.e. uncompressed squashfs file + difference of the potential kernel/initrd swap etc), and then we check against that?

#14 Updated by intrigeri over 3 years ago

  • Assignee changed from emmapeel to anonym

intrigeri wrote:

Let's say the problem is "Tails Upgrader sometimes corrupts the device to the point it does not boot anymore". We still don't know whether this bug is caused by lack of storage space. I think the easiest way to prove or disprove that hypothesis would be to install Tails 1.5.1 on a USB stick with Tails Installer, then artificially make it space-constrained by creating a large file in the system partition, and then trying to go through the automatic upgrade and reproduce the problem. One likely will have to test different file sizes and bisect to find a reproducer (if any).

emmapeel, do you feel comfortable trying that? Do you think you can do it by October 15, or to find other people at frontdesk to do so?

Looks like emmapeel / frontdesk won't do that. anonym, may you please take it as part of the time allocated for fixing regressions?

#15 Updated by anonym over 3 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Info Needed to Dev Needed

intrigeri wrote:

intrigeri wrote:

Let's say the problem is "Tails Upgrader sometimes corrupts the device to the point it does not boot anymore". We still don't know whether this bug is caused by lack of storage space. I think the easiest way to prove or disprove that hypothesis would be to install Tails 1.5.1 on a USB stick with Tails Installer, then artificially make it space-constrained by creating a large file in the system partition, and then trying to go through the automatic upgrade and reproduce the problem. One likely will have to test different file sizes and bisect to find a reproducer (if any).

emmapeel, do you feel comfortable trying that? Do you think you can do it by October 15, or to find other people at frontdesk to do so?

Looks like emmapeel / frontdesk won't do that. anonym, may you please take it as part of the time allocated for fixing regressions?

I actually did this weeks ago and told emmapeel about it. From my tests, Tails Upgrader correctly shows the error if < 3 times the IUK's size of disk space is available.

Also my comment #10239#note-13 seems invalid. For instance, the 1.5.1_to_1.6 IUK is 106 MiB compressed and 109 MiB decompressed, so the 3x requirement is definitely enough (well, overkill :)).

#16 Updated by intrigeri over 3 years ago

Hi,

(Cc'ing frontdesk since there's a question for them at the bottom, and emmapeel didn't answer my most recent question, so it seems we need to get other frontdesk people on board.)

I actually did this weeks ago and told emmapeel about it. From my tests, Tails Upgrader correctly shows the error if < 3 times the IUK's size of disk space is available.

OK, so the initial hasty guesses that lead to the branch for #10235 to be merged were wrong, and the documentation that was merged into master is incorrect. anonym, emmapeel, may you please fix this somehow?

Now, that's too bad: if that test was buggy, it would probably be easy to fix, but now I've no idea why we suddenly have this problem on 1.5->1.5.1 and 1.5.1->1.6, while we've never seen it before (?). So:

  • Dear frontdesk, can you confirm we didn't hear about such problems earlier? Please reply on the ticket.
  • In any case, we should add the sync hacks I suggested on #10239#note-10. I can provide an initial untested patch but I won't have time to test it in time for 1.7, so that'll land onto our dear RM's bugfixing plate. The thing is, to be able to check whether it fixes the problem or not, we need a reproducer, and nobody could provide any yet.

Cheers!

#17 Updated by intrigeri over 3 years ago

Also, now that the "fails to detect when there's not enough disk space is available" option was discarded, the only piece of debugging info I from frontdesk (apart of the other question I've asked today) is, after such a failed upgrade:

  • the checksum of the 1.6.iuk file;
  • the directory listing of the live directory on the system partiton.

#18 Updated by intrigeri over 3 years ago

  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 10
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:

In any case, we should add the sync hacks I suggested on #10239#note-10. I can provide an initial untested patch but I won't have time to test it in time for 1.7, so that'll land onto our dear RM's bugfixing plate. The thing is, to be able to check whether it fixes the problem or not, we need a reproducer, and nobody could provide any yet.

That's now bugfix/10239-more-sync in the upgrader's Git repo. Given the info I have I don't see what else I can do for now => please review and merge (I suggest using config/chroot_local-patches/ to save time now, and only do the upstream release + package update thing for 1.7~rc1), but likely we should not close this ticket before we see if it improves things (that is, once people upgrade to 1.8).

#19 Updated by anonym over 3 years ago

  • QA Check deleted (Ready for QA)

intrigeri wrote:

intrigeri wrote:

In any case, we should add the sync hacks I suggested on #10239#note-10. I can provide an initial untested patch but I won't have time to test it in time for 1.7, so that'll land onto our dear RM's bugfixing plate. The thing is, to be able to check whether it fixes the problem or not, we need a reproducer, and nobody could provide any yet.

That's now bugfix/10239-more-sync in the upgrader's Git repo. Given the info I have I don't see what else I can do for now => please review and merge (I suggest using config/chroot_local-patches/ to save time now, and only do the upstream release + package update thing for 1.7~rc1), but likely we should not close this ticket before we see if it improves things (that is, once people upgrade to 1.8).

I've merged that into the tails-iuk shipped in Tails 1.7~rc1. What now? I guess we'll just wait an see...

#20 Updated by anonym over 3 years ago

  • Target version changed from Tails_1.7 to Tails_1.8

#21 Updated by intrigeri over 3 years ago

  • Assignee changed from anonym to sajolida
  • QA Check set to Info Needed

After the post-1.6 (?) frenzy, were there many such reports again? We (at least I) didn't hear from frontdesk nor from other support people since a while.

#22 Updated by sajolida over 3 years ago

  • Assignee changed from sajolida to mercedes508

Assigning to frontdesk for more info. I'm not following their communications enough anymore.

#23 Updated by emmapeel over 3 years ago

After the release of Tails 1.8 there were no reports about this.

Some users received the 'not enough space on the disk to do an automatic upgrade' and wrote to frontdesk scared because their disk was big enough and they were used to upgrade before, but they didn't got a broken Tails.

#24 Updated by intrigeri over 3 years ago

  • Assignee changed from mercedes508 to anonym
  • Target version changed from Tails_1.8 to Tails_2.0
  • QA Check changed from Info Needed to Ready for QA

After the release of Tails 1.8 there were no reports about this.

Thanks!

anonym, I suggest we close this bug as resolved (and we have another ticket that tracks the huge performance regression introduced by what seems to be a fix for this one). OK?

#25 Updated by anonym over 3 years ago

  • Status changed from Confirmed to Resolved
  • Assignee deleted (anonym)
  • % Done changed from 10 to 100
  • QA Check changed from Ready for QA to Pass

intrigeri wrote:

anonym, I suggest we close this bug as resolved (and we have another ticket that tracks the huge performance regression introduced by what seems to be a fix for this one). OK?

ACK.

#26 Updated by intrigeri over 3 years ago

  • Related to Bug #10347: Troubleshooting for automatic upgrades that fail to restart added

Also available in: Atom PDF