Project

General

Profile

Feature #11131

Endless automatic upgrades

Added by anonym over 3 years ago. Updated over 1 year ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Installation
Target version:
-
Start date:
01/05/2015
Due date:
% Done:

100%

Feature Branch:
feature/11131-endless-upgrade, iuk:feature/11131-endless-upgrade, perl5lib:feature/11131-endless-upgrade
Type of work:
Research
Starter:
Affected tool:

Description

Currently automatic upgrades have the limitation that only some N number of them can be performed in a row before the system partition runs out of space and automatic upgrades cannot be done further, but a full, manual upgrade is needed. Ideally, automatic upgrades should be possible forever, which will improve the UX and eliminate the recurring need to manually verify the Tails ISO when doing these unavoidable full upgrades.


Subtasks

Feature #8534: Merge incremental upgrades to allow endless upgradingRejected


Related issues

Related to Tails - Feature #7499: Extend the upgrader to allow full (self) upgrade Confirmed 07/06/2014
Related to Tails - Feature #10478: Do statistics on upgrade paths Resolved 11/04/2015
Related to Tails - Feature #11627: Consider updating the default system partition's size Resolved 08/10/2016
Related to Tails - Bug #12392: Correct documentation regarding 'Check for updates' Resolved 03/20/2017
Related to Tails - Feature #15281: Stack one single SquashFS diff when upgrading In Progress 03/07/2014

Associated revisions

Revision 2de7788e (diff)
Added by intrigeri over 1 year ago

Start updating design doc for endless upgrades (refs: #11131)

Revision 1b558814 (diff)
Added by intrigeri over 1 year ago

Endless upgrades: update design doc wrt. new field added to the UDF file format (refs: #11131)

This is a backwards-compatible change so we don't need to bump the version of
the URL scheme / UDF file format.

Revision 10950ddf (diff)
Added by anonym over 1 year ago

Specify IUK format version 2.

In our quest for (almost) endless incremental upgrades we'll end up
with larger IUKs than before. It turns out Tails Upgrader's memory
requirement of 2x the size of the IUK becomes a problem on systems
with 2 GB of memory, so we want to decrease the factor to 1x.

We achieve this by making the IUK mountable, so we can freely copy
contents from there onto the system partition without the need to
extract it and bundled archives, as was needed with the previous
format.

Will-fix: #6876
Refs: #11131

Revision 2763439d (diff)
Added by intrigeri over 1 year ago

Test suite: add a note about a new test that's needed (refs: #11131).

History

#1 Updated by anonym over 3 years ago

  • Related to Feature #7499: Extend the upgrader to allow full (self) upgrade added

#2 Updated by anonym over 3 years ago

  • Blueprint set to https://tails.boum.org/blueprint/Endless_upgrades/

Added a blueprint.

#3 Updated by intrigeri over 3 years ago

#4 Updated by intrigeri almost 3 years ago

  • Related to Feature #11627: Consider updating the default system partition's size added

#5 Updated by Dr_Whax almost 3 years ago

  • Blocked by Feature #11679: Rethink the installation process and upgrade process added

#6 Updated by anonym over 2 years ago

  • Assignee set to intrigeri
  • QA Check set to Info Needed

While working on #11857 it occurred to me that we can provide more or less endless upgrades until Tails is rebased on a new version of Debian (for IUK size reasons) simply by generating IUKs for each release back to the first one, and extending Tails Upgrader to be able to replace IUKs so that we never stack IUKs again (so we should probably drop the "I" in "IUK" :)), but always install one that upgrades to the current version. It's a pretty obvious idea, so my realization was more about that the number of IUKs that we have to generate aren't that insane, and that they can be generated pretty much in parallel with the rest of the release process without much effort.

Example: let's say I once installed Tails 2.2 on a USB stick, and at some point updated it to 2.3 with the 2.2_to_2.3 IUK. Let's say I am thrown in prison for 6 months and when I get out the current version is 2.7.1. Then Tails Upgrader will download the 2.2_to_2.7.1 IUK and store it in addition to 2.2_to_2.3, and update Tails.modules to only list 2.2_to_2.7.1. The next time we want to upgrade (= next time we will remount the drive in read-write mode), the 2.2_to_2.3 IUK is cleaned up to make space for the new IUK (therefore the Upgrader can discount the space occupied by old IUKs when doing the disk space check).

Let's look at some IUK sizes:

569M Tails_i386_2.0_to_2.9.iuk (took 15 minutes to generate on my system, BTW) 
328M Tails_i386_2.6_to_2.9.iuk
198M Tails_i386_2.7_to_2.9.iuk
175M Tails_i386_2.7.1_to_2.9.iuk

Unexpectedly they grow significantly throught the Debian release cycle, but I have a hunch we can expect 2.0_to_XX size to not increase much beyond the current value -- after all, it generally is the same suspect packages that are upgraded throughout a Debian release.

The IUK size is involved in at least three concerns:
  • Disk space needs on mirrors: After 2.0 we have currently (including 2.9) released 11 non-rc/beta/alpha Tails upgrades, of which 3 were emergency releases. We have only three more planned releases in 2.x, so excluding emergency releases our mirrors would need 14. Let's go crazy and assume 6 more emergencies, so we're at 20 for the whole 2.x series. If IUKs average 600 MiB, that's 20*600 MiB = ~11 GiB of disk space needed when we are in the end of a Debian release cycle. [Right before I was gonna post this I realized that my data isn't perfect since 2.0 was released 6+ months into the Jessie cycle, so probably the number will be higher than 20. But we could always have a release in the middle of the Debian cycle were we reset what is considered as the first release. Or we just decide to support IUK upgrades only for the 20 last releases, which may mean that those that installed Tails early in a series might have to do a full installation closer to the end of it -- that's still a lot better than what we have now.]
  • Disk space needs on the Tails system partition: we need to be able to store two large IUKs. The worst case would be when e.g. 2.0 was the the first ISO we installed from, and that we have upgraded to 3.0-2 and want to upgrade to 3.0-1, because those two IUKs are reasonably those that will occupy the most space. Tails 2.0 uses about 1.1 GiB of the 2.5 GiB system partition. That means we'd be ok if those two IUKs are about (2.5 - 1.1)/2 = 0.7 GiB. That seems quite plausible for the 2.x series, so we could try it. And for 3.0 we can increase the system partitions size so we have a better marigin.
  • Bandwidth needs of the RM. The amount of data that the RM must upload will be 11 + 1 = 12 GiB when we are in the end of the Debian release cycle. This might be problematic for RMs other than me at the moment. This will only be a blocker until we have deterministic builds + IUK generation, since the same data then can be exactly replicated on lizard.

All this seems doable to me, especially if we introduce it in Tails 3.0 where we will have the possibility to increase the system partition's size. Of course, this would only be a temporary solution -- we still want #7499 to be part of #11679, I think.

Given that it probably is you and me, intrigeri, that have the best understanding of this domain, I'd like to hear what you have to say about this.

#7 Updated by anonym over 2 years ago

anonym wrote:

[Right before I was gonna post this I realized that my data isn't perfect since 2.0 was released 6+ months into the Jessie cycle, so probably the number will be higher than 20. But we could always have a release in the middle of the Debian cycle were we reset what is considered as the first release. Or we just decide to support IUK upgrades only for the 20 last releases, which may mean that those that installed Tails early in a series might have to do a full installation closer to the end of it -- that's still a lot better than what we have now.]

Another strategy would be to "synchronize" each n (e.g. ~10) releases. Example: let's say 2.0 is the "first" release. Let's say the next n releases puts us at 2.5 -- at that time we'll have IUKs 2.0_to_${i} for each of them. From then on, the next n releases will provide IUKs from 2.5, i.e. 2.5_to_${i}. And so on. So, to support users installing an early 2.x, like 2.0, all the way until 2.x's EOL, which is the worst case vs system partition disk needs, we would need to stack a few IUKs, i.e. floor(N/10) + 1 where N is the total number of releases during 2.x.

So, this way we'd never need to generate more than n IUKs per release. And I think we can assume that the average IUK size will go down quite a bit. On the mirrors we'll have a choice: when we release a new version that is the new IUK synchronization point we would keep all IUKs upgrading to the new release for a while. The choice I'm talking about is in how long we keep them. Either we keep them until the EOL, so we always support even the oldest version (but this will require a lot of space on the mirrors); or we optimize for users that upgrade semi-regularly, so we kill all those old IUKs after a few releases. At the very least I think we can kill IUKs for synchronization point number i when we release synchronization point number i + 2. That way the mirrors will only ever store up to 2 * n IUKs.

So, yeah, we have a lot of parameters to play with. :)

#8 Updated by anonym over 2 years ago

anonym wrote:

Another strategy would be to "synchronize" each n (e.g. ~10) releases.

Or when the IUK grows too big. The point is the concept of the synchronization points.

So, yeah, we have a lot of parameters to play with. :)

Indeed!

#9 Updated by intrigeri over 2 years ago

While working on #11857 it occurred to me that we can provide more or less endless upgrades until Tails is rebased on a new version of Debian (for IUK size reasons) simply by generating IUKs for each release back to the first one, and extending Tails Upgrader to be able to replace IUKs so that we never stack IUKs again (so we should probably drop the "I" in "IUK" :)), but always install one that upgrades to the current version.

Sound like a good idea, that's worth thinking about!

Example: let's say I once installed Tails 2.2 on a USB stick, and at some point updated it to 2.3 with the 2.2_to_2.3 IUK. Let's say I am thrown in prison for 6 months and when I get out the current version is 2.7.1. Then Tails Upgrader will download the 2.2_to_2.7.1 IUK and store it in addition to 2.2_to_2.3, and update Tails.modules to only list 2.2_to_2.7.1. The next time we want to upgrade (= next time we will remount the drive in read-write mode), the 2.2_to_2.3 IUK is cleaned up to make space for the new IUK (therefore the Upgrader can discount the space occupied by old IUKs when doing the disk space check).

Tails Upgrader could actually delete (unlink) the 2.2_to_2.3 IUK while installing the 2.2_to_2.7.1 one: it's in use so disk space won't be freed (and the system will keep working), but it might make it easier to decide which IUK can be deleted (i.e. exactly the ones we're going to remove from Tails.modules).

Unexpectedly they grow significantly throught the Debian release cycle, but I have a hunch we can expect 2.0_to_XX size to not increase much beyond the current value -- after all, it generally is the same suspect packages that are upgraded throughout a Debian release.

I suspect that a substantial part of the growth is caused by our own changes (e.g. adding packages, moving stuff around) rather than by package upgrades, but it's not important at this point.

The IUK size is involved in at least three concerns:

[...]

  • Disk space needs on the Tails system partition: we need to be able to store two large IUKs. The worst case would be when e.g. 2.0 was the the first ISO we installed from, and that we have upgraded to 3.0-2 and want to upgrade to 3.0-1, because those two IUKs are reasonably those that will occupy the most space. Tails 2.0 uses about 1.1 GiB of the 2.5 GiB system partition. That means we'd be ok if those two IUKs are about (2.5 - 1.1)/2 = 0.7 GiB. That seems quite plausible for the 2.x series, so we could try it. And for 3.0 we can increase the system partitions size so we have a better marigin.

Tails Upgrader needs more free space than that: it requires 3 times the size of the IUK it's going to install to be free on the system partition before it can download and install it. This might be fixable given enough time+energy, but let's start by reasoning about the cheapest option, that requires the smallest amount of changes to Tails Upgrader ⇒ time to redo the maths.

#10 Updated by intrigeri over 2 years ago

  • Assignee changed from intrigeri to anonym

Another strategy would be to "synchronize" each n (e.g. ~10) releases.

Or when the IUK grows too big. The point is the concept of the synchronization points.

I like it. I'd like to see the maths wrt. disk space and memory requirements, computed on the actual 2.x cycle until now.

Also, Tails Upgrader requires twice the size of the to-be-installed IUK as free memory before it downloads it. It would be good to take this into account and do the maths for that too.

#11 Updated by intrigeri over 2 years ago

  • Status changed from Confirmed to In Progress

anonym, can you please sum up this idea on the blueprint?

#12 Updated by anonym over 2 years ago

  • QA Check changed from Info Needed to Dev Needed

intrigeri wrote:

anonym, can you please sum up this idea on the blueprint?

I'll do that next time I work on this topic.

#13 Updated by u almost 2 years ago

  • Related to Bug #12392: Correct documentation regarding 'Check for updates' added

#14 Updated by intrigeri over 1 year ago

  • Related to Feature #15277: Update our survey of non-NIH system upgrade solutions added

#15 Updated by intrigeri over 1 year ago

Added to the blueprint my research about non-NIH endless automatic upgrades solutions. They have plenty of advantages over the one proposed on this ticket but are not fully ready yet and will require lots of work on our side => I'll come back to it on #15277. Meanwhile, let's see if the solution proposed on this ticket can be implemented quickly and cheaply: if it can, it could be worth it; if it can't, then perhaps we should skip this temporary solution and instead focus our efforts directly on the nicer, long-term one.

#16 Updated by anonym over 1 year ago

intrigeri wrote:

Another strategy would be to "synchronize" each n (e.g. ~10) releases.

Or when the IUK grows too big. The point is the concept of the synchronization points.

I like it. I'd like to see the maths wrt. disk space and memory requirements, computed on the actual 2.x cycle until now.

So in the end there was 14 releases for 2.x. I make IUKs from 2.0 to the last few releases:

678M Tails_i386_2.0_to_2.10.iuk
682M Tails_i386_2.0_to_2.11.iuk
634M Tails_i386_2.0_to_2.12.iuk

So, actually the last IUK is the smallest, and the most space consuming upgrade will be 2.10 to 2.11. So:

  • The 2.0 installation itself uses 1082M, so upgraded to 2.10 the total usage is 1082M + 678M = 1760M.
  • Due to the Upgrader requiring 3x the size of the IUK on the Tails system partition, the actual requirement is 1760M + 3*682M = 3806M of free disk space. That wouldn't have worked during the 2.x times, when the system partition was 2.5 GB, and it barely would work with our current 4.0 GB partition.
  • When both IUKs are present at the same time (i.e. before rebooting after having applied the upgrade) the disk usage is 1760M + 682M = 2442M.
  • After the old IUK is purged the disk usage is down to 2442M - 678M = 1764M.

Also, Tails Upgrader requires twice the size of the to-be-installed IUK as free memory before it downloads it. It would be good to take this into account and do the maths for that too.

So 2*682M = 1364M of RAM.

#17 Updated by anonym over 1 year ago

intrigeri also pointed out that we probably do not want to support upgrading very old Tails installations because our signing key probably will have been updated in the meantime, and/or some critical update to Tails Upgrader (e.g. UDF version). For instance, in the example in #11131#note-6, our Tails 2.2 (with no upgrades, so it was manually installed as 2.2) has a too old key (098f578586f1e49b6cc63431f661b9c20151916c) so it would not be able to verify the UDF letting us know about the existence of Tails 2.7.1 and its IUK. So while we provide a 2.2_to_2.7.1 IUK, it can actually only be used by those that update regularly, e.g. someone that installed Tails 2.2, and installed each automatic upgrade.

Conclusion: let's stick with out current "you can at most skip one planned release" so it in effect means "you must upgrade once every three months".

#18 Updated by intrigeri over 1 year ago

#19 Updated by intrigeri over 1 year ago

We have to implement a part of this idea in order to migrate from aufs to overlayfs; see #8415#note-35 for details.

#20 Updated by intrigeri over 1 year ago

anonym wrote:

Conclusion: let's stick with out current "you can at most skip one planned release" so it in effect means "you must upgrade once every three months".

I think we could actually do a little bit better without making the cost noticeably higher: we could support upgrading for any older releases as long as it has the current signing key and a version of the Upgrader that is compatible (i.e. there's been no major change or bugfix in it since).

#21 Updated by anonym over 1 year ago

  • Related to Feature #15279: Refresh Tails signing key before each upgrade check added

#22 Updated by intrigeri over 1 year ago

  • Feature Branch set to feature/11131-endless-upgrade, iuk:feature/11131-endless-upgrade

#23 Updated by intrigeri over 1 year ago

  • Feature Branch changed from feature/11131-endless-upgrade, iuk:feature/11131-endless-upgrade to feature/11131-endless-upgrade, iuk:feature/11131-endless-upgrade, perl5lib:feature/11131-endless-upgrade

#24 Updated by anonym over 1 year ago

  • Blocked by Feature #6876: Have the incremental upgrade process use less RAM added

#25 Updated by anonym over 1 year ago

  • Related to Feature #15281: Stack one single SquashFS diff when upgrading added

#26 Updated by intrigeri over 1 year ago

  • Blocked by deleted (Feature #6876: Have the incremental upgrade process use less RAM)

#27 Updated by intrigeri over 1 year ago

  • Blocked by deleted (Feature #11679: Rethink the installation process and upgrade process)

#28 Updated by intrigeri over 1 year ago

  • Related to deleted (Feature #15277: Update our survey of non-NIH system upgrade solutions)

#29 Updated by intrigeri over 1 year ago

  • Related to deleted (Feature #15279: Refresh Tails signing key before each upgrade check)

#30 Updated by intrigeri over 1 year ago

#31 Updated by intrigeri over 1 year ago

Deprecated for now in favour of #15281. We'll reconsider after #15277.

#32 Updated by intrigeri over 1 year ago

  • Status changed from In Progress to Rejected
  • Assignee deleted (anonym)

Also available in: Atom PDF