Project

General

Profile

Feature #6876

Feature #15281: Stack one single SquashFS diff when upgrading

Feature #15283: Implement the "one single SquashFS diff" scheme in Tails Upgrader

Have the incremental upgrade process use less RAM

Added by intrigeri over 5 years ago. Updated 3 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
03/07/2014
Due date:
% Done:

10%

Feature Branch:
feature/11131-endless-upgrade
Type of work:
Code
Blueprint:
Starter:
No
Affected tool:
Upgrader

2017Q3.stats (1.09 KB) sajolida, 09/25/2018 10:46 PM

2018Q1.stats (912 Bytes) sajolida, 09/25/2018 10:46 PM

2017Q4.stats (984 Bytes) sajolida, 09/25/2018 10:46 PM

2018Q2.stats (946 Bytes) sajolida, 09/25/2018 10:46 PM

2018Q3.stats (981 Bytes) sajolida, 09/25/2018 10:46 PM


Related issues

Related to Tails - Feature #6877: Research how the check for upgrades could require less RAM Rejected 03/07/2014
Related to Tails - Feature #10115: Add a splash screen to Tails persistence assistant Rejected 08/28/2015
Related to Tails - Feature #9373: Make tails-iuk support overlayfs In Progress 05/11/2015
Related to Tails - Feature #8415: Migrate from aufs to overlayfs In Progress 12/18/2014
Related to Tails - Bug #15955: whisperback_scripts/decrypt.sh doesn't handle Memory Hole Resolved 09/14/2018
Related to Tails - Bug #16015: Change hardware requirement for using Tails? Rejected 09/30/2018
Related to Tails - Feature #5502: Next time we bump RAM requirements: notify user at runtime if RAM requirements are not met Confirmed
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed 03/22/2019

Associated revisions

Revision 10950ddf (diff)
Added by anonym over 1 year ago

Specify IUK format version 2.

In our quest for (almost) endless incremental upgrades we'll end up
with larger IUKs than before. It turns out Tails Upgrader's memory
requirement of 2x the size of the IUK becomes a problem on systems
with 2 GB of memory, so we want to decrease the factor to 1x.

We achieve this by making the IUK mountable, so we can freely copy
contents from there onto the system partition without the need to
extract it and bundled archives, as was needed with the previous
format.

Will-fix: #6876
Refs: #11131

History

#1 Updated by intrigeri over 5 years ago

Once Tails is based on Wheezy, porting the whole things from Moose to Moo might help, but perhaps there are lower-hanging fruits to pick first.

#2 Updated by intrigeri almost 4 years ago

Memory::Usage is now in sid, which should help.

#3 Updated by intrigeri almost 4 years ago

  • Related to Feature #10115: Add a splash screen to Tails persistence assistant added

#4 Updated by intrigeri almost 2 years ago

Idea: instead of storing files to deploy inside tarballs inside iuk tarball, we could store files to deploy in a SquashFS inside a SquashFS IUK, so we can mount a couple SquashFS'es instead of extracting archives recursively. Incidentally this would drop the "needs N times the size of an IUK to install it" requirement. Note that we need 2 levels of bundling as the top level needs control files for the update.

#5 Updated by anonym over 1 year ago

intrigeri wrote:

Idea: instead of storing files to deploy inside tarballs inside iuk tarball, we could store files to deploy in a SquashFS inside a SquashFS IUK, so we can mount a couple SquashFS'es instead of extracting archives recursively.

I love it! So we could use any format we can mount (e.g. raw disk image containing a FAT fs) but the compression of first layer would be nice, so SquashFS seems like a fine choice.

Note that we need 2 levels of bundling as the top level needs control files for the update.

Let's just store the things we would put in the second layer bundle in subdirectories (e.g. ./boot.tar.bz2./boot/; ./system.tar.bz2./system/). It's simpler and should result in better (or at least no worse than equal) compression.

#6 Updated by anonym over 1 year ago

  • Priority changed from Low to Normal

Without this, the increased memory requirements of #11131 will result in inability to upgrade with 2 GB of RAM.

#7 Updated by anonym over 1 year ago

#8 Updated by anonym over 1 year ago

  • Status changed from Confirmed to In Progress

#9 Updated by intrigeri over 1 year ago

#10 Updated by intrigeri over 1 year ago

  • Related to Feature #15281: Stack one single SquashFS diff when upgrading added

#11 Updated by intrigeri 11 months ago

  • Related to Feature #9373: Make tails-iuk support overlayfs added

#12 Updated by intrigeri 11 months ago

  • % Done changed from 0 to 10
  • Feature Branch set to feature/11131-endless-upgrade

The corresponding design doc changes were made on feature/11131-endless-upgrade (tails.git).

IMO we should ship this in Tails 4.0 along with #15281 and #8415, to reduce the number of times we break automatic upgrades.

#13 Updated by intrigeri 11 months ago

  • Subject changed from Research how the incremental upgrade process could use less RAM to Have the incremental upgrade process use less RAM
  • Type of work changed from Research to Code

#14 Updated by u 11 months ago

#15 Updated by u 11 months ago

  • Target version set to SponsorT_2016_Internal

Setting target version accordingly.

#16 Updated by u 11 months ago

  • Target version changed from SponsorT_2016_Internal to Tails_4.0

Correct target version…

#17 Updated by intrigeri 11 months ago

  • Target version deleted (Tails_4.0)

u wrote:

Setting target version accordingly.

I think this is slightly premature. This may transitively block #8415 but 1. only if we care about 2GB systems in this context, which we did not decide yet; 2. #8415 was added on our 2018 roadmap only because someone was ready to do it on their volunteer time, which did not happen, so I'd rather not have this on my 4.0 plate until we update the status of this big pile of inter-related tickets at the summit. Spoiler alert: I'll propose they're explicitly added to the FT plate and we stop waiting for volunteer time to make things happen :)

#18 Updated by u 11 months ago

Ok, I was referring to what you said in #note-14.

#19 Updated by intrigeri 10 months ago

  • Blocks Feature #15392: Core work 2018Q2 → 2018Q3: User experience added

#20 Updated by intrigeri 10 months ago

  • Assignee changed from intrigeri to sajolida
  • Target version set to Tails_3.10.1
  • QA Check set to Info Needed

Dear sajolida, how bad would it be if as a consequence of #15281, we stopped supporting automatic upgrades on systems with only 2GB of RAM? Perhaps it's worth computing stats about RAM installed on systems sending WhisperBack reports (shout if you need help cooking a regexp).

Depending on the answer, we'll block on this (or not) for #15281, which itself blocks #8415, that the FT would like to complete in 2018Q4.

#21 Updated by intrigeri 10 months ago

intrigeri wrote:

Perhaps it's worth computing stats about RAM installed on systems sending WhisperBack reports (shout if you need help cooking a regexp).

Actually we already have internal.git:stats/whisperback_scripts/ram.pl :)

#22 Updated by sajolida 10 months ago

  • Assignee changed from sajolida to intrigeri

These are the reports I could get (before being hit by #15955):

 298 2017Q3.ram
 257 2017Q4.ram
  46 2018Q1.ram
   0 2018Q2.ram
 601 total
 - count:              600
 - variance:           27979743180995.6
 - standard deviation: 5289588.18633319
 - mean:                       7.0G
 - per-quartiles:
   * min:                      1.8G
   * 25th percentile:          3.8G
   * 50th percentile (median): 5.8G
   * 75th percentile:          7.8G
   * max:                      32G
 - frequency distribution:
   *   8.8%                          under 2.1G
   *   0.2% between 2.1G and 2.4G
   *   0.3% between 2.4G and 2.7G
   *   6.0% between 2.7G and 3.0G
   *   0.3% between 3.0G and 3.3G
   *   3.0% between 3.3G and 3.6G
   *  21.0% between 3.6G and 3.9G
   *   8.5% between 3.9G and 4.2G
   *   0.2% between 4.2G and 5.1G
   *   0.3% between 5.1G and 5.4G
   *   0.7% between 5.4G and 5.7G
   *   3.0% between 5.7G and 6.0G
   *   0.7% between 6.0G and 6.8G
   *   0.2% between 6.8G and 7.1G
   *   0.8% between 7.1G and 7.4G
   *   6.0% between 7.4G and 7.7G
   *  23.2% between 7.7G and 8.0G
   *   0.2% between 8.0G and 9.8G
   *   0.2% between 9.8G and 12G
   *   0.5% between 12G and 12G
   *   1.2% between 12G and 12G
   *   0.5% between 12G and 16G
   *   0.5% between 16G and 16G
   *  12.0% between 16G and 16G
   *   0.3% between 16G and 16G
   *   0.2% between 16G and 23G
   *   0.3% between 23G and 24G
   *   1.0% between 24G and 32G

Requesting 3GB might break upgrades for 15% of our audience. That's assuming that the hardware of people sending WhisperBack reports is representative from our general audience (or intended audience) but, if I were to guess I would say that they are a bit more tech-savvy, rich, and with better hardware.

15% seems a lot to me.

But I can compute stats for after 3.6.2 (April) once I solved #15955. As the progression itself is also an interesting data to see how fast this hardware is dying.

#23 Updated by intrigeri 10 months ago

These are the reports I could get (before being hit by #15955):

Thanks!

15% seems a lot to me.

Absolutely.

But I can compute stats for after 3.6.2 (April) once I solved #15955. As the progression itself is also an interesting data to see how fast this hardware is dying.

Indeed, next time it would be interesting to have separate stats per quarter, e.g. since 2017Q3.

#24 Updated by intrigeri 10 months ago

  • Target version changed from Tails_3.10.1 to Tails_3.11
  • QA Check deleted (Info Needed)

#25 Updated by sajolida 10 months ago

Here you go. It's weird that the number of people < 2GB goes back up in 2018Q2 so I added stats for what we have already in 2018Q3.

2017Q3

 - count:              297
 - variance:           21261724312256.8
 - standard deviation: 4611043.73350077
 - mean:                       6.4G
 - per-quartiles:
   * min:                      1.9G
   * 25th percentile:          3.8G
   * 50th percentile (median): 4.0G
   * 75th percentile:          7.8G
   * max:                      32G
 - frequency distribution:
   *  11.4%                          under 2.2G
   *   1.0% between 2.2G and 2.8G
   *   6.1% between 2.8G and 3.0G
   *   0.3% between 3.0G and 3.3G
   *   3.0% between 3.3G and 3.6G
   *  25.6% between 3.6G and 3.9G
   *   4.0% between 3.9G and 4.2G
   *   0.3% between 4.2G and 5.1G
   *   1.3% between 5.1G and 5.7G
   *   3.0% between 5.7G and 6.0G
   *   1.3% between 6.0G and 6.9G
   *   0.3% between 6.9G and 7.2G
   *   1.0% between 7.2G and 7.5G
   *   8.1% between 7.5G and 7.7G
   *  19.5% between 7.7G and 8.0G
   *   0.3% between 8.0G and 11G
   *   1.0% between 11G and 12G
   *   0.7% between 12G and 12G
   *   0.7% between 12G and 16G
   *   0.7% between 16G and 16G
   *   9.4% between 16G and 16G
   *   0.3% between 16G and 23G
   *   0.3% between 23G and 32G

2017Q4

 - count:              257
 - variance:           34981401127129.5
 - standard deviation: 5914507.68256577
 - mean:                       7.7G
 - per-quartiles:
   * min:                      1.8G
   * 25th percentile:          3.8G
   * 50th percentile (median): 7.5G
   * 75th percentile:          7.9G
   * max:                      32G
 - frequency distribution:
   *   5.8%                          under 2.1G
   *   0.4% between 2.1G and 2.4G
   *   0.4% between 2.4G and 2.7G
   *   5.4% between 2.7G and 3.0G
   *   0.4% between 3.0G and 3.3G
   *   3.1% between 3.3G and 3.6G
   *  19.1% between 3.6G and 3.9G
   *   9.3% between 3.9G and 4.2G
   *   0.8% between 4.2G and 5.4G
   *   3.1% between 5.4G and 6.0G
   *   0.8% between 6.0G and 7.4G
   *   6.6% between 7.4G and 7.7G
   *  23.3% between 7.7G and 8.0G
   *   0.4% between 8.0G and 9.8G
   *   1.6% between 9.8G and 12G
   *   0.4% between 12G and 16G
   *   0.4% between 16G and 16G
   *  16.7% between 16G and 16G
   *   1.9% between 16G and 32G

2018Q1

 - count:              284
 - variance:           33443635436891.2
 - standard deviation: 5783047.2449126
 - mean:                       7.3G
 - per-quartiles:
   * min:                      1.9G
   * 25th percentile:          3.8G
   * 50th percentile (median): 7.7G
   * 75th percentile:          7.8G
   * max:                      47G
 - frequency distribution:
   *   5.6%                          under 2.3G
   *   0.7% between 2.3G and 2.8G
   *   3.9% between 2.8G and 3.2G
   *   2.8% between 3.2G and 3.7G
   *  27.8% between 3.7G and 4.1G
   *   0.4% between 4.1G and 5.0G
   *   3.9% between 5.0G and 5.9G
   *   0.7% between 5.9G and 7.3G
   *   4.9% between 7.3G and 7.7G
   *  34.9% between 7.7G and 8.2G
   *   1.8% between 8.2G and 12G
   *   0.4% between 12G and 13G
   *   0.4% between 13G and 16G
   *   9.5% between 16G and 16G
   *   0.7% between 16G and 24G
   *   1.4% between 24G and 32G
   *   0.4% between 32G and 47G

2018Q2

 - count:              212
 - variance:           39142858705476.2
 - standard deviation: 6256425.39358348
 - mean:                       7.3G
 - per-quartiles:
   * min:                      1.9G
   * 25th percentile:          3.8G
   * 50th percentile (median): 5.8G
   * 75th percentile:          7.8G
   * max:                      47G
 - frequency distribution:
   *   9.0%                          under 2.3G
   *   1.4% between 2.3G and 2.8G
   *   2.8% between 2.8G and 3.2G
   *   5.7% between 3.2G and 3.7G
   *  27.8% between 3.7G and 4.1G
   *   0.5% between 4.1G and 5.5G
   *   3.8% between 5.5G and 5.9G
   *   0.9% between 5.9G and 6.8G
   *   0.5% between 6.8G and 7.3G
   *   4.2% between 7.3G and 7.7G
   *  25.9% between 7.7G and 8.2G
   *   3.8% between 8.2G and 12G
   *   0.5% between 12G and 16G
   *  10.4% between 16G and 16G
   *   0.9% between 16G and 24G
   *   1.4% between 24G and 32G
   *   0.5% between 32G and 47G

2018Q3

 - count:              167
 - variance:           38635795702148.2
 - standard deviation: 6215769.92030337
 - mean:                       8.6G
 - per-quartiles:
   * min:                      1.8G
   * 25th percentile:          3.9G
   * 50th percentile (median): 7.7G
   * 75th percentile:          16G
   * max:                      32G
 - frequency distribution:
   *   4.2%                          under 2.0G
   *   0.6% between 2.0G and 2.9G
   *   2.4% between 2.9G and 3.5G
   *   9.6% between 3.5G and 3.8G
   *  18.6% between 3.8G and 4.1G
   *   0.6% between 4.1G and 5.6G
   *   1.8% between 5.6G and 5.9G
   *   0.6% between 5.9G and 6.2G
   *   0.6% between 6.2G and 6.8G
   *   0.6% between 6.8G and 7.4G
   *   4.2% between 7.4G and 7.7G
   *  28.7% between 7.7G and 8.0G
   *   0.6% between 8.0G and 9.7G
   *   0.6% between 9.7G and 11G
   *   0.6% between 11G and 12G
   *   1.8% between 12G and 16G
   *  18.0% between 16G and 16G
   *   3.6% between 16G and 16G
   *   2.4% between 16G and 32G

#26 Updated by sajolida 10 months ago

  • Related to Bug #15955: whisperback_scripts/decrypt.sh doesn't handle Memory Hole added

#27 Updated by intrigeri 10 months ago

  • Assignee changed from intrigeri to sajolida
  • QA Check set to Info Needed

Here you go.

Thanks!

It's weird that the number of people < 2GB goes back up in 2018Q2

Well, on a sample of ~250 data points, +/- 12 reports over 3 months is sufficient to cause a 5% difference so it's barely statistically significant.

In passing, I've noticed a potential problem with our stats method: we're counting reports sent from Tails running in a VM. Our virtualization doc reads "allocate at least 2048 MB of RAM" so I suspect many VM users will have allocated exactly 2GB. This might skew our results because I guess that most such users could actually allocate 3GB of RAM to the VM just as well. If I added an option to the script that extracts these stats for ignoring reports sent from VMs, would you re-run it?

#28 Updated by sajolida 10 months ago

Your script takes as input a list of RAM amount so I don't think we can add to its logic to ignore virtual machines. But I wrote a oneline (for real!) to delete such reports from a set of decrypted reports. See bf2328a.

It's not 100% perfect but it should be good enough. Patches are welcome!

And I run again your script. See results in attachment.

Summary, amount of reports with less than 3 GB:

2017Q3 283 17%
2017Q4 243 5%
2018Q1 277 8%
2018Q2 204 11%
2018Q3 186 5%

So unfortunately, removing virtual machines doesn't really change the results...

#29 Updated by sajolida 10 months ago

  • Related to Bug #16015: Change hardware requirement for using Tails? added

#30 Updated by intrigeri 10 months ago

  • Assignee changed from sajolida to intrigeri
  • QA Check deleted (Info Needed)

sajolida wrote:

Your script takes as input a list of RAM amount so I don't think we can add to its logic to ignore virtual machines.

I don't understand why it would be hard to support a --exclude-vm option. Anyway:

So unfortunately, removing virtual machines doesn't really change the results...

OK. So I'll adjust metadata so this ticket blocks what it should. Thanks!

#31 Updated by intrigeri 10 months ago

  • Related to deleted (Feature #15281: Stack one single SquashFS diff when upgrading)

#32 Updated by intrigeri 10 months ago

  • Parent task set to #15281

#33 Updated by intrigeri 10 months ago

#34 Updated by sajolida 9 months ago

sajolida wrote:

Your script takes as input a list of RAM amount so I don't think we can add to its logic to ignore virtual machines.
I don't understand why it would be hard to support a --exclude-vm

option. Anyway:

Because your script takes as input a list of RAM amount, formatted like
that:

3787678987
4763456786
8545678987
8365456787

So at that point we can't know which amount of RAM is from a VM and
which is from real hardware. A solution would be to improve your script
to take a list of decrypt WhisperBack reports and have an --exclude-vm
option.

See also the comment on top of
internal.git:stats/whisperback_scripts/ram.pl.

So unfortunately, removing virtual machines doesn't really change the results...

OK. So I'll adjust metadata so this ticket blocks what it should. Thanks!

You're welcome :)

#35 Updated by sajolida 9 months ago

  • Blocks deleted (Feature #15392: Core work 2018Q2 → 2018Q3: User experience)

#36 Updated by intrigeri 9 months ago

  • Target version changed from Tails_3.11 to Tails_3.12

#37 Updated by intrigeri 9 months ago

  • Target version changed from Tails_3.12 to Tails_3.13

#38 Updated by sajolida 8 months ago

I found this awesome website reporting aggregated telemetry data from Firefox users:

https://data.firefox.com/

It includes hardware data and users with 2GB of RAM account for 11.4% of Firefox users.

#39 Updated by intrigeri 8 months ago

I found this awesome website reporting aggregated telemetry data from Firefox users:

https://data.firefox.com/

It includes hardware data and users with 2GB of RAM account for 11.4% of Firefox users.

Excellent, thanks :)

#40 Updated by intrigeri 8 months ago

  • Parent task changed from #15281 to #15283

#41 Updated by intrigeri 8 months ago

#42 Updated by intrigeri 8 months ago

#43 Updated by sajolida 6 months ago

  • Related to Feature #5502: Next time we bump RAM requirements: notify user at runtime if RAM requirements are not met added

#44 Updated by intrigeri 6 months ago

  • Target version changed from Tails_3.13 to 2019

#45 Updated by intrigeri 6 months ago

#46 Updated by intrigeri 6 months ago

#47 Updated by intrigeri 4 months ago

  • Assignee deleted (intrigeri)

#48 Updated by intrigeri 3 months ago

  • Assignee set to intrigeri

Most of the work will happen in tails-iuk so it would be rather wasteful to ask anyone else to do it.

Also available in: Atom PDF