Project

General

Profile

Feature #9373

Feature #8415: Migrate from aufs to overlayfs

Make tails-iuk support overlayfs

Added by intrigeri over 4 years ago. Updated about 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Installation
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
iuk:feature/9373-overlayfs
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Upgrader

Description

The IUK creation process and its test suite both rely on aufs now. Whenever we'll be ready to use overlayfs instead everywhere else, these two pieces of code will need to support overlayfs too. A first step could be to make this configurable, support both, so that there's less of a flag day.


Related issues

Related to Tails - Feature #6876: Have the incremental upgrade process use less RAM Resolved
Related to Tails - Feature #17262: Make the build of overlayfs-based IUKs reproducible Resolved
Blocked by Tails - Feature #8083: Fix automatic upgrades on Jessie Resolved 10/12/2014
Blocks Tails - Feature #12106: Adjust test suite to overlayfs In Progress 01/02/2017
Blocked by Tails - Feature #8473: Add support to live-boot to support multiple read-only lower layers with overlayfs Resolved 12/21/2014
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed
Blocked by Tails - Feature #17152: Port tails-iuk to a more lightweight set of dependencies Resolved

Associated revisions

Revision 0569f78c (diff)
Added by intrigeri 2 months ago

Release process: don't delete the (upcoming) overlayfs-based test IUK (refs: #9373)

Revision 425206fa (diff)
Added by intrigeri 2 months ago

Test suite: use an overlayfs-based test IUK (refs: #9373, #12106)

I did not create nor upload that IUK yet, nor did I generate the corresponding
UDFs and their signatures.

Revision 7cb37108 (diff)
Added by intrigeri 2 months ago

Add UDFs for 1.0~testoverlayfs (refs: #9373, #12106)

Without signatures for now as I can't generate them at the moment.

Revision 716470c5 (diff)
Added by intrigeri 2 months ago

Add UDFs for 1.1~testoverlayfs (refs: #9373, #12106)

Without signatures for now as I can't generate them at the moment.

Revision 585f412f (diff)
Added by intrigeri about 2 months ago

Test suite: verify that directory deletion via SquashFS diff is applied (refs: #9373)

Revision 3ba1676c (diff)
Added by intrigeri about 2 months ago

Test suite: update expected checksum of Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk and sign the corresponding UDFs (refs: #9373)

Revision 27a79ba4 (diff)
Added by intrigeri about 2 months ago

Test suite: update expected image (refs: #9373)

Revision d79bb63e (diff)
Added by intrigeri about 2 months ago

Test suite: adjust to test IUK (refs: #9373)

Revision 82ac56b6 (diff)
Added by intrigeri about 2 months ago

Test suite: sign the 1.1~testoverlayfs UDFs (refs: #9373)

History

#1 Updated by intrigeri over 4 years ago

  • Blocked by Feature #8083: Fix automatic upgrades on Jessie added

#2 Updated by intrigeri over 4 years ago

  • Target version set to Sustainability_M1

#3 Updated by intrigeri over 4 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Feature Branch set to iuk:feature/9373-overlayfs

Initial porting done, test suite passes.

Next step: try installing the resulting IUK on a Tails built with overlayfs support.

#4 Updated by sajolida over 4 years ago

  • Target version changed from Sustainability_M1 to 2016

#5 Updated by intrigeri over 3 years ago

  • Assignee deleted (intrigeri)
  • Target version deleted (2016)

Given we could do #10298 without migrating to overlayfs, we removed this from our roadmap at the summit this year.

#6 Updated by intrigeri about 3 years ago

Next step: try installing the resulting IUK on a Tails built with overlayfs support.

Now that we have automated tests for incremental upgrades, this will be easier; one "just" needs to:

  1. generate and upload an IUK that's essentially the same as the one we use currently in the test suite, except it must be generated with overlayfs
  2. add UDFs for 1.0~testoverlay and 1.1~testoverlay
  3. s/1.0~test/1.0~testoverlay/ in the test suite
  4. s/1.1~test/1.1~testoverlay/ in the test suite
  5. run the test suite

#7 Updated by intrigeri almost 3 years ago

Also, we'll probably need to bump the IUK format version number, and ensure we don't generate a Tails.module file that mixes aufs and overlayfs SquashFS diffs.

#8 Updated by u about 2 years ago

  • Assignee set to anonym

Looks like this is part of our roadmap. Assigning to anonym who also committed to work on the parent ticket #8415.

#9 Updated by intrigeri over 1 year ago

  • Related to Feature #6876: Have the incremental upgrade process use less RAM added

#10 Updated by intrigeri over 1 year ago

intrigeri wrote:

Also, we'll probably need to bump the IUK format version number, and ensure we don't generate a Tails.module file that mixes aufs and overlayfs SquashFS diffs.

In order to avoid breaking automatic upgrades between Tails N and N+1 more often than needed, ideally we should implement #6876 at the same time and release all this in Tails 4.0.

#11 Updated by intrigeri over 1 year ago

  • Assignee changed from anonym to intrigeri
  • Target version set to Tails_3.11

#12 Updated by intrigeri over 1 year ago

#13 Updated by intrigeri over 1 year ago

#14 Updated by intrigeri over 1 year ago

  • Blocked by Feature #8473: Add support to live-boot to support multiple read-only lower layers with overlayfs added

#15 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.11 to Tails_3.12

#16 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.12 to Tails_3.13

#17 Updated by intrigeri about 1 year ago

#18 Updated by intrigeri about 1 year ago

#19 Updated by intrigeri 12 months ago

  • Target version changed from Tails_3.13 to 2019

#20 Updated by intrigeri 12 months ago

#21 Updated by intrigeri 12 months ago

#22 Updated by intrigeri 5 months ago

  • Status changed from In Progress to Confirmed

(Not much progress lately. I'll work on this later this year or early 2020.)

#23 Updated by intrigeri 2 months ago

  • Blocked by Feature #17152: Port tails-iuk to a more lightweight set of dependencies added

#24 Updated by intrigeri 2 months ago

  • Status changed from Confirmed to In Progress

Refreshed the branch, merged #17152 into it, and brought back aufs support ⇒ the iuk.git test suite passes both with UNION_TYPE=aufs (which is the default) and UNION_TYPE=overlayfs.

Next step: have the tails.git automated test suite exercise upgrading with an overlayfs-based IUK.

#25 Updated by intrigeri 2 months ago

intrigeri wrote:

Next step: have the tails.git automated test suite exercise upgrading with an overlayfs-based IUK.

I've done everything I could do today on this front:

  • prepared Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk, that uses overlayfs, using tails-create-iuk + manual post-processing to make it closer to Tails_amd64_1.0~test_to_1.1~test.iuk
  • manually installed this IUK with tails-install-iuk, rebooted, confirmed the changes are applied as expected
  • uploaded this IUK to rsync.lizard
  • updated the test suite to use this overlayfs-based IUK
  • pushed UDFs for 1.0~testoverlayfs and 1.1~testoverlayfs to our master branch

Next steps:

  1. sign these 4 new UDFs
  2. run Scenario: Upgrading Tails with Tails Upgrader through an incremental upgrade

#26 Updated by intrigeri about 2 months ago

  • Target version changed from 2019 to Tails_4.5

The milestone for this is "March 2020" so the current goal is to have this ready in time for 4.5~rc1. We might manage to complete #8415 earlier, we'll see.

#27 Updated by intrigeri about 2 months ago

  • Priority changed from Elevated to High

#28 Updated by intrigeri about 2 months ago

Another next step: ensure the test suites (in both iuk.git and tails.git) exercise whiteouts:

  • for non-directory deletion (in theory, managed by overlayfs with a xattr to make it opaque)
  • for directory deletion (managed by overlayfs with character devices or xattr)

#29 Updated by intrigeri about 2 months ago

intrigeri wrote:

Another next step: ensure the test suites (in both iuk.git and tails.git) exercise whiteouts:

  • for non-directory deletion (in theory, managed by overlayfs with a xattr to make it opaque)

Interestingly, in my tests I see whiteouts (character device with 0/0 device number) for deleted directories too. I could not find any such xattr usage in the overlayfs-based IUKs I've generated.

Anyway, this is exercised already.

  • in tails.git: Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk deletes /usr/share/common-licenses/BSD
  • in iuk.git via Scenario: install an IUK that should delete some files
  • for directory deletion (managed by overlayfs with character devices or xattr)

AFAICT we have no test for this yet. I should write one, at least in iuk.git.

#30 Updated by intrigeri about 2 months ago

intrigeri wrote:

intrigeri wrote:

Another next step: ensure the test suites (in both iuk.git and tails.git) exercise whiteouts:

  • for non-directory deletion (in theory, managed by overlayfs with a xattr to make it opaque)

Interestingly, in my tests I see whiteouts (character device with 0/0 device number) for deleted directories too. I could not find any such xattr usage in the overlayfs-based IUKs I've generated.

Anyway, this is exercised already:

  • in tails.git: Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk deletes /usr/share/common-licenses/BSD

Confirmed → good enough.

  • in iuk.git via Scenario: install an IUK that should delete some files

Scratch that, this scenario only tests deletion of files in the system partition, not via the SquashFS diff. So while adding a test that deletes stuff via the SquashFS diff, I should test both directory and non-directory deletion.

#31 Updated by intrigeri about 2 months ago

Tails_amd64_1.0~testoverlayfs_to_1.1~testoverlayfs.iuk (just uploaded, not on the mirrors yet) now deletes a whole directory (recursively), on top of deleting a single regular file (which it did already). I'm adjusting the corresponding test in tails.git so it verifies that this new change is indeed applied upon upgrade.

So I'm back to next step: run Scenario: Upgrading Tails with Tails Upgrader through an incremental upgrade.

#32 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)

intrigeri wrote:

So I'm back to next step: run Scenario: Upgrading Tails with Tails Upgrader through an incremental upgrade.

It passes on my machine.

I've verified that the test suite in iuk.git still passes both with UNION_TYPE=aufs (which is the default) and UNION_TYPE=overlayfs. I've also verified that setting $UNION_TYPE works correctly (unloading both aufs and overlay kernel modules before running the test suite and verifying that only the required one was loaded after it has run).

So I think we're good here!

Note to the reviewer: this branch includes #17152, which anonym reviewed already, so you can skip that part and compare this topic branch with "current master + #17152 merged in".

#33 Updated by intrigeri about 2 months ago

  • Related to Feature #17262: Make the build of overlayfs-based IUKs reproducible added

#34 Updated by intrigeri about 2 months ago

  • Status changed from Needs Validation to In Progress

#35 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Needs Validation

#36 Updated by segfault about 2 months ago

  • Assignee set to segfault

#37 Updated by segfault about 2 months ago

  • Status changed from Needs Validation to Resolved
  • Assignee deleted (segfault)

Also available in: Atom PDF