Project

General

Profile

Bug #15146

Feature #8415: Migrate from aufs to overlayfs

Make memory erasure feature compatible with overlayfs

Added by intrigeri about 2 years ago. Updated about 2 months ago.

Status:
In Progress
Priority:
High
Assignee:
-
Category:
-
Target version:
Start date:
01/03/2018
Due date:
% Done:

0%

Feature Branch:
feature/8415-overlayfs+force-all-tests
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

The "Erasure of the overlayfs read-write branch on shutdown" test scenario fails. But interestingly, "Tails erases memory on DVD boot medium removal: overlayfs read-write branch" passes. Looks like live-boot behaves slightly differently on overlayfs than on aufs, which seems to explain why mountpoints are not visible and can't be unmounted and thus cleaned.

Design doc: https://tails.boum.org/contribute/design/memory_erasure/

aufs.png View (45.8 KB) intrigeri, 11/23/2019 05:45 PM

overlayfs.png View (44.1 KB) intrigeri, 11/23/2019 05:45 PM

exposed-upper-dir-log-2.png View (34.1 KB) intrigeri, 11/24/2019 06:42 AM

exposed-upper-dir-log-1.png View (47 KB) intrigeri, 11/24/2019 06:42 AM

exposed-upper-dir-log-3.png View (34.5 KB) intrigeri, 11/24/2019 06:42 AM

exposed-upper-dir-log-4.png View (58.4 KB) intrigeri, 11/24/2019 06:42 AM

exposed-upper-dir-log-5.png View (34.3 KB) intrigeri, 11/24/2019 06:42 AM

exposed-upper-dir-log-6.png View (35.9 KB) intrigeri, 11/24/2019 06:42 AM


Related issues

Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision f3ec8583 (diff)
Added by intrigeri 2 months ago

Let live-boot expose its /live/overlay as /lib/live/mount/overlay (refs: #15146)

/live/overlay (in the context of the initramfs) is the tmpfs
where the read-write branch of our union rootfs lives.

With aufs, this call to umount failed, and then live-boot would run:

mount -o move /live/overlay /root/lib/live/mount/overlay

As a result, this tmpfs mount was visible outside of the initramfs,
and our initramfs-pre-shutdown-hook could unmount it on shutdown,
which ensured the data stored in there was cleaned from memory.

But with overlayfs, for some reason this call to mount succeeds, even though the
overlayfs upper layer (/live/overlay/rw) is stored in this filesystem, which
shows that this tmpfs is still mounted. As a result, this tmpfs is not
visible anymore, and cannot be unmounted on shutdown, so the data stored
in there remains in memory, available to cold-boot attackers.

Let's not unmount this tmpfs and go back to the same behavior we had
with aufs.

This will probably require bringing back some AppArmor-related automated
tests, that were removed on the #8415 branch precisely because live-boot
did not expose the overlay branch:

12404eb883d8b68ab07242734d9a14a3d07f91ba
c6541323cb70f7c2d1af77bb407bef0d3d3e554a
a822c25f13c9673ffb39fb623e72c4d2894b112e
4ee7e8d041a29d78e15a1bbca4b5dba1c1e09296

Revision 97117dc9 (diff)
Added by intrigeri 2 months ago

WIP: on shutdown, empty the tmpfs read-write branch of the overlayfs mounted on / (refs: #15146)

Do this when stopping systemd-update-utmp.service, which is one of the last
things that happen before systemd umounts the filesystems.

WIP because ideally this would live in a dedicated unit,
properly ordered to stop at about that time.

Revision 3cdeadfe (diff)
Added by intrigeri 2 months ago

Let live-boot expose its /live/overlay as /lib/live/mount/overlay (refs: #15146)

/live/overlay (in the context of the initramfs) is the tmpfs
where the read-write branch of our union rootfs lives.

With aufs, this call to umount failed, and then live-boot would run:

mount -o move /live/overlay /root/lib/live/mount/overlay

As a result, this tmpfs mount was visible outside of the initramfs,
and our initramfs-pre-shutdown-hook could unmount it on shutdown,
which ensured the data stored in there was cleaned from memory.

But with overlayfs, for some reason this call to umount succeeds, even though the
overlayfs upper layer (/live/overlay/rw) is stored in this filesystem, which
shows that this tmpfs is still mounted. As a result, this tmpfs is not
visible anymore, and cannot be unmounted on shutdown, so the data stored
in there remains in memory, available to cold-boot attackers.

Let's not unmount this tmpfs and go back to the same behavior we had
with aufs.

This will probably require bringing back some AppArmor-related automated
tests, that were removed on the #8415 branch precisely because live-boot
did not expose the overlay branch:

12404eb883d8b68ab07242734d9a14a3d07f91ba
c6541323cb70f7c2d1af77bb407bef0d3d3e554a
a822c25f13c9673ffb39fb623e72c4d2894b112e
4ee7e8d041a29d78e15a1bbca4b5dba1c1e09296

Revision cb5921e4 (diff)
Added by segfault 2 months ago

On shutdown, empty the tmpfs read-write branch of the overlayfs mounted on / (refs: #15146)

Revision d604c714 (diff)
Added by segfault 2 months ago

Enable tails-remove-overlayfs-dirs.service (refs: #15146)

Revision 547b1560 (diff)
Added by segfault 2 months ago

Fix DefaultDependencies=no missing in tails-remove-overlayfs-dirs.service (refs: #15146)

Revision 007edc20 (diff)
Added by segfault 2 months ago

Fix tails-remove-overlayfs-dirs.service (refs: #15146)

The service is not stopped without Conflicts=shutdown.target.

History

#1 Updated by intrigeri about 2 years ago

  • Target version set to 2018

#2 Updated by intrigeri about 2 years ago

anonym, you committed to handle the parent ticket this year (and I'm supposed to be the reviewer) but right now I fancy playing a bit with this ticket. If it becomes less fun I'll reassign to you.

#3 Updated by intrigeri over 1 year ago

  • Related to Bug #15477: Consider upgrading to live-boot 1:20180328+ added

#4 Updated by intrigeri over 1 year ago

  • Related to deleted (Bug #15477: Consider upgrading to live-boot 1:20180328+)

#5 Updated by intrigeri over 1 year ago

  • Blocked by Bug #15477: Consider upgrading to live-boot 1:20180328+ added

#6 Updated by intrigeri over 1 year ago

I don't think it's worth spending time on this before #15477 is done: perhaps the revamping of how mountpoints are managed by live-boot will fix this problem, perhaps it'll make it harder to solve, but in any case better do the overlayfs-specific work only once, after that other migration is done.

#7 Updated by intrigeri over 1 year ago

  • Target version changed from 2018 to Tails_3.11

#8 Updated by intrigeri over 1 year ago

#9 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.11 to Tails_3.12

#10 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.12 to Tails_3.13

#11 Updated by intrigeri about 1 year ago

#12 Updated by intrigeri about 1 year ago

#13 Updated by intrigeri 12 months ago

  • Target version changed from Tails_3.13 to 2019

#14 Updated by intrigeri 12 months ago

#15 Updated by intrigeri 12 months ago

#16 Updated by intrigeri 10 months ago

  • Assignee deleted (intrigeri)

#17 Updated by intrigeri 2 months ago

  • Description updated (diff)

#18 Updated by intrigeri 2 months ago

  • File aufs.png View added
  • File overlayfs.png View added
  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Feature Branch set to feature/8415-overlayfs+force-all-tests

#19 Updated by intrigeri 2 months ago

To test all this, I:

  • on the kernel command line, replace quiet with debug nosplash
  • run sudo touch /run/initramfs/tails_shutdown_debugging within Tails before shutting down

On the topic branch, the tmpfs where the read-write upper dir of the overlayfs lives is exposed as /lib/live/mount/overlay; it is successfully unmounted on shutdown before we switch back to the initramfs; same for the read-only branch (SquashFS). Then, the overlayfs itself is successfully unmounted in the initrd (initramfs-pre-shutdown-hook), so from my reading of the logs everything looks good, modulo some actions we take in the initrd are not necessary anymore (as some stuff has already been unmounted) and fail. But still, the known pattern we write in "Scenario: Erasure of the overlayfs read-write branch on shutdown" is not cleaned up.

I'm not sure what to do at this point. Hunches:

  • It could be that some of the seemingly successful unmount operations are done in a lazy way, and the tmpfs we want to clean up is still seen as internally mounted by the kernel. It could be interesting to run lsof or similar at various points of the shutdown process.
  • It could be that deleting the content of the tmpfs is necessary to clean the memory (we did rm -rf /mnt/live/overlay/* for aufs but it's a no-op here because the tmpfs has already been unmounted so there's nothing visible to delete there anymore at that point). We could try to do that in a systemd unit that stops just before Unmounting /lib/live/mount/overlay. I'll try that.

#20 Updated by intrigeri 2 months ago

  • Assignee deleted (intrigeri)
  • Feature Branch changed from wip/bugfix/15146-overlayfs-memory-erasure to bugfix/15146-overlayfs-memory-erasure

Looks like I got a PoC fix that works! I'm not sure if I should polish this or switch to #15281.

#21 Updated by intrigeri 2 months ago

Also, if we go the way my PoC branch does, we need to bring back some tests, see f3ec8583001a9a90861eede18ac5f330a560cad2.

#22 Updated by intrigeri 2 months ago

  • Blocked by deleted (Bug #15477: Consider upgrading to live-boot 1:20180328+)

#23 Updated by intrigeri 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee set to intrigeri

The affected scenario now passes on the topic branch! Next steps: check that the reintroduced AppArmor-related test steps still make sense in overlayfs-world, verify that they pass, and finally merge into #8415.

#24 Updated by intrigeri 2 months ago

  • Status changed from Needs Validation to In Progress
  • Assignee deleted (intrigeri)

Next steps: check that the reintroduced AppArmor-related test steps still make sense in overlayfs-world,

They needed adjustments on the test suite side. They also showed that our AppArmor configuration needed to be adapted to the changes brought by 3cdeadfeadc28d93aed5356c5780b97dac75dc19. I did both, let's see what Jenkins thinks.

Also, unrelated to AppArmor, lots of the commands we have in initramfs-pre-shutdown-hook are now obsolete and thus fail loudly. IMO we should clean up this script: otherwise, next time we debug a problem in this area, we may get confused by all the error messages unneeded and failing operations trigger ⇒ back to "In Progress".

#25 Updated by intrigeri about 2 months ago

intrigeri wrote:

Next steps: check that the reintroduced AppArmor-related test steps still make sense in overlayfs-world,

They needed adjustments on the test suite side. They also showed that our AppArmor configuration needed to be adapted to the changes brought by 3cdeadfeadc28d93aed5356c5780b97dac75dc19. I did both, let's see what Jenkins thinks.

This now looks good on Jenkins so I've merged this branch into #8415, to make it easier to analyze test suite results there. But I'm leaving this ticket in progress as more work is needed on this front IMO:

Also, unrelated to AppArmor, lots of the commands we have in initramfs-pre-shutdown-hook are now obsolete and thus fail loudly. IMO we should clean up this script: otherwise, next time we debug a problem in this area, we may get confused by all the error messages unneeded and failing operations trigger ⇒ back to "In Progress".

#26 Updated by intrigeri about 2 months ago

  • Feature Branch changed from bugfix/15146-overlayfs-memory-erasure to feature/8415-overlayfs+force-all-tests

#27 Updated by intrigeri about 2 months ago

  • Target version changed from 2019 to Tails_4.5

#28 Updated by intrigeri about 2 months ago

  • Priority changed from Normal to High

Also available in: Atom PDF