Ship a pre-compiled AppArmor binary cache
Context: decreasing Tails startup time significantly would help us achieve our "Make it easier to switch between a Tails contextual identity and another identity outside of Tails" strategic planning goal.
One of the main reasons why Tails takes so long to start, especially on systems with few and/or not super fast CPU cores, is that we compile the AppArmor policy during every single boot. This is kinda dumb given this compilation will produce the same result every time, which should make any engineer immediately suggest "well, let's cache this, mayyyyybe?".
Caching the resulting pre-compiled policy was not straightforward until Stretch, inclusive. But all the pieces we need to do that are finally in place in Buster so we can finally do that! :)
In theory one could implement this without waiting for Tails 4.0, by backporting src:apparmor from Buster to Stretch. As the maintainer of that package in Debian, I would recommend to keep this as a Tails-only backport and not upload it to stretch-backports, to avoid the cost of maintaining it plus its dependencies that are not satisfied in Stretch for 1 bonus year after we've lost interest in Stretch.
I'm interested in implementing this myself at some point but if someone else wants to do it earlier, by all means, be my guest: reassign this ticket to you before you start so we avoid duplicating work.
I'll stop using the blueprint as I'm making progress on the implementation side, so next updates will live here.
I think we need a
config/chroot_local-hooks/99-cache-AppArmor-policy hook that does:
apparmor_parser \ --write-cache \ --skip-kernel-load \ --verbose \ --add \ /etc/apparmor.d
Also, I've thought about adding a scenario in our test suite, to verify that the cache from the ISO/IMG is still valid. But my current understanding is that it has to be valid: the only difference between build and runtime environments is the kernel version, and with a fixed
features-file and the same parser, this should not affect the resulting binary cache at all.
Actually, I don't think this requires AppArmor from Buster: running the aforementioned command line in Tails 3.11, after moving away the cache files generated in
/etc/apparmor.d/cache/ at boot, generates the exact same binary cache files. The features file is the same at runtime as at build time, the features file is the same too, only the kernel version differs which should not matter with a fixed features file, so it seems that this could actually be implemented in Tails 3.x!
The only change in Buster is per-features-file (and possibly per-parser-version, if the parser changes the way it computes the hash of the features which is used as the name as the cache dir) cache directories under
/var/cache/apparmor, but that should not be relevant for Tails.
- % Done changed from 0 to 20
- Feature Branch changed from feature/16138-pre-compiled-AppArmor-cache to feature/16138-pre-compiled-AppArmor-cache-buster
- Measured value: time from pressing ENTER in the bootloader menu, until the Greeter is ready.
- On laptops: from the same (pretty fast) USB drive, which was installed from the USB image using GNOME Disks, and booted once before the benchmarks (to avoid the repartitioning process from tainting results)
- In VM: from ISO, (pretty fast) NVMe underlying storage
- The two branches were built simultaneously in order to get the same APT snapshots and thus comparable results.
defaultcompbuild option was used in all cases, for more real-world-like results.
- The SquashFS sort file was not updated for 4.19.0-2 (used on the Stretch branches) nor for Buster (where it's probably very wrong). If this inaccuracy taints the results significantly, it'll probably be in favor of the baseline, i.e. without a pre-compiled AppArmor policy: it'll slow down I/O and will make the relative impact of CPU-bound tasks on boot time smaller; and compiling the AppArmor policy is CPU-bound. So worst case, this inaccuracy will make the product of this ticket appear worse than it would be in actual Tails releases, i.e. conservative results.
- My reasoning about the SquashFS sort file implies that slower USB drives would benefit less from the improvements brought by this ticket.
- I only did each test once, assuming that the duration of a Tails offline boot with the same system on the same hardware does not vary too much across runs.
Results on Stretch:
- feature/16390-linux-4.19.0-2+force-all-tests at 65f2805fe00cec7978723ff101ded91394d43bc4, i.e. what's needed to make current devel branch build successfully
- AppArmor enabled (baseline):
- ThinkPad X200: 90 seconds
- HP EliteBook 840 G1: 66 seconds
- AppArmor disabled (
- ThinkPad X200: 70 seconds
- HP EliteBook 840 G1: 50 seconds
- AppArmor enabled (baseline):
- feature/16138-pre-compiled-AppArmor-cache (only change on top of the 3.x baseline branch: pre-compiled AppArmor policy):
- ThinkPad X200: 90 seconds
- HP EliteBook 840 G1: 57 seconds
Conclusions for Stretch:
- This implementation does not work at all on Stretch: the cache is regenerated. It could probably made to work but if it works out of the box on Buster as I expect, I'll target that.
- The AppArmor policy compilation accounts for ~20-25% of the "boot to Greeter" duration. That's… a lot. This confirms that this ticket may be a very interesting low-hanging fruit.
Results on Buster:
- feature/buster at 7f0fddc8138d7cebc208759b74ff6365b2a9d0a2 (baseline)
- ThinkPad X200: 140 seconds
- HP EliteBook 840 G1: 104 seconds
- VM with fast underlying storage: 43 seconds
- feature/16138-pre-compiled-AppArmor-cache-buster at 9acec481f724698e5ca8391cdbea9f259974a0cb (only change on top of feature/buster: pre-compiled AppArmor policy):
- ThinkPad X200: 96 seconds
- HP EliteBook 840 G1: 62 seconds
- VM with fast underlying storage: 37 seconds
Conclusions for Buster:
- This optimization makes the boot 14-40% faster on Buster. Sweet!
- Current feature/buster boots much slower than current Tails 3.x. Even if the improvements brought by this ticket cancel it almost entirely, that seems to be a serious problem. I was looking to improve things here, not merely to avoid them getting worse! ⇒ This needs a dedicated ticket.
Measuring boot time from pressing Enter in the syslinux menu to the desktop being ready (including desktop icons), after pressing Enter quickly in the Greeter:
- HP EliteBook 840G1: 71 seconds
- ThinkPad X200: 122 seconds
- feature/buster (after updating SquashFS sort file on #16393)
- HP EliteBook 840G1: 75 seconds (+6% compared to 3.13.1)
- ThinkPad X200: 148 seconds (+21% compared to 3.13.1)
- HP EliteBook 840G1: 65 seconds (-8% compared to 3.13.1, -13% compared to feature/buster)
- ThinkPad X200: 116 seconds (-5% compared to 3.13.1, -22% compared to feature/buster)
So this branch cancels Buster's boot slowdown and improves things a little bit compared to 3.13.1.