Project

General

Profile

Bug #17320

Kernel Panic with some Macbook Pro

Added by goupille about 1 month ago. Updated 15 days ago.

Status:
Resolved
Priority:
Elevated
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

two different users with three different apple computers (one unidentified, on macbook pro 8,1, and one macbook pro 9,1) have reported this problem with Tails 4.1:

the boot process stops with the following kernel panic error message :

[    1.894460] Initramfs unpacking failed: invalid magic at start of
compressed archive
[    1.991176] Kernel panic - not syncing: VFS: Unable to mount root
fs on unknown-block(0,0)
[    1.991229] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.3.0-2-amd64
#1 Debian 5.3.9-2
[    1.991270] Hardware name: Apple Inc.
MacBookPro8,1/Mac-94245B3640C91C81, BIOS 87.0.0.0.0 06/13/2019
[    1.991319] Call Trace:
[    1.991342]  dump_stack+0x5c/0x80
[    1.991365]  panic+0x101/0x2d7
[    1.991388]  mount_block_root+0x2c4/0x2e8
[    1.991415]  prepare_namespace+0x136/0x16c
[    1.991441]  kernel_init_freeable+0x1ff/0x20a
[    1.991468]  ? rest_init+0xaa/0xaa
[    1.991491]  kernel_init+0xa/0x106
[    1.991512]  ret_from_fork+0x35/0x40
[    1.991569] Kernel Offset: 0x10400000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    1.991633] ---[ end Kernel panic - not syncing: VFS: Unable to
mount root fs on unknown-block(0,0) ]---

one of the pictures show this message instead of the initramfs one:

List of all partitions:
No filesystem could mount root, tried:

Bug report: 6c589ea7443b70b3592ec58f3f8137ca


Related issues

Duplicated by Tails - Feature #17349: Unable to start Duplicate 12/14/2019
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision 1c9e7a0a (diff)
Added by CyrilBrulebois about 1 month ago

Drop net drivers from the initramfs to shrink initramfs size drastically.

Hitting the 32 MiB mark might be the reason why so many Mac machines
can't boot 4.1 while they could boot 4.0 (refs: #17320).

Revision acadc5f2 (diff)
Added by CyrilBrulebois about 1 month ago

Only allow up to (but excluding) 32 MiB for initramfs (refs: #17320).

History

#1 Updated by goupille about 1 month ago

just resent the second report (the ticket number is in the subject line)

#2 Updated by goupille about 1 month ago

I just resent you a third report (Apple Macbook air 4,1 mid 2011)

#3 Updated by goupille about 1 month ago

more reports :

macbook pro 2012 -> fowarded
macbook air 2012 -> Bug report: e5e2e75f8dccb894ec897184ccc295b4 (this user says that a macbook air 2013 is not affected)
macbook pro mid 2012 -> forwarded

then I stopped forwarding the reports (I can do that if needed) but it seems that a lot of people are affected...

#4 Updated by goupille about 1 month ago

  • Priority changed from Normal to Elevated

I'm setting the priority on this ticket to Elevated, given the number of user reports

#5 Updated by goupille about 1 month ago

  • Assignee changed from intrigeri to CyrilBrulebois

#6 Updated by CyrilBrulebois about 1 month ago

Haven't spotted anything obvious on the Linux kernel side at this point. And I couldn't reproduce the issue with some other hardware (tails-installer'd 4.0 image upgraded to 4.1 after enabling persistence, or tails-installer'd 4.1 image).

I'm currently wondering whether some microcode update might be responsible for this apparently rather hardware-specific (even if affecting a wide range of models) issue. I couldn't spot anything obvious in the Debian BTS though.

I think it might make sense to prepare unofficial images: one with the intel-microcode package downgraded to its earlier version (in Tails 4.0), and one with the linux-image-* package downgraded to its earlier version (in Tails 4.0). Possibly a third one with both packages downgraded, just in case.

Would submitters be happy to test such images? In which case, would they prefer ISO and/or IMG images?

#7 Updated by CyrilBrulebois about 1 month ago

I've pushed three branches to git:

  1. hack/17320-revert-microcode
  2. hack/17320-revert-linux
  3. hack/17320-revert-both-linux-and-microcode

The relevant diffs of .packages files, compared to 4.1, are respectively:

1. with hack/17320-revert-microcode:

-intel-microcode    3.20191112.1~deb10u1
+intel-microcode    3.20190618.1
-libnss3:amd64    2:3.42.1-1+deb10u1
+libnss3:amd64    2:3.42.1-1+deb10u2

2. with hack/17320-revert-linux:

-libnss3:amd64    2:3.42.1-1+deb10u1
+libnss3:amd64    2:3.42.1-1+deb10u2
-linux-image-5.3.0-2-amd64    5.3.9-2
+linux-image-5.3.0-trunk-amd64    5.3.2-1~exp1

3. with hack/17320-revert-both-linux-and-microcode:

-intel-microcode    3.20191112.1~deb10u1
+intel-microcode    3.20190618.1
-libnss3:amd64    2:3.42.1-1+deb10u1
+libnss3:amd64    2:3.42.1-1+deb10u2
-linux-image-5.3.0-2-amd64      5.3.9-2
+linux-image-5.3.0-trunk-amd64    5.3.2-1~exp1

(I didn't try to get rid of the libnss3 difference.)

All resulting .img files were tested by dropping them on USB devices with gnome-disks, and can boot properly on a Thinkpad X230. I didn't try to run the test suite, and didn't test much more than booting to the desktop. But at least they aren't obviously broken, so I'd be happy to have test results from users affected by this MacBookPro booting issue.

I'm thinking about sharing the resulting files, signed with my ckb@riseup.net key, on my company's webserver and maybe via Torrent files if people are more comfortable with this way of grabbing files. Of course, the latter would likely result in fewer peers than is usual for official releases.

#8 Updated by pushbox about 1 month ago

I can test the images on macbook pro.
Using balenaEtcher, so both formats should work.
I'm okay with downloading them from web or torrent.

#9 Updated by CyrilBrulebois about 1 month ago

Here we go: https://tails.debamax.com/ has ISO+IMG images for all 3 proposed branches, with detached signatures from my personal Tails key.

If users would have enough time/bandwidth/patience, I'd be happy to have results of Tails 4.1 image deployed “from scratch” (to rule out possible troubles due to upgrading from 4.0), and with the 3 images available on my company's webserver. I have no guarantee the issue is caused by the microcode update and/or the linux update, but those are my best guess at the moment…

Also please mention which MacBookPro you're testing with (e.g. 4,1 or 8,1, etc.).

Thanks already!

#10 Updated by pushbox about 1 month ago

MacBookPro10,1
All 3 of the branches booting up without any problem.
ISO and IMG format.

The official 4.1 build is not booting, I had to recheck it after the 6 successes.

#11 Updated by Jambalaya about 1 month ago

I just registered after finding this thread and it being my savior for Tails 4.1 release. I do not have a MacBook Pro, I have an iMac (iMac11,3) that could no longer boot Tails with 4.1, no other changes. Got kernel panic screen and saw a lot of people on Reddit had the same issue. I found my way back here and this thread, tried the above and can report - after having nothing to lose - that the “Revert both Linux and microcode” img fixed my issue. The “Revert Linux” img produced the same kernel panic. Thanks for your test builds and hope this data helps.

#12 Updated by goupille about 1 month ago

@CyrilBrulebois, given the reports we're receiving, it looks like the problem is fixed with hack/17320-revert-microcode and hack/17320-revert-both-linux-and-microcode. However hack/17320-revert-linux doesn't fixe the issue. I'm forwarding you the reports

#13 Updated by geb about 1 month ago

Hi,

Just tested on a MacBook Pro (13 pouces, mi-2009) (as presented on https://checkcoverage.apple.com/) / Macbook pro 5.5 (as show in mac os / linux kernel panic trace).

- 4.1 generated the same kernel panic message other reported (have pictures if needed).
- tails-amd64-hack_17320-revert-both-linux-and-microcode-4.1.1-20191208T1518Z-84f1e5957a.img works
- tails-amd64-hack_17320-revert-linux-4.1.1-20191208T1328Z-1af7c412ed.img do not work
- tails-amd64-hack_17320-revert-microcode-4.1.1-20191208T1411Z-927a0f6f85.img works

If needed I could ask the owner of the computer to do more tests later.

#14 Updated by intrigeri about 1 month ago

Thanks everyone who's been helping debug this problem!

FWIW here's a semi-random hunch (disclaimer: I have some fever).

The Initramfs unpacking failed: invalid magic at start of compressed archive message, combined with the fact only Apple hardware seems to be affected (as opposed to specific CPU models that may be installed on MacBooks and on PCs) makes me wonder if the microcode update is directly the culprit, or if it broke stuff indirectly.

For example, in 4.0, initrd.img is 33367476 bytes large, which is less than 32 MiB; while in 4.1, it is 33817076 bytes large, which is greater than 32 MiB. I find this suspicious and I would not be surprised if Apple's EFI firmware behaved in a different way than others.

I can't easily build/download @CyrilBrulebois's test images today but I would be very curious to know if the test images that fix the problem bring back the initramfs size under 32 MiB :) If they do, it would be interesting to test images that have the updated microcodes but an initramfs smaller than 32 MiB.

Also, it would be nice if affected users could try adding the dis_ucode_ldr boot option, which disables the microcode loader.

#15 Updated by intrigeri about 1 month ago

#16 Updated by CyrilBrulebois about 1 month ago

  • Status changed from Confirmed to In Progress

#17 Updated by CyrilBrulebois about 1 month ago

Thanks everyone for your input indeed!

I had some troubles reconciling some findings, namely the absence of obvious reports on the Debian BTS regarding possible regressions on all Mac hardware after the intel-microcode update, and even if an upload to unstable happened yesterday mentioning some rollback, there wasn't anything obvious regarding our rather global problem.

Looking back at the size of our initramfs in test images:

-r--r--r-- 1 root root 33353972 Dec  8 15:09 tails-amd64-hack_17320-revert-both-linux-and-microcode-4.1.1-20191208T1518Z-84f1e5957a.iso.loop/live/initrd.img
-r--r--r-- 1 root root 33362904 Dec  8 15:09 tails-amd64-hack_17320-revert-microcode-4.1.1-20191208T1411Z-927a0f6f85.iso.loop/live/initrd.img
-r--r--r-- 1 root root 33797596 Dec  8 14:26 tails-amd64-hack_17320-revert-linux-4.1.1-20191208T1328Z-1af7c412ed.iso.loop/live/initrd.img

With 32 MiB = 33554432 meaning both images with a microcode revert are below this limit, and the one with just the linux revert is above the limit. This seems to neatly match most of our findings, except for the very first report by @pushbox who mentioned the third image booted fine as well. I'm going to concentrate on the other reports for the time being.

While wondering what we could strip from the 4.1 image, without touching anything else, we figured removing all network drivers from the initramfs would make a huge difference, possibly without having high impacts on general use cases.

I've therefore pushed a new branch called hack/17320-remove-net-drivers which implements that by removing them from the generated initramfs, also lowering the maximal allowed size from 35 MiB to 32 MiB (this check was initially intended to make sure compression happens), with a final commit adding the +x bit on the initramfs hook after I verified failing to do so triggers a build failure because the check is triggered. The resulting initramfs is way smaller than 32 MiB:

-r--r--r-- 1 root root 29197812 Dec 12 15:29 tails-amd64-hack_17320-remove-net-drivers-4.1.1-20191212T1503Z-094dbcce39.iso.loop/live/initrd.img

and I'd be happy to have confirmations from everyone able to test that this image (containing the same versions of intel-microcode and linux as 4.1) boots fine, which would likely confirm the size of the initramfs is the matter, rather than the actual versions/contents of those packages.

Direct links (add .sig for the detached signature):

#18 Updated by intrigeri about 1 month ago

Yeah! :)))

Direct links (add .sig for the detached signature):

Dear testers, please use the .img, not the .iso. Thanks!

(Rationale: the ISO image is known to not boot on MacBooks, for mostly unrelated reasons, and when it fails the error message is very similar to the one we're seeing here, so testing reports about the ISO won't provide useful info but can instead muddy the water.)

#19 Updated by CyrilBrulebois about 1 month ago

Thanks for mentioning that. I wasn't sure at first whether the ISO→IMG process might have anything to do with the issue, and some users mentioned tails-installer, that's why I published ISO+IMG for all test images, so that everyone could pick their favorite process.

#20 Updated by intrigeri about 1 month ago

Thanks for mentioning that. I wasn't sure at first whether the ISO→IMG process might have anything to do with the issue, and some users mentioned tails-installer, that's why I published ISO+IMG for all test images, so that everyone could pick their favorite process.

Indeed, you're right, ISO + Tails Installer should work. But ISO installed "ala dd/cat" (e.g. with GNOME Disks) will not.

#21 Updated by CyrilBrulebois about 1 month ago

I've got an offline report that iMac12,2 boots successfully with my latest image (shrinking initramfs without downgrading components), hardware that was not booting with Tails 4.1.

#22 Updated by cheesyvampire about 1 month ago

I tried using the .IMG with balenaEtcher and the 17320-remove-net-drivers-4.1.1 .IMG and it booted without the previous kernel panic that I was suffering. It's a MacBookPro8_1 per everymac.com.

Gracias!

#23 Updated by intrigeri about 1 month ago

  • Target version set to Tails_4.2

#24 Updated by goupille about 1 month ago

tails-amd64-hack_17320-remove-net-drivers-4.1.1-20191212T1503Z-094dbcce39.img
is booting fine on these (they were alll affected by this problem with tails 4.1):

MacBook 13-inch (Late 2009)
MacBook Pro 10.1 (mid 2012)
Macbook Air 4,1 11-inch (mid 2011)

#25 Updated by CyrilBrulebois about 1 month ago

  • Target version deleted (Tails_4.2)

Thanks, @goupille, that's very encouraging.

I'm contemplating shipping a variant of this bugfix as 4.1.1, seeing if removing drivers we blacklist anyway would be sufficient. Otherwise, double-checking we don't regress with my latest published image, on various non-Mac laptops would be great. I'll be trying to report back on this front during the week-end.

#26 Updated by abZLQrnoXoj about 1 month ago

The remove-net-drivers version boots fine on MacBookPro8,1 and MacBookPro9,1

#27 Updated by intrigeri about 1 month ago

  • Target version set to Tails_4.2

(Assuming the previous removal of Target version was a mistake caused by Redmine's buggy client-side caching of metadata.)

Awesome news! \o/

#28 Updated by intrigeri about 1 month ago

  • Related to Bug #17257: Mac Mini 2018 kernel panic added

#29 Updated by intrigeri about 1 month ago

  • Related to deleted (Bug #17257: Mac Mini 2018 kernel panic)

#30 Updated by intrigeri about 1 month ago

#31 Updated by goupille about 1 month ago

CyrilBrulebois wrote:

Thanks, @goupille, that's very encouraging.

it is also fixing the issue on a MacBook Pro 10,2 (late 2012)

I'm contemplating shipping a variant of this bugfix as 4.1.1, seeing if removing drivers we blacklist anyway would be sufficient. Otherwise, double-checking we don't regress with my latest published image, on various non-Mac laptops would be great. I'll be trying to report back on this front during the week-end.

I think that would be great

#32 Updated by geb about 1 month ago

CyrilBrulebois wrote:

Thanks, @goupille, that's very encouraging.

I'm contemplating shipping a variant of this bugfix as 4.1.1

+1 on this (especially considering that it may prevent people who did not already upgrade to need to manually upgrade). Happy to perform tests (or more precisely ask somebody to do, who would only be available this week) if that helps.

#33 Updated by CyrilBrulebois about 1 month ago

  • Target version changed from Tails_4.2 to Tails_4.1.1

#34 Updated by CyrilBrulebois about 1 month ago

  • Status changed from In Progress to Resolved

#35 Updated by CyrilBrulebois about 1 month ago

  • Target version deleted (Tails_4.1.1)

Thanks to everyone who helped test images!

#36 Updated by CyrilBrulebois 15 days ago

For the avoidance of doubt/surprise if users were to stumble upon this bug report in the future: I've removed the test images to the staging area hosted by my company (https://tails.debamax.com), following the 4.1.1 release; I've also mentioned this in its new README.txt (as I might re-use this space for other Tails purposes later on).

Also available in: Atom PDF