Project

General

Profile

Bug #16818

Impossible to start on MacBookPro 8,1

Added by sajolida about 1 month ago. Updated 15 days ago.

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Hardware support
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
bugfix/16818-buster-vagrant-box
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

That's with bc022ef71e, the Tails Mac I have with me, and the same USB stick (and copy of Tails) that I've used for all my other tests.

Behavior is a bit strange:

  • At first I had a hard time making the USB stick appear in the Mac startup disks.
  • I finally got it there, both on its own and by plugging a Tails 3.14 first and replacing it.
  • Both times I got a kernel panic right after the boot loader menu. See attachment.

The main error message in the attachment is: "Initramfs unpacking failed: junk in compressed archive".

IMG_20190617_192602.jpg View (752 KB) sajolida, 06/17/2019 05:33 PM


Related issues

Related to Tails - Bug #16755: Call for testing: feature/buster (June 2019 edition) Resolved 06/18/2019
Blocks Tails - Feature #8393: Consider replacing sound-juicer with goobox Confirmed 12/05/2014
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed 03/22/2019

Associated revisions

Revision 4e7504c7 (diff)
Added by intrigeri 19 days ago

Strip debug symbols from aufs.ko to make it smaller (refs: #16818)

Let's hope that bringing the initramfs down under 32MB will repair the boot of
feature/buster on MacBookPro 8,1. In any case, this should improve UX by:

- making boot faster for every user
- shortening the duration during which we display a black screen,
between the bootloader and the Plymouth splash screen

Revision f6360117
Added by segfault 17 days ago

Merge branch 'bugfix/16818-smaller-aufs-module' into stable (refs: #16818)

History

#1 Updated by intrigeri about 1 month ago

  • Description updated (diff)
  • Category set to Hardware support
  • Type of work changed from Code to Research

@sajolida, a few questions:

  • This machine is booting in UEFI mode, right? (grey background in the boot loader, no nice Tails custom syslinux theme)
  • Just to make sure we can reasonably eliminate a faulty USB stick or corrupted data, can you please retry this with another USB stick on which you've installed from scratch a brand new feature/buster?

Meanwhile, I'm going to briefly research the error message to see if there's another explanation.

#2 Updated by intrigeri about 1 month ago

Next step for me: compare the size of the initrd in 3.14 and feature/buster.

@sajolida, can you also confirm that 3.14 boots fine on this machine?

#3 Updated by intrigeri about 1 month ago

  • Related to Bug #16755: Call for testing: feature/buster (June 2019 edition) added

#4 Updated by intrigeri about 1 month ago

initrd.img is 34730980 bytes (34M) on current feature/buster vs. 32292792 bytes (31M) on 3.14. This might explain "Initramfs unpacking failed: junk in compressed archive", e.g. in case the interaction between syslinux and the Apple-style EFI firmware has trouble loading this much data in memory. It could be that there's a 32M limit somewhere. I've compared their content and there's no huge difference. We could try:

  • trimming some useless stuff, e.g. cryptsetup and friends, ntfs-3g, kernel modules that are useless in the initrd (sound, net, scsi, infiniband)
  • making our aufs.ko smaller (it's 20MB uncompressed!)

Then, once we have a feature/buster -based branch with an initramfs that's not bigger than 3.14's, we could ask sajolida to retry. Let's do all this once the faulty USB stick hypothesis has been fully invalidated.

Of course, this does not explain "At first I had a hard time making the USB stick appear in the Mac startup disks", which I suspect is an entirely different bug (I've seen this happen occasionally on various Macs including this one and could never find a way to reliably reproduce it).

#5 Updated by intrigeri about 1 month ago

  • Priority changed from Normal to Elevated

#6 Updated by sajolida about 1 month ago

  • Blocks Feature #8393: Consider replacing sound-juicer with goobox added

#7 Updated by sajolida about 1 month ago

Different USB, same kernel panic. After only a quick look, many of the error messages are similar. The "Kernel Offset" is different.
Tell me if I should inspect it more in depth.

And yes, 3.14 starts fine. It's the first time I see something like this on this machine.

#8 Updated by intrigeri about 1 month ago

  • Subject changed from Impossible to start on Mac to Impossible to start on MacBookPro 8,1

#9 Updated by intrigeri about 1 month ago

  • Status changed from New to Confirmed

Thank you.

The "faulty USB stick" hypothesis has been fully invalidated ⇒ time to try & fix this. #16818#note-4 has my ideas.

#10 Updated by intrigeri 19 days ago

  • Assignee set to intrigeri

So:

  • the aufs.ko we build ourselves from upstream sources is 20MB and running strings on it yields 290564 lines, including strings I would not expect there, such as scsi (35 times) and sndbuf_policy (34 times); file says that it is compiled with debug_info and not stripped; gzip'ing it with default settings gives me a 5.8MB file
  • the aufs.ko compiled on my sid by dkms is 572KB and running strings on it yields 3319 lines; not stripped, but no debug info; gzip'ing it → 147KB

So it looks like we could save a few MB on the initramfs size (therefore making boot faster for every user, and shortening the time between the bootloader and the plymouth splash, on top of hopefully fixing the problem this ticket is about), by removing this debug_info. I'll look into this option before starting to remove kernel modules.

#11 Updated by intrigeri 19 days ago

  • Feature Branch set to bugfix/16818-smaller-aufs-module

#12 Updated by intrigeri 19 days ago

  • Status changed from Confirmed to In Progress

#13 Updated by intrigeri 19 days ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to sajolida

initrd.img is down to 29MB on this branch => merged into feature/buster so I can ask sajolida to test it (my branch is based on stable so Jenkins won't build it for Buster).

@sajolida, can you please try booting a feature/buster USB image dated from tomorrow (earliest) on this MacBookPro 8,1 and tell me if it helps?

#14 Updated by intrigeri 19 days ago

And to be clear, regardless of sajolida's feedback, I'll request a proper review and merge into stable once I've seen Jenkins results.

#15 Updated by intrigeri 19 days ago

intrigeri wrote:

initrd.img is down to 29MB on this branch

… and to 31MB (31858656 bytes) on feature/buster with this branch merged, which is smaller than the initramfs we shipped in 3.14.2, so if initramfs size is the problem, then this should definitely fix it, and I'll want to update the limit we set in config/binary_local-hooks/05-check_initramfs-size. But if my branch does not fix this bug, then we'll know we have a different problem and I will want to try feature/buster with Stretch's syslinux.

#16 Updated by intrigeri 19 days ago

@sajolida, if current feature/buster does not fix this, I'm still interested in the answer to this question: "This machine is booting in UEFI mode, right? (grey background in the boot loader, no nice Tails custom syslinux theme)"

#17 Updated by intrigeri 19 days ago

@alant, please try booting current feature/buster on the MacBookPro you have access to.

#18 Updated by intrigeri 18 days ago

  • Assignee deleted (sajolida)
  • Target version changed from Tails_4.0 to Tails_3.15

intrigeri wrote:

And to be clear, regardless of sajolida's feedback, I'll request a proper review and merge into stable once I've seen Jenkins results.

Jenkins is happy enough (https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_bugfix-16818-smaller-aufs-module/, https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-buster/) so please review and merge into stable; but don't mark this ticket as "Fix Committed" until we have sajolida's or alant's feedback. @segfault, @anonym, @hefee: anyone?

#19 Updated by segfault 17 days ago

  • Assignee set to segfault

#20 Updated by segfault 17 days ago

  • Status changed from Needs Validation to In Progress

#21 Updated by segfault 17 days ago

  • Assignee changed from segfault to sajolida

Merged into stable -> devel -> feature/buster. Assigning to sajolida to provide feedback.

#22 Updated by sajolida 16 days ago

@sajolida, if current feature/buster does not fix this, I'm still interested in the answer to this question: "This machine is booting in UEFI mode, right? (grey background in the boot loader, no nice Tails custom syslinux theme)"

Correct!

#23 Updated by sajolida 15 days ago

  • Assignee changed from sajolida to intrigeri

#25 Updated by intrigeri 15 days ago

  • Target version changed from Tails_3.15 to Tails_3.16

#26 Updated by intrigeri 15 days ago

  • Feature Branch deleted (bugfix/16818-smaller-aufs-module)

#27 Updated by sajolida 15 days ago

The same in public:

I'm coming back with bad news: it got worse :(

Here is what I did:

  • Install 3.14.2 on a USB stick
    • Starts fine on the Mac.
  • Install 5712f23dbe from June 19 on another USB stick
    • As reported initially, it shows up in the Mac startup disks, then displays the UEFI mode bootloader (gray background) and the kernel panic
  • Install 2007a18fa5 from July 8 on another USB stick
    • This time it doesn't even appear in the Mac startup disks.
    • The same USB stick starts fine on a X200.

I've used systematically my most reliable technique, which is to get to the startup disks with no USB stick plugged in (only the internal hard disk appears) and then only plug the USB stick. If I unplug the USB stick it disappears and I can plug another one (I have only 1 USB port). Both 3.14.2 and 5712f23dbe appear and disappear fine from the startup disks, with 2007a18fa5 the USB stick itself blinks but doesn't appear in the list.

Sorry!

NB: You changed the target version from 4.0 to 3.15 while this bug is only affecting feature/buster, but you probably have a good reason for that :)

#28 Updated by intrigeri 15 days ago

  • Target version changed from Tails_3.16 to Tails_4.0

I'm coming back with bad news: it got worse :(

Ouch!

  • Install 5712f23dbe from June 19 on another USB stick
    • As reported initially, it shows up in the Mac startup disks, then displays the UEFI mode bootloader (gray background) and the kernel panic
  • Install 2007a18fa5 from July 8 on another USB stick
    • This time it doesn't even appear in the Mac startup disks.
    • The same USB stick starts fine on a X200.

I've looked at the changes between these 2 images:

  • on our side (5712f23dbe..2007a18fa5):
    • no change to partitioning or MBR -related stuff
    • only update between 3.14 and 3.15 APT suites: Thunderbird
  • on Debian side: no syslinux update

So I really don't understand how behavior can have changed for you :/

You initially reported "At first I had a hard time making the USB stick appear in the Mac startup disks", about an earlier version (bc022ef71e, June 8), which suggests that there's some amount of non-determinism around all this, and makes me wonder why an intermediary version (5712f23dbe, June 19) works better than that one. So I'd like to know how many times you did both tests (the successful one and the failing one) and whether you did them in exactly the same way: all kinds of small details can matter here, for example "was the USB stick already started on another computer before?", "did you reboot or power off?", "was that 'another USB stick' the same one in both cases?", etc.

Also, it would be useful to know whether a feature/buster USB stick cloned with Tails Installer displays the same behavior or not: there's a small chance that the bootloader bits on the resulting USB stick are 100% Buster while the ones you've tested are a mix between Stretch and Buster (see below for technical details if curious). This would help tell if the problem comes from the ISO to USB image conversion process.

In any case, it seems clear that you've never had any issue with 3.x but you see trouble with feature/buster, so let's try some things that may help us understand where the problem comes from:

  • Buster with Stretch's syslinux
  • Buster built in a Buster Vagrant box: I'm concerned that since 6e097195f13e44a4664051cad79f32dc2f134c9f, auto/scripts/create-usb-image-from-iso uses the syslinux binary from the (Buster) chroot, but it's not run with chroot, so there's a chance that it uses files from the Vagrant box (e.g. the MBR or whatnot), which might create inconsistencies and trouble

NB: You changed the target version from 4.0 to 3.15 while this bug is only affecting feature/buster, but you probably have a good reason for that :)

That was because I wanted bugfix/16818-smaller-aufs-module into 3.15 which is now done ⇒ reverting.

#29 Updated by intrigeri 15 days ago

  • Feature Branch set to bugfix/16818-buster-vagrant-box

#30 Updated by sajolida 15 days ago

  • Status changed from In Progress to Resolved

OMG! You rock! It's working now :)

This morning I missed the fact that I had USB images for bc022ef71e and 5712f23dbe but an ISO image for 2007a18fa5. D'oh!

Now I downloaded an USB image for 7523d0b2a0 and it starts fine :)))

Also, in my previous tests I managed to get rid of the non-determinism by always getting to the startup disks menu before plugging the USB stick.

#31 Updated by intrigeri 15 days ago

#32 Updated by intrigeri 15 days ago

I've moved my WIP to #16868.

Also available in: Atom PDF