Project

General

Profile

Feature #10298

Upgrade to Linux 4.x

Added by intrigeri over 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Hardware support
Target version:
Start date:
08/11/2015
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
feature/10298-linux-4.x-aufs
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Tails should work on recent hardware. If we don't do anything special, we're stuck with Linux 3.16, missing out on the Linux kernel's hardware support improvements.

Our preferred way to upgrade Tails to Linux 4.x would be to migrate from aufs to overlayfs (#8415). We have done a lot of the needed work on our side, but before we go on we are blocked by work that must be done upstream (AppArmor support for overlayfs in the Linux kernel, aka. #9045).

If this plan works in time for Tails 3.0 (Debian Stretch), fine: we'll then have to finish porting Debian Live and Tails -specific bits to overlayfs.

If it does not, then we will have to stick to aufs, which implies building out-of-tree aufs modules ourselves (note that the support patches have been re-introduced in Debian's Linux kernel, so that is possible now.

In any case, we will also have to deal with other new issues we'll discover with recent kernels (see subtasks).


Subtasks

Bug #9969: Re-enable dkms modules building with Linux 4.xResolved

Feature #11511: Drop custom value for kernel.perf_event_paranoid sysctlResolved


Related issues

Related to Tails - Bug #11441: Stretch's X.Org cannot start with Linux 3.16's qxl Resolved 05/18/2016
Related to Tails - Bug #11518: Linux 4.x QXL 64-bit kernel modesetting breaks 32-bit X.Org Resolved 06/08/2016
Blocks Tails - Feature #7649: Include a grsecurity-patched kernel Rejected 01/07/2015
Blocks Tails - Bug #10289: Tails based on Debian Stretch Resolved 09/27/2015
Blocks Tails - Feature #11281: Enable kASLR Resolved 03/24/2016
Blocks Tails - Bug #8485: Consider including an x86 kernel with SMP support Resolved 12/25/2014

Associated revisions

Revision 4eb604e7 (diff)
Added by intrigeri almost 3 years ago

Upgrade to Linux 4.5.

This is essentially a squashed version of feature/8415-overlayfs, purged
of its overlayfs-specific bits, and that fetches the kernel from
jessie-backports (instead of from sid) to ease compiling out-of-tree and
dkms modules. At this stage, the resulting ISO won't boot since it'll
lack both aufs and overlayfs.

refs: #10298

To achieve that:

  • Install the 686 kernel flavour instead of the obsolete 586 one.
  • Rename the amd64 kernel via binary_local-hooks.

    lb_binary_syslinux depends on --linux-flavours, but we can't pass
    amd64 in there anymore (due to lack of multiarch support) so we need
    to rename the second (amd64) kernel ourselves.

  • APT, dpkg: add amd64 architecture. The amd64 kernel flavour is not
    built anymore for the i386 architecture, so we need to use
    multiarch now.
  • Temporary disable dkms modules building.

    Building them for the amd64 kernel requires installing an amd64
    toolchain, that conflicts with the i386 one and with lots of packages
    we need. Let's come back to it later (refs: #9969).

Revision b3e0e57b (diff)
Added by intrigeri almost 3 years ago

Build and install the out-of-tree aufs4 module.

refs: #10298

Note that we disable aufs internal debug code, since it's disabled in
Jessie's kernel, and Kconfig reads: "It will have a negative impact to
the performance".

Revision e7fdfa2d (diff)
Added by intrigeri almost 3 years ago

Test suite: use a VGA graphics adapter, instead of the QXL one.

X.Org cannot start with Linux 4.x and QXL.

refs: #11441, #10298

Revision 6c3d9e8b (diff)
Added by intrigeri almost 3 years ago

Don't force using the vboxvideo X.Org driver.

According to anonym's tests, this forced setting is:

  • harmful: it breaks X startup when the vboxvideo kernel driver
    is loaded;
  • useless: X.Org now autodetects the vboxvideo X.Org driver and uses it
    when running in VirtualBox and the vboxvideo kernel is not present.

refs: #10298

Revision 388b019f (diff)
Added by intrigeri almost 3 years ago

Bump BUILD_SPACE_REQUIREMENT: at least one of us needed that to build feature/10298-linux-4.x-aufs with the gzipcomp option.

refs: #10298

Revision fd6904d2 (diff)
Added by intrigeri almost 3 years ago

Fix regexp to disable aufs internal debugging code.

refs: #10298

Revision bd961bd8
Added by anonym almost 3 years ago

Merge remote-tracking branch 'origin/feature/from-intrigeri-for-2.6' into devel

Fix-committed: #5650, #6729, #6850, #8485, #10190, #10298, #10733, #10733, #11281, #11588, #11582, #11590

History

#1 Updated by intrigeri over 3 years ago

#2 Updated by intrigeri over 3 years ago

  • Description updated (diff)

#3 Updated by intrigeri over 3 years ago

  • Blocks Feature #7649: Include a grsecurity-patched kernel added

#4 Updated by intrigeri over 3 years ago

  • Blocks Bug #10289: Tails based on Debian Stretch added

#5 Updated by intrigeri about 3 years ago

#6 Updated by intrigeri about 3 years ago

  • Related to Bug #11441: Stretch's X.Org cannot start with Linux 3.16's qxl added

#7 Updated by intrigeri about 3 years ago

  • Related to Feature #11421: Change kernel.perf_event_paranoid sysctl to 2 added

#8 Updated by intrigeri almost 3 years ago

  • Description updated (diff)

#9 Updated by Dr_Whax almost 3 years ago

  • Blocks Feature #11511: Drop custom value for kernel.perf_event_paranoid sysctl added

#10 Updated by intrigeri almost 3 years ago

  • Blocks deleted (Feature #11511: Drop custom value for kernel.perf_event_paranoid sysctl)

#11 Updated by intrigeri almost 3 years ago

  • Related to deleted (Feature #11421: Change kernel.perf_event_paranoid sysctl to 2)

#12 Updated by intrigeri almost 3 years ago

  • Description updated (diff)

#13 Updated by intrigeri almost 3 years ago

  • Feature Branch set to feature/10298-linux-4.x-aufs

Here's a branch that builds and includes the out-of-tree aufs modules. It seems to boot fine (except with QXL) both on 32-bit and 64-bit (virtual) CPUs but I didn't test any further. It shares some of the issues of feature/8415-overlayfs but at least it supports AppArmor, and should require little additional porting work, that also would be needed with overlayfs anyway (see subtasks). Let's see what Jenkins thinks of it.

#14 Updated by intrigeri almost 3 years ago

  • Blocks Bug #8485: Consider including an x86 kernel with SMP support added

#15 Updated by intrigeri almost 3 years ago

  • Description updated (diff)

#16 Updated by intrigeri almost 3 years ago

  • Status changed from Confirmed to In Progress

#17 Updated by intrigeri almost 3 years ago

  • Related to Bug #11518: Linux 4.x QXL 64-bit kernel modesetting breaks 32-bit X.Org added

#18 Updated by intrigeri almost 3 years ago

  • Target version changed from 2016 to Tails_2.6
  • QA Check set to Ready for QA

I've seen all test suite scenarios pass, Jenkins doesn't particularly dislike it, and I see no blockers left. anonym, how about merging this for 2.6?

#19 Updated by anonym almost 3 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

When I built this branch (merged into devel) I ran out of space (RAM) when creating the ISO image (at 97.5%) and genisoimage was OOM killed. I guess you used defaultcomp, which still should work, as opposed to gzipcomp, which I used. IMHO branches should remain buildable with gzipcomp since it makes the review'n'merge work less tedious. Bumping BUILD_SPACE_REQUIREMENT by 128 fixes it, which I was ready to do myself so I could merge this now...

... which I'd like to do and actually was ready to, because this branch is otherwise great and I have no remarks, except that it breaks vboxvideo in a 32-bit VirtualBox VM (64-bit is still ok). Boot stops at:

[  OK  ] Started GNOME Display Manager.

If I login as root in a different VT I can see this in the Xorg log after the vboxvideo X driver is ~loaded:
[...]
(II) vboxvideo: kernel driver found, not loading.
(EE) No devices detected.
[...]

If I remove the /etc/X11/xorg.conf.d file created by live-{boot,config} (or whatever) which has a section that (force) loads the vboxvideo Xorg driver, and restart Xorg, then everything works as expected (e.g. VirtualBox's auto-resolution feature). Xorg will then use the fbdevhw Xorg driver, which is the correct thing when the vboxvideo kernel driver is loaded.

So, it would seem to me that we could fix this if we prevent the /etc/X11/xorg.conf.d file from being created by live-{boot,config} (or whatever).

(Note: on 64-bit we still want the vboxbideo Xorg driver since we don't have the kernel driver, but it will be properly autodetected by Xorg without the /etc/X11/xorg.conf.d file, so good riddance. :))

So, in conclusion, please:
  • bump Vagrant's BUILD_SPACE_REQUIREMENT by 128 or so, and make sure that this branch builds with gzipcomp, and
  • prevent the /etc/X11/xorg.conf.d file (that (force) loads the vboxvideo driver) from being created by live-{boot,config} (or whatever).

#20 Updated by cypherpunks almost 3 years ago

When we get 4.x, we'll also be stuck with unprivileged USERNS (which cannot be disabled with anything other than a custom kernel). Are we going to do anything about that? Since we're not using grsecurity yet, we're going to be stuck with all the horrible bugs that unprivileged USERNS pumps out on a regular basis, like these two from just today: http://www.openwall.com/lists/oss-security/2016/06/24/5

#21 Updated by intrigeri almost 3 years ago

When we get 4.x, we'll also be stuck with unprivileged USERNS (which cannot be disabled with anything other than a custom kernel).

Indeed, these have a long history of security issues. https://anonscm.debian.org/cgit/kernel/linux.git/tree/debian/patches/debian/add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by-default.patch is applied to the Debian 4.x kernel. If we used this sysctl to disable unprivileged CLONE_NEWUSER, would this address your concern?

#22 Updated by cypherpunks almost 3 years ago

intrigeri wrote:

Indeed, these have a long history of security issues. https://anonscm.debian.org/cgit/kernel/linux.git/tree/debian/patches/debian/add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by-default.patch is applied to the Debian 4.x kernel. If we used this sysctl to disable unprivileged CLONE_NEWUSER, would this address your concern?

Yeah, a lot better than nothing!

#23 Updated by intrigeri almost 3 years ago

When I built this branch (merged into devel) I ran out of space (RAM) when creating the ISO image (at 97.5%) and genisoimage was OOM killed. I guess you used defaultcomp, which still should work, as opposed to gzipcomp, which I used. IMHO branches should remain buildable with gzipcomp since it makes the review'n'merge work less tedious.

Sorry about that. I used gzipcomp as usual, and it works fine for me here, so I'm not quite sure why it doesn't for you. I'll bump BUILD_SPACE_REQUIREMENT anyway :)

... which I'd like to do and actually was ready to, because this branch is otherwise great and I have no remarks, except that it breaks vboxvideo in a 32-bit VirtualBox VM (64-bit is still ok). Boot stops at:

Thanks for the detailed analysis! It seems to me that you found a bug in live-config, that should be fixed in there instead of merely workaround'ed on our side, so I've reported it. Meanwhile, I'll patch out the faulty code via config/chroot_local-patches/. In doubt, I'll only drop the bits that force the video driver, but I'll leave the ones about vboxmouse in place. I'd appreciate if you could test it again (both 32-bit and 64-bit) and check if there's anything wrong with the input devices. Then I'll upstream the patch :)

So, in conclusion, please:

Sure! I'm on it.

#24 Updated by intrigeri almost 3 years ago

Status: update the code according to anonym's review, running the full test suite. Then we'll need another review and more testing on VirtualBox as requested above.

#25 Updated by intrigeri almost 3 years ago

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Dev Needed to Ready for QA

I've seen all tests pass (except #11571) so please review, re-do the VirtualBox tests, and merge :)

#26 Updated by intrigeri almost 3 years ago

I'd like to ease reviewing for the 2.6 RM, and to get automated tests running about the combination of all these changes ASAP in the 2.6 dev cycle. So, I've merged this work, along with the other major branches I'm proposing for 2.6, into the feature/from-intrigeri-for-2.6 integration branch (Jenkins builds and tests).

#28 Updated by anonym almost 3 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • QA Check changed from Ready for QA to Pass

#29 Updated by itiamo over 2 years ago

This build confirmed working on my GeForce 940MX GPU, ThinkPad T460p laptop.

#30 Updated by anonym over 2 years ago

  • Status changed from Fix committed to Resolved

#31 Updated by intrigeri over 2 years ago

  • Related to deleted (Feature #8415: Migrate from aufs to overlayfs)

Also available in: Atom PDF