Project

General

Profile

Bug #9262

Port our ISO build system to Jessie

Added by intrigeri over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Build system
Target version:
Start date:
04/19/2015
Due date:
% Done:

100%

Feature Branch:
feature/9262-build-on-jessie, puppet-tails:feature/9262-build-on-jessie
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

It's unclear whether the Vagrant one can/will be ported, or if we'll instead focus on migrating to Docker. This shall perhaps not block migrating the manual build instructions and Puppet module to Jessie.

diffoscope-wheezy_to_jessie-no-timestamps.html.xz (280 KB) intrigeri, 02/22/2016 11:52 AM


Related issues

Blocked by Tails - Bug #9523: Discrepancy between eatmydata versions breaks Jessie build Resolved 06/03/2015
Blocks Tails - Bug #10592: Upgrade i18nspector to 0.20+ on the Jenkins slave that runs check_PO_master Resolved 11/20/2015
Blocks Tails - Bug #11006: Reinstall Lizard's isobuilders from scratch Resolved 01/27/2016

Associated revisions

Revision 14606a49 (diff)
Added by intrigeri over 4 years ago

Adjust build doc to run on Jessie.

Will-fix: #9262

Revision 16cf67b8
Added by anonym over 3 years ago

Merge remote-tracking branch 'origin/feature/9262-build-on-jessie'

Fix-committed: #9262

Revision 91ebb316 (diff)
Added by anonym over 3 years ago

Upgrade the vagrant-libvirt tails-builder to Debian Jessie.

Refs: #9262

History

#1 Updated by intrigeri over 4 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Initial test results: our build system seems to work just fine on Jessie, which is no big surprise given that most of the build process happens in a Wheezy chroot anyway.

Next step: compare the resulting ISO with one built on Wheezy. The preliminary work made on #5630, and the debbindiff tool, should help a lot.

#2 Updated by intrigeri over 4 years ago

  • Assignee changed from intrigeri to anonym
  • Target version changed from Tails_1.4.1 to Tails_1.5
  • QA Check set to Info Needed

I doubt I'll have time to work on this during the 1.4.1 cycle (and same the next one, possibly).

anonym, assuming ISO images built on Wheezy and on Jessie don't essentially differ, do you think that we should block this task on the corresponding upgrade of our Vagrant basebox? If yes, then we'll need to find out who/when does that (or completes the migration to Docker :)

#3 Updated by anonym over 4 years ago

  • Assignee changed from anonym to intrigeri
  • % Done changed from 10 to 30
  • Feature Branch set to feature/9262-update-vagrant-to-jessie

intrigeri wrote:

I doubt I'll have time to work on this during the 1.4.1 cycle (and same the next one, possibly).

anonym, assuming ISO images built on Wheezy and on Jessie don't essentially differ, do you think that we should block this task on the corresponding upgrade of our Vagrant basebox? If yes, then we'll need to find out who/when does that (or completes the migration to Docker :)

I did the upgrade (incl. building and pushing a new basebox) while procrastinating today. However, there are some issues:

  • eatmydata doesn't work in the chroot and we get spammed with:
    ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded: ignored.
    Seems related to #9523, right?
  • Builds fail due to monkeysphere not installing properly. It looks exactly like Debian bug #778833 which apparently wasn't fixed in Wheezy. I wonder what's different compared to the Wheezy-based builder, since this error occurs in the Wheezy chroot used by live-build. Any ideas?

It should be said that I managed to build a seemingly working image by omitting monkeysphere, so it's quite close to being useful.

#4 Updated by intrigeri over 4 years ago

I did the upgrade (incl. building and pushing a new basebox) while procrastinating today.

\o/

  • eatmydata doesn't work in the chroot and we get spammed with:
    ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded: ignored.
    Seems related to #9523, right?

Exactly. Expect a pull request for that one in less that one hour.

  • Builds fail due to monkeysphere not installing properly. It looks exactly like Debian bug #778833 which apparently wasn't fixed in Wheezy. I wonder what's different compared to the Wheezy-based builder, since this error occurs in the Wheezy chroot used by live-build. Any ideas?

I've noticed that too, and I'll take a look.

#5 Updated by intrigeri over 4 years ago

  • Blocked by Bug #9523: Discrepancy between eatmydata versions breaks Jessie build added

#6 Updated by intrigeri over 4 years ago

  • QA Check deleted (Info Needed)

#7 Updated by anonym over 4 years ago

intrigeri wrote:

  • eatmydata doesn't work in the chroot and we get spammed with: [...] Seems related to #9523, right?

Exactly. Expect a pull request for that one in less that one hour.

The fix for #9523 fixes this as expected...

  • Builds fail due to monkeysphere not installing properly. It looks exactly like Debian bug #778833 which apparently wasn't fixed in Wheezy. I wonder what's different compared to the Wheezy-based builder, since this error occurs in the Wheezy chroot used by live-build. Any ideas?

I've noticed that too, and I'll take a look.

... but it also (quite unexpectedly) fixes this issue. So, like we theorized on #tails-dev, there seems to be a race condition in monkeyspehere's post-inst script that eatmydata prevents from triggering. Or something.

#8 Updated by anonym over 4 years ago

  • QA Check set to Info Needed

Ok, so this branch works well for me now. I've managed to build both stable and feature/jessie seemingly with great success. Is anything else required from me for this ticket?

#9 Updated by intrigeri over 4 years ago

  • Assignee changed from intrigeri to anonym

Ok, so this branch works well for me now.

Woohoo!

Is anything else required from me for this ticket?

Let me have a look at the branch before you disappear in vacation, then. It looks very good overall, here are a few nitpicks (nothing that would really block me from merging TBH, but still):

d-i passwd/root-login boolean false
+d-i passwd/root-password password vagrant
+d-i passwd/root-password-again password vagrant

Assuming this results in a working root account, can it be ssh'ed into from the host system with that password? Or does the new "no password auth for root over SSH" default Jessie restriction apply?

+d-i passwd/user-default-groups string audio cdrom video admin

Do we really have an admin group on Jessie? I don't have it on my system, so this looks suspiciously like something that was copied'n'pasted from Ubuntu-specific docs ;)

Also, why does the default user account need to be in the audio, cdrom and video groups?

# Add jessie-backports for apt-cacher-ng. The version in jessie is older
# than the one in wheezy-backports that we used to install

That comment seems incorrect to me: wheezy-backports has 0.8.0-3~bpo70+1, and Jessie has 0.8.0-3.
(And in general, this is always incorrect because it would break upgrades; that's why Debian also has sloppy backports.)
Is there any other reason why we really need acng 0.8.3-1, or?

Otherwise, looks good!

#10 Updated by anonym over 4 years ago

  • Assignee changed from anonym to intrigeri

intrigeri wrote:

d-i passwd/root-login boolean false
+d-i passwd/root-password password vagrant
+d-i passwd/root-password-again password vagrant

Assuming this results in a working root account, can it be ssh'ed into from the host system with that password? Or does the new "no password auth for root over SSH" default Jessie restriction apply?

No, the root account is disabled so the added lines are NOOP:s...

+d-i passwd/user-default-groups string audio cdrom video admin

Do we really have an admin group on Jessie? I don't have it on my system, so this looks suspiciously like something that was copied'n'pasted from Ubuntu-specific docs ;)

Almost. :)

Also, why does the default user account need to be in the audio, cdrom and video groups?

Obviously we do not.

I just made our template conform exactly with veewee's jessie template modulo comments and the lines we actually want to diverge on. I had issues with the d-i/preseeding stuff failing at some unspecified step, and I have no idea how to debug it properly, so this seemed like the thing to do that would waste the least amount of time.

# Add jessie-backports for apt-cacher-ng. The version in jessie is older
# than the one in wheezy-backports that we used to install

That comment seems incorrect to me: wheezy-backports has 0.8.0-3~bpo70+1, and Jessie has 0.8.0-3.
(And in general, this is always incorrect because it would break upgrades; that's why Debian also has sloppy backports.)
Is there any other reason why we really need acng 0.8.3-1, or?

Ah, I must have looked at wheezy-backports-sloppy for whatever reason. :S Fixed in 9c8776a.

Anything else?

#11 Updated by intrigeri over 4 years ago

  • QA Check changed from Info Needed to Dev Needed

Anything else?

No, that's great. Thanks!

#12 Updated by intrigeri over 4 years ago

  • Feature Branch changed from feature/9262-update-vagrant-to-jessie to feature/9262-build-on-jessie

(Let's share a single branch, whose name is not Vagrant-specific.)

#13 Updated by intrigeri over 4 years ago

  • Feature Branch changed from feature/9262-build-on-jessie to feature/9262-build-on-jessie, puppet-tails:feature/9262-build-on-jessie

#15 Updated by intrigeri about 4 years ago

  • % Done changed from 30 to 40

I've built two ISO images from the topic branch, one in a Wheezy VM, the other one in a Jessie VM, and I've looked at the output of debbindiff + diff'ed the initramfs and SquashFS manually:

  • package list is the same
  • differing initrd's size (a bit smaller on Jessie), but only difference in content is mtimes
  • Number ​of inodes in SquashFS: 141721 vs. 141718, explained since three files are in the SquashFS only when built on Wheezy:
    • /dev/xconsole: created at runtime if missing
    • /var/lib/urandom/random-seed: created at runtime if missing (and #7642 seems to indicate that we should not ship it in the ISO anyway)
    • /run/sendsigs.​omit.​d/rsyslog: created at runtime if missing
  • /run/motd.​dynamic content differs because it embeds the kernel version of the build system

I saw no other relevant non-mtime-related differences, so IMO next steps are:

  • run the automated test suite on an ISO built on Jessie;
  • make a decision on #7642, so we can be comfortable not shipping that random seed in the ISO.

#16 Updated by intrigeri about 4 years ago

  • Blocked by Feature #7642: Investigate whether we should resume shipping a static random seed added

#17 Updated by intrigeri about 4 years ago

Left to see pass: features/erase_memory.feature:7 features/erase_memory.feature:21 features/erase_memory.feature:35 features/erase_memory.feature:49

#18 Updated by intrigeri about 4 years ago

  • % Done changed from 40 to 50

intrigeri wrote:

I saw no other relevant non-mtime-related differences, so IMO next steps are:

  • run the automated test suite on an ISO built on Jessie;

Done, so the only blocker left is:

  • make a decision on #7642, so we can be comfortable not shipping that random seed in the ISO.

#19 Updated by intrigeri about 4 years ago

  • Target version changed from Tails_1.5 to Tails_1.6

Too late to reach a conclusion on #7642 before the freeze => postponing this one.

#20 Updated by intrigeri about 4 years ago

  • Target version changed from Tails_1.6 to Tails_1.7

I don't think it's realistic to expect bertagaz to have time to resolve the remaining blocker (#7642) within a week => let's postpone.

#21 Updated by intrigeri about 4 years ago

  • Target version changed from Tails_1.7 to Tails_1.8

intrigeri wrote:

I don't think it's realistic to expect bertagaz to have time to resolve the remaining blocker (#7642) within a week => let's postpone.

This is again the case.

#22 Updated by intrigeri almost 4 years ago

  • Target version changed from Tails_1.8 to 246

#7642 likely won't be addressed during this cycle so moving this out of my radar for now. The real deadline is: before Wheezy is EOL'ed (hint: Wheezy may have a LTS).

#23 Updated by intrigeri almost 4 years ago

  • Blocks Bug #10592: Upgrade i18nspector to 0.20+ on the Jenkins slave that runs check_PO_master added

#24 Updated by sajolida almost 4 years ago

  • Target version changed from 246 to Tails_2.0

#25 Updated by intrigeri almost 4 years ago

  • Blocks Feature #6297: Save list of packages used at ISO build time added

#26 Updated by intrigeri almost 4 years ago

intrigeri wrote:

  • /var/lib/urandom/random-seed: created at runtime if missing (and #7642 seems to indicate that we should not ship it in the ISO anyway)

For the record, upgrading debootstrap to 1.0.73~bpo8+1 on a Wheezy build system is enough to trigger this change. I guess it is because of / thanks to https://anonscm.debian.org/cgit/d-i/debootstrap.git/commit/?id=29cd6beebf4a35bf178028e13917eb275fe6c66e, that probably prevents the urandom initscript from creating the random-seed.

#27 Updated by intrigeri almost 4 years ago

  • Blocks deleted (Feature #6297: Save list of packages used at ISO build time)

#28 Updated by intrigeri almost 4 years ago

  • Target version changed from Tails_2.0 to Tails_2.2

#29 Updated by bertagaz over 3 years ago

  • Blocks Bug #11006: Reinstall Lizard's isobuilders from scratch added

#30 Updated by intrigeri over 3 years ago

  • Target version changed from Tails_2.2 to Tails_2.3

This is blocked by #7642, that I'm told should move forward by early March, which will likely be too late for me to complete this in time for Tails 2.2.

#31 Updated by intrigeri over 3 years ago

  • Blocked by deleted (Feature #7642: Investigate whether we should resume shipping a static random seed)

#32 Updated by intrigeri over 3 years ago

  • Target version changed from Tails_2.3 to Tails_2.2

We're done on #7642 as far as this ticket is concerned, and anonym and I will try to get this merged today.

#33 Updated by intrigeri over 3 years ago

  • % Done changed from 50 to 60

anonym did most of the work, I reviewed the code at 01e930550598ba70a2bae7e12d336e400d0bcff4 and updated the Puppet stuff. Next steps are:

  • to compare two ISO images (one built on Wheezy, the other built on Jessie)
  • to run some automated tests on an ISO built on Jessie

... and then we can merge in Git, and deploy on our infra + upgrade our isobuilders on lizard.

#34 Updated by intrigeri over 3 years ago

Here's a diffoscope report with two ISO images built respectively on a Wheezy and a Jessie VM, from the same source, that has a hack to find / -exec touch --no-create -t 197001010000 {} \; as a chroot hook, to eliminate file timestamp variations. I'll provide another one, with --no-dereference passed to touch as well so we also eliminate these variations for symlinks.

#35 Updated by intrigeri over 3 years ago

  • Assignee changed from intrigeri to anonym
  • % Done changed from 60 to 70
  • QA Check changed from Dev Needed to Ready for QA

I say let's merge.

#36 Updated by anonym over 3 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 70 to 100
  • QA Check changed from Ready for QA to Pass

Merged! A full automated test suite run of an image built from this branch has passed!

#37 Updated by intrigeri over 3 years ago

  • Status changed from Fix committed to Resolved

(This was merged into master already.)

Also available in: Atom PDF