Project

General

Profile

Bug #14459

Some branches fail to build an ISO because we merge their base branch too late

Added by intrigeri about 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Start date:
08/28/2017
Due date:
% Done:

100%

Feature Branch:
bugfix/14459-merge-base-branch-earlier
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Our ISO build system errors out when building from a branch based on master:

> + apt-snapshots-serials prepare-build
> + apt-mirror debian
> Not building from a tag, but last version in changelog was released
> + DEBIAN_MIRROR=
> + apt-mirror debian-security
> Not building from a tag, but last version in changelog was released
> + DEBIAN_SECURITY_MIRROR=
> + apt-mirror torproject
> Not building from a tag, but last version in changelog was released
> + TORPROJECT_MIRROR=
> + [ -n  ]
> + exit 1

I believe this is caused by auto/config running apt-snapshots-serials prepare-build before auto/build had a chance to merge the base branch (generally: stable) into the topic branch that's being built: a branch based on master generally satisfies the "Not building from a tag, but last version in changelog was released" error condition until their base branch is merged into it.

The simplest, ad-hoc fix would probably be to move a chunk of code from auto/config to auto/build. I'll handle this.

But thinking about the root cause, a more fundamental issue becomes obvious: merging the base branch happens too late, and by running a fair amount of (build) code before we do that, we create a frankenstein ISO whose build system partly comes from the current status of the topic branch, and partly comes from the current status of it base branch. IMO we should do the base branch merge first thing in the build process, in a manner that's isolated from other build steps, so that all build code (including Rakefile) except the tiny script that performs the merge will be. anonym, bertagaz: if you agree we have a fundamental problem here, please file a ticket about it.


Related issues

Related to Tails - Bug #12557: The mergebasebranch option does the base branch merge too late Confirmed 05/17/2017
Related to Tails - Bug #14857: Some doc branches fail to build an ISO because the base branch of master is "testing" after a major release Resolved 10/17/2017
Blocks Tails - Feature #11355: Re-enable Jenkins notifications on ISO build/test failure In Progress 08/28/2017
Blocks Tails - Feature #13234: Core work 2017Q3: Foundations Team Resolved 06/29/2017
Blocks Tails - Bug #12641: Comment changes in POT files make ISO builds non-reproducible Resolved 06/05/2017

Associated revisions

Revision 7396e42e (diff)
Added by intrigeri about 2 years ago

Merge base branch earlier, i.e. in auto/config instead of auto/build (refs: #14459).

Previously, a given build from a topic branch would mix inconsistent versions
of things. Most changes done in $topic_branch..$base_branch would be taken into
account, but some would not, e.g. changes that affect:

  • everything we set up in auto/config, such as
    - copying tails-transform-mirror-url,
    gnome-shell-extension-florence-indicator
    - APT snapshots
    - debian/changelog, used by `apt-snapshots-serials prepare-build'
  • auto/build

In practice, we've been suffering from some consequences of this problem
e.g.:

  • ISO build failed on documentation branches based on master;
  • topic branches fail to build building once the APT snapshots they encode
    disappears, even though their base branch encodes newer & valid APT
    snapshots.

This commit implements a cheap and partial fix: as stated on
https://labs.riseup.net/code/issues/14459, the base branch merge still happens
too late e.g. to take into account auto/config changes done in
$topic_branch..$base_branch. Ideally we should do the base branch merge first
thing in the build process, in a manner that's isolated from other build steps,
so that all build code except the tiny script that performs the merge will be
in the correct state.

Revision 6a9c3899 (diff)
Added by intrigeri about 2 years ago

Repair build reproducibility (refs: #14459).

In most cases, $BUILD_BASENAME contains a timestamp, so when building twice in
a row from the same commit, we could not have got the same ISO anymore.

Revision f5ed6d9a
Added by anonym about 2 years ago

Merge remote-tracking branch 'origin/bugfix/14459-merge-base-branch-earlier' into master

Fix-committed: #14459

Revision 1bef9097 (diff)
Added by anonym about 2 years ago

Don't use variables that are not set yet.

LB_* variables has not been set yet, so we cannot use them.

This is a quick'n'dirty fixup on the fix for #14459 which already has
been merged.

Refs: #14459

History

#1 Updated by intrigeri about 2 years ago

  • Blocks Feature #11355: Re-enable Jenkins notifications on ISO build/test failure added

#2 Updated by intrigeri about 2 years ago

  • Subject changed from Some documentation branches fail to build on Jenkins to Some documentation branches fail to build an ISO

#3 Updated by intrigeri about 2 years ago

BTW, the same root cause is shared by another recurring painful problem: topic branches stop building once the APT snapshots they encode disappears, even though their base branch encodes newer, valid APT snapshots.

#4 Updated by intrigeri about 2 years ago

  • Description updated (diff)

#5 Updated by intrigeri about 2 years ago

  • Subject changed from Some documentation branches fail to build an ISO to Some branches fail to build an ISO because we merge their base branch too late

#6 Updated by intrigeri about 2 years ago

#7 Updated by intrigeri about 2 years ago

  • Related to Bug #12557: The mergebasebranch option does the base branch merge too late added

#8 Updated by intrigeri about 2 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA
  • Feature Branch set to bugfix/14459-merge-base-branch-earlier

I've tested my fix this way a branch forked off current master, whose base branch is set to stable, fails to build as expected (it has commits on top of the last entry in debian/changelog); cherry-picking my commit on top of that branch fixes the problem.

I wanted to also test a topic branch whose APT snapshots have expired, but this will still fail even with my fix unless the corresponding Vagrant box already exists: we're still building it before we merge the base branch. So something more involved (#12557) is needed to address this problem.

FWIW I've also verified that (locally) after merging my branch into stable and devel doesn't break the build; I didn't wait for the build to finish, I've only waited for lb build to have been run by auto/build.

The topic branch is based on master on purpose, as it should be merged into master (to fix the original problem this ticket was about) in addition to the usual stable & devel. (Once this is done we may have to also merge its base branch into every active topic branch; we'll see.)

#9 Updated by intrigeri about 2 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

Hold on! This branch breaks reproducibility on Jenkins (not for actual releases though, I think) because it encodes BUILD_BASENAME, that contains a timestamp, in /usr/share/amnesia/build/variables.

#10 Updated by intrigeri about 2 years ago

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Dev Needed to Ready for QA

#11 Updated by intrigeri about 2 years ago

  • Blocks Bug #12641: Comment changes in POT files make ISO builds non-reproducible added

#12 Updated by anonym about 2 years ago

  • Status changed from In Progress to Fix committed
  • % Done changed from 50 to 100

#13 Updated by anonym about 2 years ago

  • Assignee deleted (anonym)
  • QA Check changed from Ready for QA to Pass

I agree this isn't ideal and we at some point will need #12557, but the feature branch still clearly improves the situation significantly, so => merged!

#14 Updated by anonym about 2 years ago

  • Assignee set to intrigeri
  • % Done changed from 100 to 90
  • QA Check changed from Pass to Ready for QA

So while I verified that your branch could successfully build an ISO image, I didn't look closely at their file names. I've already pushed a fixup with 1bef9097d16d94752e6a0657a2230fc09c544691. Please have a look!

#15 Updated by anonym about 2 years ago

  • Status changed from Fix committed to In Progress

#16 Updated by intrigeri about 2 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100
  • QA Check changed from Ready for QA to Pass

Looks OK for now as we hardcode amd64 in a few other places anyway. Thanks!

#17 Updated by intrigeri almost 2 years ago

  • Related to Bug #14857: Some doc branches fail to build an ISO because the base branch of master is "testing" after a major release added

Also available in: Atom PDF