Some branches fail to build an ISO because we merge their base branch too late
Our ISO build system errors out when building from a branch based on master:
> + apt-snapshots-serials prepare-build > + apt-mirror debian > Not building from a tag, but last version in changelog was released > + DEBIAN_MIRROR= > + apt-mirror debian-security > Not building from a tag, but last version in changelog was released > + DEBIAN_SECURITY_MIRROR= > + apt-mirror torproject > Not building from a tag, but last version in changelog was released > + TORPROJECT_MIRROR= > + [ -n ] > + exit 1
I believe this is caused by
apt-snapshots-serials prepare-build before
auto/build had a chance to merge the base branch (generally:
stable) into the topic branch that's being built: a branch based on master generally satisfies the "Not building from a tag, but last version in changelog was released" error condition until their base branch is merged into it.
The simplest, ad-hoc fix would probably be to move a chunk of code from
auto/build. I'll handle this.
But thinking about the root cause, a more fundamental issue becomes obvious: merging the base branch happens too late, and by running a fair amount of (build) code before we do that, we create a frankenstein ISO whose build system partly comes from the current status of the topic branch, and partly comes from the current status of it base branch. IMO we should do the base branch merge first thing in the build process, in a manner that's isolated from other build steps, so that all build code (including
Rakefile) except the tiny script that performs the merge will be. anonym, bertagaz: if you agree we have a fundamental problem here, please file a ticket about it.
Merge base branch earlier, i.e. in auto/config instead of auto/build (refs: #14459).
Previously, a given build from a topic branch would mix inconsistent versions
of things. Most changes done in $topic_branch..$base_branch would be taken into
account, but some would not, e.g. changes that affect:
- everything we set up in auto/config, such as
- copying tails-transform-mirror-url,
- APT snapshots
- debian/changelog, used by `apt-snapshots-serials prepare-build'
In practice, we've been suffering from some consequences of this problem
- ISO build failed on documentation branches based on master;
- topic branches fail to build building once the APT snapshots they encode
disappears, even though their base branch encodes newer & valid APT
This commit implements a cheap and partial fix: as stated on
https://labs.riseup.net/code/issues/14459, the base branch merge still happens
too late e.g. to take into account auto/config changes done in
$topic_branch..$base_branch. Ideally we should do the base branch merge first
thing in the build process, in a manner that's isolated from other build steps,
so that all build code except the tiny script that performs the merge will be
in the correct state.
Repair build reproducibility (refs: #14459).
In most cases, $BUILD_BASENAME contains a timestamp, so when building twice in
a row from the same commit, we could not have got the same ISO anymore.
Merge remote-tracking branch 'origin/bugfix/14459-merge-base-branch-earlier' into master
#8 Updated by intrigeri over 2 years ago
- Status changed from Confirmed to In Progress
- Assignee changed from intrigeri to anonym
- % Done changed from 0 to 50
- QA Check set to Ready for QA
- Feature Branch set to bugfix/14459-merge-base-branch-earlier
I've tested my fix this way a branch forked off current master, whose base branch is set to
stable, fails to build as expected (it has commits on top of the last entry in
debian/changelog); cherry-picking my commit on top of that branch fixes the problem.
I wanted to also test a topic branch whose APT snapshots have expired, but this will still fail even with my fix unless the corresponding Vagrant box already exists: we're still building it before we merge the base branch. So something more involved (#12557) is needed to address this problem.
FWIW I've also verified that (locally) after merging my branch into stable and devel doesn't break the build; I didn't wait for the build to finish, I've only waited for
lb build to have been run by
The topic branch is based on master on purpose, as it should be merged into master (to fix the original problem this ticket was about) in addition to the usual stable & devel. (Once this is done we may have to also merge its base branch into every active topic branch; we'll see.)
#9 Updated by intrigeri over 2 years ago
- Assignee changed from anonym to intrigeri
- QA Check changed from Ready for QA to Dev Needed
Hold on! This branch breaks reproducibility on Jenkins (not for actual releases though, I think) because it encodes
BUILD_BASENAME, that contains a timestamp, in
#14 Updated by anonym over 2 years ago
- Assignee set to intrigeri
- % Done changed from 100 to 90
- QA Check changed from Pass to Ready for QA
So while I verified that your branch could successfully build an ISO image, I didn't look closely at their file names. I've already pushed a fixup with 1bef9097d16d94752e6a0657a2230fc09c544691. Please have a look!