Feature #5630: Reproducible builds
reproducibly_build_Tails_ISO_stable Jenkins job always fails
We're quite unlucky:
build_Tails_ISO_stable is scheduled daily, and with Jenkins' hashing system it happens to run at 07:15 so
reproducibly_build_Tails_ISO_stable starts around 08:00. We update the Debian security APT snapshot at 7:41 so in practice,
reproducibly_build_Tails_ISO_stable always gets a different snapshot and thus always produces a different ISO.
This prevents us from easily noticing when an update or branch merge broke reproducibility. And possibly worse, it teaches us to ignore email notifications from Jenkins (this has been going on for a week and nobody noticed or if they did, then they kept it to themselves for some reason).
Due to the way our Jenkins jobs are generated, we have little flexibility here: we can't simply reschedule
build_Tails_ISO_stable (and only this one) to run at a different time. And if we start playing with the H symbol we'll be lucky if we don't break devel or testing.
The simplest option we have seems to slightly reschedule the APT snapshot update. Now, of course this will likely break "reproducibly" (sic) jobs for other branches, but until we have a better solution, not breaking stable/testing/devel gets higher priority by far. While implementing this I'll need to be careful not to break testing or devel, but at least I can easily reason about it, while I don't know how the Jenkins hash thing works exactly.
Long term, the only sane option is probably to teach our build system how to use snapshots specified on the command line (instead of "latest"), and to teach Jenkins to pass the correct parameters when trying to rebuild a given ISO image. Whoever feels responsible for this, please file a ticket about it.
#3 Updated by intrigeri almost 2 years ago
- % Done changed from 0 to 50
- QA Check set to Ready for QA
Let's assume a 70 min delay between the time when the 1st build resolves the "latest" snapshot pointer and the time when the 2nd build does (currently we're around 52 minutes but let's be on the safe side: the longer this delay, the more chances the problem this ticket is about happens). Let's assume this pointer resolution happens 1st thing during a build to simplify (in practice it's more ~2 minutes later, whatever).
- daily build of stable starts at 07:15 => the debian-security repo should not be updated between 07:15 and 08:25
- daily build of testing starts at 15:23 => the debian-security repo should not be updated between 15:23 and 16:33
- daily build of devel starts at 06:56 => no repo should be updated between 06:56 and 08:06
- daily build of feature/buster starts at 09:15 => no repo should be updated between 09:15 and 10:25
- daily build of feature/tor-nightly-master starts at 08:15 => no repo should be updated between 08:15 and 09:25
So ideally no repo should be updated between 06:56 and 10:25, and the debian-security repo should not be updated between 15:23 and 16:33. I've reworked the way we schedule these cronjobs so this is now guaranteed: all updates now start between 05:00 and 05:45, between 11:00 and 11:45, between 17:00 and 17:45, and between 23:00 and 23:45 (UTC).
The catch is that I've assumed the repo updates take 0 time, which is of course incorrect. But as long as no such update takes more than ~1 hour we should be good. Anyway, I think that this change will already improve things in the vast majority of cases. If practice proves me wrong I'll come back to it.
If Jenkins ever changes its hashing algorithm that determines when "daily" jobs are run for a given branch, we'll need to come back to it and possibly adjust
I'll now wait ~36 hours so that all common cases have been tested.
#8 Updated by intrigeri almost 2 years ago
- % Done changed from 50 to 60
Daily build of devel is OK too. For
feature/tor-nightly-master I'll wait for #14946 to be fixed before fixing the FTBFS (=> saving storage space). We'll see if this happens early enough (i.e. today) for me to have data here before the 3.3 release.
#9 Updated by intrigeri almost 2 years ago
- Target version changed from Tails_3.3 to Tails_3.5
Daily build of devel is OK too. For
feature/tor-nightly-masterI'll wait for #14946 to be fixed before fixing the FTBFS (=> saving storage space). We'll see if this happens early enough (i.e. today) for me to have data here before the 3.3 release.
The merges happened too late for me to get data already, so postponing.