Project

General

Profile

Feature #6220

Automated Debian package build infrastructure

Added by intrigeri about 6 years ago. Updated about 1 year ago.

Status:
Confirmed
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
-
Start date:
08/07/2013
Due date:
% Done:

0%

Starter:
No
Affected tool:

Description

It's been done before, no need to reinvent the wheel. See the blueprint for potentially useful tools.

Team: bertagaz


Related issues

Related to Tails - Feature #7036: Move custom software to main git repo Confirmed
Related to Tails - Feature #6090: Automated builds Resolved 07/26/2013 02/28/2015
Related to Tails - Feature #14455: Reproducible Builds Stage 2 Confirmed 08/26/2017
Related to Tails - Feature #6219: Document our autobuild setup Rejected 08/07/2013

History

#1 Updated by Dr_Whax over 5 years ago

The Debile link is not working, this one is: http://anonscm.debian.org/gitweb/?p=pkg-debile/debile.git

#2 Updated by intrigeri over 5 years ago

  • Description updated (diff)

#3 Updated by intrigeri over 5 years ago

  • Description updated (diff)
  • Blueprint set to https://tails.boum.org/blueprint/automated_builds_and_tests/#resources-debian-pkg

#4 Updated by intrigeri about 5 years ago

  • Blueprint changed from https://tails.boum.org/blueprint/automated_builds_and_tests/#resources-debian-pkg to https://tails.boum.org/blueprint/automated_builds_and_tests/Debian_packages/

#5 Updated by intrigeri about 5 years ago

  • Related to Feature #7036: Move custom software to main git repo added

#6 Updated by intrigeri about 5 years ago

#7 Updated by sajolida almost 5 years ago

  • Target version set to Sustainability_M1

#8 Updated by sajolida about 4 years ago

  • Description updated (diff)

#9 Updated by sajolida almost 4 years ago

  • Target version changed from Sustainability_M1 to 2016

#10 Updated by Dr_Whax almost 3 years ago

  • Description updated (diff)
  • Assignee set to bertagaz
  • Target version changed from 2016 to 2018

#11 Updated by intrigeri about 2 years ago

At this point I'm happy we did not set this up server-side, in a centralized manner, because I've entirely changed my mind wrt. how we should approach the problem at hand.

IMO we should build our custom packages as part of building an ISO (using good ol' sbuild/pbuilder or something that better isolates package builds from one another, such as https://gitlab.com/uhoreg/whalebuilder or one of the other options), and track their source code as Git submodules i.e. #7036 (for non-native packages, likely we want to track the branch that contains the Debian packaging).

This advantages I see with this approach, compared to what we currently do, and to extending our central infrastructure to build packages automatically:

  1. We get more value out of reproducible ISO builds, by removing binary input and the required trust in the content of these packages, in the developer who built+uploaded them, and in our custom APT repo infrastructure.
  2. Developers who don't have Git commit access can work on our custom software and build an ISO with their modified version in a straightforward way (no need to have a pbuilder setup nor to copy .deb's to config/chroot_local-packages, and the result of the work can be handled as any other Git pull request).
  3. No need for all developers to set up and maintain a local pbuilder setup with N chroots.
  4. We can get rid of our custom APT repository and, perhaps more importantly, of the complex & error-prone workflow that comes with it (e.g. for merging/resetting APT suites). Everything related to config/APT_overlays.d/ can go away, which simplifies our codebase and infrastructure.
  5. More parameters that affect the resulting ISO are encoded in Git, which makes them easier to audit, log, and diff. The only remaining thing that's not encoded in Git is the content of the APT snapshots we use (we already encode in Git which set of APT snapshots we use).
  6. We simplify what it means to have Git commit access: developers who have Git commit access don't also need upload rights to our custom APT repo. This simplifies sysadmin (no need to keep OpenPGP keys up-to-date) and makes it easier for developers to understand how the whole thing works.
  7. The apt-cacher-ng cache can be shared between ISO build process and .deb packages build process.
  8. We get finer-grained control over what exact state of the pbuilder/sbuild/whalebuilder chroot we use for building packages. E.g. we might want to use exactly the same APT snapshots that are used for building the rest of the ISO, which can matter a lot if we switch to Debian testing (#12615): building against current Debian testing can create packages that are not installable on the (older) snapshot of Debian testing we use in the SquashFS.
  9. More of us have the skills and access needed to work, within our Git tree, on such a Debian packages build system, than to work on a centralized package build system hosted our infrastructure. This implies that we have more options wrt. who implements, reviews & maintains the new thing, instead of increasing our reliance on a tiny sysadmin team that would use a totally different stack (Puppet, Jenkins, jenkins-debian-glue, etc.).
  10. Compared to what we do today, we have the option to enforce some quality checks on the packaging and upstream code (e.g. Lintian or autopkgtests).
  11. No need to design & implement yet another mapping between Git branches, Jenkins jobs and APT suites.
  12. No race conditions / async design skills required: we're guaranteed that the expected version of every custom package we want has been built when we need it.

#12 Updated by intrigeri almost 2 years ago

  • Target version deleted (2018)

(as per updated roadmap)

#14 Updated by intrigeri almost 2 years ago

#15 Updated by u about 1 year ago

#16 Updated by u about 1 year ago

Also available in: Atom PDF