Project

General

Profile

Feature #6218

Run our custom programs test suite on interesting commits

Added by intrigeri about 6 years ago. Updated about 2 years ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
-
Start date:
08/07/2013
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
No
Affected tool:

Description

For now the scope of this ticket is limited to our 3 Perl code bases (perl5lib, persistence-setup, iuk)… simply because our other custom programs (sadly) have no test suite.

One way to do this would be to run our custom programs' test suite, using Vagrant, on our existing isobuilders (that already have Vagrant installed and some room for storing baseboxes) or isotesters (which would feel more logical, because "testers") or both. What it takes is:

  • at the root of each Git repo we want to test:
    • add a suitable Vagrantfile that installs all build/test deps during provisioning
    • add a Makefile/Rakefile/script that starts the Vagrant VM, runs the test suite in there, and tears down the VM
  • for each Git repo we want to test:
    • create a Jenkins job that runs the aforementioned script / rake test / make test

Open questions:

  • how to deal with inter-dependencies between these code bases? e.g. persistence-setup and iuk depend on perl5lib; our options:
    • start by testing perl5lib that has no such problem, gain experience with the general idea during this first iteration, and think about this problem later iff. the general idea is validated
    • install current perl5lib master in the VM with dzil install: this works iff. we don't delegate to the Debian packaging anything that's required to use perl5lib (currently we install MO files via debian/rules, but perhaps we can do without them here)
    • set up the APT source for our devel custom APT repo and treat perl5lib just like any other build dependency
  • how to avoid creating yet another complex setup for managing baseboxes? our options:
    • don't keep any basebox persistently on the Jenkins "slaves" (sic :/) that run the tests, instead download & provision them every time; we can probably leverage our existing apt-cacher-ng to speed things up (see file:///usr/share/doc/apt-cacher-ng/html/howtos.html#ssluse to make caching of HTTPS downloads work). This option implies we use Vagrant baseboxes published by third-parties (otherwise we're just moving the basebox management problem elsewhere) which is probably acceptable here; an official Debian Stretch Vagrant box is available, and we can dist-upgrade it during provisioning to whatever version of Debian we want to test on.
    • use the baseboxes we already use & manage for building ISO images: this probably requires quite some refactoring to extract code from tails.git, and forces us to test only on whatever snapshot of Debian we currently use, while we would like to test on current Debian sid too.
    • other ideas?
  • how to store or compute build & test dependencies? We already store them in two different places (dist.ini on the master branch, debian/control on the debian branch) so it would be nice to avoid adding a 3rd one. Likely the easiest way is to extract build deps from the debian/control file and install them with something like controlfile=$(mktemp) ; git show origin/debian:debian/control > $controlfile ; mk-build-deps -i -r --root-cmd sudo $controlfile ; rm $controlfile
  • how does this relate to #7036 and #6220#note-11? Can we implement this ticket in a way that also makes us progress on that other front? At least having to deal with custom APT sources, APT snapshots, build deps, will teach us a thing or three about the issues we'll face when we tackle that other project, but perhaps we can do better.

Ideally we would run these tests both on Debian sid (to detect breakage in advance and make our RM's life easier), whatever snapshot of Debian we currently use, and an unfrozen version of the Debian release we use. But already running them on one of them (probably sid) would be a pretty good start!


Subtasks

Feature #6217: Set up a Perl tester Jenkins slaveRejected


Related issues

Blocked by Tails - Feature #7036: Move custom software to main git repo In Progress

History

#1 Updated by intrigeri about 2 years ago

  • Description updated (diff)

#14674 prompted me to think about this topic: if we had this, then I would automatically get actionable bug reports from Jenkins about test suite fragility, breakage caused by updates in Debian sid, and of course actual bugs in the software under testing. And then, hopefully I'll be able to resolve these problems before anonym faces them.

#2 Updated by intrigeri about 2 years ago

anonym & bertagaz, please take a look at the plan outlined in the description, and shout if it feels like the wrong approach, if you have better ideas (be it for the big picture or specific implementation details).

#4 Updated by intrigeri about 2 years ago

  • Description updated (diff)

#5 Updated by anonym about 2 years ago

intrigeri wrote:

anonym & bertagaz, please take a look at the plan outlined in the description, and shout if it feels like the wrong approach [...]

I think it sounds great! I feel I'm too unfamiliar with how things work on Jenkins to comment about specifics. However, I'd love if it was integrated in our existing rake setup some how so I can also use it locally, but this would probably only make sense to do if we also do #6220 and solve it with rake + Vagrant.

#6 Updated by intrigeri 28 days ago

  • Blocked by Feature #7036: Move custom software to main git repo added

Also available in: Atom PDF