Project

General

Profile

Feature #7036

Move custom software to main git repo

Added by intrigeri over 5 years ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
-
Start date:
Due date:
% Done:

67%

Feature Branch:
Type of work:
Code
Starter:
No
Affected tool:

Description

See the blueprint.

  • pythonlib
  • greeter
  • WhisperBack
  • installer
  • perl5lib
  • persistence-setup
  • iuk

Subtasks

Feature #16912: Move greeter source to main git repoResolvedanonym

Feature #16935: Move tailslib to main repoResolvedintrigeri

Feature #16936: Move whisperback source to main repoConfirmedsegfault


Related issues

Related to Tails - Feature #5506: Split Git Rejected 09/11/2013
Related to Tails - Feature #7087: Do not bundle Tor Launcher in the main Tails Git repository Resolved 04/15/2014
Related to Tails - Feature #9171: Refactor tails-custom-apt-sources and tails-diff-suite duplicated code Confirmed 04/07/2015
Related to Tails - Feature #7221: Write a script that deletes old merged Git branches Resolved
Related to Tails - Feature #15864: Make onboarding of new developers easier In Progress 08/30/2018
Related to Tails - Bug #15778: Make the Tails Installer upstream tarball DFSG-free Confirmed 08/09/2018
Related to Tails - Feature #16095: Curate the list of languages in Tails Greeter Resolved 11/04/2018
Related to Tails - Feature #6442: Run Cucumber and Test::Spec tests at tails-iuk Debian package build time Confirmed 11/27/2013
Related to Tails - Feature #10085: Port Tails Installer to Python 3 Needs Validation 08/24/2015
Duplicated by Tails - Feature #16678: Integrate Tails-specific Debian packages into the main git repo Duplicate
Blocks Tails - Feature #12111: APT snapshots: add arm64 architecture Confirmed 01/03/2017
Blocks Tails - Feature #6218: Run our custom programs test suite on interesting commits Confirmed 08/07/2013
Blocks Tails - Feature #6220: Automated Debian package build infrastructure Confirmed 08/07/2013

History

#1 Updated by intrigeri over 5 years ago

#2 Updated by intrigeri over 5 years ago

  • Related to Feature #7087: Do not bundle Tor Launcher in the main Tails Git repository added

#3 Updated by intrigeri over 5 years ago

  • Subject changed from Manage custom software as Git submodules to Manage custom software as Git submodules or similar
  • Type of work changed from Code to Research

#4 Updated by intrigeri over 5 years ago

  • Blueprint set to https://tails.boum.org/blueprint/Git_sub-repositories/

#5 Updated by intrigeri over 5 years ago

  • Description updated (diff)

#6 Updated by intrigeri over 5 years ago

  • Related to Feature #6220: Automated Debian package build infrastructure added

#7 Updated by intrigeri over 4 years ago

  • Related to Feature #9171: Refactor tails-custom-apt-sources and tails-diff-suite duplicated code added

#8 Updated by intrigeri over 4 years ago

  • Related to Feature #7221: Write a script that deletes old merged Git branches added

#9 Updated by intrigeri 8 months ago

A discussion with @segfault convinced me that at least some of the components that currently live in their own Git repo, and that we include as custom Debian packages in the images we build, should instead live in our main tails.git repo. For example, there's very little value in having Tails Greeter in its own repo, while it does significantly increase friction and overhead when working on its codebase.

#10 Updated by intrigeri 8 months ago

  • Related to Feature #15864: Make onboarding of new developers easier added

#11 Updated by intrigeri 8 months ago

  • Duplicated by Feature #16678: Integrate Tails-specific Debian packages into the main git repo added

#12 Updated by intrigeri 8 months ago

segfault wrote on #16678:

We maintain some Debian packages which are solely used in Tails and cannot reasonably used outside of Tails. IMO, maintaining these as Debian packages makes our lives unnecessarily harder, because we have to release, build, and upload a new Debian package every time we change something in these packages.

We also share code between the main repo and some packages, which makes it hard to find usages/implementations.

The two packages I currently have in mind are tails-greeter and tails-persistence-setup.

I propose that we integrate them into the main git repo by copying the files that are installed by the package into config/chroot_local_includes.

Fully agreed in principle for these 2 repos. Potential difficulties:

  • tails-persistence-setup might require special care to keep its own test suite working
  • PO/POT files management: currently these are translated under their own Transifex resources; I don't know if/how we can preserve existing translations across this migration

I think WhisperBack and Tails Installer would be good candidates for this first batch too.

Some other candidates (tails-iuk, tails-perl5lib) might be a bit less simple since IIRC RMs need to install the resulting packages on their system, or similar. Let's do some easier ones first :)

Next step: try this for one repo and see how it goes wrt. POT/PO files.

#13 Updated by segfault 5 months ago

  • Assignee set to segfault

#14 Updated by segfault 5 months ago

  • Subject changed from Manage custom software as Git submodules or similar to Move custom software to main git repo

#15 Updated by segfault 5 months ago

I would like to start with the greeter, and then move pythonlib to config/chroot_local_includes.

#16 Updated by intrigeri 4 months ago

  • Related to Bug #15778: Make the Tails Installer upstream tarball DFSG-free added

#17 Updated by sajolida 4 months ago

  • Related to Feature #16095: Curate the list of languages in Tails Greeter added

#18 Updated by intrigeri 3 months ago

#19 Updated by intrigeri 3 months ago

  • Blocks Feature #6218: Run our custom programs test suite on interesting commits added

#20 Updated by intrigeri 3 months ago

  • Related to deleted (Feature #6220: Automated Debian package build infrastructure)

#21 Updated by intrigeri 3 months ago

  • Blocks Feature #6220: Automated Debian package build infrastructure added

#22 Updated by intrigeri 2 months ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress
  • Type of work changed from Research to Code

#23 Updated by intrigeri 2 months ago

intrigeri wrote:

Some other candidates (tails-iuk, tails-perl5lib) might be a bit less simple since IIRC RMs need to install the resulting packages on their system, or similar.

I thought about this again while working in these 2 repositories yesterday and remembering how painful it is to validate such work in the context of a real Tails system, anticipating on how my work on #15281 would be easier if we integrated these code bases in a different manner.

So I've looked into this problem a bit and our 3 Perl codebases (perl5lib, iuk, persistence) are in similar situations:

  • The main added complexity in the upstream and Debian build systems comes from the localization stuff, which we would simply get rid of if we moved these files to tails.git: we already have all the bits in place to manage Perl + Locale::gettext there. So we lose nothing on this front, yeah.
  • Regarding automated tests run by developers, reviewers, RMs, and/or some kind of CI, this is the most unclear part for me at the moment. It depends on when/where we want to run these tests.
    • Dist::Zilla generates some static analysis tests automatically: code & POD formatting. Perhaps we can drop a dist.ini file at the root of tails.git and configure/tweak things so that we don't have to reinvent this wheel or to switch to a totally different system.
    • On top of the static code analysis mentioned above, 2 of these 3 codebases have quite extensive test suites (rspec and cucumber -style). We currently run them whenever we prepare new releases of these codebases. Some of the tests need to do privileged operations which might be tricky in container environments (thinking of GitLab CI). They might occasionally leave the system in a dirty state when something goes really wrong. This set of constraints looks suspiciously like the Tails test suite's ones so dedicated tester VMs like we have on Jenkins could fit the bill. But ideally we want to run these tests on the same version of Debian as the one which the target Tails system is based on (we currently get part of that via pbuilder/sbuild, for the tests that can run in there); as it happens, currently our isotesters run Stretch but we target Buster; this has happened in the past and will happen again. This seems to call for a Vagrant-like system to run these test suites in a suitable environment.
    • For the first iteration, IMO we should only aim to provide means for developers, reviewers, and RMs to run these tests locally, sometimes with different versions of the dependencies than what ends up in Tails, just like they've been doing so far. Running this in CI can come later.
  • Regarding RMs and dependency tracking, there are two things:
    • In our release process we simply point the PERL5LIB environment variable to wherever the RM has cloned this repo, so if we move these files to tails.git, we can get make this a constant; great.
    • RMs run code that lives in the tails-iuk package, which currently depends on the tails-perl5lib package, so they get the dependencies they need via APT, because we currently maintain the list of these dependencies in debian/control. If we move perl5lib to tails.git, then we need a different way for RMs to know which dependencies they need to install. The option I'm thinking of at the moment is to list these dependencies in a new config/chroot_local-packageslists/tails-perl5lib.list file. Compared to the current debian/control and dist.ini dependency management:
      • We lose the ability to declare versioned dependencies, but I'm willing to take the risk.
      • We lose the ability to declare dependencies that are only needed to run the tests. I can think of various solutions already, depending on where/when we want to run the tests, so I'm not too worried about this.

While working on #15281, if I get too annoyed by the current state of things, I might give a try to the cheapest possible way to do this migration, and we'll see how it goes.

#24 Updated by segfault 2 months ago

intrigeri wrote:

  • On top of the static code analysis mentioned above, 2 of these 3 codebases have quite extensive test suites (rspec and cucumber -style). We currently run them whenever we prepare new releases of these codebases. Some of the tests need to do privileged operations which might be tricky in container environments (thinking of GitLab CI).

GitLab CI also supports VM executors. VirtualBox is supported natively [1], and using libvirt is possible via the custom executor [2] (but I never used the latter, so I don't know how well that works).

[1] https://docs.gitlab.com/runner/executors/virtualbox.html
[2] https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html

While working on #15281, if I get too annoyed by the current state of things, I might give a try to the cheapest possible way to do this migration, and we'll see how it goes.

Cool!

#25 Updated by intrigeri about 2 months ago

  • Related to Feature #6442: Run Cucumber and Test::Spec tests at tails-iuk Debian package build time added

#26 Updated by intrigeri about 1 month ago

Also available in: Atom PDF