Project

General

Profile

Feature #7036

Move custom software to main git repo

Added by intrigeri almost 6 years ago. Updated 14 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
-
Start date:
Due date:
% Done:

67%

Feature Branch:
Type of work:
Code
Starter:
No
Affected tool:

Description

See the blueprint.

  • pythonlib
  • greeter
  • WhisperBack
  • installer
  • perl5lib: done via #15281
  • persistence-setup
  • iuk: done via #15281

Subtasks

Feature #16912: Move greeter source to main git repoResolvedanonym

Feature #16935: Move tailslib to main repoResolvedintrigeri

Feature #16936: Move whisperback source to main repoConfirmedsegfault


Related issues

Related to Tails - Feature #5506: Split Git Rejected 09/11/2013
Related to Tails - Feature #7087: Do not bundle Tor Launcher in the main Tails Git repository Resolved 04/15/2014
Related to Tails - Feature #9171: Refactor tails-custom-apt-sources and tails-diff-suite duplicated code Confirmed 04/07/2015
Related to Tails - Feature #7221: Write a script that deletes old merged Git branches Resolved
Related to Tails - Feature #15864: Make onboarding of new developers easier In Progress 08/30/2018
Related to Tails - Bug #15778: Make the Tails Installer upstream tarball DFSG-free Confirmed 08/09/2018
Related to Tails - Feature #16095: Curate the list of languages in Tails Greeter Resolved 11/04/2018
Related to Tails - Feature #6442: Run Cucumber and Test::Spec tests at tails-iuk Debian package build time Duplicate 11/27/2013
Related to Tails - Feature #10085: Port Tails Installer to Python 3 Needs Validation 08/24/2015
Duplicated by Tails - Feature #16678: Integrate Tails-specific Debian packages into the main git repo Duplicate
Blocks Tails - Feature #12111: APT snapshots: add arm64 architecture Confirmed 01/03/2017
Blocks Tails - Feature #6218: Run our custom programs test suite on interesting commits Confirmed 08/07/2013
Blocks Tails - Feature #6220: Automated Debian package build infrastructure Confirmed 08/07/2013

Associated revisions

Revision 9d600e6a (diff)
Added by intrigeri 29 days ago

Import perl5lib (refs: #7036)

From branch feature/15281-single-squashfs-diff at commit
0bfdd05cc094a08d60f07034afcb71344ac4ec14.

Revision c81d9984 (diff)
Added by intrigeri 29 days ago

perl5lib: remove Git and PO integration that don't make sense in tails.git (refs: #7036)

Revision 70b434d4 (diff)
Added by intrigeri 28 days ago

WIP: adjust to perl5lib being imported into tails.git (refs: #7036)

This also updates the iuk.git snapshot (refs: #15281) with patches come from
iuk.git:feature/15281-single-squashfs-diff at
commit 38ee60cc71761a5b6024e81a52e1c36be051c1ec.

Revision 415ef146 (diff)
Added by intrigeri 28 days ago

Adjust to perl5lib being imported into tails.git (refs: #7036)

This also updates the iuk.git snapshot (refs: #15281) with patches come from
iuk.git:feature/15281-single-squashfs-diff at
commit 38ee60cc71761a5b6024e81a52e1c36be051c1ec.

Revision bd29ea12 (diff)
Added by intrigeri 27 days ago

Import iuk.git (refs: #7036)

Copied from the feature/15281-single-squashfs-diff branch in iuk.git
at commit abe82504d03bacbc3e803c9a9d53d4f359488239.

On top of that:

- Removed PO files as translations will be managed with tails.git's system
- Removed obsolete PO and Git integration
- Moved most dependency management from dist.ini to our "release" doc

Revision 8ad35573 (diff)
Added by intrigeri 27 days ago

Adjust to the tails-iuk codebase having been imported into tails.git (refs: #7036)

Revision bcb5f367 (diff)
Added by intrigeri 27 days ago

Update repository URL (refs: #7036)

Revision 9b80cd13 (diff)
Added by intrigeri 27 days ago

Clean up after installing dists with Dist::Zilla (refs: #7036)

History

#1 Updated by intrigeri almost 6 years ago

#2 Updated by intrigeri over 5 years ago

  • Related to Feature #7087: Do not bundle Tor Launcher in the main Tails Git repository added

#3 Updated by intrigeri over 5 years ago

  • Subject changed from Manage custom software as Git submodules to Manage custom software as Git submodules or similar
  • Type of work changed from Code to Research

#4 Updated by intrigeri over 5 years ago

  • Blueprint set to https://tails.boum.org/blueprint/Git_sub-repositories/

#5 Updated by intrigeri over 5 years ago

  • Description updated (diff)

#6 Updated by intrigeri over 5 years ago

  • Related to Feature #6220: Automated Debian package build infrastructure added

#7 Updated by intrigeri almost 5 years ago

  • Related to Feature #9171: Refactor tails-custom-apt-sources and tails-diff-suite duplicated code added

#8 Updated by intrigeri over 4 years ago

  • Related to Feature #7221: Write a script that deletes old merged Git branches added

#9 Updated by intrigeri 10 months ago

A discussion with @segfault convinced me that at least some of the components that currently live in their own Git repo, and that we include as custom Debian packages in the images we build, should instead live in our main tails.git repo. For example, there's very little value in having Tails Greeter in its own repo, while it does significantly increase friction and overhead when working on its codebase.

#10 Updated by intrigeri 10 months ago

  • Related to Feature #15864: Make onboarding of new developers easier added

#11 Updated by intrigeri 9 months ago

  • Duplicated by Feature #16678: Integrate Tails-specific Debian packages into the main git repo added

#12 Updated by intrigeri 9 months ago

segfault wrote on #16678:

We maintain some Debian packages which are solely used in Tails and cannot reasonably used outside of Tails. IMO, maintaining these as Debian packages makes our lives unnecessarily harder, because we have to release, build, and upload a new Debian package every time we change something in these packages.

We also share code between the main repo and some packages, which makes it hard to find usages/implementations.

The two packages I currently have in mind are tails-greeter and tails-persistence-setup.

I propose that we integrate them into the main git repo by copying the files that are installed by the package into config/chroot_local_includes.

Fully agreed in principle for these 2 repos. Potential difficulties:

  • tails-persistence-setup might require special care to keep its own test suite working
  • PO/POT files management: currently these are translated under their own Transifex resources; I don't know if/how we can preserve existing translations across this migration

I think WhisperBack and Tails Installer would be good candidates for this first batch too.

Some other candidates (tails-iuk, tails-perl5lib) might be a bit less simple since IIRC RMs need to install the resulting packages on their system, or similar. Let's do some easier ones first :)

Next step: try this for one repo and see how it goes wrt. POT/PO files.

#13 Updated by segfault 6 months ago

  • Assignee set to segfault

#14 Updated by segfault 6 months ago

  • Subject changed from Manage custom software as Git submodules or similar to Move custom software to main git repo

#15 Updated by segfault 6 months ago

I would like to start with the greeter, and then move pythonlib to config/chroot_local_includes.

#16 Updated by intrigeri 6 months ago

  • Related to Bug #15778: Make the Tails Installer upstream tarball DFSG-free added

#17 Updated by sajolida 5 months ago

  • Related to Feature #16095: Curate the list of languages in Tails Greeter added

#18 Updated by intrigeri 4 months ago

#19 Updated by intrigeri 4 months ago

  • Blocks Feature #6218: Run our custom programs test suite on interesting commits added

#20 Updated by intrigeri 4 months ago

  • Related to deleted (Feature #6220: Automated Debian package build infrastructure)

#21 Updated by intrigeri 4 months ago

  • Blocks Feature #6220: Automated Debian package build infrastructure added

#22 Updated by intrigeri 3 months ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress
  • Type of work changed from Research to Code

#23 Updated by intrigeri 3 months ago

intrigeri wrote:

Some other candidates (tails-iuk, tails-perl5lib) might be a bit less simple since IIRC RMs need to install the resulting packages on their system, or similar.

I thought about this again while working in these 2 repositories yesterday and remembering how painful it is to validate such work in the context of a real Tails system, anticipating on how my work on #15281 would be easier if we integrated these code bases in a different manner.

So I've looked into this problem a bit and our 3 Perl codebases (perl5lib, iuk, persistence) are in similar situations:

  • The main added complexity in the upstream and Debian build systems comes from the localization stuff, which we would simply get rid of if we moved these files to tails.git: we already have all the bits in place to manage Perl + Locale::gettext there. So we lose nothing on this front, yeah.
  • Regarding automated tests run by developers, reviewers, RMs, and/or some kind of CI, this is the most unclear part for me at the moment. It depends on when/where we want to run these tests.
    • Dist::Zilla generates some static analysis tests automatically: code & POD formatting. Perhaps we can drop a dist.ini file at the root of tails.git and configure/tweak things so that we don't have to reinvent this wheel or to switch to a totally different system.
    • On top of the static code analysis mentioned above, 2 of these 3 codebases have quite extensive test suites (rspec and cucumber -style). We currently run them whenever we prepare new releases of these codebases. Some of the tests need to do privileged operations which might be tricky in container environments (thinking of GitLab CI). They might occasionally leave the system in a dirty state when something goes really wrong. This set of constraints looks suspiciously like the Tails test suite's ones so dedicated tester VMs like we have on Jenkins could fit the bill. But ideally we want to run these tests on the same version of Debian as the one which the target Tails system is based on (we currently get part of that via pbuilder/sbuild, for the tests that can run in there); as it happens, currently our isotesters run Stretch but we target Buster; this has happened in the past and will happen again. This seems to call for a Vagrant-like system to run these test suites in a suitable environment.
    • For the first iteration, IMO we should only aim to provide means for developers, reviewers, and RMs to run these tests locally, sometimes with different versions of the dependencies than what ends up in Tails, just like they've been doing so far. Running this in CI can come later.
  • Regarding RMs and dependency tracking, there are two things:
    • In our release process we simply point the PERL5LIB environment variable to wherever the RM has cloned this repo, so if we move these files to tails.git, we can get make this a constant; great.
    • RMs run code that lives in the tails-iuk package, which currently depends on the tails-perl5lib package, so they get the dependencies they need via APT, because we currently maintain the list of these dependencies in debian/control. If we move perl5lib to tails.git, then we need a different way for RMs to know which dependencies they need to install. The option I'm thinking of at the moment is to list these dependencies in a new config/chroot_local-packageslists/tails-perl5lib.list file. Compared to the current debian/control and dist.ini dependency management:
      • We lose the ability to declare versioned dependencies, but I'm willing to take the risk.
      • We lose the ability to declare dependencies that are only needed to run the tests. I can think of various solutions already, depending on where/when we want to run the tests, so I'm not too worried about this.

While working on #15281, if I get too annoyed by the current state of things, I might give a try to the cheapest possible way to do this migration, and we'll see how it goes.

#24 Updated by segfault 3 months ago

intrigeri wrote:

  • On top of the static code analysis mentioned above, 2 of these 3 codebases have quite extensive test suites (rspec and cucumber -style). We currently run them whenever we prepare new releases of these codebases. Some of the tests need to do privileged operations which might be tricky in container environments (thinking of GitLab CI).

GitLab CI also supports VM executors. VirtualBox is supported natively [1], and using libvirt is possible via the custom executor [2] (but I never used the latter, so I don't know how well that works).

[1] https://docs.gitlab.com/runner/executors/virtualbox.html
[2] https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html

While working on #15281, if I get too annoyed by the current state of things, I might give a try to the cheapest possible way to do this migration, and we'll see how it goes.

Cool!

#25 Updated by intrigeri 3 months ago

  • Related to Feature #6442: Run Cucumber and Test::Spec tests at tails-iuk Debian package build time added

#26 Updated by intrigeri 2 months ago

#27 Updated by intrigeri 28 days ago

  • Description updated (diff)

#28 Updated by intrigeri 28 days ago

  • Description updated (diff)

#29 Updated by intrigeri 26 days ago

  • Description updated (diff)

OK, so perl5lib and iuk are done via #15281. I'm glad I found a way to get the benefits from the move (this greatly simplified the work I did on #15281 and friends in the last few days), without losing the benefits of storing each of those codebases as a standard Perl source dist managed with Dist::Zilla. Wrt. Perl code, for me it's the best of both worlds :) I did not look at persistence-setup yet; likely the same recipe will work, but there might be something special; we'll see; chances are that I migrate it next time I have to work on this codebase.

#30 Updated by segfault 14 days ago

  • Assignee deleted (segfault)

intrigeri wrote:

OK, so perl5lib and iuk are done via #15281. I'm glad I found a way to get the benefits from the move (this greatly simplified the work I did on #15281 and friends in the last few days), without losing the benefits of storing each of those codebases as a standard Perl source dist managed with Dist::Zilla. Wrt. Perl code, for me it's the best of both worlds :) I did not look at persistence-setup yet; likely the same recipe will work, but there might be something special; we'll see; chances are that I migrate it next time I have to work on this codebase.

Awesome! I'm unassigning myself since I'm not the only one working on this :)

Also available in: Atom PDF