Project

General

Profile

Feature #16935

Feature #7036: Move custom software to main git repo

Move tailslib to main repo

Added by segfault about 2 months ago. Updated 7 days ago.

Status:
Needs Validation
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
feature/16935-move-tailslib-to-main-repo+force-all-tests, puppet-tails:feature/16935-move-tailslib-to-main-repo
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Related issues

Blocks Tails - Feature #16664: Simplify "Tor has bootstrapped" status check Needs Validation
Blocks Tails - Feature #17048: Make the tails::pythonlib Puppet class track the stable branch of tails.git Confirmed
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017

Associated revisions

Revision 31e7bee1 (diff)
Added by segfault about 2 months ago

Move pythonlib to main repo and remove the submodule (refs: #16935)

Revision 6087ca6f (diff)
Added by segfault 10 days ago

Move pythonlib to main repo and remove the submodule (refs: #16935)

Revision 5acfdb84 (diff)
Added by segfault 10 days ago

Import latest changes from submodules/pythonlib (refs: #16935)

This imports the changes of the following pythonlib commits:

8dc57383e02c20a9c34d9582db30bedf14bc6468
67347b20000871ee6f01b04bf2bbdb81d46f62e2
29a894a320237f6d186f7cc4e0212f4ecd0beb95
cd812663fc47ca2c93257ae8968917282c419089

Revision aeb2f629 (diff)
Added by intrigeri 10 days ago

Reintroduce lib/python3/tailslib convenience symlink (refs: #16935).

Our release process doc relies on this symlink. Also, folks may have set up
stuff locally that relies on this symlink as well. I don't see any
serious drawback in keeping it around.

Fixup against 6087ca6f124995d91864ba847fe258ffee5e3411.

Revision dfb9f49d
Added by segfault 7 days ago

Merge branch 'feature/16935-move-tailslib-to-main-repo+force-all-tests' into devel (Closes: #16935)

History

#1 Updated by intrigeri about 2 months ago

If this is about pythonlib.git: it has other consumers than tails.git. I can think of one (our Jenkins jobs generator). I don't know by heart if there are more.

#2 Updated by intrigeri about 2 months ago

  • Parent task set to #7036

#3 Updated by segfault about 2 months ago

  • Parent task deleted (#7036)
  • Feature Branch set to feature/16935-move-tailslib-to-main-repo+force-all-tests

This seems to work. A few notes though:

  • The feature branch is based on feature/16912-move-greeter-to-main-git-repo+force-all-tests
  • Moving tailslib to config/chroot_local_includes/usr/local/lib/python3/ caused the build to fail with this error:
P: Begin building root filesystem image...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  squashfs-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 125 kB of archives.
After this operation, 305 kB of additional disk space will be used.
Get:1 http://deb.tails.boum.org stable/main amd64 squashfs-tools amd64 1:4.3-3.0tails4 [125 kB]
Traceback (most recent call last):
  File "/usr/local/sbin/tails-additional-software", line 16, in <module>
    from tailslib import LIVE_USERNAME
ImportError: No module named 'tailslib'

I moved it to config/chroot_local_includes/usr/lib/python3/ instead, which works fine. The cause of the error is probably that /usr/local/lib/python3/dist-packages is a non-standard directory for Python packages, and config/chroot_local-includes/etc/environment, where we add this directory to the PYTHONPATH, wasn't sourced at the point where tails-additional-software is executed (but I don't know why tails-additional-software is executed during build at all). We had issues with the non-standard Python directory in the past (#15732) and I don't really see why we prefer it over /usr/lib/python3. I'd suggest we just move all our Python packages to /usr/lib/python3/dist-packages.

  • I don't know where the jenkins.py and git.py modules are being used. Maybe those should be extracted into a separate repository or, if they are only being used in one repo, be moved to that repo.

#4 Updated by segfault about 2 months ago

  • Status changed from Confirmed to In Progress

#5 Updated by segfault about 2 months ago

  • Parent task set to #7036

#6 Updated by segfault about 2 months ago

intrigeri wrote:

If this is about pythonlib.git: it has other consumers than tails.git. I can think of one (our Jenkins jobs generator).

Yeah, I thought so, because of the jenkins.py and git.py. So those modules should probably be moved there.

I don't know by heart if there are more.

The greeter also uses it, that's why this branch is based on the one for #16912. I couldn't find any other repos which use it.

#7 Updated by intrigeri 21 days ago

pythonlib.git: it has other consumers than tails.git. I can think of one (our Jenkins jobs generator).

Yeah, I thought so, because of the jenkins.py and git.py. So those modules should probably be moved there.

I think that moving these 2 modules elsewhere than to tails.git would cause too much trouble:

  • git.py is used by tails.git:bin/delete-merged-git-branches so we can't simply move it elsewhere. We could move bin/delete-merged-git-branches to that same place but it makes the RM's jobs more complex and error-prone, which I'd rather avoid for destructive operations.
  • The main class in jenkins.py inherits from the one from git.py, so having these 2 modules in different repos would make some operations needlessly complex. So if we move git.py to tails.git, IMO we should move jenkins.py there as well.

I don't know by heart if there are more.

I've checked in our Puppet codebase and the only user of tailslib in there is our Jenkins jobs generator. It could use the library from its own clone of tails.git just like it's currently using it from a clone of pythonlib.git.

⇒ IMO, the way to go is to move git.py and jenkins.py to tails.git, along with the rest of pythonlib.

(Another option would be to keep these two files in the pythonlib repo and leave it as a submodule of tails.git, but then we lose some of the benefits of what this ticket is about, which would be a bit sad.)

What do you think?

#8 Updated by segfault 10 days ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (segfault)

⇒ IMO, the way to go is to move git.py and jenkins.py to tails.git, along with the rest of pythonlib.

Great! Then this is ready for review now.

#9 Updated by intrigeri 10 days ago

  • Assignee set to intrigeri

#10 Updated by intrigeri 10 days ago

  • Target version set to Tails_4.0

#11 Updated by intrigeri 10 days ago

Code review passes. FTR I've verified that the imported code is identical to the one in the submodule (cd812663fc47ca2c93257ae8968917282c419089).

I'll now build + run the test suite (locally, without waiting for our shared Jenkins to be back up).

Regarding how to coordinate the merge in tails.git with our Puppet stuff, my plan is:

  1. Merge this branch into devel.
  2. Adjust Puppet code to clone tails.git's devel branch and use the library from there.
  3. File a ticket to make that Puppet code instead use the stable branch, once 4.0 is out (I initially thought we would use the master branch there, but it's often awkward when we have to backport code changes to master for similar reasons).

#12 Updated by intrigeri 10 days ago

  • Blocks Feature #16664: Simplify "Tor has bootstrapped" status check added

#13 Updated by intrigeri 10 days ago

  • Status changed from Needs Validation to In Progress

#14 Updated by intrigeri 10 days ago

  • Status changed from In Progress to Needs Validation
  • Feature Branch changed from feature/16935-move-tailslib-to-main-repo+force-all-tests to feature/16935-move-tailslib-to-main-repo+force-all-tests, puppet-tails:feature/16935-move-tailslib-to-main-repo

#15 Updated by intrigeri 10 days ago

  • Blocks Feature #17048: Make the tails::pythonlib Puppet class track the stable branch of tails.git added

#16 Updated by intrigeri 10 days ago

@segfault, I've pushed aeb2f6292259589bb035920cb80a25b1a08f16ec on top. Can you please review it?

@zen, could you please review https://git.tails.boum.org/puppet-tails/log/?h=feature/16935-move-tailslib-to-main-repo? Things to consider:

  • Do not merge it: I need to coordinate the merge with another one (in tails.git) which is not ready yet.
  • Context: our pythonlib Git repo is being deprecated so our infra code that uses libraries that lived there now needs to instead use these libraries from tails.git.
  • I have a doubt about environment => "PYTHONPATH='${tails::pythonlib::pythonpath}'", ← not sure I can access the $pythonpath variable from tails::pythonlib here; worst case, I'll move it to a class parameter and that will work (see e.g. how we reuse tails::borgbackup::borgpassphrase in tails::borgbackup::{fs,lv}).
  • For now I'm tracking the devel branch of tails.git, since that's where segfault's branch will be merged into initially. As part of the Tails 4.0 release process, devel will be merged into stable, and then we should use the stable branch of tails.git in tails::pythonlib → tracked by #17048.

Thanks in advance!

#17 Updated by intrigeri 9 days ago

  • Assignee changed from intrigeri to segfault

@segfault, the full test suite passed locally (except #17007) so feel free to merge into devel if you're happy with my extra commit, then reassign for review to zen for the sysadmin part. Thanks!

#18 Updated by segfault 7 days ago

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

#19 Updated by segfault 7 days ago

  • Status changed from Resolved to Needs Validation
  • Assignee changed from segfault to zen
  • % Done changed from 100 to 0

#20 Updated by intrigeri 4 days ago

  • Blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

Also available in: Atom PDF