Project

General

Profile

Feature #17377

Weblate: Generate list of Python dependencies automatically

Added by zen about 2 months ago. Updated about 2 months ago.

Status:
Needs Validation
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
-
Start date:
Due date:
% Done:

0%

Feature Branch:
https://salsa.debian.org/hefee/puppet-tails/:hefee/update_puppet_dependecy_files
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Translation Platform

Description

For maintaining the Weblate installation, we chose to:

  • install Python dependencies from Debian repo as much as possible; and
  • pin dependencies' versions and install from upstream repositories whenever they are not available in Debian. (The reason for installing from repositories is that the pip repo distributes binaries which have no way to be verified.)

Currently, we're manually maintaining the Puppet code for installing Weblate dependencies either as Debian packages or Python modules, with the aid of a script. That is hard to maintain and error prone, so we should ideally generate the list of dependencies automatically.

We need have a script that, given a Weblate version and a list of extra packages (example: redis is needed for celery which is needed -- see #17347), will traverse the whole dependency trees and generate whatever is needed to have a fixed list of Debian packages and Python modules installed -- and nothing else.

History

#1 Updated by hefee about 2 months ago

  • Affected tool set to Translation Platform

#2 Updated by zen about 2 months ago

  • Assignee changed from Sysadmins to hefee

#3 Updated by hefee about 2 months ago

  • Status changed from Confirmed to In Progress
  • Feature Branch set to https://salsa.debian.org/hefee/puppet-tails/:hefee/update_puppet_dependecy_files
  • Type of work changed from Translate to Code

version installed is: `pip3 freeze -l`
the script is: https://salsa.debian.org/hefee/puppet-tails/blob/hefee/update_puppet_dependecy_files/files/weblate/external_scripts/update_puppet_dependecy_files.py

Dependency Version installed script Status
amqp 2.5.2 2.5.2 # <2.6,>=2.5.2 ok
asgiref -- 3.2.3 # ~=3.2 dependency for django
billiard 3.6.1.0 3.6.1.0 # <4.0,>=3.6.1 ok
celery 4.3.0 4.4.0 # >=4.0 ok
celery-batches 0.2 0.2 # >=0.2 ok
defusedxml 0.5.0 -- (installed by Debian package) ok
diff-match-patch 20121119 -- (installed by Debian package) ok
django 1.11.27 3.0.1 # >=1.11 not ok
django-appconf 1.0.2 1.0.3 # >=1.0 ok
django-compressor 2.2 2.3 # >=2.1.1 ok
django-crispy-forms 1.7.2 1.8.1 # >=1.6.1 ok
djangorestframework 3.8.2 3.11.0 # >=3.8 ok
et-xmlfile 1.0.1 -- can be removed
filelock 3.0.4 3.0.12 # >=3.0.1 ok
importlib-metadata 1.3.0 1.3.0 # >=0.18 ok
jdcal 1.4.1 -- can be removed
jellyfish 0.7.2 0.7.2 # >=0.6.1 ok
kombu 4.6.7 4.6.7 # <4.7,>=4.6.7 ok
more-itertools 8.0.2 8.0.2 ok
oauthlib 3.1.0 3.1.0 # >=3.0.0 ok
openpyxl 2.6.0 3.0.2 # >=2.5.0 ok
phply 1.2.4 -- can be removed, we don't handle php files in Weblate
ply 3.11 -- can be removed
python3-openid 3.1.0 -- can be removed
pytz 2019.3 -- (installed by Debian package) ok
rcssmin 1.0.6 -- (installed by Debian package) ok
redis 3.3.11 3.3.11 ok
requests-oauthlib 1.0.0 -- can be removed
rjsmin 1.0.12 1.1.0 # ==1.1.0 ok
ruamel.yaml 0.16.5 --
ruamel.yaml.clib 0.2.0 --
siphashc 1.0 1.3 # >=0.8 ok
social-auth-app-django 3.1.0 3.1.0 # >=3.1.0 ok
social-auth-core 3.1.0 3.2.0 # >=3.1.0 ok
translate-toolkit 2.3.1 2.4.0 # >=2.3.1 ok
translation-finder 1.7 1.7 # >=1.0 ok
ua-parser 0.8.0 0.8.0 # >=0.8.0 ok
user-agents 1.1.0 2.0 # >=1.1.0 ok
vine 1.3.0 1.3.0 # ==1.3.0 ok
zipp 0.6.0 0.6.0 # >=0.5 ok

#4 Updated by hefee about 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from hefee to zen

The first version of the script is ready for review.
Please check the output of git show -w 408e4db6, that is the output of the script.

Things that need changes:
- I would only list non weblate debian dependencies in debian_dependencies.pp, so this file, won't needs to be changed by the script and add the weblate debian depdendencies in pip_dendendencies.pp.
- rename {pip,weblate}_dendendencies.pp
- express the pip dependencies as a requirements.txt and run:

pip3 install --no-deps --ignore-installed -r weblate_requirements.txt
#maybe this works
pip3 install --no-deps --upgrade-strategy "eager" -r weblate_requirements.txt

Also available in: Atom PDF