Project

General

Profile

Bug #15185

Feature #10034: Translation web platform

Feature #15079: Integrate the platform with our Git and ikiwiki infrastructure

Feature #15082: Have the Weblate Git communicate with our main Git repository

Automatically update Weblate components on mdwn file add/rename/delete in tails.bo git

Added by emmapeel over 1 year ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Internationalization
Target version:
-
Start date:
02/27/2019
Due date:
% Done:

100%

Spent time:
(Total: 0.50 h)
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Commits that make weblate suffer are exemplified with:

5de9ad30fc30213bacbce79bddf8d81eee3d83a7

in which we have many lines like:

Rename PO files from renamed source pages

diff --git a/wiki/src/install/debian/clone/overview.de.po b/wiki/src/install/debian/clone-overview.de.po
similarity index 100%
rename from wiki/src/install/debian/clone/overview.de.po
rename to wiki/src/install/debian/clone-overview.de.po
diff --git a/wiki/src/install/debian/clone/overview.fa.po b/wiki/src/install/debian/clone-overview.fa.po
similarity index 100%
rename from wiki/src/install/debian/clone/overview.fa.po
rename to wiki/src/install/debian/clone-overview.fa.po
diff --git a/wiki/src/install/debian/clone/overview.fr.po b/wiki/src/install/debian/clone-overview.fr.po
similarity index 100%
rename from wiki/src/install/debian/clone/overview.fr.po
rename to wiki/src/install/debian/clone-overview.fr.po
diff --git a/wiki/src/install/debian/clone/overview.it.po b/wiki/src/install/debian/clone-overview.it.po
similarity index 100%

This creates a problem in Weblate, as there is an old component (for example wiki/src/install/debian/clone/overview.*.po in the first line) that now is 'orphaned', specially in the case of files that are not part of the main repo, as in the other languages that are not part of this original commit (i.e. wiki/src/install/debian/clone/overview.tr.po )

Care should be taken to move these files in Weblate and see no translations are being lost when moving files, and also delete any stale files before importing the translations.

update-weblate-components.py View (5.28 KB) hefee, 06/14/2018 12:27 PM


Subtasks

Bug #16490: translation-server: detect weblate components correctly.Resolved


Related issues

Related to Tails - Bug #15356: Automate adding new components to Weblate Duplicate 03/01/2018
Related to Tails - Bug #15402: update hook for Main git when handling push from weblate Resolved
Related to Tails - Bug #15679: update puppet-tails to have scripts in weblate vm Resolved 06/20/2018
Related to Tails - Feature #15190: Cronjobs for translate.lizard are not running Resolved 01/17/2018
Related to Tails - Bug #15674: Change all Weblate components to use correct committer name and email Resolved 06/19/2018
Related to Tails - Feature #16225: merge hefee/dev/translation-server Resolved 12/14/2018
Blocked by Tails - Bug #15624: Please install python-git, python3-git, python-pathlib on Weblate VM Resolved 05/29/2018

History

#1 Updated by emmapeel over 1 year ago

  • Blocks Feature #15079: Integrate the platform with our Git and ikiwiki infrastructure added

#2 Updated by intrigeri over 1 year ago

Looks like weblate should detect that itself (probably based on Git's own detection of it).

#3 Updated by emmapeel over 1 year ago

  • Description updated (diff)

intrigeri wrote:

Looks like weblate should detect that itself (probably based on Git's own detection of it).

It detects it for the language files that are in master, but the problem are the ones only in weblate.

I have corrected the description to make it more clear.

#4 Updated by emmapeel over 1 year ago

Related to this I reported https://github.com/WeblateOrg/weblate/issues/1200 a while ago

#5 Updated by intrigeri over 1 year ago

When doing #15192 we noticed two more classes of related problems in the branch emmapeel submitted:

  • 2 pages that were removed from our website still had .es.po files
  • 50 pages that are not translatable anymore since 6-24 months still had .es.po files, which broke the News section on our English homepage

I don't know if you want to track these two problems on this ticket or elsewhere, but IMO this is a blocker for calling the Weblate setup ready for production use.

#6 Updated by u over 1 year ago

@emmapeel: would this kind of problem be nonexistent with the new weblate version? If yes, fine. If not, we could write some scripts run as a cronjob to check for such issues. What do you think?

#7 Updated by u over 1 year ago

  • Assignee set to emmapeel
  • QA Check set to Info Needed

@emmapeel: See my previous note. If you don't want to handle it, you can reassign this to me so that we don't lose track of this problem.

#8 Updated by emmapeel over 1 year ago

u wrote:

@emmapeel: would this kind of problem be nonexistent with the new weblate version? If yes, fine. If not, we could write some scripts run as a cronjob to check for such issues. What do you think?

AFAIK this will not be fixed with all its particularities in the new version.

I see four complementary approaches:

  • Yesterday intrigeri quickly scratched a script that removes .es.po files if they don't have a corresponding .fr.po file, this could be a cronjob in weblate:
    
    find . -type f -name *.es.po | while read ES ; do PAGE=$(echo $ES | perl -pE 's,\.es\.po,,') ; [ -f $PAGE.fr.po ] || [ -f $PAGE.fr.po ] || echo $ES ; done 2>/dev/null  | xargs git rm
    
    
  • We could also make git hooks, looking for deleted files when weblate pulls from tails/master
  • This one is ugly but unfortunately possible: we train website writers to delete componens in weblate when deleting files in tails/master

#9 Updated by emmapeel over 1 year ago

I have added intri's script to the gitlabCI checks I am testing, and each time I push a language branch i get this message at the end:

I run it after building the wiki with tails/master and the pertinent language files copied shamelessly on top of it.

$ git status
HEAD detached at 8f403b1e6e
nothing to commit, working tree clean
$ find . -type f -name *.ar.po | while read ES ; do PAGE=$(echo $ES | perl -pE 's,\.ar\.po,,') ; [ -f $PAGE.fr.po ] || [ -f $PAGE.fr.po ] || echo $ES ; done |ccze -A
./wiki/src/misc/unsafe_browser_warning.ar.po 
./wiki/src/install/win/usb/overview.ar.po 
./wiki/src/install/download/openpgp.ar.po 
Job succeeded

It is only information, and it is not affecting the actual weblate master, but it can be used to raise alarm for now.

the gitlab config commit is
https://0xacab.org/emmapeel/weblate/blob/8f403b1e6ebffaf5fdc7ff46ae1238b0a6dfafc7/.gitlab-ci.yml

#10 Updated by emmapeel over 1 year ago

  • Assignee changed from emmapeel to u

#11 Updated by u over 1 year ago

  • Parent task set to #10034

#12 Updated by u over 1 year ago

  • Related to Bug #15356: Automate adding new components to Weblate added

#13 Updated by u over 1 year ago

  • Subject changed from weblate components need to be moved when moving files in git. to Weblate components need to be moved when files have been git mv'ed or deleted
  • Description updated (diff)
  • QA Check deleted (Info Needed)

#14 Updated by u over 1 year ago

  • Subject changed from Weblate components need to be moved when files have been git mv'ed or deleted to Automatically update Weblate components on mdwn file add/modify/delete in tails.bo git
  • Parent task changed from #10034 to #15082

This needs to happen between Weblate.git and Weblate (python got hook).

#15 Updated by u over 1 year ago

  • Related to Bug #15402: update hook for Main git when handling push from weblate added

#16 Updated by u over 1 year ago

  • Blocked by Bug #15624: Please install python-git, python3-git, python-pathlib on Weblate VM added

#17 Updated by u over 1 year ago

  • Category set to Internationalization
  • Assignee changed from u to hefee

#18 Updated by hefee over 1 year ago

  • Subject changed from Automatically update Weblate components on mdwn file add/modify/delete in tails.bo git to Automatically update Weblate components on mdwn file add/rename/delete in tails.bo git

#19 Updated by u over 1 year ago

  • Status changed from Confirmed to In Progress

#20 Updated by hefee over 1 year ago

  • % Done changed from 0 to 60

This script should do the handling of the components adding/deleting/renaming.
The rename part is not tested yet.

import git                                                                                                                                                                                                                          
import logging                                                                                                                                                                                                                      
import os                                                                                                                                                                                                                           
import os.path                                                           
import pathlib
import sys  

def weblateComponentFile(filename, tree):                                               
    """is this file a weblate component"""                                                                      
    if not filename.startswith("wiki/src"):
        return False                                                     

    basename, *extensions = os.path.basename(filename).split(".")             
    if extensions and extensions[-1] == "po":                                  
        return False                 

    detectionFile = "{}.{}.po".format(os.path.splitext(filename)[0],LANG)                                                                                                                                                          
    try:
        tree/detectionFile
        return True
    except KeyError:
        return False
    return False

# Format logging    
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)  # Change to DEBUG if you wnat to see debug messages

formatter = logging.Formatter('%(levelname)s: %(message)s')
ch.setFormatter(formatter)                                             

logger = logging.getLogger('')
logger.setLevel(logging.DEBUG)                                   
logger.addHandler(ch)                                    

# DJANGO/WEBLATE setup to be able to change stuff within weblate
sys.path.insert(0,"/usr/local/share/weblate")          
os.environ["DJANGO_SETTINGS_MODULE"] = "weblate.settings" 
os.environ["DJANGO_IS_MANAGEMENT_COMMAND"] = "1"    

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

from weblate.trans import models
from weblate.trans.management.commands import import_project                    

# weblate project name
PROJECT =  models.Project.objects.first().name

# repo path ( should be start with weblate:// )                            
REPO = models.SubProject.objects.first().repo

# Where is the git repository located?                 
PATH = pathlib.Path("/var/lib/weblate/repositories/vcs/tails/wikisrcindexpo")

# One langague that is activated and handled within the main git
LANG = "fr" 

if not REPO.startswith("weblate://"):                             
    logger.error("We need a weblate:// url for searching for new components and not '{}'.".format(REPO))
    sys.exit(-1)

repo = git.Repo(str(PATH))                          

# fetch first, to see what we need to update
repo.remotes.origin.fetch()

# the object to orgin master branch
remote_master = repo.remotes.origin.refs.master

# create a diff                            
diff = repo.index.diff(remote_master.commit)

# perform pull                     
logger.info("HEAD before pull is at {}".format(repo.head.commit.hexsha))
repo.remotes.origin.pull()             

for deleted_file in diff.iter_change_type('D'):
    if not weblateComponentFile(deleted_file.a_path, tree=repo.tree()):
        logger.debug("deleted file (ignoring): {}".format(deleted_file.a_path))
        continue
    filemask = os.path.splitext(deleted_file.a_path)[0] + ".*.po" 
    logger.info("deleted component: {}".format(filemask))
    subproject = models.SubProject.objects.filter(filemask=filemask).first()
    if subproject is None:
        logger.warning("\t... can't find component :(")
        continue
    if not subproject.repo.startswith("weblate://"):
        logger.warning("Do not delete {}, because it has no weblate url {}.".format(filemask, subproject.repo))
        continue
    subproject.delete()

# Handle added files
# as we rescan the whole dictonary anyways, we only need this once per dictonary

rescanDirs=set()
for added_file in diff.iter_change_type('A'):
    if not weblateComponentFile(added_file.a_path, tree=remote_master.commit.tree):
        logger.debug("added file (ignoring): {}".format(added_file.a_path))
        continue
    filemask = os.path.splitext(added_file.a_path)[0] + ".*.po" 
    logger.info("added component: {}".format(filemask))
    fname = pathlib.Path(added_file.a_path)
    dirname = fname.parent
    rescanDirs.add(dirname)

for directory in rescanDirs:
    # Use import_project as used for inital creation of components
    # see /usr/local/sbin/weblate-setup-project.sh

    args=["--component-regex", str(directory/"(?P<name>.*)\.(?P<language>.+)\.po"),
        "--name-template", str(directory/"%s.*.po"),
        PROJECT,
        REPO,
        "master",
        str(directory/"**.*.po"),
        ]

    command = import_project.Command()
    parser = command.create_parser("",None)
    options = parser.parse_args(args=args)
    logger.info("searching for new components in {}".format(directory))
    command.handle(**vars(options))

# Handle renamed files (not tested yet)
for renamed_file in diff.iter_change_type('R'):
    if not weblateComponentFile(renamed_file.rename_to, tree=remote_master.commit.tree):
        logger.debug("renamed file (ignoring) {} -> {}".format(renamed_file.rename_from,renamed_file.rename_to))
        continue
    filemaskOld = os.path.splitext(renamed_file.rename_from)[0] + ".*.po" 
    filemaskNew = os.path.splitext(renamed_file.rename_to)[0] + ".*.po" 
    logger.info("renamed component {} -> {}".format(filemaskOld, filemaskNew))
    subproject = models.SubProject.objects.filter(filemask=filemaskOld).first()
    subproject.filemask = filemaskNew
    component.create_translations()  # Thats what loadpo is doing ...

#21 Updated by hefee over 1 year ago

The only renaming of po files do not trigger nothing, so we need to take into account the move of the mdwn/html files, that were done at 58ffc9606694835150ad98183382e8a89a3c5e90:

 wiki/src/doc.mdwn                                                    | 16 ++++++++--------
 wiki/src/doc/first_steps/upgrade.mdwn                                |  2 +-        
 wiki/src/install/debian.html                                         |  6 +++---    
 wiki/src/install/debian/{clone/overview.html => clone-overview.html} |  0  
 wiki/src/install/debian/{usb/overview.html => usb-overview.html}     |  0           
 wiki/src/install/expert/{usb/overview.html => usb-overview.html}     |  0       
 wiki/src/install/inc/steps/debian_requirements.inline.mdwn           |  2 +-  
 wiki/src/install/linux.html                                          |  4 ++--  
 wiki/src/install/linux/{clone/overview.html => clone-overview.html}  |  0      
 wiki/src/install/linux/{usb/overview.html => usb-overview.html}      |  0             
 wiki/src/install/mac.html                                            |  6 +++---                                     
 wiki/src/install/mac/{clone/overview.html => clone-overview.html}    |  0      
 wiki/src/install/mac/{dvd/overview.html => dvd-overview.html}        |  0             
 wiki/src/install/mac/{usb/overview.html => usb-overview.html}        |  0           
 wiki/src/install/win.html                                            |  4 ++--
 wiki/src/install/win/{clone/overview.html => clone-overview.html}    |  0           
 wiki/src/install/win/{usb/overview.html => usb-overview.html}        |  0        
 wiki/src/news/report_2016_09.mdwn                                    |  2 +-     
 wiki/src/upgrade.html                                                |  4 ++--   
 wiki/src/upgrade/{clone/overview.html => clone-overview.html}        |  0  
 wiki/src/upgrade/{tails/overview.html => tails-overview.html}        |  0        
 21 files changed, 23 insertions(+), 23 deletions(-)

#23 Updated by hefee over 1 year ago

I updated the file update-weblate-components.py to process renames correctly.
I used the update from 60c37bf4926763199c2d19bd8c0ec9e8db9d7cc0 -> 7d555467817e849beb756dae4b72de9167320bc7 and git detected following renames:

INFO: renamed component wiki/src/install/linux/clone/overview.*.po -> wiki/src/install/debian/clone-overview.*.po                                                                  
INFO: renamed component wiki/src/install/debian/usb/overview.*.po -> wiki/src/install/debian/usb-overview.*.po                          
INFO: renamed component wiki/src/install/expert/usb/overview.*.po -> wiki/src/install/expert/usb-overview.*.po      
INFO: renamed component wiki/src/install/win/clone/overview.*.po -> wiki/src/install/linux/clone-overview.*.po                                     
INFO: renamed component wiki/src/install/linux/usb/overview.*.po -> wiki/src/install/linux/usb-overview.*.po                         
INFO: renamed component wiki/src/install/mac/clone/overview.*.po -> wiki/src/install/mac/clone-overview.*.po                                                                 
INFO: renamed component wiki/src/install/mac/dvd/overview.*.po -> wiki/src/install/mac/dvd-overview.*.po                         
INFO: renamed component wiki/src/install/mac/usb/overview.*.po -> wiki/src/install/mac/usb-overview.*.po                          
INFO: renamed component wiki/src/install/vm.*.po -> wiki/src/install/vm-download.*.po                   
INFO: renamed component wiki/src/install/debian/clone/overview.*.po -> wiki/src/install/win/clone-overview.*.po                                                                                                                                                                                                  
INFO: renamed component wiki/src/install/win/usb/overview.*.po -> wiki/src/install/win/usb-overview.*.po                                                                           
INFO: renamed component wiki/src/upgrade/clone/overview.*.po -> wiki/src/upgrade/clone-overview.*.po                                   
INFO: renamed component wiki/src/upgrade/tails/overview.*.po -> wiki/src/upgrade/tails-overview.*.po

This result in renaming the weblate component(name,slug and filemask) and an update of the translations:

INFO: renamed component wiki/src/install/vm.*.po -> wiki/src/install/vm-download.*.po                         
INFO tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.de.po (de) [1/6]                         
INFO: tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.de.po (de) [1/6]             
INFO tails/wikisrcinstallvm-downloadpo/de: processing wiki/src/install/vm-download.de.po, revision has changed
INFO: tails/wikisrcinstallvm-downloadpo/de: processing wiki/src/install/vm-download.de.po, revision has changed
INFO tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.es.po (es) [2/6]               
INFO: tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.es.po (es) [2/6]         
INFO tails/wikisrcinstallvm-downloadpo/es: processing wiki/src/install/vm-download.es.po, revision has changed
INFO: tails/wikisrcinstallvm-downloadpo/es: processing wiki/src/install/vm-download.es.po, revision has changed
INFO tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.fa.po (fa) [3/6]
INFO: tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.fa.po (fa) [3/6]
INFO tails/wikisrcinstallvm-downloadpo/fa: processing wiki/src/install/vm-download.fa.po, revision has changed
INFO: tails/wikisrcinstallvm-downloadpo/fa: processing wiki/src/install/vm-download.fa.po, revision has changed        
INFO tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.fr.po (fr) [4/6]                                                                                                                                                
INFO: tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.fr.po (fr) [4/6]                                    
INFO tails/wikisrcinstallvm-downloadpo/fr: processing wiki/src/install/vm-download.fr.po, revision has changed         
INFO: tails/wikisrcinstallvm-downloadpo/fr: processing wiki/src/install/vm-download.fr.po, revision has changed        
INFO tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.it.po (it) [5/6]
INFO: tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.it.po (it) [5/6]     
INFO tails/wikisrcinstallvm-downloadpo/it: processing wiki/src/install/vm-download.it.po, revision has changed
INFO: tails/wikisrcinstallvm-downloadpo/it: processing wiki/src/install/vm-download.it.po, revision has changed
INFO tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.pt.po (pt) [6/6]                         
INFO: tails/wikisrcinstallvm-downloadpo: checking wiki/src/install/vm-download.pt.po (pt) [6/6]
INFO tails/wikisrcinstallvm-downloadpo/pt: processing wiki/src/install/vm-download.pt.po, revision has changed
INFO: tails/wikisrcinstallvm-downloadpo/pt: processing wiki/src/install/vm-download.pt.po, revision has changed
INFO tails/wikisrcinstallvm-downloadpo: updating completed           
INFO: tails/wikisrcinstallvm-downloadpo: updating completed

#24 Updated by hefee about 1 year ago

  • Related to Bug #15679: update puppet-tails to have scripts in weblate vm added

#25 Updated by emmapeel about 1 year ago

Nice script! I haven't tested it yet, but I have a couple of suggestions for the default values when creating a new component:


file_format = u'po'
license = u'GPL v3'
committer_email = u'tails-l10n@boum.org'
committer_name = u'Tails translators'
i.name = i.filemask
suggestion_voting = True  # this is what makes our review
suggestion_autoaccept = 2 # process work by now
enable_suggestions = True #
allow_translation_propagation = True

#26 Updated by emmapeel about 1 year ago

  • Related to Feature #15190: Cronjobs for translate.lizard are not running added

#27 Updated by hefee 10 months ago

  • Related to Bug #15674: Change all Weblate components to use correct committer name and email added

#28 Updated by hefee 9 months ago

#29 Updated by hefee 9 months ago

  • Status changed from In Progress to Resolved

#30 Updated by hefee 9 months ago

  • % Done changed from 60 to 100

#31 Updated by emmapeel 7 months ago

I see that the release notes since Tails 3.9 are missing, probably other files too.

Could you please have a look?

I don't want to add them 'by hand' if you are working on the automation...

#32 Updated by u 7 months ago

  • Status changed from Resolved to In Progress

I'm reopening this ticket for the last question.

#33 Updated by hefee 6 months ago

  • Status changed from In Progress to Resolved

This should be solved with #16490 got solved.

I know there are still some quirks around that make my scripts stop working. But those need to be addressed if they occur.

#34 Updated by intrigeri 3 months ago

  • Assignee deleted (hefee)

Also available in: Atom PDF