Project

General

Profile

Feature #12223

Feature #10034: Translation web platform

Feature #11759: Install & configure a fresh weblate on the VM

Puppetize machine translation service on translate.lizard

Added by emmapeel almost 3 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
-
Start date:
02/13/2017
Due date:
% Done:

50%

Feature Branch:
emmapeel:feature/12223-tmserver
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

Weblate uses the translate toolkit to provide real time suggestions for translations from a sqlite database that can be fed with .po files.

https://docs.weblate.org/en/weblate-2.10.1/admin/machine.html#tmserver

This is important because it saves time for the translators, specially in cumbersome documents, and helps us to be consistent not only with our translations but, for example, with the Debian locales if we feed them to the machine.

It is a very subtle way of increasing the quality of our translations.

So, the debian package provides `tmserver` http://docs.translatehouse.org/projects/translate-toolkit/en/latest/commands/tmserver.html

tmserver is a Translation Memory service that can be queried via HTTP using a simple REST like URL/http and data is exchanged between server and client encoded in JSON.

So, first you feed the database (I have been breeding a little one myself) with
different files:

build_tmdb -d /var/lib/tm/db -s en -t cs locale/cs/LC_MESSAGES/django.po

and then you can run it like this:

tmserver -d /var/lib/tm/db

and query it like this:

http://HOST:PORT/tmserver/SOURCE_LANG/TARGET_LANG/unit/STRING
http://localhost:8080/tmserver/en/de/unit/contribute

I was thinking on maybe creating a systemd.service to run it.

Also, what about the database? shall we add it to the mutable_data dir?
Let me know what you think and then I move on

update_tm.sh View (567 Bytes) emmapeel, 11/13/2018 10:35 AM


Related issues

Related to Tails - Feature #15359: List parts of code/packages/configs to be puppetized for translation platform & its clone Resolved 03/02/2018
Blocks Tails - Feature #15180: Backups for the tmserver in translate.lizard Resolved 01/17/2018

History

#1 Updated by emmapeel almost 3 years ago

  • Parent task set to #11759

#2 Updated by emmapeel almost 3 years ago

  • Description updated (diff)

#3 Updated by emmapeel almost 3 years ago

  • Description updated (diff)

#4 Updated by intrigeri almost 3 years ago

  • Assignee changed from intrigeri to emmapeel

So, first you feed the database (I have been breeding a little one myself) with
different files:

> build_tmdb -d /var/lib/tm/db -s en -t cs locale/cs/LC_MESSAGES/django.po
> 

I guess you can do this yourself. Let me know if you need anything from me. I suspect a dedicated user to run tmserver, and /var/lib/tm that would be owned by the tmserver user. Anything else?

and then you can run it like this:

> tmserver -d /var/lib/tm/db
> 

I was thinking on maybe creating a systemd.service to run it.

Sure, I can do that once the above point is clarified.

Also, what about the database? shall we add it to the mutable_data dir?

Given tmserver has a web API, it can run as a dedicated user, so it's better if its DB is in a dedicated directory.

#5 Updated by intrigeri almost 3 years ago

  • Subject changed from Puppetize machine translation service on [translate] to Puppetize machine translation service on translate.lizard
  • Assignee changed from emmapeel to intrigeri

#6 Updated by intrigeri almost 3 years ago

  • Tracker changed from Bug to Feature
  • Status changed from New to Confirmed
  • Assignee changed from intrigeri to emmapeel
  • QA Check set to Info Needed

#7 Updated by emmapeel almost 2 years ago

  • Assignee changed from emmapeel to intrigeri

intrigeri wrote:

I guess you can do this yourself. Let me know if you need anything from me. I suspect a dedicated user to run tmserver, and /var/lib/tm that would be owned by the tmserver user. Anything else?

[...]

Given tmserver has a web API, it can run as a dedicated user, so it's better if its DB is in a dedicated directory.

Ok, I have pushed some suggestions for the tmserver user and its folder at
git.tails.boum.org:emmapeel/puppet-tails.git

branch: feature-15077-tmservice

But still needs to create the database and run the service at boot. Could you please have a look?

#8 Updated by emmapeel almost 2 years ago

  • Blocks Feature #15074: Set up and configure the web interface of the translation platform added

#9 Updated by intrigeri almost 2 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to emmapeel
  • % Done changed from 0 to 10
  • QA Check changed from Info Needed to Dev Needed

WIP, we're working on it privately: user, group and homedir are managed. Next step: systemd unit file.

#10 Updated by emmapeel almost 2 years ago

  • Blocked by Feature #15180: Backups for the tmserver in translate.lizard added

#11 Updated by emmapeel almost 2 years ago

  • Blocked by deleted (Feature #15180: Backups for the tmserver in translate.lizard)

#12 Updated by emmapeel almost 2 years ago

  • Blocks Feature #15180: Backups for the tmserver in translate.lizard added

#13 Updated by emmapeel almost 2 years ago

Ok, I have created a database with the command:

build_tmdb -d /var/lib/tmserver/db -s en -t es /var/lib/tmserver/allspanish.po

(and I got the spanish file with all the translations from current directory with the l10n trick
"Build a translation memory to use with Poedit" at:
https://tails.boum.org/contribute/l10n_tricks/#index11h1 )

#14 Updated by emmapeel almost 2 years ago

Ok, I am able to run the service now with the command

tmserver -d /var/lib/tmserver/db

#15 Updated by emmapeel almost 2 years ago

  • Assignee deleted (emmapeel)
  • % Done changed from 10 to 50

OK, now somebody should create the service.

#16 Updated by u over 1 year ago

  • Related to Feature #15359: List parts of code/packages/configs to be puppetized for translation platform & its clone added

#17 Updated by u over 1 year ago

  • Assignee set to emmapeel
  • QA Check changed from Dev Needed to Info Needed

Hi emmapeel,

I'm reassigning this to you so you can clarify what is missing here and if you provided all the information intrigeri requested?

#18 Updated by emmapeel over 1 year ago

  • QA Check changed from Info Needed to Dev Needed

The information is ready for the service to be created.

#19 Updated by emmapeel over 1 year ago

  • Assignee changed from emmapeel to intrigeri

The information is ready for the service to be created.

#20 Updated by emmapeel over 1 year ago

  • Blocks deleted (Feature #15074: Set up and configure the web interface of the translation platform)

#21 Updated by intrigeri over 1 year ago

  • Assignee changed from intrigeri to groente

FYI I'm not involved in this project, groente is handling anything Puppet.

#22 Updated by u over 1 year ago

  • Parent task changed from #11759 to #10034

any news on this groente?

#23 Updated by groente about 1 year ago

  • Assignee changed from groente to emmapeel
  • QA Check changed from Dev Needed to Info Needed

So, tmserver is now puppetised and running, but weblate doesn't seem to like it very much yet. Any suggestions?

#24 Updated by emmapeel about 1 year ago

  • Assignee changed from emmapeel to drebs

I am not sure what is happening, I can get suggestions from the machine with

wget http://127.0.0.1:8080/tmserver/en/es/unit/Contact

But weblate does not seem to understand. Drebs had found out something in the code, passing it to him

#25 Updated by emmapeel about 1 year ago

  • Assignee changed from drebs to groente
  • QA Check changed from Info Needed to Ready for QA
  • Feature Branch set to emmapeel:feature/12223-tmserver

I got it! it's the stupid link, it missed a part of the URL.

it is working with the changes proposed at git.tails.boum.org:emmapeel/puppet-tails.git
[feature/12223-tmserver d0eaa4c] correct link. part of #12223
1 file changed, 1 insertion(+), 1 deletion(-)

#26 Updated by emmapeel about 1 year ago

Incidentally, we need to give someone permissions to edit /var/lib/tmserver/db to be able to add more suggestions.

Currently doing

sudo -u weblate build_tmdb -d /var/lib/tmserver/db -s en -t es  wiki/src/donate/thanks.es.po

gives a sqlite3.OperationalError: unable to open database file

#27 Updated by groente about 1 year ago

  • Assignee changed from groente to emmapeel

i've changed the link in the settings template and loosened the permissions on the /var/lib/tmserver directory so user weblate is now allowed to access the db file. can you verify if everything works now?

#28 Updated by emmapeel about 1 year ago

  • Assignee changed from emmapeel to groente

i was not able to open the database repeating the command above.

#29 Updated by groente about 1 year ago

  • Assignee changed from groente to emmapeel
  • QA Check changed from Ready for QA to Info Needed

Okay, so apparently having access to the db file is not enough, it wants write access to the whole /var/lib/tmserver directory to play around with db-journal files. I see two possible paths:

- invoke posix extended acl's, which is going to be somewhat painful
- change the group ownership of /var/lib/tmserver to weblate instead of weblate_admin, this has my preference

Is there any actual reason this directory has group weblate_admin? I see some po files there, but they do not look like they need to be in this particular directory.

#30 Updated by emmapeel about 1 year ago

groente wrote:

Okay, so apparently having access to the db file is not enough, it wants write access to the whole /var/lib/tmserver directory to play around with db-journal files. I see two possible paths:

- change the group ownership of /var/lib/tmserver to weblate instead of weblate_admin, this has my preference

Is there any actual reason this directory has group weblate_admin? I see some po files there, but they do not look like they need to be in this particular directory.

I think intrigeri wanted the tmservice to be run apart from weblate, to isolate them. But I am not sure.

#31 Updated by emmapeel about 1 year ago

  • Assignee changed from emmapeel to groente

#32 Updated by groente about 1 year ago

  • Assignee changed from groente to emmapeel

well, i changed the group ownership there, that should fix things. can you let me know if the script works now?

#33 Updated by emmapeel about 1 year ago

  • File update_tm.sh View added
  • Assignee changed from emmapeel to groente
  • QA Check changed from Info Needed to Ready for QA

It works!

It says it is not finding Python-Levenshtein so it does a slower matching, even when Python-Levenshtein is installed. But it adds the component!

Yay!

I have a small script that gos through the files and adds them to the translation memory, maybe it would be nice to schedule a monthly run of it to update our suggestions. what do you think?

here attached

#34 Updated by groente about 1 year ago

  • Assignee changed from groente to emmapeel

okay, great, script deployed!
is it time to close this ticket now?

#35 Updated by emmapeel about 1 year ago

  • Status changed from In Progress to Resolved
  • Assignee changed from emmapeel to groente
  • QA Check changed from Ready for QA to Pass

yeah! kill it with fire!

#36 Updated by u 9 months ago

  • Parent task changed from #10034 to #11759

#37 Updated by intrigeri 5 months ago

  • Assignee deleted (groente)

Also available in: Atom PDF