Project

General

Profile

Feature #15086

Feature #15085: Document our translation platform infrastructure

Translation platform: Write design documentation

Added by u almost 2 years ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

100%

Spent time:
Feature Branch:
Type of work:
Contributors documentation
Starter:
Affected tool:
Translation Platform

Description

This includes:

  • Check the doc that's in the translate-server repo, make sure it's up-to-date and correct, and ideally move whatever can be published to tails.git.

Subtasks

Feature #16979: Clarify how to enable a new language on our translation platformResolvedintrigeri

Bug #16995: translate.lizard: cron.sh fail with 'Lock wait timeout exceeded; try restarting transaction'Resolved


Related issues

Related to Tails - Bug #17063: Cleanup translate-server.git Resolved
Related to Tails - Bug #17129: rename puppet-tails:update_weblate_git.py → merge_canonical_changes.py Confirmed
Blocks Tails - Feature #15089: Write public technical report about the Translation platform Resolved 12/19/2017

Associated revisions

Revision 161b3436 (diff)
Added by intrigeri 2 months ago

Translation platform design doc: misc post-review improvements (refs: #15086).

This includes:

- fix Markdown formatting
- fix typos
- fix broken links
- tiny structure improvements
- improve unclear phrasing
- unify duplicated content
- update obsolete info

Revision c8ae45e9 (diff)
Added by Sandro Knauß 2 months ago

Translation platform design doc: some improvements (refs: #15086).

This includes:
- rewrite, why we are doing commits in our
scripts and remove empty commits later again.
- get rid of the logical "then" relationship there "Weblate's Git repository
is not bare. Hence"
- remove the last XXX and create a issue for that: #17129.

Revision 52ec8eb2 (diff)
Added by intrigeri 2 months ago

Move translation platform ops doc under its (future) role page (refs: #15086).

Revision 01915911 (diff)
Added by intrigeri 2 months ago

Move translation platform specs to a subpage of the upcoming design doc (refs: #15086).

Revision 5d4cf93f (diff)
Added by intrigeri 2 months ago

Move translation platform design doc under contribute/design (refs: #15086).

While doing so:

- implement some structure improvements that we discussed
on the ticket
- various small polishing

And accordingly:

- adjust internal links
- adjust backlinks

History

#1 Updated by u over 1 year ago

  • Blueprint set to https://tails.boum.org/blueprint/translation_platform

#2 Updated by u over 1 year ago

  • Status changed from Confirmed to In Progress

#3 Updated by intrigeri over 1 year ago

  • Target version changed from Tails_3.8 to Tails_3.9

#4 Updated by u over 1 year ago

  • Target version changed from Tails_3.9 to Tails_3.11

#5 Updated by u about 1 year ago

  • Target version changed from Tails_3.11 to Tails_3.12

I don't see this happening as early as 3.11, but we'll work on this together with hefee.

#6 Updated by anonym 11 months ago

  • Target version changed from Tails_3.12 to Tails_3.13

#7 Updated by u 9 months ago

  • Subject changed from Write design documentation for other projects to reuse our work to Translation platform: Write design documentation for other projects to reuse our work

#8 Updated by u 9 months ago

  • Target version changed from Tails_3.13 to Tails_3.16

I'd rather do that once this has been implemented :)

#9 Updated by u 5 months ago

  • Assignee changed from u to hefee
  • Affected tool set to Translation Platform

#10 Updated by u 5 months ago

  • Status changed from In Progress to Confirmed

#11 Updated by u 5 months ago

  • Subject changed from Translation platform: Write design documentation for other projects to reuse our work to Translation platform: Write design documentation

Some information that could go into translate-server.git for future reference:

- clean the VM because files are shattered in various & unpredictible
locations

@intrigeri wrote:

I just did what I felt was good enough for these 2 things:

- Move logs to /var/log
- Turn class parameters into constants in all cases where we've never
supported setting them to a non-default value (and have no reason
to ever do so).

ulrike: I believe this fixes:

intrigeri:

I'd like to broaden the scope a bit
to also include: "make all scripts and configuration files managed by
Puppet honor the parameters of the tails::weblate class" (currently we
pass parameters to this class, honor them in some places, but
hard-code their default value in various scripts, so if someone ever
passes non-default values for these parameters, well, things won't
work™).

- Use the remaining class parameters in templates instead of
hard-coding their default value.

The end result is definitely not perfect but improving things further
would start having a pretty high cost/benefit ratio so I don't think
it's worth the effort

#12 Updated by u 5 months ago

  • Blocks Feature #15089: Write public technical report about the Translation platform added

#13 Updated by intrigeri 5 months ago

  • Description updated (diff)

#14 Updated by u 5 months ago

@hefee: I think a good start is to merge our blueprint with all information from translate-server.git that can be made public.

#15 Updated by intrigeri 4 months ago

  • Description updated (diff)
  • Type of work changed from Communicate to Contributors documentation

#16 Updated by hefee 4 months ago

  • Status changed from Confirmed to Needs Validation
  • Assignee changed from hefee to u
  • Feature Branch set to hefee/15086-desgin-documentation

I updated now the blueprint and added some paragraphs about the scripts that we are running.

Additionally I scanned through the translation-server.git and removed not needed stuff and updated the remaining files.

Do you see more stuff that should end up in the blueprint?

The README.mdwn is another (older) approch to describe the scripts, what do you find more readable?

#17 Updated by CyrilBrulebois 3 months ago

  • Target version changed from Tails_3.16 to Tails_3.17

#18 Updated by intrigeri 3 months ago

  • Target version changed from Tails_3.17 to Tails_4.0

#20 Updated by u 3 months ago

I'm on it!

#21 Updated by u 3 months ago

  • Status changed from Needs Validation to In Progress
  • Assignee changed from u to hefee
  • Feature Branch deleted (hefee/15086-desgin-documentation)

Hi hefee!

I've done a first review and tried to rewrite some parts that I found hard to understand. Can you please go through the document and improve the parts that I marked with "XXX"? Then reassign to me again for review please.

Thank you!

Also, you can do this directly in the master branch. Blueprints are editable for everyone, so no need to create a branch.

#22 Updated by u 3 months ago

PS I have not deleted your branch (hefee/15086-desgin-documentation). Can you do this yourself please?

#23 Updated by u 3 months ago

Some other things I've noticed:

I've started to cleanup the translate-server.git repository with information that is either obsolete or should live in our public documentation.
There are some files left though, can you please move them to a public place, after having deleted the sections that are outdated? You might want to coordinate this with intrigeri who might have some ideas on where to store this information instead.

Concerned files:

- sysadmin.mdwn → Not sure where this contents could go?
- weblate.mdwn
- tmserver.mdwn (I think this one can be deleted)

Also, I've noticed that there is a documentation in contribute/l10n_tricks that explains how to add a new language to weblate. I've now linked to this documentation from the blueprint. But I'm not sure that l10_tricks.mdwn is the right place for this documentation. I've also not checked this documentation for accuracy. Can you please do this as part of the design documentation?

#24 Updated by u 3 months ago

hefee wrote:

Additionally I scanned through the translation-server.git and removed not needed stuff and updated the remaining files.

As said, I did this again, and there is still stuff that needs to be moved somewhere else.

Do you see more stuff that should end up in the blueprint?

The README.mdwn is another (older) approch to describe the scripts, what do you find more readable?

I think we can get rid of the README now.

Please note that if I want to re-review the doc + have intrigeri do a tech review + write the call for translation before Oct 22nd, we have not that much time left to finish this work.

#25 Updated by hefee 2 months ago

  • Related to Bug #17063: Cleanup translate-server.git added

#26 Updated by hefee 2 months ago

  • Assignee changed from hefee to u

u wrote:

I've started to cleanup the translate-server.git repository with information that is either obsolete or should live in our public documentation.
There are some files left though, can you please move them to a public place, after having deleted the sections that are outdated? You might want to coordinate this with intrigeri who might have some ideas on where to store this information instead.

Done. As removing all the files are not necessary the part of this issue. I created a new issue #17063, to separate the discussions. In the end it is only `sysadmin.mdwn` is left.

Concerned files:

- sysadmin.mdwn → Not sure where this contents could go?

→ move discussion to #17063.

- weblate.mdwn

→ moved to blueprint

- tmserver.mdwn (I think this one can be deleted)

→ moved to blueprint

Also, I've noticed that there is a documentation in contribute/l10n_tricks that explains how to add a new language to weblate. I've now linked to this documentation from the blueprint. But I'm not sure that l10_tricks.mdwn is the right place for this documentation. I've also not checked this documentation for accuracy. Can you please do this as part of the design documentation?

This part was written by intrigeri and I reviewed it.

I have not deleted your branch (hefee/15086-desgin-documentation). Can you do this yourself please?

done

I've done a first review and tried to rewrite some parts that I found hard to understand. Can you please go through the document and improve the parts that I marked with "XXX"? Then reassign to me again for review please.

I gone through the document and improved the parts you marked with "XXX".

Also, you can do this directly in the master branch. Blueprints are editable for everyone, so no need to create a branch.

Well at the state, when I created the branch, I had no commit access to the master branch, so I needed to create a branch as far I know. Now this is different, as I got full commit access to tails.git.

#27 Updated by u 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from u to intrigeri

Thanks! I've reviewed the new version and will now let intrigeri add any bits that we might have overlooked.

@hefee: blueprints are world editable via the website.

#28 Updated by u 2 months ago

  • Parent task deleted (#15085)

unparenting

#29 Updated by u 2 months ago

  • Parent task set to #15085

Reverting

#30 Updated by intrigeri 2 months ago

Thank you both. I'll try to make time this week for this review but I'm very busy with urgent+important things that have to be done in time for 4.0, so I can't promise I'll be able to work on this on short notice ⇒ it may have to wait a few weeks. So at this point, I unfortunately can't tell you anything more precise, that could help you schedule time for the next steps (e.g. acting on my review if needed, moving this to the final location, sending the link wherever it makes sense, etc.).

#31 Updated by u 2 months ago

I don't think this is a blocker.

But, in order to get closure rather sooner than later on this project on my side, instead of doing back and forth reviews between the three of us ("acting on your review") I would instead ask you to please implement your fixes right away, and only send it back to hefee or myself if there is something you cannot answer or act upon yourself.

#32 Updated by intrigeri 2 months ago

But, in order to get closure rather sooner than later on this project on my side, instead of doing back and forth reviews between the three of us ("acting on your review") I would instead ask you to please implement your fixes right away, and only send it back to hefee or myself if there is something you cannot answer or act upon yourself.

Sure thing! This will be smoother for me as well :)

#33 Updated by u 2 months ago

Sure thing! This will be smoother for me as well :)

Great :)

#34 Updated by intrigeri 2 months ago

  • Status changed from Needs Validation to In Progress

#35 Updated by intrigeri 2 months ago

  • Assignee changed from intrigeri to hefee

Hi!

I've reviewed this piece of doc. First, congrats: this work is amazing! By reading it, I understood a few things that were previously not 100% clear to me.

As agreed beforehand, I've pushed a number of (mostly minor) improvements: 161b3436732e950ae3c84897bc7db0a7d3a3fac3. I'll let you review them if you'd like to.

There were a few things I was not sure about so I've not acted on them myself. I'll split them between questions directed particularly to hefee, and questions that also affect u.

For hefee and u:

  • There's a "XXX" left. It seems to me that this comment is correct and the script should be renamed. I'm not sure how important this is and given we're likely running out of budget, I would be fine with leaving things as-is.
  • The bits about Docker are outdated, no? I propose we leave them in the blueprint when we'll move the rest of it elsewhere.
  • I think I would move the specification ("Choosing a translation web platform") to a dedicated sub-page. If that's fine by you, I volunteer to do it myself.
  • More generally, this blueprint mixes rather different kinds of information together. I think it's been fine so far, as a single place to write all this made this work simpler; but I don't think it would be good enough to simply move it to contribute/design as-is. I proposed a rough plan on #17063#note-6. Should I take care of splitting the blueprint and moving pieces to contribute/*, or would you folks like us to first discuss together what should go where, or would one of you want to take care of thinking this through by themselves, or something else?

For hefee:

  • The blueprint claims that touch and changing mtime triggers a Git commit, which is incorrect AFAICT: Git does not care about such filesystem metadata. So the explanation of why we need to avoid creating useless merge commits seems buggy to me. hefee, if you agree, could you please fix it?
  • I don't understand the logical "then" relationship there "Weblate's Git repository is not bare. Hence […]". Even if it were a bare repo, we would have to merge its changes, no?
  • Structure-wise, it's not clear to me why "Machine translation" is in its own section instead of being part of "Setup of our translation platform". For example, I don't understand why this section is different in nature from the one about the staging website, which is in "Setup of our translation platform". I would move it there, if you agree.

#36 Updated by u 2 months ago

intrigeri wrote:

I've reviewed this piece of doc. First, congrats: this work is amazing! By reading it, I understood a few things that were previously not 100% clear to me.

Cool!

As agreed beforehand, I've pushed a number of (mostly minor) improvements: 161b3436732e950ae3c84897bc7db0a7d3a3fac3. I'll let you review them if you'd like to.

There were a few things I was not sure about so I've not acted on them myself. I'll split them between questions directed particularly to hefee, and questions that also affect u.

For hefee and u:

  • There's a "XXX" left. It seems to me that this comment is correct and the script should be renamed. I'm not sure how important this is and given we're likely running out of budget, I would be fine with leaving things as-is.

I put the XXX, so I cannot be the person replying to this question :)

  • The bits about Docker are outdated, no? I propose we leave them in the blueprint when we'll move the rest of it elsewhere.
  • I think I would move the specification ("Choosing a translation web platform") to a dedicated sub-page. If that's fine by you, I volunteer to do it myself.

@intrigeri: That's a very good idea → please go ahead.

  • More generally, this blueprint mixes rather different kinds of information together. I think it's been fine so far, as a single place to write all this made this work simpler; but I don't think it would be good enough to simply move it to contribute/design as-is. I proposed a rough plan on #17063#note-6. Should I take care of splitting the blueprint and moving pieces to contribute/*, or would you folks like us to first discuss together what should go where, or would one of you want to take care of thinking this through by themselves, or something else?

@intrigeri I'm fine with you making this choice and implement it as you see fit.

  • Structure-wise, it's not clear to me why "Machine translation" is in its own section instead of being part of "Setup of our translation platform". For example, I don't understand why this section is different in nature from the one about the staging website, which is in "Setup of our translation platform". I would move it there, if you agree.

@intrigeri this part was moved from translate-server.git. I agree with you, it should be part of our setup. Please move it there.

#37 Updated by hefee 2 months ago

I've reviewed this piece of doc. First, congrats: this work is amazing! By reading it, I understood a few things that were previously not 100% clear to me.

great ;D

As agreed beforehand, I've pushed a number of (mostly minor) improvements: 161b3436732e950ae3c84897bc7db0a7d3a3fac3. I'll let you review them if you'd like to.

ACK

  • There's a "XXX" left. It seems to me that this comment is correct and the script should be renamed. I'm not sure how important this is and given we're likely running out of budget, I would be fine with leaving things as-is.

lets create a ticket for this and remove the XXX.

  • The bits about Docker are outdated, no? I propose we leave them in the blueprint when we'll move the rest of it elsewhere.

ACK

  • I think I would move the specification ("Choosing a translation web platform") to a dedicated sub-page. If that's fine by you, I volunteer to do it myself.

ACK

  • More generally, this blueprint mixes rather different kinds of information together. I think it's been fine so far, as a single place to write all this made this work simpler; but I don't think it would be good enough to simply move it to contribute/design as-is. I proposed a rough plan on #17063#note-6. Should I take care of splitting the blueprint and moving pieces to contribute/*, or would you folks like us to first discuss together what should go where, or would one of you want to take care of thinking this through by themselves, or something else?

ACK

  • The blueprint claims that touch and changing mtime triggers a Git commit, which is incorrect AFAICT: Git does not care about such filesystem metadata. So the explanation of why we need to avoid creating useless merge commits seems buggy to me. hefee, if you agree, could you please fix it?

@intrigeri: This was my explanation, why git allows me to commit, because from cmd line I know that git stops me from commiting empty commits. I see this is a highlevel check, that is not activated within python-git.

  • I don't understand the logical "then" relationship there "Weblate's Git repository is not bare. Hence […]". Even if it were a bare repo, we would have to merge its changes, no?

@intrigeri: This logical "then" is about that we need to pull and can't push, not about the integration of commits from Weblate. But yeah I'm not happy with this paragraph.

  • Structure-wise, it's not clear to me why "Machine translation" is in its own section instead of being part of "Setup of our translation platform". For example, I don't understand why this section is different in nature from the one about the staging website, which is in "Setup of our translation platform". I would move it there, if you agree.

Move it.

#38 Updated by hefee 2 months ago

  • Related to Bug #17129: rename puppet-tails:update_weblate_git.py → merge_canonical_changes.py added

#39 Updated by hefee 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from hefee to intrigeri

hefee wrote:

As agreed beforehand, I've pushed a number of (mostly minor) improvements: 161b3436732e950ae3c84897bc7db0a7d3a3fac3. I'll let you review them if you'd like to.

I reviewed this one and others by u on top of mine.

  • There's a "XXX" left. It seems to me that this comment is correct and the script should be renamed. I'm not sure how important this is and given we're likely running out of budget, I would be fine with leaving things as-is.

lets create a ticket for this and remove the XXX.

created #17129.

  • The blueprint claims that touch and changing mtime triggers a Git commit, which is incorrect AFAICT: Git does not care about such filesystem metadata. So the explanation of why we need to avoid creating useless merge commits seems buggy to me. hefee, if you agree, could you please fix it?

intrigeri: This was my explanation, why git allows me to commit, because from cmd line I know that git stops me from commiting empty commits. I see this is a highlevel check, that is not activated within python-git.

improved the paragraph.

  • I don't understand the logical "then" relationship there "Weblate's Git repository is not bare. Hence […]". Even if it were a bare repo, we would have to merge its changes, no?

intrigeri: This logical "then" is about that we need to pull and can't push, not about the integration of commits from Weblate. But yeah I'm not happy with this paragraph.

improved the paragraph.

Please review c8ae45e9bd656a1005d63e9b6df926b973df5ef1.

#40 Updated by intrigeri 2 months ago

  • Status changed from Needs Validation to In Progress

hefee wrote:

Please review c8ae45e9bd656a1005d63e9b6df926b973df5ef1.

LGTM! I'll now move stuff around and then we should be done here \o/

#41 Updated by intrigeri 2 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)

Also available in: Atom PDF