Project

General

Profile

Feature #15077

Feature #10034: Translation web platform

Feature #15079: Integrate the platform with our Git and ikiwiki infrastructure

Have a staging website to build planned languages, with a resilient build

Added by u almost 2 years ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
03/14/2017
Due date:
% Done:

100%

Spent time:
(Total: 2.20 h)
Feature Branch:
feature/15077-add-staging-website
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

This is part of the point "Integrate the platform with our Git and ikiwiki infrastructure". We want to have a publicly accessible copy of our website running on the translation platform VM. This copy shall use Weblate's Git repository and regularly build from it's master branch our website.

This ticket is about
- installing the website on the machine
- make it accessible under some sort of subdomain (preprod.tails.boum.org) - or something similar
- set up a cron job do build planned languages


Subtasks

Feature #12340: [translate.vm] give weblate permission to write to /var/lib/weblate/stagingDuplicate

Feature #12341: [translate.vm]apache2 config for staging websiteResolved

Feature #12342: [translate.vm]Decide if we merge 'weblate git' with 'staging git'Resolved

Feature #15078: Review staging website and resilient buildResolved

Bug #15568: Script create staging wiki with suggestionsResolved

Bug #15625: Create ikiwiki-staging.setup for staging websiteResolved

Bug #16302: Syslog and staging websiteRejected

Bug #16309: ikiwiki tries to pull/push from gitResolved


Related issues

Duplicated by Tails - Feature #12311: Create weblate staging wiki Duplicate 03/14/2017
Blocks Tails - Feature #15084: Review & Rubberducking Resolved 12/19/2017

History

#1 Updated by u almost 2 years ago

  • Blocks Feature #15078: Review staging website and resilient build added

#2 Updated by u almost 2 years ago

  • Parent task changed from #10034 to #15079

#4 Updated by u almost 2 years ago

#5 Updated by u almost 2 years ago

#6 Updated by u almost 2 years ago

#7 Updated by u over 1 year ago

We might want to reuse work done on #12311.

#8 Updated by emmapeel over 1 year ago

#9 Updated by u over 1 year ago

We've decided at our first meeting yesterday that we would want the wiki to be built once a day and also allow translators to trigger a rebuile (eventually only for a single file) by clicking some button.

#10 Updated by u over 1 year ago

Probably before each build we would also need to run "git clean -fdx".

#11 Updated by bertagaz over 1 year ago

At this same meeting, we also raised the question of the hosting of this staging website: should it be in the translation VM or a in dedicated one? This has to be considered before starting to implement it.

#12 Updated by groente over 1 year ago

  • Assignee changed from groente to u
  • QA Check set to Info Needed

Personally, I'm in favour of hosting the staging site on the same machine. It will make it easier to check if a staging-build is already running (so the setup doesn't go beserk when a user is clicking 'rebuild' several times in a row) and it reduces resource overhead. What do you think?

#13 Updated by u over 1 year ago

I agree.

We will need to build it with a dedicated ikiwiki-staging.setup file - which will have all languages enabled. I wonder if this file can be stored in Git without disturbing other builds though.

#14 Updated by bertagaz over 1 year ago

  • Target version changed from Tails_3.7 to Tails_3.8

#15 Updated by u over 1 year ago

  • Assignee changed from u to groente
  • QA Check changed from Info Needed to Dev Needed

Hi groente!

In #15568 respectively in the translate-server.git repository, under scripts/, you will find a Python script that can update a given Git clone (/var/lib/weblate/repositories/vcs/staging on the production machine) with uncommitted suggestions from Weblate. You can also find this script in /var/lib/weblate/repositories/home/save-suggestions.py. You need to execute this file as 'weblate' user.

This script should probably be run by cron once a day and when it's done, the wiki should be built from this folder. This wiki needs their proper ikiwiki.setup file - which I will commit to master shortly.

Now, this build should also be run by a daily cronjob.

You will also need to make this accessible through a subdomain like https://staging.tails.boum.org

That's it! :)

#16 Updated by u over 1 year ago

This script should probably be run by cron once a day and when it's done, the wiki should be built from this folder. This wiki needs their proper ikiwiki.setup file - which I will commit to master shortly.

Done in #15625.

#17 Updated by u over 1 year ago

Hi groente: any ETA for this? Please let me know if you need help.

#18 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.8 to Tails_3.9

#19 Updated by groente about 1 year ago

  • QA Check changed from Dev Needed to Info Needed

There's a first peak on https://translate.tails.boum.org/staging , a dedicated domainname has been requested. Turning this into a cronjob is on the todo.

One question, though: do you want/need apache to serve the appropriate index.XX.html file based on user language settings or is it okay like this?

#20 Updated by groente about 1 year ago

  • Assignee changed from groente to u

#21 Updated by groente about 1 year ago

Another question: how did the save-suggestions.py script end up in /var/lib/weblate/repositories/home/ ? It seems to differ (atleast in the comments) from the one in git..

I've added a small shellscript to run as cronjob which sets the right environment variables, runs the save-suggestions script and builds the wiki, but I'm not sure whether to save it in the translate-server repo or in puppet..

#22 Updated by u about 1 year ago

groente wrote:

Another question: how did the save-suggestions.py script end up in /var/lib/weblate/repositories/home/ ? It seems to differ (atleast in the comments) from the one in git..

This file ended up there because we wanted to test it (it worked).
In Git there are two versions of this script, one works with plain Python2. See git log for more details. Otherwise they are the same.

I've added a small shellscript to run as cronjob which sets the right environment variables, runs the save-suggestions script and builds the wiki, but I'm not sure whether to save it in the translate-server repo or in puppet..

That sounds great. I think we should save that in puppet. I've put it in translate-server because that's where I had unlimited access. But if it's in puppet, we can point there and delete it from translate-server.

#23 Updated by u about 1 year ago

  • Assignee changed from u to groente
  • QA Check deleted (Info Needed)

groente wrote:

There's a first peak on https://translate.tails.boum.org/staging , a dedicated domainname has been requested. Turning this into a cronjob is on the todo.

Wooohoo! :)

Great!

One question, though: do you want/need apache to serve the appropriate index.XX.html file based on user language settings or is it okay like this?

Right now, there is no file selected, there is only the directory listing.
User language settings would be good, but if it's too complicated, we don't really need that.
But: I think it would be good to at least have the english version selected by default, i.e. point to index.en.html.

#24 Updated by groente about 1 year ago

  • Assignee changed from groente to u
  • QA Check set to Dev Needed

Okay, the default index is now set to english. The shellscript to build the staging site is now deployed through puppet and a daily cronjob has been set to run it at night.

Currently, the script breaks at the build-website part, where it fails a sanity check:

wiki/src/news/version_3.1.fr.po: 'meta date' msgstr contains non-rfc2822 date: Martes, 08 de agosto de 2017 12:34:56 +0000

Any chance you can fix that?

Apart from that, it's now a matter of waiting for the domainname to be created and setting up the reverse proxy, shouldn't take more than a week.

#25 Updated by groente about 1 year ago

  • Assignee changed from u to intrigeri
  • QA Check changed from Dev Needed to Ready for QA
  • Feature Branch set to feature/15077-add-staging-website

Hey intri, can you check if I guessed the right way to add a vhost to our nginx (commit 5c3af6dc24a807a2b2a1381ade13e8b8b1fe6883) ? Thanks!

#26 Updated by intrigeri about 1 year ago

  • Assignee changed from intrigeri to groente
  • QA Check deleted (Ready for QA)

Hey intri, can you check if I guessed the right way to add a vhost to our nginx (commit 5c3af6dc24a807a2b2a1381ade13e8b8b1fe6883) ? Thanks!

  • nginx::included { 'tails_weblate_reverse_proxy' will trigger a duplicate declaration error: it's already declared in tails::weblate::reverse_proxy. You'll need to give that resource a unique name.
  • You're lucky that tails/weblate/nginx/reverse_proxy.vhost.erb seems to be suitable here. But reusing it as-is will be confusing: someone might change that template, thinking it's only used for weblate ("This file is managed by the `::tails::weblate::reverse_proxy` class"). So either duplicate or share, but if you share, find some way to make it clear. Same for tails/weblate/nginx/reverse_proxy.inc.erb. In both cases, for #14588 I'll need similar templates for reverse proxying to an ikiwiki website. So perhaps you can copy the stuff to tails/ikiwiki/nginx/reverse_proxy.{vhost,inc}.erb, which eliminates the confusion and paves the way for the work we'll have to do soon anyway?

Other than that, LGTM!

#27 Updated by groente about 1 year ago

  • Assignee changed from groente to intrigeri
  • QA Check set to Ready for QA

hey, can you check commit 622a0de1542ff41ff75c2104450181b84a4444d8 ?

i've tried to re-use the template between limesurvey, translate, and staging in such a way that you can easily add your ikiwiki site aswell. i wouldn't be surprised if i'm violating some puppet syntax, though, and i'm a bit unsure about the directory naming in the templates directory, so some feedback would be very welcome :)

#28 Updated by emmapeel about 1 year ago

groente wrote:

Currently, the script breaks at the build-website part, where it fails a sanity check:

wiki/src/news/version_3.1.fr.po: 'meta date' msgstr contains non-rfc2822 date: Martes, 08 de agosto de 2017 12:34:56 +0000

I fixed that string, now the ikiwiki build script (/var/lib/weblate/repositories/home/suggestions-cronjob.sh) is failing with:

/var/www/staging//sandbox.html independently created, not overwriting with version from sandbox

I suggest to run ./build-website --rebuild instead of ./build-website to avoid this problem.

#29 Updated by emmapeel about 1 year ago

Actually, I did change that command and now it seems to be working

#30 Updated by groente about 1 year ago

hey emmapeel,

i've added the --rebuild flag to the suggestions-cronjob.sh in puppet. let me know if you still encounter errors in the cron output!

#31 Updated by intrigeri about 1 year ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to groente
  • QA Check deleted (Ready for QA)

groente wrote:

hey, can you check commit 622a0de1542ff41ff75c2104450181b84a4444d8 ?

i've tried to re-use the template between limesurvey, translate, and staging in such a way that you can easily add your ikiwiki site aswell.

Good idea!

i wouldn't be surprised if i'm violating some puppet syntax, though, and i'm a bit unsure about the directory naming in the templates directory, so some feedback would be very welcome :)

It looks good! Two comments:

  • I'm not very happy with the name of the $includename variable, but I think that's mostly because it's exposed as a parameter and thus becomes API. Maybe make it a non-parameter variable and be done with it?
  • We now have 3 reverse proxy classes that are basically the same and I see we have 2 more that diverge in minor ways. Granted, I started the copy'n'paste dance but looking at the big picture now, I'll want to refactor this at some point. You don't have to do it and that's not a blocker for this ticket, but in case you want to use this as a learning opportunity, here's what I would do: I would create a tails::nginx::reverse_proxy defined resource and then most of the existing & future tails::*::reverse_proxy classes become thin wrappers around it, whose job is merely to 1. provide a higher level API; and 2. manage and pass parameters to tails::nginx::reverse_proxy. Maybe the one for reprepro is too special, would require to make t::n::reverse_proxy too complex, and should therefore remain ad hoc.

Finally, sooner or later you'll need to set up things to be able to test this kind of work locally. This may be better done during a sprint.

#32 Updated by groente about 1 year ago

  • Assignee changed from groente to intrigeri
  • QA Check set to Ready for QA

Okay, thanks for the feedback!
Does commit a4191f27140ba2bb6bd6dfcbb6d8b0421f2754a6 look acceptable for merging?

Let's work on getting local deploys for testing purposes in the next sprint and then I'll continue refactoring a bit further.

#33 Updated by intrigeri about 1 year ago

  • Assignee changed from intrigeri to groente
  • QA Check changed from Ready for QA to Pass

Okay, thanks for the feedback!

My pleasure.

Does commit a4191f27140ba2bb6bd6dfcbb6d8b0421f2754a6 look acceptable for merging?

Absolutely.

#34 Updated by groente about 1 year ago

  • Status changed from In Progress to Resolved

#35 Updated by intrigeri 3 months ago

  • Assignee deleted (groente)

Also available in: Atom PDF