Project

General

Profile

Feature #14588

Self-host our website

Added by intrigeri over 1 year ago. Updated 28 days ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
10/03/2018
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

New design

ikiwiki and web hosting

ikiwiki would run on www.lizard which also serves the generated content and runs ikiwiki.cgi (until #9174 is done):

  • ikiwiki on www.lizard pushes changes back to the master/central tails.git repo (PO files updates, changes done in the web interface aka. ikiwiki.cgi). That's exactly what the current production setup does.
  • Ensure logging policy is OK:
    • nginx' own logs: no IPs: OK
    • Journal, if relevant: OK, I've not found anything nginx-related in the Journal, even after hitting a 404
  • There's currently a cronjob that extracts connection stats from access.log (Tails boots, downloads of the detached ISO signature) and emails them to . We need to import that too.
  • The current setup is heavily based on Apache features, while we usually run nginx on our infra:

language negotiation

I got language negotiation to work fine with this included in the http {} block:

map $http_accept_language $lang {
    default en;
    ~*^de de;
    ~*^fa fa;
    ~*^es es;
    ~*^fr fr;        
    ~*^it it;
    ~*^pt pt;
}

… and this included in the vhost:

location / {
    try_files $uri $uri/index.$lang.html $uri/index.en.html $uri/index.html =404;
}

Our initial options were:

Master/central tails.git repo

The master/central tails.git repo would move to git.puppet.t.b.o (sic) i.e. hosted on lizard's gitolite:

  • we'll be able to use gitolite to manage ACLs which is better than the current setup at b.o, e.g. we can manage users & keys ourselves, give access to some more people to a subset of branches (e.g. we could allow some developers to push to feature/* and bugfix/* so their stuff is built and tested in Jenkins, but they would not be allowed to push to protected branches such as master, stable, testing and devel)
  • maybe we can use a better CNAME that does not say "puppet"; but anyway, unless using the Tor onion service, a SSH config is needed because that Gitolite runs behind a non-standard port, so contributors with push access can as well use the onion service and we don't bother about DNS names
  • Git validation hook (file size, obsolete rewritten history, etc.) copied from the one that's set up at b.o
  • Git hook that triggers an ikiwiki update with the pingee plugin
  • post-update hook that pushes to all mirrors (copied from b.o)

Migration plan

migrate the master/central tails.git repo

  1. copy everything over to lizard and have gitolite push updates to all mirrors (see design above); mirrors, Git validation hook
  2. have b.o disable their Git hook that pushes updates to mirrors, except lizard
  3. migrate the other repos hosted at boum.org (mirror-pool-dispatcher, mirror-pool, promotion-material, uui-binary) and push to boum.org and immerda on update
  4. announce downtime to Git committers
  5. forbid Git committers write access to the former master/central tails.git
  6. drop the ACL that allows the former repo to push to the new one
  7. have b.o disable their Git hook that pushes updates to lizard
  8. have the new master/central tails.git push to b.o on updates so that our website, still running at b.o at this point, is updated
  9. update contribute/git doc
  10. tell Git committers they can push again and ask them to update their config

bugfixes

  1. fix updating master Git repo upon PO files updates / edit on the live website

prepare the new web hosting setup

  1. set up the basic web server (including LE) and ikiwiki stuff on www.lizard under some temporary vhost name
  2. CGI: tweak fcgiwrap.socket so that only the www-data user has access to the socket
  3. have the master/central repo trigger an ikiwiki update (on Git push) on www.lizard as well, using the pingee plugin => we can test how it behaves
  4. deal with the other repos needed to build our website (mirror-pool-dispatcher, mirror-pool, promotion-material, uui-binary): give them each a post-update hook that triggers a git pull in a non-bare clone on www.lizard and then calls the pingee plugin
  5. import relevant stuff from the Apache vhost at b.o (e.g. HSTS)
  6. try to set Content-Language header correctly (not needed since we specify this in our HTML)
  7. logging
  8. htaccess
  9. avoid HTTP cache issues such as #16049
  10. see design above for more things to do
  11. ensure there's enough space in www.lizard:/var/log/nginx to host the amount of logs we need to keep around (even without hosting our website, at some point that directory took too much space: #12425)
  12. set up rss2email
  13. prevent ikiwiki.cgi from recording IP addresses in commit messages

migrate to the new web hosting setup

Once happy with the new hosting setup:

  1. allow ikiwiki on www.lizard to push to the canonical repo's master branch
  2. rename/copy it to support tails.b.o, adjust ikiwiki.setup accordingly, rebuild
  3. copy X.509 cert+key from the old website to the new one
  4. point tails.b.o in /etc/hosts to lizard on a test machine and ensure the website hosted there works as expected
  5. point DNS to lizard
  6. update /etc/hosts:
    1. on www: commit e4b20b4
    2. on other systems: look for jenkins.tails.boum.org in manifests/classes.pp
  7. update Git hooks: s/new\.tails\.boum\.org/tails\.boum\.org/ (or maybe just use www.lizard)
  8. drop/update everything (e.g. hooks) about new.tails.b.o
  9. ensure Let's Encrypt certificate renewal has a chance to work

And once we're convinced the new hosting setup works well enough:

  1. cronjob that emails stats from logs
  2. have b.o disable their rss2email feed, then set the email recipient for ours to the production one
  3. replace content of .htaccess with a pointer to https://git-tails.immerda.ch/puppet-tails/tree/templates/website/nginx/rewrite_rules.conf.erb and notify the translation platform team (who runs a staging version of our website on Apache)
  4. drop the temporary vhost and its files (whose name include news.tails.boum.org)
  5. drop permissions from b.o to all Git repos on lizard and immerda
  6. ask b.o to delete our website, Git repo, cronjob, and the corresponding SSH keys
  7. If time allows, do #9174 and/or #12408 (time does not allow)

Related issues

Related to Tails - Feature #10034: Translation web platform Confirmed 08/14/2015
Related to Tails - Bug #6907: ikiwiki po plugin does not play well with inline directives In Progress 12/03/2018
Related to Tails - Feature #11815: Have Tails::Download::HTTPS require TLS 1.2+ In Progress 09/20/2016
Related to Tails - Feature #16028: /mirrors.json is not synced with gitolite@d53ykjpeekuikgoq.onion Resolved 10/03/2018
Related to Tails - Bug #16123: Test suite broken on Jenkins since we self-host our website Resolved 11/13/2018
Related to Tails - Bug #16124: URLs without explicit .html are not redirected anymore Resolved 11/13/2018
Related to Tails - Bug #16142: Consider giving sajolida access to the web logs of our website Confirmed 11/19/2018
Blocks Tails - Feature #12408: Ensure our website is ready for temporary surge of new users Confirmed 03/29/2017
Blocks Tails - Feature #9174: Migrate our blueprints to blueprints.tails.boum.org Confirmed 04/07/2015
Blocks Tails - Feature #13284: Core work 2017Q2→2019Q2: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017
Blocks Tails - Bug #13450: Implement CSP HTTP header In Progress 07/10/2017
Blocks Tails - Feature #16091: Rethink our caching of CSS files In Progress 11/02/2018
Blocks Tails - Bug #12113: Make sure tails.boum.org is available over IPv6 Confirmed 01/05/2017

Associated revisions

Revision d8d3618b (diff)
Added by intrigeri 3 months ago

promotion-material.git was moved (refs: #14588)

Revision 3c6c928f (diff)
Added by intrigeri 3 months ago

Update doc: our main Git repository has moved (refs: #14588).

Revision 730ecb82 (diff)
Added by intrigeri about 1 month ago

Deprecate .htaccess (refs: #14588)

We're not going to maintain two versions of these rewrite rules,
one in Apache language, the other in nginx-speak.

Revision 8be6538d (diff)
Added by intrigeri about 1 month ago

Update who shall be notified in order to have ikiwiki.setup changes deployed on our production website (refs: #14588).

Revision 716c2bdc (diff)
Added by intrigeri about 1 month ago

Document our various ikiwiki configuration files and how to modify them (refs: #14588).

This page is explicitly linked from contribute/how/documentation
and will be automatically linked from contribute/how/website.

History

#1 Updated by intrigeri over 1 year ago

  • Blocks Feature #12408: Ensure our website is ready for temporary surge of new users added

#2 Updated by sajolida 11 months ago

  • Blocks Feature #9174: Migrate our blueprints to blueprints.tails.boum.org added

#3 Updated by u 10 months ago

#4 Updated by intrigeri 8 months ago

  • Blocks Feature #13284: Core work 2017Q2→2019Q2: Sysadmin (Adapt our infrastructure) added

#5 Updated by intrigeri 8 months ago

  • Target version changed from 2019 to Tails_3.10.1

#6 Updated by intrigeri 8 months ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress

#7 Updated by intrigeri 8 months ago

  • Description updated (diff)

#8 Updated by intrigeri 8 months ago

  • Description updated (diff)

#9 Updated by intrigeri 8 months ago

  • Description updated (diff)

#10 Updated by intrigeri 8 months ago

  • Description updated (diff)

#11 Updated by intrigeri 8 months ago

  • Description updated (diff)

#12 Updated by intrigeri 8 months ago

  • Description updated (diff)

#13 Updated by intrigeri 8 months ago

  • Description updated (diff)

#14 Updated by intrigeri 8 months ago

  • Description updated (diff)

#15 Updated by intrigeri 8 months ago

  • Description updated (diff)

#16 Updated by intrigeri 8 months ago

  • Description updated (diff)

#17 Updated by intrigeri 8 months ago

  • Description updated (diff)

#18 Updated by intrigeri 8 months ago

  • Description updated (diff)

#19 Updated by intrigeri 8 months ago

  • Description updated (diff)

#20 Updated by intrigeri 8 months ago

  • Description updated (diff)

#21 Updated by intrigeri 8 months ago

  • Description updated (diff)

#22 Updated by intrigeri 8 months ago

  • Description updated (diff)

#23 Updated by intrigeri 8 months ago

  • Description updated (diff)

#24 Updated by intrigeri 5 months ago

#25 Updated by intrigeri 4 months ago

  • Blocks Bug #13450: Implement CSP HTTP header added

#26 Updated by u 3 months ago

  • Related to Bug #12113: Make sure tails.boum.org is available over IPv6 added

#27 Updated by intrigeri 3 months ago

#28 Updated by intrigeri 3 months ago

  • Description updated (diff)

#29 Updated by intrigeri 3 months ago

  • Description updated (diff)

#30 Updated by intrigeri 3 months ago

  • Description updated (diff)

#31 Updated by intrigeri 3 months ago

  • Description updated (diff)

#32 Updated by intrigeri 3 months ago

  • Description updated (diff)

#33 Updated by intrigeri 3 months ago

  • Description updated (diff)

#34 Updated by intrigeri 3 months ago

  • Description updated (diff)

#35 Updated by intrigeri 3 months ago

  • Blocks deleted (Feature #15202: Onboard new members to the mirror team)

#36 Updated by intrigeri 3 months ago

  • Description updated (diff)

#37 Updated by intrigeri 3 months ago

  • Description updated (diff)

#38 Updated by intrigeri 3 months ago

  • Description updated (diff)

#39 Updated by intrigeri 3 months ago

  • Description updated (diff)
  • % Done changed from 0 to 20

#40 Updated by intrigeri 3 months ago

  • Description updated (diff)

#41 Updated by intrigeri 3 months ago

  • Description updated (diff)

#42 Updated by intrigeri 3 months ago

  • Description updated (diff)

#43 Updated by intrigeri 3 months ago

  • Description updated (diff)

#44 Updated by bertagaz 3 months ago

While fixing some unrelated stuff on isobuilder4, I noticed that apache2 was now installed and running on our isobuilders. My research pointed to commit puppet-tails:b24dd2d6ee58d500367f8d5ffc849a260b7f89b3 which I think is the root cause. The timing of apache2 installation shown by etckeeper seems to point to that commit. It's probably due to the fact that the xapian-omega package recommends apache2 | httpd-cgi.

#45 Updated by intrigeri 2 months ago

While fixing some unrelated stuff on isobuilder4, I noticed that apache2 was now installed and running on our isobuilders. My research pointed to commit puppet-tails:b24dd2d6ee58d500367f8d5ffc849a260b7f89b3 which I think is the root cause. The timing of apache2 installation shown by etckeeper seems to point to that commit. It's probably due to the fact that the xapian-omega package recommends apache2 | httpd-cgi.

Good catch, thanks! Fixed.

#46 Updated by intrigeri 2 months ago

  • Description updated (diff)

#47 Updated by intrigeri 2 months ago

  • Target version changed from Tails_3.10.1 to Tails_3.11

I've not scheduled my next work session on this yet but it's clear it won't be before 3.10.

#48 Updated by intrigeri 2 months ago

#49 Updated by intrigeri 2 months ago

  • Related to Bug #6907: ikiwiki po plugin does not play well with inline directives added

#50 Updated by intrigeri about 2 months ago

  • Description updated (diff)

#51 Updated by intrigeri about 1 month ago

  • Related to Feature #11815: Have Tails::Download::HTTPS require TLS 1.2+ added

#52 Updated by intrigeri about 1 month ago

#53 Updated by u about 1 month ago

  • Related to deleted (Bug #12113: Make sure tails.boum.org is available over IPv6)

#54 Updated by u about 1 month ago

  • Blocks Bug #12113: Make sure tails.boum.org is available over IPv6 added

#55 Updated by intrigeri about 1 month ago

  • Related to Feature #16028: /mirrors.json is not synced with gitolite@d53ykjpeekuikgoq.onion added

#56 Updated by intrigeri about 1 month ago

  • Priority changed from Normal to Elevated
  • % Done changed from 100 to 10

#57 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#58 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#59 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#60 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#61 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#62 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#63 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#64 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#65 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#66 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#67 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#68 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#69 Updated by intrigeri about 1 month ago

  • Description updated (diff)
  • % Done changed from 10 to 20

#70 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#71 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#72 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#73 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#74 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#75 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#76 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#77 Updated by intrigeri about 1 month ago

  • Description updated (diff)

#78 Updated by intrigeri about 1 month ago

  • Description updated (diff)
  • Assignee changed from intrigeri to groente
  • % Done changed from 20 to 50
  • QA Check set to Ready for QA

Seems to work fine. All relevant commits in puppet-tails.git should reference this ticket.

#80 Updated by intrigeri about 1 month ago

FTR I took a good look at Munin (both zooming in www.lizard -specific metrics and global lizard ones) and it's hard to notice any impact of switching the production website to lizard. The only significant change I could spot is that our outgoing network traffic has increased by ~1-2 sustained MBit/s, which is expected and matches the decrease we can see on boum.org's side. I'm now very tempted to stress-test the new setup and adjust if/as needed for #12408 but I'll take it easy: preliminary testing with ab(1) showed that with 10k connections and concurrency=500, the new setup is tremendously more efficient and reliable than the previous one (as in: the new setup can handle this load nicely while the old one would fall apart), which is a pretty good first step :)

#81 Updated by intrigeri about 1 month ago

  • Related to Bug #16123: Test suite broken on Jenkins since we self-host our website added

#82 Updated by groente about 1 month ago

  • Assignee changed from groente to intrigeri
  • QA Check changed from Ready for QA to Pass

#83 Updated by intrigeri 29 days ago

  • Related to Bug #16124: URLs without explicit .html are not redirected anymore added

#84 Updated by intrigeri 28 days ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100

#85 Updated by intrigeri 25 days ago

  • Related to Bug #16142: Consider giving sajolida access to the web logs of our website added

Also available in: Atom PDF