Project

General

Profile

Feature #11966

Feature #5630: Reproducible builds

Reproducible website build

Added by intrigeri about 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Build system
Target version:
-
Start date:
11/19/2016
Due date:
% Done:

100%

Feature Branch:
web/11966-deterministic-website
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

We include a local copy of the website in the ISO image. It must build reproducibly, somehow. ikiwiki has a deterministic option that should help.


Related issues

Related to Tails - Bug #11987: ikiwiki includes "Posted" timestamps in some generated web pages Resolved 11/22/2016
Related to Tails - Feature #12051: Use ikiwiki 3.20161219+ instead of our patched one Resolved 12/21/2016
Related to Tails - Bug #13531: Use ikiwiki 3.20161219+, again Duplicate 07/29/2017
Blocks Tails - Bug #12032: The SquashFS creation is not deterministic Resolved 12/14/2016

Associated revisions

Revision a459578c (diff)
Added by intrigeri about 3 years ago

Enable ikiwiki's "deterministic" option, and require it when building.

refs: #11966

Revision 00b542cd (diff)
Added by intrigeri about 3 years ago

page.tmpl: drop "Created" comment.

It causes reproducibility issues, and we don't display it anyway.

refs: #11966

Revision aa3feb64
Added by intrigeri about 3 years ago

Merge remote-tracking branch 'origin/web/11966-deterministic-website' into feature/5630-deterministic-builds

refs: #11966

Revision d8fc848a (diff)
Added by intrigeri about 3 years ago

Web site: remove the "Last edited" block (refs: #11966).

It causes reproducibility issues and brings little value: e.g.
for non-English pages, it boils down to "when the website was last
refreshed", which is mostly useless information.

Revision 224fe905 (diff)
Added by intrigeri about 3 years ago

Install ikiwiki from our builder-jessie APT suite, to make the pagestats plugin output deterministic (refs: #11966).

Patch submitted upstream: https://ikiwiki.info/bugs/pagestats_output_is_not_deterministic/

Revision 378680f4
Added by anonym about 3 years ago

Merge remote-tracking branch 'origin/web/11966-deterministic-website'

Fix-committed: #11966

Revision d38e385b (diff)
Added by intrigeri over 2 years ago

Add an explicit publication date to the 3.0 announcement.

refs: #11966

Revision 82706717 (diff)
Added by intrigeri over 2 years ago

Fix publication date of old calls for testing (refs: #11966).

Buggy dates make the website fail to build reproducibly.

Revision e992c455 (diff)
Added by intrigeri over 2 years ago

Fix "meta date" directives that were translated by mistake.

This is metadata, not strings displayed as-is to users. It must be in a format
that Date::Parse can read, which means translating dates into arbitrary language
won't cut it.

refs: #11966

History

#1 Updated by intrigeri about 3 years ago

  • Feature Branch set to web/11966-deterministic-website

#2 Updated by intrigeri about 3 years ago

  • Assignee set to intrigeri

#3 Updated by intrigeri about 3 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Progress!

  • ikiwiki's "deterministic" option was enabled; not sure if it buys us much, but it should only improve things;
  • the pagestats output issue was reported with a patch: https://ikiwiki.info/bugs/pagestats_output_is_not_deterministic/ -- next step is probably to upload a patched ikiwiki to our builder-jessie overlay APT suite;
  • the email address mangling issue was fixed on the topic branch (by using a new "discount" package), that I'll merge into feature/5630-deterministic-builds once anonym has fixed his Git history there; same for the "Created" timestamp issue.

#4 Updated by intrigeri about 3 years ago

Once my branch will be merged and a patched ikiwiki uploaded, it looks like the "Last edited" timestamp may be the only remaining problem.

#5 Updated by intrigeri about 3 years ago

  • % Done changed from 10 to 20

intrigeri wrote:

Uploaded, and web/11966-deterministic-website ensures it is installed in the Vagrant build box.

it looks like the "Last edited" timestamp may be the only remaining problem.

Fixed on the topic branch.

#6 Updated by intrigeri about 3 years ago

  • Assignee changed from intrigeri to anonym
  • Target version set to Tails_2.9.1
  • % Done changed from 20 to 50
  • QA Check set to Ready for QA

I did two ISO builds with cleanall from feature/5630-deterministic-builds, that has web/11966-deterministic-website merged in, and the resulting SquashFS'es contain exactly the same website (empty diff output). So please merge the topic branch into master.

#7 Updated by anonym about 3 years ago

  • Status changed from In Progress to 11
  • Assignee deleted (anonym)
  • % Done changed from 50 to 100
  • QA Check changed from Ready for QA to Pass

Looks great!

#8 Updated by intrigeri about 3 years ago

  • Status changed from 11 to Resolved

This was merged into master.

#9 Updated by intrigeri about 3 years ago

  • Related to Bug #11987: ikiwiki includes "Posted" timestamps in some generated web pages added

#10 Updated by sajolida about 3 years ago

I'm quite happy we got rid of the "Last edited" timestamp!

#11 Updated by intrigeri almost 3 years ago

  • Related to Feature #12051: Use ikiwiki 3.20161219+ instead of our patched one added

#12 Updated by intrigeri over 2 years ago

  • Status changed from Resolved to In Progress
  • Assignee set to intrigeri
  • Target version deleted (Tails_2.9.1)
  • % Done changed from 100 to 50
  • QA Check deleted (Pass)

Actually it's still not 100% reproducible: some of the pages that list other pages (with inline) produce different results, apparently depending on the mtime of source files. I'm on it.

#13 Updated by intrigeri over 2 years ago

  • Blocks Bug #12032: The SquashFS creation is not deterministic added

#14 Updated by intrigeri over 2 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100

I've fixed all issues I've seen happen so far. Will reopen if more problems show up.

#15 Updated by intrigeri over 2 years ago

  • Related to Bug #13531: Use ikiwiki 3.20161219+, again added

Also available in: Atom PDF