Project

General

Profile

Bug #12053

Website build takes 10-20 minutes longer since upgrade to 3.20160905~bpo8+1

Added by intrigeri almost 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Build system
Target version:
Start date:
12/21/2016
Due date:
% Done:

100%

Feature Branch:
web/12053-faster-build
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

On November 19 we upgraded ikiwiki on our isobuilders from 3.20141016.3 to 3.20160905~bpo8+1 (a459578, for #11966, but it was going to happen at some point anyway). Since then, ISO build times went crazy because the web site takes much more time to build.

Associated revisions

Revision 29d3d41b (diff)
Added by intrigeri almost 3 years ago

Website: explicitly add sort="age" to all inline directives I can easily & mechanically patch (refs: #12053).

A serious performance regression was brought by ikiwiki 3.20150614. One of the
few changes introduced in that release was to change the default sorting, for
inline directives, from "age" to "age title". Presumably the latter is
substantially slower, possibly since it has to process the content of all
included pages in order to retrieve their title, which may mean that a subset of
pages are processed more than once.

Revision 33a45971 (diff)
Added by intrigeri almost 3 years ago

Add sort="age" to inline directives in PO files, to avoid making hundreds of strings fuzzy (refs: #12053).

Revision 845bf059
Added by intrigeri almost 3 years ago

Merge branch 'web/12053-faster-build' (Closes: #12053)

History

#1 Updated by sajolida almost 3 years ago

I get very similar build times on my Tails with different versions of ikiwiki:

|        3.20161219 | 34m5.039s |
| 3.20160905~bpo8+1 | 34m7.126s |

That's also similar to the times I reported with earlier versions in #11151.

#2 Updated by intrigeri almost 3 years ago

I get very similar build times on my Tails with different versions of ikiwiki:

> |        3.20161219 | 34m5.039s |
> | 3.20160905~bpo8+1 | 34m7.126s |
> 

So, the regression introduced between 3.20141016.3 and 3.20160905 isn't fixed in 3.20161219 (and probably still isn't fixed).
That's no big surprise to me, but still a useful data point :)

#3 Updated by intrigeri almost 3 years ago

On my machine, at 8044f99f7632f0bff87cee7f4dad37efba1fec06:

  • 3.20161219, deterministic: 16:25
  • 3.20161219, non-deterministic: 17:21 => no big difference, so next results all have deterministic enabled
  • 3.20160121: 17:10
  • 3.20150614: 15:57
  • 3.20150610: 6:19
  • 3.20150329: 6:12
  • 3.20141016: 6:09

#4 Updated by intrigeri almost 3 years ago

On my machine, 29d3d41b8d4ca7680e2074c0988c9710d44be20f, with ikiwiki 3.20161219, gets the build back down to 6:30.

#5 Updated by intrigeri almost 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to sajolida
  • Target version changed from Tails_2.11 to Tails 2.10
  • % Done changed from 0 to 30
  • QA Check set to Ready for QA
  • Feature Branch set to web/12053-faster-build
  • Type of work changed from Research to Code

Please review'n'merge into master. I only gave a cursory look at the resulting built website, so it would be nice if you made sure that non-obvious parts (e.g. the IA) are still OK.

#6 Updated by sajolida almost 3 years ago

  • Assignee changed from sajolida to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

Woh, that's sooo cool that you found that! My build time went down from ~34 minutes to ~14. You're my savior! Now I wonder whether the huge difference I observed between build in Tails and in Debian still exists :)

Regarding the work itself, I couldn't spot any problems by having 'sort="age"' everywhere, including in the IA. Still, I have two concerns:

  • You added a missing @[[!meta date@ to one post (1bc6bf2) but doing grep -L '!meta date="' news/*.mdwn I still spot 29 reports without dates. Would this be problematic?
  • Building with PO enable brings in several hundreds of broken translation string. Could you cast a sed spell to fix them?
  • The merge with master now conflicts after my work on #11297. I don't mind fixing this myself if you think I should be the one responsible for that :)

#7 Updated by sajolida almost 3 years ago

I've added some date with e2362df :)

#8 Updated by intrigeri almost 3 years ago

  • You added a missing @[[!meta date@ to one post (1bc6bf2) but doing grep -L '!meta date="' news/*.mdwn I still spot 29 reports without dates. Would this be problematic?

Apparently you've fixed that already after reassigning to me. Thanks. I'll skip my usual rant about limiting risks of duplicating work :p

  • Building with PO enable brings in several hundreds of broken translation string. Could you cast a sed spell to fix them?

I don't know what you mean with "broken translation string" so I'll build and try to figure it myself :) Stay tuned!

  • The merge with master now conflicts after my work on #11297. I don't mind fixing this myself if you think I should be the one responsible for that :)

I've just solved that problem, which took less time than arguing about who should resolve merge conflicts introduced by working on master before merging ready for QA branches that affect the same files.

Note that there was one more conflict due to a3a1d83 (I had done this on the master branch already to avoid new reports creating more problems in case this branch wasn't merged promptly).

#9 Updated by intrigeri almost 3 years ago

  • Assignee changed from intrigeri to sajolida
  • % Done changed from 30 to 80
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:

  • Building with PO enable brings in several hundreds of broken translation string. Could you cast a sed spell to fix them?

I don't know what you mean with "broken translation string" so I'll build and try to figure it myself :) Stay tuned!

Fixed on the branch (I had in mind to do that immediately after the merge, but I didn't want to do it earlier in case the review took time, otherwise we would have had to deal with potentially lots of painful merge conflicts).

So, I think we're good to go: please review and merge :)

#10 Updated by anonym almost 3 years ago

  • Target version changed from Tails 2.10 to Tails_2.11

#11 Updated by spriver almost 3 years ago

  • Assignee changed from sajolida to spriver

#12 Updated by spriver almost 3 years ago

  • Assignee changed from spriver to intrigeri

I reviewed the changes, I'm fine with them. But when I'm issuing a --rebuild of the wiki, I'll have 168 .po files changed (generating fuzzy entries, at least with some I checked). Is this correct?

#13 Updated by intrigeri almost 3 years ago

I reviewed the changes, I'm fine with them. But when I'm issuing a --rebuild of the
wiki, I'll have 168 .po files changed (generating fuzzy entries, at least with some
I checked). Is this correct?

Good catch!

I've manually unfuzzied them all and am now going to proceed with merging! :)

#14 Updated by intrigeri almost 3 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 80 to 100
  • QA Check changed from Ready for QA to Pass

Also available in: Atom PDF