Project

General

Profile

Feature #15342

Add option to our build system for caching the wiki

Added by anonym almost 2 years ago. Updated 3 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Start date:
02/22/2018
Due date:
% Done:

100%

Feature Branch:
feature/15342-cache-wiki
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Related issues

Related to Tails - Feature #15864: Make onboarding of new developers easier In Progress 08/30/2018
Blocks Tails - Bug #17439: Consider enabling the cachewebsite build option on our CI Confirmed

Associated revisions

Revision 5d07d53a (diff)
Added by anonym almost 2 years ago

Build system: add opt-in caching of the wiki.

This commit is dedicated to segfault! ;)

Will-fix: #15342

Revision 7ee190b2 (diff)
Added by intrigeri 18 days ago

Replace all new occurrences of "wiki" with "website" (refs: #15342)

Our website has not exactly been a wiki for many years now, and thus we've been
slowly moving from this terminology to "website".

Revision 0bc00bbc (diff)
Added by intrigeri 18 days ago

Build system: when building a release with cachewebsite set, disable it instead of aborting (refs: #15342)

Let's be nicer to our RMs and don't punish them for having cachewebsite
set in their environment, shell history, or muscle memory :)

Revision 23c13910 (diff)
Added by intrigeri 18 days ago

Build system: when building a release with fastcomp set, disable it instead of aborting (refs: #15342)

Let's be nicer to our RMs and don't punish them for having fastcomp
set in their environment, shell history, or muscle memory :)

Revision dccbff0e (diff)
Added by intrigeri 17 days ago

Build system: key the website build cache on the input parameters that primarily determine its output (refs: #15342)

As said when I reviewed anonym's initial implementation
(5d07d53a0a70a51bfb43d9fa7f317855362a0706), I was not satisfied by a cache that
came with no automated invalidation mechanism at all, thus leaving up to each
individual developer the "is this weird behavior caused by the cache?" analysis,
and the manual invalidation of the cache.

So, IMO we need to know when we can invalidate the cache, which is equivalent
to: we need to know when a fresh build of the website would yield
a significantly different output. This is what this commit does, by maintaining
multiple cache directories, keyed to the input parameters that primarily
determine the output of the website build. Note that I'm definitely not aiming
for perfection here: for example, I did not list all the ikiwiki
dependencies recursively.

Finally, I'm also moving this cache to a dedicated storage volume, in order to
support caching the website even when not using the 'vmproxy' option:

- To start with, I don't use 'vmproxy' myself, but I would like to benefit from
caching the website.
- Furthermore, once we're confident enough in this caching mechanism, we could
consider enabling it for our CI builds, that don't use 'vmproxy' either.

Revision f5c75d4e
Added by intrigeri 4 days ago

Merge branch 'feature/15342-cache-wiki' into stable (Closes: #15342, #17288, #16607, #17289, #11411)

History

#1 Updated by anonym almost 2 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from anonym to bertagaz
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA
  • Feature Branch set to feature/15342-cache-wiki

5d07d53a0a70a51bfb43d9fa7f317855362a0706 is dedicated to you, segfault! ;)

bertagaz, please review'n'merge into stable! I took some effort to confine the code so it should only fire when really wanted, to keep you Jenkins ops calm. :)

#2 Updated by bertagaz almost 2 years ago

  • Target version changed from Tails_3.6 to Tails_3.7

#3 Updated by intrigeri almost 2 years ago

  • Assignee changed from bertagaz to intrigeri

(Switching to the new RM/FT task split.)

#4 Updated by intrigeri almost 2 years ago

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Ready for QA to Dev Needed

I'm not convinced that "caching" the wiki directory makes sense: do I understand correctly that the source of the last successfully built website will be re-used, thus ignoring any change made to the website in the branch being built? I see the point of caching (saving build time) but building something different from the source that one is building from feels very wrong to me. What's the expected benefit of doing that as opposed to only caching the website build artifacts?

This is weird:

+            mkdir -p "${target}" 
+            rm -rf "${target}" 

Maybe you want something like mkdir -p $(dirname $target)? Or I misunderstand the intent?

Also, I'd rather not introduce new instances of the "wiki" nomenclature, that we're been slowly moving from for years in favour of "website": please s/wiki/website/.

Finally, is there any compelling reason to target stable? I'm not opposed to it in principle (I see that the code should be a no-op as long as the cachewiki option is not set) but I thought this was in great part meant for segfault, who's mostly working on branches based on devel anyway, so perhaps the (small) risk is not worth it.

#5 Updated by intrigeri almost 2 years ago

  • Target version deleted (Tails_3.7)

#6 Updated by intrigeri 10 months ago

  • Related to Feature #15864: Make onboarding of new developers easier added

#7 Updated by intrigeri about 2 months ago

segfault and I discussed this a bit a few days ago. Here's what I recall from our conclusions. This is probably heavily biased by my own views on this topic so please assume it's my own opinion only.

The caching problem is a well-known one. One of its hardest part is to be able to detect when a cache is stale and must be invalidated. That is, in order to cache the website in a way that produces the expected results, we need to know when we can invalidate the cache. This is equivalent to: we need to know when a fresh build of the website would yield a different output. It follows that we need to encode, along with the cached built website, the status of anything that affects such output. This includes:

  • last commit on wiki/src and ikiwiki.setup (so before reusing the cached built website, we can check if anything changed there); we do something similar for the Vagrant build box.
  • version of the tools used to build the cached version, such as ikiwiki and its most important dependencies (e.g. po4a and the library used to render Markdown to HTML)

IMO we should not aim for perfection here (e.g. I doubt it's worth listing all ikiwiki dependencies recursively) but for some trade-off that gives us something that's good enough (for some value of good enough that we can agree on).

(And while I'm on this topic, a meta point: I'm personally exposed to lots of input wrt. the pain points of our development process. I'm missing a way to collectively prioritize them. Right now I'm commenting here, and I'm not filing tickets for the various other ideas/problems that I've heard of, because I believe that the lack of this build feature is the most important pain point in our dev process; I venture this is confirmed by the fact some of us have local, custom hacks to avoid building the website when they build Tails images; if this is not resolved once we're there, I'd like to propose this ticket as a candidate for FT work for our next roadmap.)

#8 Updated by intrigeri 18 days ago

  • Assignee changed from anonym to intrigeri

Feeling like structured procrastination today ⇒ I'll give it a shot and we'll see how far I go.

#9 Updated by intrigeri 17 days ago

  • Status changed from In Progress to Needs Validation

#10 Updated by intrigeri 17 days ago

  • Assignee deleted (intrigeri)
  • Target version set to Tails_4.2

#11 Updated by CyrilBrulebois 11 days ago

  • Target version changed from Tails_4.2 to Tails_4.3

#12 Updated by segfault 9 days ago

  • Assignee set to segfault

#13 Updated by segfault 9 days ago

  • Assignee changed from segfault to intrigeri

LGTM, nice Python code again :) I pushed a commit fixing some minor style issues, please review and then feel free to merge the branch.

#14 Updated by segfault 9 days ago

The branch fixes some issues which qualify as FT work and some which do not. I only tracked the time I spent reviewing the whole branch, not each commit / issue. Would it be ok if I just clocked half of the time as FT work?

#15 Updated by intrigeri 9 days ago

Hi @segfault,

I only tracked the time I spent reviewing the whole branch, not each commit / issue. Would it be ok if I just clocked half of the time as FT work?

Reviewing code submissions is FT work, regardless of whether the development work was paid or not ⇒ clock it all!

#16 Updated by intrigeri 9 days ago

LGTM, nice Python code again :)

/me blushes again!

I pushed a commit fixing some minor style issues, please review and then feel free to merge the branch.

LGTM! I'll happily merge once 4.2.1 is out and Jenkins has successfully built from the updated branch.

#17 Updated by intrigeri 4 days ago

  • Blocks Bug #17439: Consider enabling the cachewebsite build option on our CI added

#18 Updated by intrigeri 4 days ago

Dear segfault, now that 4.2.2 is behind us I'm going to merge this. Just to be sure I tried to build with cachewebsite enabled first, and… it failed. A part of 2f5785fde147dcc5eadba93f9f64f52ab4c638d8 broke the build, so I've reverted that part in b1a68a6349cbd3ee47004fc54926e774a87483dc. If there's a more clever way to fix that, or if I misunderstood something, please let me know :)

And while we're at it, segfault and @anonym, I'd love your input on #17439.

#19 Updated by intrigeri 4 days ago

  • Status changed from Needs Validation to Resolved
  • % Done changed from 50 to 100

#20 Updated by segfault 3 days ago

intrigeri wrote:

Dear segfault, now that 4.2.2 is behind us I'm going to merge this. Just to be sure I tried to build with cachewebsite enabled first, and… it failed. A part of 2f5785fde147dcc5eadba93f9f64f52ab4c638d8 broke the build, so I've reverted that part in b1a68a6349cbd3ee47004fc54926e774a87483dc.

Oh shit, sorry about that!

#21 Updated by segfault 3 days ago

I also forgot something else I wanted to note after the review:

I would also like to have an option to not rebuild the website even if the wiki changed. Most of the time I build a Tails image, I don't care about the website at all, I just want to test my feature / bugfix. In fact, I'm almost always using this script I wrote to build Tails, which simply patches out building the website, and the only issue I had with this is that the Unsafe Browser doesn't start if /usr/share/doc/tails/website/misc/unsafe_browser_warning doesn't exist. That would be fixed if any cached website build would be used.

Also available in: Atom PDF