Project

General

Profile

Bug #17439

Bug #16960: Make our CI feedback loop shorter

Enable the cachewebsite build option by default, including on our CI

Added by intrigeri 3 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
feature/17439-enable-cachewebsite-by-default
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Enabling the new cachewebsite build option (#15342) by default should lower by 25% the total runtime of a Tails images build job on our CI.
It would take up to 5 extra GiB on isobuilders' /var/lib/libvirt/images.
Note that this option is automatically disabled when building a release (i.e. from a tag), to be on the safe side.

Here's the plan:

  1. Ask our developers to use the cachewebsite option locally, in order to iron out whatever bugs are left in the implementation (I think I'll do that anyway) done (2020-02-04)
  2. Ensure our isobuilders have enough space to store the cache done
  3. While waiting for feedback from developers, enable this on my local Jenkins for a few weeks, in order to learn about potential Jenkins-specific problems this may cause
  4. On a topic branch, enable cachewebsite by default and add a cachewebsite=no option to allow disabling this feature

Related issues

Blocked by Tails - Feature #15342: Add option to our build system for caching the wiki Resolved 02/22/2018
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed
Blocked by Tails - Bug #17481: Refresh our patches to apply on top of thunderbird 1:68.5.0-1~deb10u1 Resolved

Associated revisions

Revision 21717530 (diff)
Added by intrigeri about 2 months ago

Build system: enable website caching by default, with a way option to disable it (refs: #17439)

Revision e4c8259d (diff)
Added by intrigeri about 2 months ago

Build system: enable website caching by default, with a way option to disable it (refs: #17439)

Revision 79517875
Added by segfault about 1 month ago

Merge branch 'feature/17439-enable-cachewebsite-by-default' into stable (Closes: #17439)

Revision d1102c94 (diff)
Added by intrigeri about 1 month ago

Build system: enable website caching by default, with a way option to disable it (refs: #17439)

History

#1 Updated by intrigeri 3 months ago

  • Blocked by Feature #15342: Add option to our build system for caching the wiki added

#2 Updated by intrigeri 3 months ago

  • Description updated (diff)

#3 Updated by anonym 3 months ago

intrigeri wrote:

Enabling the new cachewebsite build option (#15342) should lower by 25% the total runtime of a Tails images build job on our CI.
It would take up to 5 extra GiB on isobuilders' /var/lib/libvirt/images.

A great trade-off, IMHO!

Any reasons why we should not enable this option?

I don't think so. Well, this adds another instance of where a caching bug can occur, which generally is tricky to even notice, but I don't think that should be a deterrent (imagine where we would be without any caching). It is very clear to me that the code that disables this cache for releases is correct, and that's the most import bit. Let's just keep it that way!

If not, I could:

  • ask our developers to use the cachewebsite option locally, in order to iron out whatever bugs are left in the implementation (I think I'll do that anyway)
  • enable this on my local Jenkins for a few weeks, in order to learn about potential Jenkins-specific problems this may cause

Sounds good! In fact, I think we should consider making cachewebsite the default since it will improve the developer UX story and we get less fragmented options among developers. In my experience, opt-in for a nice feature generally results in more fragmentation compared to opt-out -- with opt-in for a desirable feature, only users that read the docs will know about it and thus possibly/likely enable it.

#4 Updated by intrigeri 3 months ago

In fact, I think we should consider making cachewebsite the default since it will improve the developer UX story and we get less fragmented options among developers. In my experience, opt-in for a nice feature generally results in more fragmentation compared to opt-out -- with opt-in for a desirable feature, only users that read the docs will know about it and thus possibly/likely enable it.

Agreed. I made it opt-in to start with only out of humility and because I did not want to affect our CI so soon.
So perhaps this ticket becomes "Consider enabling the cachewebsite build option by default, including on our CI".
And then we'll need a nocachewebsite option, I guess.

#5 Updated by segfault 3 months ago

anonym wrote:

intrigeri wrote:

Enabling the new cachewebsite build option (#15342) should lower by 25% the total runtime of a Tails images build job on our CI.
It would take up to 5 extra GiB on isobuilders' /var/lib/libvirt/images.

A great trade-off, IMHO!

Any reasons why we should not enable this option?

I don't think so. Well, this adds another instance of where a caching bug can occur, which generally is tricky to even notice, but I don't think that should be a deterrent (imagine where we would be without any caching).

I agree.

#6 Updated by intrigeri 2 months ago

  • Subject changed from Consider enabling the cachewebsite build option on our CI to Enable the cachewebsite build option by default, including on our CI
  • Description updated (diff)
  • Assignee set to intrigeri
  • Target version set to Tails_4.4

#7 Updated by intrigeri 2 months ago

#8 Updated by intrigeri 2 months ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress

intrigeri wrote:

Enable this on my local Jenkins for a few weeks, in order to learn about potential Jenkins-specific problems this may cause

Enabled this today and already enjoying the benefits :)

Ensure our isobuilders have enough space to store the cache

They don't: during a build, there's only 2.9G free space there.

#9 Updated by intrigeri 2 months ago

  • Description updated (diff)

#10 Updated by intrigeri 2 months ago

  • Description updated (diff)

#11 Updated by intrigeri 2 months ago

  • Parent task set to #16960
  • Type of work changed from Sysadmin to Code

(The sysadmin part is done.)

#12 Updated by intrigeri about 2 months ago

  • Feature Branch set to feature/17439-enable-cachewebsite-by-default

#13 Updated by intrigeri about 2 months ago

  • Blocked by Bug #17481: Refresh our patches to apply on top of thunderbird 1:68.5.0-1~deb10u1 added

#14 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)

Looks good so far. It indeed cuts the build time down by almost 30% on Jenkins, in the best case scenario: https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-17439-enable-cachewebsite-by-default/buildTimeTrend

#15 Updated by segfault about 1 month ago

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF