Project

General

Profile

Bug #15993

Feature #15292: Distribute a USB image

Adjust infrastructure and CI for USB images

Added by u 9 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Start date:
11/27/2018
Due date:
% Done:

100%

Estimated time:
24.00 h
Spent time:
(Total: 0.30 h)
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

Adjust CI wrt. more outputs from the build jobs => more inputs for the test suite and reproducibility tests

This includes at least:
- store .img* artifacts too when building
- copy .img* artifacts too to other jobs that need them
- garbage-collect .img* too
- symlink duplicate .img* too
- Deal with increased disk space requirements on Jenkins and in the ISO archive
- probably more stuff that we'll discover along the way
- post-release bugfixing

This work is supposed to be equally shared by groente and intrigeri.


Subtasks

Bug #16155: increase jenkins and iso-archive diskspaceResolved

Bug #16157: Re-enable USE_LAST_RELEASE_AS_OLD_ISO=yes on JenkinsResolved

Bug #16158: Ensure that old Jenkins artifacts are cleaned upResolved

Bug #16159: reproducibly_build_Tails_ISO_* jobs are brokenResolved

Bug #16160: test_Tails_ISO_* fail on Jenkins due to lack of disk spaceRejected

Bug #16162: Test reproducibility of USB images for all branchesResolvedgroente

Bug #16174: reproducibly_build_Tails_ISO_* jobs leave *.img around even when successfulResolved

Bug #16377: bittorrent.lizard lacks space for uploading both .img and .isoResolvedgroente

Bug #16395: rsync.lizard lacks space for uploading both .img and .iso + IUKsResolvedgroente

History

#1 Updated by u 9 months ago

  • Parent task set to #15929

#2 Updated by u 9 months ago

  • Parent task changed from #15929 to #15292

#3 Updated by u 9 months ago

@intrigeri: I trust you create corresponding subtickets for this task yourself and assign them between yourself and groente. Thanks!

#4 Updated by intrigeri 7 months ago

  • Subject changed from Adjust infrastructure and sysadmin for USB images to Adjust infrastructure and CI for USB images

#5 Updated by intrigeri 7 months ago

  • Category set to Continuous Integration

#6 Updated by intrigeri 7 months ago

  • Assignee changed from intrigeri to groente

During the sprint I'll act as a mentor & rubber-duck both for groente (on this ticket) and kibi (on #16002), while hopefully finding time to do my own work => formalizing who will take the lead on this ticket.

#7 Updated by intrigeri 7 months ago

I expect the bulk of the work that's needed is in:

  • jenkins-jobs.git (mostly in macros; don't touch auto-generated stuff)
  • puppet-tails.git

#8 Updated by intrigeri 7 months ago

- store .img* artifacts too when building

We do this already. Note that the artifacts file naming scheme is changing a bit:

  • before we had .iso and .iso.{apt-sources,buildlog,etc.}
  • once #16154 is merged we'll have .{img,iso,apt-sources,buildlog,etc.} i.e. the extra artifacts (build log and friends) lose the .iso component in their filename: that component does not make sense anymore since these files apply just as well to the .img

#9 Updated by groente 7 months ago

  • Assignee changed from groente to intrigeri
  • QA Check set to Ready for QA
  • Feature Branch set to jenkins-jobs:bugfix-15993-adjust-ci-for-usb-images

i've added img where it seemed to make sense in jenkins-jobs macros, can you check if it actually makes sense? :)

#10 Updated by intrigeri 7 months ago

  • Assignee changed from intrigeri to groente
  • QA Check deleted (Ready for QA)

groente wrote:

i've added img where it seemed to make sense in jenkins-jobs macros, can you check if it actually makes sense? :)

Perfect! I've pushed a minor nitpicking commit on top. Please merge into master once I'm done with #16154.

#11 Updated by groente 7 months ago

  • Assignee changed from groente to intrigeri
  • QA Check set to Ready for QA
  • Feature Branch changed from jenkins-jobs:bugfix-15993-adjust-ci-for-usb-images to puppet-tails:bugfix-15993-adjust-ci-for-usb-images

and some more changes to review!

#12 Updated by intrigeri 7 months ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to groente
  • QA Check changed from Ready for QA to Dev Needed
  • deduplicate_reproducible_build_jobs_upstream_ISOs:
    • -regex '*/archive/build-artifacts/1/*.i(so|mg)' does not do what you mean, I think (. and * mean something else). All in all I suspect that -path '*/archive/build-artifacts/1/*.iso' -o -path '*/archive/build-artifacts/1/*.img' will end up being more readable.
    • mixed tabs & space indentation; there were already some similar issues so I've fixed them all at once
  • I've pushed another improvement (writing the commit message was just as simple as explaining it here).
  • compare_artifacts: please DRY the duplicated code and make echo "Running diffoscope" tell which kind of things we're running it on. Or ask me too, I have some time booked on this budget line too and I'm fine dealing with my own expectations myself here :)

#13 Updated by groente 7 months ago

  • Assignee changed from groente to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

regexp should be fixed now, as well as the repetition in compare_artifacts

#14 Updated by intrigeri 7 months ago

  • Assignee changed from intrigeri to groente

Pushed 1 bugfix + 1 nitpicking commit, please review and deploy if happy.

#15 Updated by groente 7 months ago

  • Assignee changed from groente to intrigeri
  • QA Check deleted (Ready for QA)
  • Feature Branch deleted (puppet-tails:bugfix-15993-adjust-ci-for-usb-images)

merged & deployed

#16 Updated by intrigeri 7 months ago

Grown each isotesterN-data by 3GB, otherwise once it's got its upstream job's artifacts downloaded, there's no disk space left to wget the ISO + IMG from our ISO history repo.

#17 Updated by intrigeri 7 months ago

  • Target version changed from Tails_3.11 to Tails_3.13

Status update: all necessary adjustments we've spotted are done and we've spotted and solved all known regressions before other Tails folks reported them to us. The remaining follow-up subtasks are tiny things that can only be done later (e.g. revert temporary hack XYZ once Tails 3.12 is out).

#18 Updated by intrigeri 5 months ago

  • Target version changed from Tails_3.13 to 2019

#19 Updated by intrigeri 5 months ago

  • Target version changed from 2019 to Tails_3.13

#20 Updated by intrigeri 4 months ago

  • Assignee changed from intrigeri to groente

(All remaining subtasks are on groente's plate.)

#21 Updated by groente 4 months ago

  • Status changed from In Progress to Resolved

i think we're done here.

#22 Updated by intrigeri 4 months ago

Woohoo, mission accomplished \o/ :)))

Also available in: Atom PDF