Project

General

Profile

Bug #15999

Feature #15292: Distribute a USB image

Integrate USB image in the release process

Added by u 9 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
11/28/2018
Due date:
% Done:

100%

Estimated time:
3.00 h
Feature Branch:
doc/15999-integrate-usb-image-in-the-release-process
Type of work:
Contributors documentation
Blueprint:
Starter:
Affected tool:

Description

Review & rubber-duck: kibi (3h)

This includes at least:

  • adjust manual test suite
  • what about UDFs? the full upgrade path only includes the ISO it's not used anyway so we don't care, see details in 2d7cf8ac4308e290c4a3872fd80fd60b5e6ed459
  • upload to iso-archive
  • upload to rsync server
  • 2 torrent files
  • wiki/src/inc/* as requested by doc writers, see #16163
  • IDF:
    • adapt IDF generation
    • git grep install/v1
    • coordinate v1 vs. v2 IDF generation (both for some time?) with the release of the updated Verification Extension + the Installation Assistant version check for that extension
  • probably more stuff that we'll discover along the way

Subtasks

Bug #16164: Adjust release process for new build artifacts naming schemeResolved

Bug #16171: Get our master branch ready for IDF v2 and new Tails Verification extensionResolved


Related issues

Blocked by Tails - Bug #16163: Clarify updated release process requirements for USB image Resolved 11/28/2018
Blocked by Tails - Bug #16112: Confusing signature date on website doc Resolved 11/09/2018

Associated revisions

Revision a4c66f88 (diff)
Added by intrigeri 7 months ago

Update manual test suite to support USB images (refs: #15999)

Revision 2d7cf8ac (diff)
Added by intrigeri 7 months ago

Update incremental upgrades design doc to mention USB images (refs: #15999)

For now at least, the full upgrade target file will remain the ISO image.
If our Upgrader ever supports full upgrades, this might need to be adjusted.
Let's not spend more time on it now because that's unlikely to happen:

- Our plans for changing our upgrade process are rather leaning towards
dropping our own installer.
- Even if we do implement full upgrade support in Tails Upgrader,
chances are that it'll be easier to use an ISO than a USB image
as input.

Revision 2cc183f3 (diff)
Added by intrigeri 7 months ago

Update most of the reproducibility testing doc to support USB images (refs: #15999)

"most" because for now I'm leaving the IDF check as-is and I'll deal with
everything IDF-related separately later.

Revision 66497cc5 (diff)
Added by intrigeri 7 months ago

Rewrap a bit (refs: #15999)

Revision 0e38dd94 (diff)
Added by intrigeri 7 months ago

Integrate USB image in the release process (refs: #15999)

The IDF part remains to be done but everything else in this document
should be ready to be tested during the 3.12 release process.

Revision 63bf10ac (diff)
Added by intrigeri 7 months ago

Adjust release process for new build artifacts naming scheme (refs: #15999)

Since feature/15292-generate-usb-image was merged into master, stable and devel,
the build artifacts that are common to the ISO and USB images lost the ".iso"
part of their filename.

Revision a39ca5ee (diff)
Added by intrigeri 7 months ago

Generate two Torrents: one for the ISO and one for the USB image (refs: #15999)

Revision 8626335e (diff)
Added by intrigeri 7 months ago

Refactoring: use a variable instead of repeating myself (refs: #15999)

Revision dcf9d895 (diff)
Added by intrigeri 7 months ago

Update manual test suite to support USB images (refs: #15999)

Revision 607519fa (diff)
Added by intrigeri 7 months ago

Update incremental upgrades design doc to mention USB images (refs: #15999)

For now at least, the full upgrade target file will remain the ISO image.
If our Upgrader ever supports full upgrades, this might need to be adjusted.
Let's not spend more time on it now because that's unlikely to happen:

- Our plans for changing our upgrade process are rather leaning towards
dropping our own installer.
- Even if we do implement full upgrade support in Tails Upgrader,
chances are that it'll be easier to use an ISO than a USB image
as input.

Revision 609d0f77 (diff)
Added by intrigeri 7 months ago

Update most of the reproducibility testing doc to support USB images (refs: #15999)

"most" because for now I'm leaving the IDF check as-is and I'll deal with
everything IDF-related separately later.

Revision eb1a2e96 (diff)
Added by intrigeri 7 months ago

Rewrap a bit (refs: #15999)

Revision 5273b9cf (diff)
Added by intrigeri 7 months ago

Integrate USB image in the release process (refs: #15999)

The IDF part remains to be done but everything else in this document
should be ready to be tested during the 3.12 release process.

Revision dc65b233 (diff)
Added by intrigeri 7 months ago

Adjust release process for new build artifacts naming scheme (refs: #15999)

Since feature/15292-generate-usb-image was merged into master, stable and devel,
the build artifacts that are common to the ISO and USB images lost the ".iso"
part of their filename.

Revision 9d916c91 (diff)
Added by intrigeri 7 months ago

Generate two Torrents: one for the ISO and one for the USB image (refs: #15999)

Revision 5ac29859 (diff)
Added by intrigeri 7 months ago

Refactoring: use a variable instead of repeating myself (refs: #15999)

Revision 832ed9a4 (diff)
Added by intrigeri 7 months ago

Release process: stop generating IDF v1 (refs: #15999)

By the time we release a Tails version with this branch merged in, our
installation doc will require a new version of the Tails Verification
extension that only supports IDF v2.

Revision 15b14d12 (diff)
Added by intrigeri 7 months ago

Delete obsolete IDF v1 (refs: #15999)

Revision 99a2df72 (diff)
Added by intrigeri 7 months ago

Release process: automatically generate wiki/src/inc/*.html (refs: #15999)

Previously, some of these files were automatically generated (some in this
section plus some later on in the release process) but for some reason, we've
left updating the rest (manually) to the release manager. This mostly worked,
except when it didn't. Now that we need to maintain almost twice as many files
in there, so let's automate it, eventually.

Closes: #16163

Revision 7198319e (diff)
Added by intrigeri 7 months ago

Stop supporting the case when we release only an ISO image (refs: #15999)

The rest of this branch assumes we release both ISO and USB images
so let's simplify this piece of code.

Revision 7bdce620 (diff)
Added by intrigeri 5 months ago

Test suite doc: don't advertise non-existing --old-img flag (refs: #15999).

I wrote the initial draft before the corresponding code actually existed.
In the end we decided to guess the filename of the .img based on the filename of
the .iso and thus we did not add a --old-img flag.

Revision 2c17bb9d (diff)
Added by intrigeri 5 months ago

Add missing filename component (refs: #15999)

Revision 21d60735 (diff)
Added by intrigeri 5 months ago

Fix typo (refs: #15999)

I want a search'n'replace feature that also updates grammar accordingly.
M-x what?

Revision 57d3c93f (diff)
Added by intrigeri 5 months ago

Oxford comma-ify (refs: #15999)

Revision 6a79c74a
Added by intrigeri 5 months ago

Merge branch 'doc/15999-integrate-usb-image-in-the-release-process' into feature/15292-usb-image

The follow-up fixes after the review of this branch were pushed
but not reviewed yet. Regardless, we need this branch merged
so that 3.12~rc1 can be prepared.

refs: #15999

History

#1 Updated by intrigeri 9 months ago

  • Estimated time set to 3.00 h
  • Type of work changed from Wait to Contributors documentation

#2 Updated by intrigeri 9 months ago

  • Description updated (diff)

#3 Updated by intrigeri 9 months ago

  • Subject changed from Release Process for USB Images to Integrate USB image in the release process

#4 Updated by intrigeri 7 months ago

  • Description updated (diff)

#5 Updated by intrigeri 7 months ago

  • Description updated (diff)

#6 Updated by intrigeri 7 months ago

  • Description updated (diff)

#7 Updated by intrigeri 7 months ago

  • Description updated (diff)

#8 Updated by intrigeri 7 months ago

  • Status changed from Confirmed to In Progress

#9 Updated by intrigeri 7 months ago

  • Description updated (diff)
  • Feature Branch set to doc/15999-integrate-usb-image-in-the-release-process

#10 Updated by intrigeri 7 months ago

  • Blocked by Bug #16163: Clarify updated release process requirements for USB image added

#11 Updated by intrigeri 7 months ago

  • Description updated (diff)

#12 Updated by intrigeri 7 months ago

  • Description updated (diff)

#13 Updated by intrigeri 7 months ago

  • Description updated (diff)

#14 Updated by intrigeri 7 months ago

  • Blocked by Bug #16112: Confusing signature date on website doc added

#15 Updated by intrigeri 7 months ago

  • Description updated (diff)

#16 Updated by intrigeri 7 months ago

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check set to Ready for QA

OK, I've done all I could think of or spot via git grep'ing the world. This branch should not be merged anywhere yet (the other bits of the USB image project are not ready and we have no integration branch yet, where we would merge doc + test suite + this kind of updates). So please first focus on reviewing my work on the 2 subtasks, that needs to be merged more urgently. Then you can review the branch for this very ticket and once happy, reassign to me: I'll coordinate merging with #16112 and with the rest of the USB image.

If you have nitpicks or improvements in mind, feel free to commit and push them directly to the topic branch, to save both of us some time :)

Thanks in advance!

#17 Updated by intrigeri 7 months ago

  • Description updated (diff)

#18 Updated by intrigeri 7 months ago

intrigeri wrote:

once happy, reassign to me: I'll coordinate merging with #16112 and with the rest of the USB image.

The branch for #16112 was merged so I'll merge this here tomorrow and will resolve merge conflicts, but that should not prevent you from reviewing the branch if you're faster than me.

#19 Updated by intrigeri 7 months ago

(And in any case, this can wait a little bit longer than #16164, no big hurry here, feel free to postpone to post-3.11 if needed.)

#20 Updated by intrigeri 7 months ago

intrigeri wrote:

intrigeri wrote:

once happy, reassign to me: I'll coordinate merging with #16112 and with the rest of the USB image.

Still the case.

The branch for #16112 was merged so I'll merge this here tomorrow and will resolve merge conflicts

Done!

#21 Updated by intrigeri 7 months ago

  • Target version changed from Tails_3.11 to Tails_3.12

Can wait until post-3.11 :)

#22 Updated by u 6 months ago

Hi Cyril, could you try to review this for January 21st latest delay please?
We need this to be done on February 1st and I would like to leave some time for intrigeri in case something needs fixing after the review. Thanks!

#23 Updated by CyrilBrulebois 5 months ago

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

In dcf9d89534648d0e876ad670fbf276e1aab72122 wrong flag:

--old-img doesn't exist; setting --old-iso is sufficient (speaking of that, the sub call used to get the IMG filename out of the ISO one could probably check for '$'…)

In 609d0f7778c0c3c349e87ea2f7454f2ee37dee6d extra 'a':

+Build and compare a Tails ISO and USB images

and this is strange:

-        mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.iso*" \
+        mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.*" \

Are we sure we're capturing everything we want here? See dc65b233bb47a2e502c06c0bc73837c7da6c6efd for stuff I had in mind while writing this very question.

In 5273b9cf0545c6148275f3d547bab11d6e92c8e5 I'd use the Oxford comma for Publish the ISO, IMG and IUKs over HTTP and ISO, USB image and IUK (and add an 's' to the latter); but colours and tastes…

In 9d916c91c76c6608e40f6e9c368756b7e948443f I think the transmission-show call is missing the .${type:?} addition?

Also, having distinct stable_amd64_iso_gpg_signature_output.html and stable_amd64_img_gpg_signature_output.html files is likely to be an issue with the current state of the website, given the latter only includes:

   <p class="pre">[[!inline pages="inc/stable_amd64_gpg_signature_output" raw="yes" sort="age"]]</p>

but that's maybe tracked in another ticket? (I see plenty of other occurrences about the ISO only, be it in install or upgrade pages.)

Switching back to “Dev Needed” for the --old-img and the transmission-show points at least.

#24 Updated by intrigeri 5 months ago

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check changed from Dev Needed to Ready for QA

Thanks for the review!

--old-img doesn't exist; setting --old-iso is sufficient

Good catch! Failed attempt at documentation driven development :) Fixed.

In 609d0f7778c0c3c349e87ea2f7454f2ee37dee6d extra 'a':

Fixed.

and this is strange:

> -        mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.iso*" \
> +        mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.*" \
> 

Are we sure we're capturing everything we want here? See dc65b233bb47a2e502c06c0bc73837c7da6c6efd for stuff I had in mind while writing this very question.

Yes, I think we're capturing what we need here (and probably more e.g. files that previously had no .iso component in their name). This doc has not been tested yet, bet it in its previous form or in the new one. I'm supposed to be the TR for 3.11 but I'm failing (#12629) so it's likely that I'll play the TR for 3.12 and will thus test this updated doc. If I'm guessing wrong here, I'll notice and fix it. Deal?

In 5273b9cf0545c6148275f3d547bab11d6e92c8e5 I'd use the Oxford comma

Done.

In 9d916c91c76c6608e40f6e9c368756b7e948443f I think the transmission-show call is missing the .${type:?} addition?

Indeed, fixed.

Also, having distinct stable_amd64_iso_gpg_signature_output.html and stable_amd64_img_gpg_signature_output.html files is likely to be an issue with the current state of the website, given the latter only includes:

>    <p class="pre">[[!inline pages="inc/stable_amd64_gpg_signature_output" raw="yes" sort="age"]]</p>
> 

This was fixed already on our integration branch (as part of the doc update work), which I hadn't merged into my topic branch for a while. After updating my topic branch:

$ git grep gpg_signature_output wiki/src/**/*.{mdwn,html}
wiki/src/contribute/release_process.mdwn:          "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_amd64_iso_gpg_signature_output.html" && \
wiki/src/contribute/release_process.mdwn:          "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_amd64_img_gpg_signature_output.html" 
wiki/src/contribute/release_process.mdwn:               inc/stable_amd64_img_gpg_signature_output.html
wiki/src/contribute/release_process.mdwn:               inc/stable_amd64_iso_gpg_signature_output.html
wiki/src/install/expert/usb.mdwn:   <p class="pre">[[!inline pages="inc/stable_amd64_img_gpg_signature_output" raw="yes" sort="age"]]</p>
wiki/src/install/inc/steps/download.inline.html:   <p class="windows linux mac upgrade-tails pre">[[!inline pages="inc/stable_amd64_img_gpg_signature_output" raw="yes" sort="age"]]</p>
wiki/src/install/inc/steps/download.inline.html:   <p class="dvd vm pre">[[!inline pages="inc/stable_amd64_iso_gpg_signature_output" raw="yes" sort="age"]]</p>

… looks good!

Note that I had to resolve a few conflicts during this merge operation.

(I see plenty of other occurrences about the ISO only, be it in install or upgrade pages.)

I think that was caused by the same reason (topic branch lagging behind the integration branch and thus not having the doc update merged in). Can you please check again and point me to specific issues that might be left?

#25 Updated by intrigeri 5 months ago

Merged into feature/15292-usb-image because this is already better than what we had there.

#26 Updated by u 5 months ago

(I see plenty of other occurrences about the ISO only, be it in install or upgrade pages.)

I think that was caused by the same reason (topic branch lagging behind the integration branch and thus not having the doc update merged in). Can you please check again and point me to specific issues that might be left?

This is the last remaining bit on this ticket it seems. I'm fine with resolving this after the release.

#27 Updated by u 5 months ago

  • Target version changed from Tails_3.12 to Tails_3.13

#28 Updated by u 4 months ago

@CyrilBrulebois can you please verify what's really left to do here? It seems not much and we might already close this ticket. Thanks!

#29 Updated by intrigeri 3 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (CyrilBrulebois)
  • QA Check deleted (Ready for QA)

@CyrilBrulebois, please reopen if you spot any USB image -related issue during the 3.13 release process.

Also available in: Atom PDF