Project

General

Profile

Bug #16226

Most branches FTBFS on Jenkins since the 3.11 release

Added by intrigeri 9 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Build system
Target version:
Start date:
12/15/2018
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

It seems that the 3.11 release process was left in a state that breaks the build of most branches: the only branch that currently builds on Jenkins is the devel one. If you need help to get things back into working order, let me know :)


Related issues

Related to Tails - Bug #16060: Improve ASP code: configuration window In Progress 10/16/2018
Blocks Tails - Bug #16110: Button to remove package in ASP GUI has a wrong label Resolved 11/08/2018

Associated revisions

Revision a48e2487 (diff)
Added by intrigeri 9 months ago

Clarify when freeze exceptions must be lifted (refs: #16226).

Revision 2a413478 (diff)
Added by intrigeri 9 months ago

Release process: clarify where never released versions must be removed (refs: #16226)

Revision 535a2d38 (diff)
Added by CyrilBrulebois 9 months ago

Revert "Revert "Freeze exception: linux 4.18.20-2"" (refs: #16226).

This reverts commit 049701430a24bc3ebe430837acc0b00f9f2ed8ee.

I failed to capture the post-release instructions; let's keep the freeze
exceptions on this stable/frozen branch. Sorry for the confusion and the
revert-the-revert dance…

Revision 53094b0e (diff)
Added by CyrilBrulebois 9 months ago

Revert "Revert "Revert "Freeze exception: linux 4.18.20-2"" (refs: #16226)."

This reverts commit 535a2d38d9e00f12b0597563ecaa61b7fc9c50ce.

On this devel (= non-frozen) branch, we would like to get rid of the
freeze exception that was granted to linux 4.18.20-2. So besides the
revert-the-revert dance explained in 535a2d38d9, this commit simply
lifts the freeze exception that was granted for the 3.11 release.

History

#1 Updated by CyrilBrulebois 9 months ago

If I didn't screw up too much, apt repositories and git branches should be fine by now:

  • devel was already fine, as noted.
  • stable no longer fails immediately (but the build is still running).
  • with this updated stable branch, I suppose other branches getting merged will get better over time.

Notes about the Prepare for the next development cycle section:

  • given #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.
  • for the same reason, I decided to create a 3.11.1 dummy changelog entry in stable instead of the customary 3.13 one.

Keeping this ticket assigned to me until Jenkins has had a chance to catch up, and until there's agreement we should delete 3.10.1 bits entirely.

Another thing that needs doing is merging devel into feature/buster, but that was filed already (#16212).

#2 Updated by CyrilBrulebois 9 months ago

  • Status changed from Confirmed to In Progress

#3 Updated by CyrilBrulebois 9 months ago

Apparently I failed at understanding how to revert freeze exceptions (release_process.mdwn and friends have some instructions, but the actual config/chroot_apt/preferences had different instructions for the kernel)…

+ apt-get install --yes build-essential libelf-dev linux-headers-4.18.0-3-amd64                                      
E: Unable to correct problems, you have held broken packages.                                                        
Reading package lists...                                                                                             
Building dependency tree...                                                                                          
Reading state information...                                                                                         
Some packages could not be installed. This may mean that you have                                                    
requested an impossible situation or if you are using the unstable                                                   
distribution that some required packages have not yet been created                                                   
or been moved out of Incoming.                                                                                       
The following information may help to resolve the situation:                                                         

The following packages have unmet dependencies:                                                                      
 linux-headers-4.18.0-3-amd64 : Depends: linux-kbuild-4.18 (>= 4.18.20-2) but 4.18.10-2 is to be installed           
E: Unable to correct problems, you have held broken packages.                                                        
E: config/chroot_local-hooks/12-kernel-modules-build-environment failed (exit non-zero). You should check for errors.

#4 Updated by alant 9 months ago

  • Blocks Bug #16034: ASP: Fix race condition when writing to packages file added

#5 Updated by alant 9 months ago

  • Blocks Bug #16110: Button to remove package in ASP GUI has a wrong label added

#6 Updated by alant 9 months ago

  • Related to Bug #16060: Improve ASP code: configuration window added

#7 Updated by intrigeri 9 months ago

  • Assignee changed from CyrilBrulebois to intrigeri
  • QA Check set to Info Needed

I kinda remember this part of our doc is, let's say, "suboptimal" :/ I'll check that doc and will either fix it or provide here the info you're missing.

#8 Updated by intrigeri 9 months ago

  • given #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.

Now that a workaround was documented, please go ahead.

  • for the same reason, I decided to create a 3.11.1 dummy changelog entry in stable instead of the customary 3.13 one.

Great, that's actually the right thing to do. I'll check our release process doc and will clarify/fix the relevant bits. That's a PITA and I'm sorry about it, but I'm confident we're currently improving that doc much faster than we're adding unclear/buggy bits to it, thanks to your input!

#9 Updated by intrigeri 9 months ago

I kinda remember this part of our doc is, let's say, "suboptimal" :/ I'll check that doc and will either fix it or provide here the info you're missing.

I've tried to fix the doc about freeze exceptions; feedback welcome! So I think we should now:

  1. revert 049701430a24bc3ebe430837acc0b00f9f2ed8ee on stable: as I've tried to clarify in a48e24875bc609255b2c784ba3b5d5606074d7e9, we shall not lift freeze exceptions on branches that are still frozen
  2. merge stable into master
  3. merge stable into devel
  4. on devel, revert the revert of 049701430a24bc3ebe430837acc0b00f9f2ed8ee

#10 Updated by intrigeri 9 months ago

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check changed from Info Needed to Dev Needed
  • for the same reason, I decided to create a 3.11.1 dummy changelog entry in stable instead of the customary 3.13 one.

Great, that's actually the right thing to do. I'll check our release process doc and will clarify/fix the relevant bits.

May I ask what lead you to think it was customary to add a 3.13 changelog entry on stable?

I see we have:

* increment the version number in stable's `debian/changelog` to match
the next bugfix release, so that
next builds from the `stable` branch do not use the APT suite meant
for the last release:

cd "${RELEASE_CHECKOUT}" && \
git checkout stable && \
dch --newversion "${NEXT_PLANNED_BUGFIX_VERSION:?}" \
"Dummy entry for next release." && \
git commit debian/changelog \
-m "Add dummy changelog entry for ${NEXT_PLANNED_BUGFIX_VERSION:?}." 

And

* `NEXT_PLANNED_BUGFIX_VERSION`: set to the version number of the next
 *bugfix* Tails release; if the next release is a bugfix release, use
that one; otherwise, use `${VERSION}.1`

So given the next release (3.12) is a major one, NEXT_PLANNED_BUGFIX_VERSION should be set to 3.11.1, no?

Happy to fix the doc once I understand where the confusion comes from :)

#11 Updated by CyrilBrulebois 9 months ago

Failure to read the documentation, I suppose. :/

Train of thoughts was probably: Planned = look at calendar, skip 3.12 as that's major, and encode 3.13. Wasn't caught by peer review.

#12 Updated by intrigeri 9 months ago

Train of thoughts was probably: Planned = look at calendar, skip 3.12 as that's major, and encode 3.13. Wasn't caught by peer review.

OK, thanks! Well, this happens and the impact is basically non-existent (you've self-corrected it where it mattered later) so let's say the doc is good enough.

#13 Updated by CyrilBrulebois 9 months ago

intrigeri wrote:

  • given #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.

Now that a workaround was documented, please go ahead.

Apparently we're getting reports it's not working, so I won't be deleting it right away?

#14 Updated by CyrilBrulebois 9 months ago

intrigeri wrote:

I've tried to fix the doc about freeze exceptions; feedback welcome! So I think we should now:

  1. revert 049701430a24bc3ebe430837acc0b00f9f2ed8ee on stable: as I've tried to clarify in a48e24875bc609255b2c784ba3b5d5606074d7e9, we shall not lift freeze exceptions on branches that are still frozen
  2. merge stable into master
  3. merge stable into devel
  4. on devel, revert the revert of 049701430a24bc3ebe430837acc0b00f9f2ed8ee

Thanks for the commit, I probably wouldn't have made that mistake with this devel vs. stable clarification.

Should I go ahead with the steps you proposed, build things locally, and push branches if everything seems to build fine?

(As a side note, and as you probably noticed already, linux 4.19.9-1 landed in unstable yesterday.)

#15 Updated by intrigeri 9 months ago

  • given #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.

Now that a workaround was documented, please go ahead.

Apparently we're getting reports it's not working, so I won't be deleting it right away?

It's a tough one. In general we never leave obsolete images around because we don't want people to use a Tor Browser with known security vulns, and finding + verifying the old image is just too hard for the vast majority of our users anyway. Most people would be safer using an up-to-date Tor Browser outside of Tails than using Tails 3.10.1, but there are obviously downsides. I would let our help desk decide: if they are actively suggesting users to downgrade their Tails to 3.10.1 (I would be surprised), then keep it around; otherwise, delete.

#16 Updated by intrigeri 9 months ago

Should I go ahead with the steps you proposed, build things locally, and push branches if everything seems to build fine?

Just go ahead and push, no need to build locally first.

(As a side note, and as you probably noticed already, linux 4.19.9-1 landed in unstable yesterday.)

Yep.

#17 Updated by CyrilBrulebois 9 months ago

Pushed all three branches; I had triggered builds past night anyway, and from checking build manifests, the stable one looks good (only a bump in serial for debian-security), as well as the devel one (a bunch of binaries from a single source package got upgraded compared to the last devel build in jenkins, because of an update in backports).

#18 Updated by intrigeri 9 months ago

Pushed all three branches; I had triggered builds past night anyway, and from checking build manifests, the stable one looks good (only a bump in serial for debian-security), as well as the devel one (a bunch of binaries from a single source package got upgraded compared to the last devel build in jenkins, because of an update in backports).

woohoo! \o/

#19 Updated by CyrilBrulebois 9 months ago

Jenkins looks way greener (or blue-er) now, sorry it took so long; just didn't want to risk having to revert the revert of the revert… etc. by rushing anything. The commit history is messy enough already.

#20 Updated by alant 9 months ago

CyrilBrulebois wrote:

Jenkins looks way greener (or blue-er) now,

Thanks a lot!

sorry it took so long; just didn't want to risk having to revert the revert of the revert… etc. by rushing anything. The commit history is messy enough already.

Don't worry.

#21 Updated by alant 9 months ago

  • Blocks deleted (Bug #16034: ASP: Fix race condition when writing to packages file)

#22 Updated by intrigeri 8 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (CyrilBrulebois)
  • QA Check deleted (Dev Needed)

intrigeri wrote:

  • given #16224 I decided not to delete 3.10.1 from rsync.lizard and from bittorrent.lizard at the moment.

Now that a workaround was documented, please go ahead.

Apparently we're getting reports it's not working, so I won't be deleting it right away?

It's a tough one. In general we never leave obsolete images around because we don't want people to use a Tor Browser with known security vulns, and finding + verifying the old image is just too hard for the vast majority of our users anyway. Most people would be safer using an up-to-date Tor Browser outside of Tails than using Tails 3.10.1, but there are obviously downsides. I would let our help desk decide: if they are actively suggesting users to downgrade their Tails to 3.10.1 (I would be surprised), then keep it around; otherwise, delete.

Help Desk has not suggested that but anyway, it's too late to delete that ISO: it'll go away on Tuesday at the same time as 3.11.

The main problem this ticket was about is resolved.

Also available in: Atom PDF