Project

General

Profile

Feature #16059

Improve UX for scheduling builds for newly pushed branches on the CI

Added by segfault about 1 year ago. Updated 2 months ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
-
Start date:
10/16/2018
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Currently, builds have to be scheduled manually, once the jobs appear on Jenkins. The cronjob to update the jobs runs every 5 min, and the jobs update takes 1-2 min. The resulting worflow is:

1. Push branch with changes you want to test
2. Wait 2-7 minutes until the corresponding job appears on Jenkins
3. Manually schedule a build for the job on Jenkins


Related issues

Related to Tails - Feature #9741: New jobs in Jenkins should be built immediately Rejected 07/14/2015
Related to Tails - Bug #16959: Gather usability data about our current CI In Progress

History

#1 Updated by intrigeri about 1 year ago

  • Category set to Continuous Integration

#2 Updated by intrigeri about 1 year ago

  • Subject changed from Improve UX for scheduling builds on the CI to Improve UX for scheduling builds for newly pushed branches on the CI

#3 Updated by intrigeri 12 months ago

  • Related to Feature #9741: New jobs in Jenkins should be built immediately added

#4 Updated by intrigeri 12 months ago

Note that this idea surfaced in #9741 a while ago and was rejected back then. I think that was for wrong reasons: my arguments that lead bertagaz to close the ticket are both mistaken.

#5 Updated by bertagaz 12 months ago

I agree it's a bit of a suboptimal UX for developers. Now I've think a bit about it in the past, and the conclusions were mitigated.

First, it's not that easy to implement. That's the way Jenkins behaves natively. There might be plugins to workaround this, but I doubt it.

So to have that behaviour, we'd have to hack something in our jenkins job creation scripts. That would not be in the jenkin-job-builder part, I don't think there's a way to do that here. So that would be in the pythonlib code, where we parse the Tails Git to get the new and merged branches to generate the job list. At that time, we could probably make a HTTP request to the URL that trigger a build, except the job does not exist yet. Bummer. So that would be another piece of code, after the JJB run, that find somewhere the information about which jobs are new, in order to do the HTTP requests to trigger the builds. Maybe by running again the pythonlib code, which takes some times.

OTOH, in the past I've been aware of this, so when I pushed a new branch, I knew if I wanted to have at that point feedbacks from Jenkins I had to push a button in the web interface. Or before working, I forked the new branch, pushed it so that itsjob gets created in Jenkins, and then hacked on it and pushed the new commits, so they get tested right away.

So all in all, I've been wondering if this little UX problem was worth the development time and added code and infra complexity. Once a developer knows that, she also knows how to workaround it. And now that more Tails people have access to the Jenkins GUI, it leverages a bit more the need for that code. Maybe some documentation would be enough?

I also think that your second argument in #9741#note-2 is reasonable.

#6 Updated by intrigeri 12 months ago

Thanks for the input!

So that would be another piece of code, after the JJB run, that find somewhere the information about which jobs are new, in order to do the HTTP requests to trigger the builds.

Yep: a Git post-receive hook in jenkins-jobs.git would have access to all the info it needs to tell which jobs were created and send the corresponding HTTP requests. This may require that we move our post-update code to post-receive too so we have guaranteed ordering (apparently, whether post-receive executes before or after post-update is undefined behaviour, but I did not research this much).

Or before working, I forked the new branch, pushed it so that itsjob gets created in Jenkins, and then hacked on it and pushed the new commits, so they get tested right away.

Interestingly, this was my exact argument last time we had this conversation, and you nicely explained why this can't possibly work: if the new branch has no commit on top of its base branch, our automation won't create a job for it yet.

So all in all, I've been wondering if this little UX problem was worth the development time and added code and infra complexity.

On the one hand, the solution described above seems pretty cheap in terms of all these cost metrics. Also, I suspect the UX cost of every such little papercut adds up in a non-linear fashion because one resents a UX bug differently, depending on whether it's isolated or the third one in a row they have to ask me on XMPP why the system won't do its job as expected. So I'm not sure if it makes sense to do UX cost/benefit analysis in a non-holistic way.

OTOH, now is probably not the best time to invest a lot into our current job generation setup:

  • If we stick to Jenkins: with Jenkins 2+ it seems there are good options to fully replace the way we generate jobs, in a way that Jenkins understands and can react sensibly to (as opposed to "hi Jenkins! things happened, we won't tell you what, but here's your new set of jobs, deal with it" that we're doing now and which causes this sort of problems).
  • We want to move to GitLab at least for merge requests and quite possibly for issues tracking. At this point it's not clear to me whether the cost of migrating the CI too will be greater than the benefits it would bring.

So well, for now, IMO let's keep this open in case one of us feels like doing some structured procrastination on a rainy day, but unassigned and not urgent.

Maybe some documentation would be enough?

This would be great!

#7 Updated by intrigeri 3 months ago

  • Status changed from Confirmed to Needs Validation
  • Assignee set to segfault
  • Feature Branch set to pythonlib:faster-jenkins-jobs-generation

Here's a branch that should improve things a bit: it should lower by a few minutes the delay until the jobs for a newly pushed branch appear on Jenkins.

I propose we do this:

  1. You review & merge this branch, then reassign this ticket to me.
  2. I'll deal with updating the submodule in tails.git.
  3. After confirming this makes the cronjob run in much less time on Jenkins, I'll make that cronjob run more often, e.g. every 1 or 2 minutes ⇒ the total waiting time before jobs are created should then be down to 0-3 minutes. While I'm at it, I'll probably add a lock on the cronjob: I just noticed it has none, which is risky for something that runs every 5 minutes and generally takes more than 3 minutes.

Then there's still the "automatically trigger a run for newly created build jobs" problem to solve, but that'll be for another Sunday. And anyway, we'll upgrade to Jenkins 2 later this month so I'll first want to check if there's a better, cheap solution in newer Jenkins.

#8 Updated by intrigeri 3 months ago

  • Related to Bug #16959: Gather usability data about our current CI added

#9 Updated by segfault 3 months ago

  • Status changed from Needs Validation to In Progress
  • Assignee changed from segfault to intrigeri

I think there is a typo in the commit message, shouldn't one of the two lines which start with "With active_days=10" have a different active_days value? Beside that it LGTM.

#10 Updated by intrigeri 2 months ago

I think there is a typo in the commit message, shouldn't one of the two lines which start with "With active_days=10" have a different active_days value?

Indeed, the 2nd one should read "With active_days=50". Fixed locally. Good catch!

Beside that it LGTM.

OK, I'll proceed with the next steps.

#11 Updated by intrigeri 2 months ago

  • Status changed from In Progress to Confirmed
  • Assignee deleted (intrigeri)
  • Feature Branch deleted (pythonlib:faster-jenkins-jobs-generation)

intrigeri wrote:

I'll deal with updating the submodule in tails.git.

Done.

After confirming this makes the cronjob run in much less time on Jenkins

It now takes 35 s instead of several minutes. Nothing unexpected happened in jenkins-jobs.git. I've pushed a new branch and the jobs were created; I've deleted it and the jobs were deleted; so it seems to work :)

I'll make that cronjob run more often, e.g. every 1 or 2 minutes
⇒ the total waiting time before jobs are created should then be down to 0-3 minutes.

Done (every 2 minutes).

While I'm at it, I'll probably add a lock on the cronjob

Done.

Next (and last) step here:

Then there's still the "automatically trigger a run for newly created build jobs" problem to solve, but that'll be for another Sunday. And anyway, we'll upgrade to Jenkins 2 later this month so I'll first want to check if there's a better, cheap solution in newer Jenkins.

Also available in: Atom PDF