Project

General

Profile

Feature #9350

Feature #6196: Build all active branches

Jenkins workspaces for disabled jobs are not removed from ISO builders

Added by intrigeri over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
Start date:
05/06/2015
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

See /var/lib/jenkins/workspace on isobuilder{1,2}.lizard. This will grow out of control once we build all active branches. There's surely a Jenkins config pref or plugin to handle that. And worst case, we can simply cron the removal of all workspaces that are not being used by a currently running build (the workspace directory is kept open for the entire build duration, as soon as build-tails is started, apparently -- but perhaps there'll be race conditions wrt. what happens before or after, so it would probably be more robust to implement this in another way).


Related issues

Related to Tails - Bug #8912: Nightly built artifacts for disabled jobs are not removed Resolved 02/17/2015
Blocks Tails - Feature #8658: Deploy the "build all active branches" system Resolved 01/09/2015

History

#1 Updated by intrigeri over 4 years ago

  • Related to Bug #8912: Nightly built artifacts for disabled jobs are not removed added

#2 Updated by intrigeri over 4 years ago

  • Description updated (diff)

#3 Updated by intrigeri over 4 years ago

  • Blocks Feature #8658: Deploy the "build all active branches" system added

#5 Updated by bertagaz over 4 years ago

  • Assignee changed from bertagaz to intrigeri

When we last discussed about this topic, we thought about a script that would publish somewhere the current list of active jobs, list that would be retrieved by another script running on the builders to compare with the current list of job workspaces it has and delete those who don't have a corresponding job.

While that's still an option, there might be an easier way to remove this obsolete workspaces, more integrated into Jenkins: we have installed the workspace-cleanup plugin, but didn't configure it. It enables us to setup a post-build step for each job that delete the workspace after the build. That way, workspaces would be deleted every time a job is finished, and there won't be one left when the job is deleted. We already delete the workspaces at the beginning of the build to build from a clean tree, so that won't change much I think.

If you agree, I'll test this implementation first.

#6 Updated by bertagaz over 4 years ago

Correction: we do use the workspace-cleanup, but we did put it in the wrappers section of the job definition, where it should be in the publishers one. I'll fix that ASAP, and it should fix this bug hopefully.

#7 Updated by intrigeri over 4 years ago

If you agree, I'll test this implementation first.

This sounds like an excellent idea to me.

#8 Updated by intrigeri over 4 years ago

  • Assignee changed from intrigeri to bertagaz

#9 Updated by bertagaz over 4 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Ready for QA

I've fixed the workspace-cleanup plugin configuration in our jenkins-jobs repo. Pushed live and the workspaces are getting deleted after the builds ended as they should. Jobs that has run since then don't have them left.

I've also fixed our generate_build_tails_iso_jobs script in our tails puppet module (commit 7265dba).

A way to test it is: delete all the workspaces on a builder and start a job on it. See if it remains after the job is completed.
Another one: find a job that has a workspace left, run it, check the workspace has been deleted.

This should fix this bug. If you agree, I'll delete all the workspaces on the isobuilders.

#10 Updated by intrigeri over 4 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

I've fixed the workspace-cleanup plugin configuration in our jenkins-jobs repo. Pushed live and the workspaces are getting deleted after the builds ended as they should. Jobs that has run since then don't have them left.

Excellent!

This should fix this bug. If you agree, I'll delete all the workspaces on the isobuilders.

I've confirmed that both a failed job and a successful one have their workspace deleted post-build => please go ahead and then mark this ticket as resolved :)

(Side note: apparently non-ISO-build jobs are not affected, but these have a way lower turnaround so IMO it's fine this way.)

#11 Updated by bertagaz over 4 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • % Done changed from 0 to 100
  • QA Check deleted (Dev Needed)

intrigeri wrote:

I've confirmed that both a failed job and a successful one have their workspace deleted post-build => please go ahead and then mark this ticket as resolved :)

(Side note: apparently non-ISO-build jobs are not affected, but these have a way lower turnaround so IMO it's fine this way.)

They are, but they have to be ran on the right builder that have a previous job workspace left to delete it for real. I've deleted all the remaining workspaces, and even the non iso building job don't let their workspace after completion.

#12 Updated by intrigeri over 4 years ago

:)

Also available in: Atom PDF