Project

General

Profile

Bug #10334

Fragile automated builds on lizard

Added by intrigeri over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
Start date:
10/03/2015
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

We've seen many build failures recently. Almost all of them happen on isobuilder2, I had a look and it's not surprising things fail:

$ df
df: `/tmp/tails-build.4ZSfX1Cx': Stale NFS file handle
df: `/tmp/tails-build.9zHHWL68': Stale NFS file handle
[...]
tmpfs                      tmpfs     8.8G  5.6G  3.3G  64% /tmp/tmpfs.MtxhVedc
tmpfs                      tmpfs     8.8G  782M  8.1G   9% /tmp/tmpfs.NQz5b14m

$ free
             total       used       free     shared    buffers     cached
Mem:          8992       6962       2029          0         10       6487
-/+ buffers/cache:        464       8527
Swap:            0          0          0

So there are two problems there:

  • I don't know what the NFS error come from, I see a NFS line in fstab, I thought we were not supposed to have any such thing anymore, but perhaps I'm confused.
  • The build wrapper doesn't always clean up properly.

I've rebooted isobuilder2 so the avalanche of notifications should stop.


Related issues

Related to Tails - Feature #6090: Automated builds Resolved 07/26/2013 02/28/2015
Related to Tails - Bug #10772: Next ISO builds fail on Jenkins when a previous job was aborted uncleanly Resolved 12/17/2015

History

#1 Updated by intrigeri over 3 years ago

#2 Updated by intrigeri over 3 years ago

  • Target version set to Tails_1.7

This happened for the 2nd time (that I noticed) in 2 weeks: all builds on isobuilder2 were failing since a few hours today, apparently for the same reason as last time => rebooted that slave.

As you can guess, since I raised all hell about it 2 weeks ago, my main concern here is the rate of false positive "build failed" notifications sent to developers (last time it took days before the bug was identified and I rebooted the slave), so setting a target version not far away. Formally speaking that's a follow-up to #6090, so it's on your plate.

#4 Updated by bertagaz over 3 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 30

I've resintalled the faulty isobuilder2 tonight from scratch. Seems easier in the end than tracking what's the root of this bug. There's a first build going on, we'll see in the coming ones if this strange bug reappear.

#5 Updated by intrigeri over 3 years ago

I've resintalled the faulty isobuilder2 tonight from scratch.

Thanks! And yay for disposable Jenkins slaves.

#6 Updated by bertagaz over 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 30 to 70
  • QA Check set to Ready for QA

It's been 3 days that it has been re-installed, and so far the stale NFS file handle bug does not seem to be the cause of the failing builds "on this isobuilder:https://jenkins.tails.boum.org/computer/isobuilder2/builds

So I think this ticket can be marked as resolved.

#7 Updated by intrigeri over 3 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 70 to 100
  • QA Check changed from Ready for QA to Pass

It's been 3 days that it has been re-installed, and so far the stale NFS file handle
bug does not seem to be the cause of the failing builds "on this
isobuilder:https://jenkins.tails.boum.org/computer/isobuilder2/builds

Great!

#8 Updated by intrigeri over 3 years ago

  • Assignee deleted (intrigeri)

#10 Updated by bertagaz over 3 years ago

  • Related to Bug #10772: Next ISO builds fail on Jenkins when a previous job was aborted uncleanly added

Also available in: Atom PDF