Fragile automated builds on lizard
We've seen many build failures recently. Almost all of them happen on isobuilder2, I had a look and it's not surprising things fail:
$ df df: `/tmp/tails-build.4ZSfX1Cx': Stale NFS file handle df: `/tmp/tails-build.9zHHWL68': Stale NFS file handle [...] tmpfs tmpfs 8.8G 5.6G 3.3G 64% /tmp/tmpfs.MtxhVedc tmpfs tmpfs 8.8G 782M 8.1G 9% /tmp/tmpfs.NQz5b14m $ free total used free shared buffers cached Mem: 8992 6962 2029 0 10 6487 -/+ buffers/cache: 464 8527 Swap: 0 0 0
So there are two problems there:
- I don't know what the NFS error come from, I see a NFS line in fstab, I thought we were not supposed to have any such thing anymore, but perhaps I'm confused.
- The build wrapper doesn't always clean up properly.
I've rebooted isobuilder2 so the avalanche of notifications should stop.
#2 Updated by intrigeri over 3 years ago
- Target version set to Tails_1.7
This happened for the 2nd time (that I noticed) in 2 weeks: all builds on isobuilder2 were failing since a few hours today, apparently for the same reason as last time => rebooted that slave.
As you can guess, since I raised all hell about it 2 weeks ago, my main concern here is the rate of false positive "build failed" notifications sent to developers (last time it took days before the bug was identified and I rebooted the slave), so setting a target version not far away. Formally speaking that's a follow-up to #6090, so it's on your plate.
#4 Updated by bertagaz over 3 years ago
- Status changed from Confirmed to In Progress
- % Done changed from 0 to 30
I've resintalled the faulty isobuilder2 tonight from scratch. Seems easier in the end than tracking what's the root of this bug. There's a first build going on, we'll see in the coming ones if this strange bug reappear.
#6 Updated by bertagaz over 3 years ago
- Assignee changed from bertagaz to intrigeri
- % Done changed from 30 to 70
- QA Check set to Ready for QA
It's been 3 days that it has been re-installed, and so far the stale NFS file handle bug does not seem to be the cause of the failing builds "on this isobuilder:https://jenkins.tails.boum.org/computer/isobuilder2/builds
So I think this ticket can be marked as resolved.
#7 Updated by intrigeri over 3 years ago
- Status changed from In Progress to Resolved
- % Done changed from 70 to 100
- QA Check changed from Ready for QA to Pass
It's been 3 days that it has been re-installed, and so far the stale NFS file handle
bug does not seem to be the cause of the failing builds "on this