Project

General

Profile

Bug #15647

PuppetDB is broken since 2018-06-10

Added by intrigeri 12 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Infrastructure
Target version:
-
Start date:
06/11/2018
Due date:
% Done:

100%

QA Check:
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

Tons of errors in puppet-git:/var/log/puppetdb which fills the root FS there which in turns breaks the Git service which in turn breaks ISO builds.


Related issues

Related to Tails - Feature #11834: Migrate our infrastructure to Puppet 4 Resolved 09/24/2016
Related to Tails - Bug #15648: puppet-git needs more diskspace Resolved 06/11/2018
Related to Tails - Bug #16083: APT snapshot used for libjetty9 is unreliable Resolved 10/31/2018
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017

History

#1 Updated by intrigeri 12 months ago

Actually I think that PuppetDB was broken because there was not enough disk space to create its shared memory file and then it logged noisily about it which made the problem worse. Deleted the logs that were about this recursive problem, restarted PuppetDB, things are a bit better now but requests to PuppetDB still fail. Investigating.

Regardless, the root FS on puppet-git.lizard is too tight. I'll file a dedicated ticket about it for groente or will recycle this one once I'm done.

#2 Updated by intrigeri 12 months ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

intrigeri wrote:

requests to PuppetDB still fail. Investigating.

OK, so apparently one of the recent upgrades of Java packages we pin from sid broke stuff. Not a big surprise, it had to happen at some point. I've downgraded them all manually (added an APT source from snapshot.d.o and apt install libmail-java=1.6.1-2 libxz-java=1.8-1 libxalan2-java=2.7.2-1 libpostgresql-jdbc-java=9.4.1212-1 libxerces2-java=2.11.0-8 libderby-java=10.14.1.0-1 libjetty9-extra-java=9.2.23-1 libjetty9-java=9.2.23-1 libecj-java=3.11.1-2 libyaml-snake-java=1.20-1 libjoda-time-java=2.9.9-1 libactivemq-java=5.15.3-2) and things are back in order. I'll now try to upgrade them one after the other in order to identify which one(s) we should freeze to older versions. And if/when that does not work anymore I think we'll have to run Buster on this system.

#3 Updated by intrigeri 12 months ago

Found it! Things start breaking when we upgrade libjetty9-java and libjetty9-extra-java from 9.2.23-1 to 9.2.24-1. I'll freeze them to the older version.

#4 Updated by intrigeri 12 months ago

  • Assignee changed from intrigeri to groente
  • % Done changed from 10 to 50
  • QA Check set to Ready for QA

intrigeri wrote:

Found it! Things start breaking when we upgrade libjetty9-java and libjetty9-extra-java from 9.2.23-1 to 9.2.24-1. I'll freeze them to the older version.

Done: https://git-tails.immerda.ch/puppet-tails/commit/manifests/puppetdb.pp?id=96c3d4ba15a9710e6350839307e551ff1a709da3

Also, please give yourself a ticket about growing the root FS on puppet-git.lizard.

#5 Updated by intrigeri 12 months ago

  • Related to Feature #11834: Migrate our infrastructure to Puppet 4 added

#6 Updated by intrigeri 12 months ago

  • Blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#7 Updated by groente 12 months ago

  • Related to Bug #15648: puppet-git needs more diskspace added

#8 Updated by groente 12 months ago

  • Assignee changed from groente to intrigeri
  • QA Check changed from Ready for QA to Pass

looks good, go for it!

#9 Updated by intrigeri 12 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100

#10 Updated by groente 7 months ago

  • Status changed from Resolved to In Progress
  • Assignee set to intrigeri
  • Priority changed from High to Elevated
  • Target version changed from Tails_3.8 to Tails_3.12
  • QA Check changed from Pass to Info Needed

the freeze is starting to become problematic:

E: The repository 'http://snapshot.debian.org/archive/debian/20180505T160451Z sid Release' does no longer have a Release file.

any idea whether the libjetty breakage is fixed by now?

#11 Updated by intrigeri 7 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • Target version deleted (Tails_3.12)
  • QA Check deleted (Info Needed)

Thanks for the heads up. Filed #16083 because repurposing a ticket that was closed ages ago has a number of problems.

#12 Updated by intrigeri 7 months ago

  • Related to Bug #16083: APT snapshot used for libjetty9 is unreliable added

Also available in: Atom PDF