PuppetDB is broken since 2018-06-10
Tons of errors in
puppet-git:/var/log/puppetdb which fills the root FS there which in turns breaks the Git service which in turn breaks ISO builds.
Actually I think that PuppetDB was broken because there was not enough disk space to create its shared memory file and then it logged noisily about it which made the problem worse. Deleted the logs that were about this recursive problem, restarted PuppetDB, things are a bit better now but requests to PuppetDB still fail. Investigating.
Regardless, the root FS on
puppet-git.lizard is too tight. I'll file a dedicated ticket about it for groente or will recycle this one once I'm done.
- Status changed from Confirmed to In Progress
- % Done changed from 0 to 10
requests to PuppetDB still fail. Investigating.
OK, so apparently one of the recent upgrades of Java packages we pin from sid broke stuff. Not a big surprise, it had to happen at some point. I've downgraded them all manually (added an APT source from snapshot.d.o and
apt install libmail-java=1.6.1-2 libxz-java=1.8-1 libxalan2-java=2.7.2-1 libpostgresql-jdbc-java=9.4.1212-1 libxerces2-java=2.11.0-8 libderby-java=10.14.1.0-1 libjetty9-extra-java=9.2.23-1 libjetty9-java=9.2.23-1 libecj-java=3.11.1-2 libyaml-snake-java=1.20-1 libjoda-time-java=2.9.9-1 libactivemq-java=5.15.3-2) and things are back in order. I'll now try to upgrade them one after the other in order to identify which one(s) we should freeze to older versions. And if/when that does not work anymore I think we'll have to run Buster on this system.
- Assignee changed from intrigeri to groente
- % Done changed from 10 to 50
- QA Check set to Ready for QA
Found it! Things start breaking when we upgrade
libjetty9-extra-javafrom 9.2.23-1 to 9.2.24-1. I'll freeze them to the older version.
Also, please give yourself a ticket about growing the root FS on puppet-git.lizard.
- Status changed from Resolved to In Progress
- Assignee set to intrigeri
- Priority changed from High to Elevated
- Target version changed from Tails_3.8 to Tails_3.12
- QA Check changed from Pass to Info Needed
the freeze is starting to become problematic:
E: The repository 'http://snapshot.debian.org/archive/debian/20180505T160451Z sid Release' does no longer have a Release file.
any idea whether the libjetty breakage is fixed by now?