Project

General

Profile

Bug #11543

bitcoin.lizard cannot connect to our Puppet master

Added by bertagaz about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
06/23/2016
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
No
Affected tool:

Description

We've received reports since a while that the puppet agent runs on bitcoin.lizard fail:

~$ sudo puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Find /puppet/v3/node/bitcoin.lizard?environment=production&transaction_uuid=935f55d0-c55c-4d0e-9cf4-0b... \
resulted in 404 with the message: Not Found: Could not find environment 'puppet'
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///pluginfacts
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///plugins
Notice: /File[/var/lib/puppet/lib/facter]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/facter]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/facter/acpi_available.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
[...]
Warning: /File[/var/lib/puppet/lib/puppet/parser/functions/any2array.rb]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/augeas.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/parser/functions/augeas.rb]: Skipping because of failed dependencies
[...]
Notice: /File[/var/lib/puppet/lib/puppet/provider]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/provider]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/provider/file_line]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/provider/file_line]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/provider/file_line/ruby.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
[...]
Notice: /File[/var/lib/puppet/lib/puppet/type]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/type]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/type/anchor.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
[...]
Notice: /File[/var/lib/puppet/lib/puppet/util]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/util]: Skipping because of failed dependencies
Notice: /File[/var/lib/puppet/lib/puppet/util/external_iterator.rb]: Dependency File[/var/lib/puppet/lib] has failures: true
Warning: /File[/var/lib/puppet/lib/puppet/util/external_iterator.rb]: Skipping because of failed dependencies
[...]
Info: Loading facts
Error: Could not retrieve catalog from remote server: Find /puppet/v3/catalog/bitcoin.lizard?environment=production&facts_format=pson&facts=%257B%2522name%2... \
resulted in 404 with the message: Not Found: Could not find environment 'puppet'
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

It may be that the recent upgrade to puppet 4.x in Sid introduced this.


Related issues

Related to Tails - Feature #11834: Migrate our infrastructure to Puppet 4 Resolved 09/24/2016

History

#1 Updated by intrigeri about 3 years ago

  • Subject changed from Sid puppet agent not compatible with Jessie puppet master to bitcoin.lizard cannot connect to our Puppet master
  • Priority changed from Normal to Elevated

Note that Puppet 4 agents are not compatible with Puppet 3 masters, and we're nowhere ready to upgrade our master to Puppet 4, so it seems to me that we need:

1. a short-term solution for our sid VM; options:

  • downgrade to Puppet 3 on bitcoin.lizard (that would be my preferred option I think)
  • maintain a private backport of bitcoind, and reinstall that VM on Jessie (given how we handle upgrading stuff like our Puppet modules and Jenkins plugins, I'm not confident in this solution, at least not in the current state of our sysadmin duty work sharing)
  • kill that service and let the accounting team handle our bitcoin themselves
  • others?

IMO this is the part that should be solved ASAP (I don't know why this wasn't done during your 4 weeks of sysadmin duty in June, but whatever: 2.5 target version is good enough; bumping priority though because this is now more than 1 month old, and setting the relevant "deliverable for").

2. a migration plan to Puppet 4, that will take quite a bit longer (and some bits should be done before the Stretch freeze, i.e. about now)

  • our manifests and modules need to be ported (I would start with enabling the "future" parser)
  • we need to push puppetdb-termini to Debian (see the puppetdb RFP for details)
  • we need to handle the fact that we won't upgrade everything to Stretch at the same time
  • https://fnord.no/2016/05/28/puppet4-debian-unstable/ has useful info for the migration

If you agree, then I'll create tickets to track the migration plan, and in parallel we can discuss the short-term fix that this ticket is about (retitling to make it more specific). OK?

#2 Updated by bertagaz almost 3 years ago

  • Target version changed from Tails_2.5 to Tails_2.6

#3 Updated by intrigeri almost 3 years ago

  • Priority changed from Elevated to High

(This problem is now more than two months old, so bumping priority to make it show up higher than less critical stuff on your radar.)

#4 Updated by bertagaz almost 3 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 20

intrigeri wrote:

1. a short-term solution for our sid VM; options:

  • downgrade to Puppet 3 on bitcoin.lizard (that would be my preferred option I think)
  • maintain a private backport of bitcoind, and reinstall that VM on Jessie (given how we handle upgrading stuff like our Puppet modules and Jenkins plugins, I'm not confident in this solution, at least not in the current state of our sysadmin duty work sharing)
  • kill that service and let the accounting team handle our bitcoin themselves
  • others?

IMO this is the part that should be solved ASAP (I don't know why this wasn't done during your 4 weeks of sysadmin duty in June, but whatever: 2.5 target version is good enough; bumping priority though because this is now more than 1 month old, and setting the relevant "deliverable for").

So I've downgraded bitcoin's Puppet to 3.x a while ago. There was still an email sent per Puppet run because of some warnings, but it seems to have disappeared lately.

2. a migration plan to Puppet 4, that will take quite a bit longer (and some bits should be done before the Stretch freeze, i.e. about now)

  • our manifests and modules need to be ported (I would start with enabling the "future" parser)
  • we need to push puppetdb-termini to Debian (see the puppetdb RFP for details)
  • we need to handle the fact that we won't upgrade everything to Stretch at the same time
  • https://fnord.no/2016/05/28/puppet4-debian-unstable/ has useful info for the migration

If you agree, then I'll create tickets to track the migration plan, and in parallel we can discuss the short-term fix that this ticket is about (retitling to make it more specific). OK?

Agreed.

#5 Updated by anonym almost 3 years ago

  • Target version changed from Tails_2.6 to Tails_2.7

#6 Updated by intrigeri almost 3 years ago

intrigeri wrote:

1. a short-term solution for our sid VM; options:

  • downgrade to Puppet 3 on bitcoin.lizard (that would be my preferred option I think)
    [...]

So I've downgraded bitcoin's Puppet to 3.x a while ago.

Great!

There was still an email sent per Puppet run because of some warnings, but it seems to have disappeared lately.

As said elsewhere a few days ago: that's because I've fixed it manually (and told you over email).

2. a migration plan to Puppet 4, that will take quite a bit longer (and some bits should be done before the Stretch freeze, i.e. about now)
[...]
If you agree, then I'll create tickets to track the migration plan, and in parallel
we can discuss the short-term fix that this ticket is about (retitling to make it
more specific). OK?

Agreed.

Then I'll go ahead with the ticket fun.

#7 Updated by intrigeri almost 3 years ago

  • Assignee changed from bertagaz to intrigeri

#8 Updated by intrigeri almost 3 years ago

  • Related to Feature #11834: Migrate our infrastructure to Puppet 4 added

#9 Updated by intrigeri almost 3 years ago

intrigeri wrote:

Then I'll go ahead with the ticket fun.

Please have a look at #11834 and children, and check that it makes sense.

#10 Updated by intrigeri almost 3 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 20 to 100

Also available in: Atom PDF