Project

General

Profile

Feature #15510

Feature #15500: Update Puppet modules: 2018Q4 → 2019Q2 edition

Switch to the puppetlabs/apt module

Added by intrigeri over 1 year ago. Updated 3 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
Infrastructure
Target version:
Start date:
04/09/2018
Due date:
% Done:

100%

Feature Branch:
puppet-lizard-manifests:feature15510, puppet-tails:feature15510
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

The shared apt module is deprecated and the shared Puppet modules group decided to switch to https://github.com/puppetlabs/puppetlabs-apt.

See https://gitlab.com/shared-puppet-modules-group/apt/blob/master/UPGRADING.md for migration info. Lots of features we use are missing so this is going to take time. Let's try to make it happen by the end of 2019Q1.

This migration can be split into steps:

  1. migrate away from functionality provided by the shared apt module but that's not supported by puppetlabs/apt: either switch to already supported alternatives, or to "plugins", or submit PRs to puppetlabs/apt
    1. listchanges done
    2. apt::apticronAPT::Periodic + monitoring check done
    3. apt::reboot_required_notify → install reboot-notifier ourselves done
    4. apt::dpkg_statoverride → import it in puppet-tails
  2. reach the point where the only functionality we use from the shared apt module is also supported either by puppetlabs/apt directly or by modules that depend on it
  3. switch to puppetlabs/apt; changes that must happen in lockstep when doing so:
    1. apt::apt_confapt::conf
    2. apt::custom_key_dirapt class' keys parameter but implementation looks scary (do the red flags apply to us?); worst case, follow https://wiki.debian.org/DebianRepository/UseThirdParty and drop the non-ascii-armorded key in /usr/share/keyrings/example-archive-keyring.gpg + point to it from sources.list
    3. apt::cronhttps://github.com/voxpupuli/puppet-unattended_upgrades
    4. apt::packagepackage + apt::pin
    5. apt::preferences_snippetapt::pin
    6. apt::proxy*apt class' proxy parameter
    7. apt::sources_list, apt::repos, apt::use_next_release, apt::use_volatile, tails::apt::repository::* → puppetlabs/apt provides several was to manage sources.list

To list the apt:: things we use:

git grep -h --only-matching --color=never \
  --recurse-submodules -E '\bapt::[^ ]+\b' -- \
  hieradata/ manifests/ modules/reprepro/ modules/tails* \
  2>/dev/null \
  | sort -u

Related issues

Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017

History

#1 Updated by intrigeri over 1 year ago

  • Blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#2 Updated by intrigeri over 1 year ago

  • Description updated (diff)

#3 Updated by intrigeri over 1 year ago

  • Target version changed from Tails_3.9 to Tails_3.10.1

#4 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.10.1 to Tails_3.11

#5 Updated by intrigeri about 1 year ago

  • Parent task changed from #15499 to #15500

#6 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.11 to Tails_3.12

I've booked time to work on this around Dec 17-31.

#7 Updated by intrigeri 11 months ago

  • Target version changed from Tails_3.12 to Tails_3.13

#8 Updated by intrigeri 9 months ago

  • Description updated (diff)

#9 Updated by intrigeri 8 months ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

Listed all the apt:: thing we use and what I think we should do with them. Migrated a few small things away from the old apt module.

#10 Updated by intrigeri 8 months ago

  • Description updated (diff)

Migrated the last thing I could migrate without switching to puppetlabs/apt.

Next steps: prepare a topic branch (= environment) that switches to puppetlabs/apt + adjusts the bare minimum, put a couple non-critical nodes into that environment, fix stuff, and progressively port more code and nodes to that environment until we're done and we can merge this branch into production and put all nodes back into the production environment.

#11 Updated by intrigeri 8 months ago

  • Description updated (diff)

#12 Updated by intrigeri 8 months ago

  • Target version changed from Tails_3.13 to Tails_3.14

I've been busy elsewhere and we're now too close to a release for such a potentially disruptive change.

#13 Updated by intrigeri 8 months ago

  • Feature Branch set to puppet-lizard-manifests:feature15510

#14 Updated by intrigeri 7 months ago

  • Priority changed from Normal to Elevated

#15 Updated by intrigeri 6 months ago

  • Target version changed from Tails_3.14 to Tails_3.15

#16 Updated by intrigeri 5 months ago

  • Feature Branch changed from puppet-lizard-manifests:feature15510 to puppet-lizard-manifests:feature15510, puppet-tails:feature15510

#17 Updated by intrigeri 5 months ago

  • Description updated (diff)

#18 Updated by intrigeri 5 months ago

  • Description updated (diff)

#19 Updated by intrigeri 5 months ago

  • Description updated (diff)

#20 Updated by intrigeri 5 months ago

  • Description updated (diff)

#21 Updated by intrigeri 5 months ago

  • Description updated (diff)

#22 Updated by intrigeri 5 months ago

  • Description updated (diff)

#23 Updated by intrigeri 5 months ago

  • % Done changed from 10 to 20

I think I've ported all the code we use to puppetlabs/apt and it works on local VMs that use tiny subsets of our Puppet codebase. Next steps: put each node, one after the other, into the feature15510 environment, from the least critical to the most, fix stuff as I go, until I'm done and we can merge this branch into production and put all nodes back into the production environment.

#24 Updated by intrigeri 5 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to groente

Deployed! @bertagaz, @groente, anyone wants to review the code changes?

#25 Updated by intrigeri 5 months ago

  • Status changed from Needs Validation to In Progress
  • Assignee changed from groente to intrigeri

/var/log/unattended-upgrades/unattended-upgrades.log on apt.lizard says:

2019-06-14 06:00:34,065 INFO Packages that will be upgraded: dbus dbus-x11 libdbus-1-3
2019-06-14 06:00:34,066 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2019-06-14 06:00:45,561 INFO All upgrades installed

… and the corresponding monitoring checked recovered. So upgrades are applied, which is good. But the "apt" and "upgreadable" monitoring check respectively went to critical and warning state before that and that's been noisy. I suspect our monitoring checks are scheduled in a way that matched what we did before (upgrading every 6 hours) but does not match unattended-upgrades' scheduling. I'll fix that.

#26 Updated by intrigeri 5 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to groente

Tried to fix the monitoring scheduling issue. It may need some more fine tuning, I'll keep an eye on it, but that shouldn't block reviewing the rest of this work.

#27 Updated by intrigeri 5 months ago

Next issue for me to triage/investigate: "Subject: [package on hold][reboot required] unattended-upgrades result for 'puppet-git': False".

#28 Updated by intrigeri 5 months ago

intrigeri wrote:

Next issue for me to triage/investigate: "Subject: [package on hold][reboot required] unattended-upgrades result for 'puppet-git': False".

Should be fixed by https://git-tails.immerda.ch/puppet-tails/commit/?id=a65ade061a5f739a4e5647b211c6f8d0b0fb56d8.

#29 Updated by intrigeri 5 months ago

intrigeri wrote:

Tried to fix the monitoring scheduling issue. It may need some more fine tuning,

It does: https://icingaweb2.tails.boum.org/monitoring/service/show?host=isobuilder1.lizard&service=apt%40isobuilder1.lizard. This time, I'll (re×4)read Icinga check scheduling doc and will base whatever fix I try next on actual maths, instead of fiddling with it like I did 2 days ago.

#30 Updated by intrigeri 5 months ago

intrigeri wrote:

intrigeri wrote:

Tried to fix the monitoring scheduling issue. It may need some more fine tuning,

It does: https://icingaweb2.tails.boum.org/monitoring/service/show?host=isobuilder1.lizard&service=apt%40isobuilder1.lizard. This time, I'll (re×4)read Icinga check scheduling doc and will base whatever fix I try next on actual maths, instead of fiddling with it like I did 2 days ago.

In the end, I figured it was better to fix the root cause of the problem (upgrades applied less often than they used to be) rather than tweaking the monitoring, so I tried it (6a715c3363d35c7ef694fd8b4c12177a79a2b0f4 + ceac8b1890de68ec42bd8cf9ca753e233ee61dc9) and reverted my earlier fiddling with the monitoring config.

#31 Updated by CyrilBrulebois 4 months ago

  • Target version changed from Tails_3.15 to Tails_3.16

#32 Updated by groente 4 months ago

  • Status changed from Needs Validation to In Progress
  • Assignee changed from groente to intrigeri
  • % Done changed from 20 to 40

heya,
sorry for the long delay, but the code looks good!
one last issue is that the monitoring still triggers quite often before the upgrade is run. perhaps your calculations didn't take the 60 minute randomisation on the upgrade timer into account?

#33 Updated by intrigeri 3 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to groente

groente wrote:

one last issue is that the monitoring still triggers quite often before the upgrade is run. perhaps your calculations didn't take the 60 minute randomisation on the upgrade timer into account?

I think you're right! https://git.tails.boum.org/puppet-tails/commit/?id=951713c6c11053fe5a6acfe94250f747ff1dbda9 should fix that. Please let me know if this sort of problems still happens :)

#34 Updated by groente 3 months ago

  • Status changed from Needs Validation to Resolved

intrigeri wrote:

groente wrote:

one last issue is that the monitoring still triggers quite often before the upgrade is run. perhaps your calculations didn't take the 60 minute randomisation on the upgrade timer into account?

I think you're right! https://git.tails.boum.org/puppet-tails/commit/?id=951713c6c11053fe5a6acfe94250f747ff1dbda9 should fix that. Please let me know if this sort of problems still happens :)

Didn't happen anymore during the last updates, I'm quite confident we're good, so closing this ticket \o/

#35 Updated by groente 3 months ago

  • % Done changed from 40 to 100

Also available in: Atom PDF