Project

General

Profile

Feature #16165

Bug #15071: Make our server backup process more usable

make puppet-lizard-manifests suitable for masterless puppet

Added by groente 10 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
11/28/2018
Due date:
% Done:

100%

Spent time:
(Total: 1.00 h)
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

since stone will be running masterless, we need to refactor puppet-lizard-manifests a bit to accomodate this.


Subtasks

Bug #16166: move secrets from nodes.pp to hieraResolved

Bug #16167: move classes.pp from manifests to its own moduleResolved

Feature #16196: fix puppet-sshd for masterless puppetResolved

Feature #16197: fix puppet-tails' vpn for masterless puppetResolved

Bug #16213: user module raises warning about real_gidResolved


Related issues

Blocked by Tails - Bug #16172: Upgrade the puppet-concat module Resolved 11/29/2018
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017

History

#1 Updated by intrigeri 10 months ago

Both subtasks are resolved. What's next apart of changing the secrets that got moved?

#2 Updated by groente 10 months ago

intrigeri wrote:

Both subtasks are resolved. What's next apart of changing the secrets that got moved?

Next up is:

- turning hieradata/node into a git submodule
- changing all the secrets in hierdata/node/*
- adding a masterless_manifests directory with a stone.pp
- creating a stone branch on puppet-lizard-manifests
- adjusting the gitmodules needed for stone on this new branch
- start testing on stone

#3 Updated by groente 10 months ago

hierdata/node is now a submodule, its repo is called puppet-lizard-hiera-node

#4 Updated by groente 10 months ago

all secrets in hieradata/node have been changed \o/

#5 Updated by groente 10 months ago

  • Blocked by Bug #16172: Upgrade the puppet-concat module added

#6 Updated by groente 10 months ago

there's now a branch stone in puppet-lizard-manifests
stone has read access to all the required repositories

first tests broke on puppet-concat, see #16172

#7 Updated by groente 10 months ago

  • Assignee changed from groente to intrigeri
  • QA Check set to Info Needed

deploying the tails base class mostly works, but monitoring, ssh, and vpn want to make use of exported resources.

i'm contemplating on the feasibility of making puppetdb available to stone.

would you mind giving me your first thoughts on this (considering stone and puppet-git should never be able to get RCE on eachother), before i invest time more in exploring the option? thank you!

#8 Updated by intrigeri 10 months ago

  • Assignee changed from intrigeri to groente
  • QA Check changed from Info Needed to Dev Needed

deploying the tails base class mostly works,

I've just pushed some refactoring of the quick ad-hoc fixes you've pushed there for stone yesterday. If I guessed wrong about the cpupower thing, let me know. BTW, how should I run Puppet by hand on stone to test such changes myself?

but monitoring, ssh, and vpn want to make use of exported resources.

  • monitoring: I don't think we can manage monitoring on stone, using our existing code, without giving RCE-on-stone power to PuppetDB.
  • SSH: I think we don't need most of that. I think the little we need on stone can be handled without exported resources, by storing what we need in Hiera.
  • VPN: maybe stone does not need to be part of the VPN?

i'm contemplating on the feasibility of making puppetdb available to stone.

would you mind giving me your first thoughts on this (considering stone and puppet-git should never be able to get RCE on eachother), before i invest time more in exploring the option? thank you!

tl;dr: mayyyybe feasible but probably too dangerous given the amount of Puppet code that will run on stone and our Puppet skills. I recommend using Hiera instead to share data among stone and the rest of our infra.

Starting point: none of our systems currently has direct access to the PuppetDB port (8080), without the mediation of the Puppet master, except the one that runs the Puppet master.

About "avoid giving PuppetDB RCE on stone": I think that's feasible but will require extreme carefulness, when writing/modifying code that affects stone's catalog, wrt. what data coming from PuppetDB it trusts and to what extent. I'm not sure we have sufficiently good Puppet skills widespread on our team to do this right consistently.

About "avoid giving stone RCE on our other systems" and "avoid giving stone access to secrets we want to hide from it", it boils down to what kind of authentication/authorization PuppetDB does when replying queries. Connecting standalone Puppet nodes to PuppetDB is a supported and documented configuration. Client authentication is a thing but I see nothing there about authorization, fine-grained access control or security. So I don't know if/how PuppetDB restricts access per-node to the data it holds. E.g. I don't know if a node that connects to PuppetDB without the mediation of the Puppet master will have access to all resources of all other nodes, or only to the exported ones; if the former, then I'm pretty sure we can't do that without giving RCE to stone on other nodes; if the latter, then it may be worth looking closer. I guess it's easy to test this by querying PuppetDB from the command line on puppet-git.lizard (see also the curl tips).

#9 Updated by groente 10 months ago

I've just pushed some refactoring of the quick ad-hoc fixes you've pushed there for stone yesterday. If I guessed wrong about the cpupower thing, let me know. BTW, how should I run Puppet by hand on stone to test such changes myself?

energy-perf-policy is unhappy again, i'll look into this later. running puppet by hand on stone goes as follows:

cd /etc/puppet/code
git pull
git submodule update
puppet apply ./masterless_manifests/stone.pp

  • monitoring: I don't think we can manage monitoring on stone, using our existing code, without giving RCE-on-stone power to PuppetDB.

hm, okay, i'll think about a workaround and get back to this later.

  • SSH: I think we don't need most of that. I think the little we need on stone can be handled without exported resources, by storing what we need in Hiera.

indeed, ssh works fine as it is now, without actually using any of the exported resources.

  • VPN: maybe stone does not need to be part of the VPN?

it would need to be part of the vpn for monitoring, no? that said, it doesn't look undoable to get vpn working without the exported resources.

tl;dr: mayyyybe feasible but probably too dangerous given the amount of Puppet code that will run on stone and our Puppet skills. I recommend using Hiera instead to share data among stone and the rest of our infra.

you've convinced me this is too risky :) guess we'll have to live with a bunch of warnings when running puppet apply on stone...

#10 Updated by intrigeri 10 months ago

energy-perf-policy is unhappy again, i'll look into this later.

Fixed.

  • VPN: maybe stone does not need to be part of the VPN?

it would need to be part of the vpn for monitoring, no?

Indeed, right.

guess we'll have to live with a bunch of warnings when running puppet apply on stone...

I don't see why. The warnings we're talking about come from:

  • sshd: if that's not possible yet, we should make it possible to opt-in from collecting/exporting exported resources.
  • tails::vpn, tails::monitoring: for now we should not apply these classes on stone; they won't do anything useful there anyway in the current state of this code. The latter can already be disabled by passing monitoring_agent => false to tails::base; the former could simply be removed from stone's own manifest.

#11 Updated by intrigeri 10 months ago

  • Blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#12 Updated by intrigeri 9 months ago

  • Assignee changed from groente to intrigeri
  • Target version set to Tails_3.12
  • QA Check changed from Dev Needed to Ready for QA

All subtasks are done so I'll assume this is done too. If not, please clarify what's missing :)

#13 Updated by intrigeri 8 months ago

  • Assignee changed from intrigeri to groente
  • QA Check changed from Ready for QA to Dev Needed

I won't reopen #16196 just for this but please clean this up:

root@stone:/etc/puppet/code# git status
On branch stone
Your branch is up-to-date with 'origin/stone'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)

        modified:   modules/sshd (modified content)

And in passing, it would be nice to run the latest version of https://gitlab.com/shared-puppet-modules-group/sshd/merge_requests/29 both on the stone and production branches, to ensure it works fine in all cases :)

Also, we need a documented process to update the stone branch: it's not been updated since Dec 20. I guess it boils down to "whenever pushing to the production branch, merge it into the stone branch, then SSH to stone, cd /etc/puppet/code && git pull && git submodule update && puppet apply /etc/puppet/code/masterless_manifests/stone.pp".

I'll do a more complete review once the stone branch is up-to-date wrt. the production one but from what I've followed so far, it looks good!

#14 Updated by anonym 8 months ago

  • Target version changed from Tails_3.12 to Tails_3.13

#15 Updated by CyrilBrulebois 6 months ago

  • Target version changed from Tails_3.13 to Tails_3.14

#16 Updated by CyrilBrulebois 4 months ago

  • Target version changed from Tails_3.14 to Tails_3.15

#17 Updated by CyrilBrulebois 2 months ago

  • Target version changed from Tails_3.15 to Tails_3.16

#18 Updated by groente 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from groente to intrigeri

From what I can tell, the stone branch is now properly up to date. I've added some basic instructions in the repo's README about keeping the branch up-to-date and applying puppet on stone. Do you agree it's time to close this ticket now?

#19 Updated by intrigeri about 2 months ago

  • Status changed from Needs Validation to In Progress
  • Assignee changed from intrigeri to groente

From what I can tell, the stone branch is now properly up to date. I've added some basic instructions in the repo's README about keeping the branch up-to-date and applying puppet on stone. Do you agree it's time to close this ticket now?

I've followed the instructions in this new README and it worked fine.

One small thing I find slightly confusing though: why do we have masterless_manifests/stone.pp on the production branch? If I understand the setup clearly, it's bound to increasingly lag behind the version on the stone branch, because we're never going to merge the stone branch back into production. If that's correct, then I suggest deleting this file on the production branch, merging into stone, resolving the merge conflict in favour of the file that's on the stone branch, and boom!

And then yeah, please close this ticket! :)

#20 Updated by groente about 1 month ago

  • Status changed from In Progress to Resolved

intrigeri wrote:

From what I can tell, the stone branch is now properly up to date. I've added some basic instructions in the repo's README about keeping the branch up-to-date and applying puppet on stone. Do you agree it's time to close this ticket now?

I've followed the instructions in this new README and it worked fine.

One small thing I find slightly confusing though: why do we have masterless_manifests/stone.pp on the production branch? If I understand the setup clearly, it's bound to increasingly lag behind the version on the stone branch, because we're never going to merge the stone branch back into production. If that's correct, then I suggest deleting this file on the production branch, merging into stone, resolving the merge conflict in favour of the file that's on the stone branch, and boom!

My idea was all changes to the masterless_manifests should be done primarily in the production branch and then merged to their specific masterless branches (in our case, sofar only the stone branch). I've added an extra line about this in the README.

And then yeah, please close this ticket! :)

yay \o/

Also available in: Atom PDF