Project

General

Profile

Feature #8647

Feature #5734: Monitor servers

Install an OS on the machine that will host the production monitoring setup

Added by intrigeri over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
12/15/2015
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

This includes the basic setup of a platform, with our base Puppet code applied, that is everything that's not specific to the monitoring setup.


Subtasks

Bug #10761: Add missing ecours connection info to sysadmins Git repoResolved

Bug #11342: Stop daily spam from ecours to sysadmins due to disabled IPv6Resolved


Related issues

Related to Tails - Bug #11181: Better abstraction of ::local::node In Progress 02/29/2016
Related to Tails - Bug #11213: Abstract common shorewall puppet bits Resolved 03/10/2016
Blocked by Tails - Feature #8646: Research and decide where to host the monitoring software Resolved 01/09/2015
Blocks Tails - Feature #9484: Deploy the monitoring setup to production Resolved 01/09/2015
Blocked by Tails - Feature #10760: Decide how to manage ecours and other systems with Puppet Resolved 12/15/2015

History

#1 Updated by intrigeri over 4 years ago

  • Blocked by Feature #8646: Research and decide where to host the monitoring software added

#2 Updated by intrigeri over 4 years ago

  • Blocks Feature #8648: Initial set up of the monitoring software added

#3 Updated by Dr_Whax over 4 years ago

Initial VM to experiment with monitoring software and resetting of the VM to a clean state.

#5 Updated by intrigeri over 4 years ago

Dr_Whax wrote:

Initial VM to experiment with monitoring software and resetting of the VM to a clean state.

No. To clarify, this ticket is about the production monitoring system, not about the VM that will be used for experiments and development.

#6 Updated by intrigeri over 4 years ago

  • Subject changed from Initial set up of the system that will host the monitoring software to Install an OS on the machine that will host the production monitoring setup

#7 Updated by intrigeri over 4 years ago

  • Blocks deleted (Feature #8648: Initial set up of the monitoring software)

#8 Updated by intrigeri over 4 years ago

  • Blocks Feature #9484: Deploy the monitoring setup to production added

#9 Updated by intrigeri almost 4 years ago

This has chances to be postponed to 1.9 (to be done by January 15) if our current hosting plans don't work out.

#10 Updated by intrigeri almost 4 years ago

  • Status changed from Confirmed to In Progress

The OS has been installed on ecours but it now needs to be managed by Puppet somehow. How?

#11 Updated by bertagaz almost 4 years ago

  • Target version changed from Tails_1.8 to Tails_2.0

Postponing

#12 Updated by intrigeri almost 4 years ago

  • Blocked by Feature #10760: Decide how to manage ecours and other systems with Puppet added

#13 Updated by intrigeri almost 4 years ago

intrigeri wrote:

The OS has been installed on ecours but it now needs to be managed by Puppet somehow. How?

This is now tracked by #10760.

#14 Updated by bertagaz almost 4 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Ready for QA

intrigeri wrote:

intrigeri wrote:

The OS has been installed on ecours but it now needs to be managed by Puppet somehow. How?

This is now tracked by #10760.

Then I'm setting this ticket as RfQA. You should now be able to connect to it with the informations added in #10761.
I've sent your password over email. You should find there a ssh server answering to your knock knock, and thus an OS installed on this machine.

#15 Updated by intrigeri almost 4 years ago

  • Description updated (diff)
  • Assignee changed from intrigeri to bertagaz
  • QA Check deleted (Ready for QA)

#16 Updated by intrigeri almost 4 years ago

  • Description updated (diff)

(Still blocked by #10760, updated description to clarify the scope of this ticket.)

#17 Updated by bertagaz over 3 years ago

  • Target version changed from Tails_2.0 to Tails_2.2

#18 Updated by bertagaz over 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Ready for QA

Now that #11094 is done, I've added our monitoring host to our puppet manifest and set it up from there. The puppet agent now runs fine on the monitoring host, and it has our base classes applied. There's still some cleanup and refactoring to do, but at least now we can go on with the parent ticket of this very one.

#19 Updated by intrigeri over 3 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Dev Needed

A few initial comments, I didn't look closely yet though:

  • This needs a firewall.
  • Do we want backups?
  • System information is not in internal.git.
  • These forgotten bits lead me to conclude that we lack a checklist for setting up hosts that are not VMs on lizard. Please start it so that it includes at least the bits that were forgotten this time, and we can improve it later. Let's not duplicate all info we have in similar, existing docs though -- better extract the common bits. I'm sorry this "adds" work but if we don't write such a doc at this time, we will never write it, and I'm tired of dealing with the consequences of undocumented processes in our sysadmin stuff (setting up isotesters recently, and having to reverse-engineer quite some stuff, reinforced this feeling quite a bit).
  • Regarding the Puppet bits that need refactoring: please file a ticket so it doesn't fall through the cracks.

#20 Updated by bertagaz over 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Dev Needed to Info Needed

intrigeri wrote:

A few initial comments, I didn't look closely yet though:

  • This needs a firewall.

Pushed a first version in ecours's node declaration.

  • Do we want backups?

Good question. So far I don't really see what we may want to backup. The monitoring host will just have a particular setup, but puppet is here for that. The rest will only be statistics, so not something we may want to bother. What we may want to save is the different key material, like the puppet cert, the Tor hidden service private key and the sshd ones? What do you think?

  • System information is not in internal.git.

Right, will push that a bit later.

  • These forgotten bits lead me to conclude that we lack a checklist for setting up hosts that are not VMs on lizard. Please start it so that it includes at least the bits that were forgotten this time, and we can improve it later. Let's not duplicate all info we have in similar, existing docs though -- better extract the common bits. I'm sorry this "adds" work but if we don't write such a doc at this time, we will never write it, and I'm tired of dealing with the consequences of undocumented processes in our sysadmin stuff (setting up isotesters recently, and having to reverse-engineer quite some stuff, reinforced this feeling quite a bit).

ACK, will be minimal given we already have good documentation for basic installation.

  • Regarding the Puppet bits that need refactoring: please file a ticket so it doesn't fall through the cracks.

Thought you did. Will do.

#21 Updated by bertagaz over 3 years ago

bertagaz wrote:

intrigeri wrote:

  • Do we want backups?

Pushed a commit into our backup repo that seem to generate the right configuration. Didn't check much more.

  • System information is not in internal.git.

Pushed.

  • These forgotten bits lead me to conclude that we lack a checklist for setting up hosts that are not VMs on lizard. Please start it so that it includes at least the bits that were forgotten this time, and we can improve it later. Let's not duplicate all info we have in similar, existing docs though -- better extract the common bits. I'm sorry this "adds" work but if we don't write such a doc at this time, we will never write it, and I'm tired of dealing with the consequences of undocumented processes in our sysadmin stuff (setting up isotesters recently, and having to reverse-engineer quite some stuff, reinforced this feeling quite a bit).

Pushed a first draft. New hosts intallation (hopefully the failover one) will help to refine it.

  • Regarding the Puppet bits that need refactoring: please file a ticket so it doesn't fall through the cracks.

Created #11181

#22 Updated by bertagaz over 3 years ago

  • QA Check changed from Info Needed to Ready for QA

#23 Updated by intrigeri over 3 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Info Needed
  • This needs a firewall.

Pushed a first version in ecours's node declaration.

Cool!

So, these are about 60 lines copy'n'pasted from lizard's node definition. Are there any plans to deal with the code duplication? (It can perhaps be argued that it's good enough as-is, I didn't think of it much yet; for now I just want to understand what your plans are, and what's the rationale behind them.)

  • Do we want backups?

Good question. So far I don't really see what we may want to backup. The monitoring host will just have a particular setup, but puppet is here for that.

Yes.

Pushed a commit into our backup repo that seem to generate the right configuration. Didn't check much more.

That commit comments out the parts of the Makefile that actually do the backups. Please double-check if it's indeed what you wanted to do.

The rest will only be statistics, so not something we may want to bother.

OK. We can always reconsider if we ever start using such historical data, e.g. to evaluate how well our systems work.

What we may want to save is the different key material, like the puppet cert, the Tor hidden service private key and the sshd ones? What do you think?

Yes, the standard stuff we almost always back up, sounds good.

  • These forgotten bits lead me to conclude that we lack a checklist for setting up hosts that are not VMs on lizard.

ACK, will be minimal given we already have good documentation for basic installation.

Pushed a first draft. New hosts intallation (hopefully the failover one) will help to refine it.

Great, thanks. I've pushed some changes to that doc, to make it easier (for me at least) to follow.

  • Regarding the Puppet bits that need refactoring: please file a ticket so it doesn't fall through the cracks.

Thought you did. Will do.

Created #11181

Thanks :) I'll start a discussion there, then, because the current state of that ticket doesn't really address "so it doesn't fall through the cracks", the way I see it.

#24 Updated by intrigeri over 3 years ago

  • QA Check deleted (Info Needed)

Also, please enable AppArmor there, or justify why we don't want it on that system :)

#25 Updated by bertagaz over 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Ready for QA

intrigeri wrote:

  • This needs a firewall.

So, these are about 60 lines copy'n'pasted from lizard's node definition. Are there any plans to deal with the code duplication? (It can perhaps be argued that it's good enough as-is, I didn't think of it much yet; for now I just want to understand what your plans are, and what's the rationale behind them.)

Didn't think much about it too. I've created #11213 to take care of that.

  • Do we want backups?

Pushed a commit into our backup repo that seem to generate the right configuration. Didn't check much more.

That commit comments out the parts of the Makefile that actually do the backups. Please double-check if it's indeed what you wanted to do.

Woops, fixed.

Pushed a first draft. New hosts intallation (hopefully the failover one) will help to refine it.

Great, thanks. I've pushed some changes to that doc, to make it easier (for me at least) to follow.

Thanks!

Also, please enable AppArmor there, or justify why we don't want it on that system :)

That was because of #11181, as it required to duplicate some code while this other ticket is not resolved, I didn't take care of enabling apparmor. Now it's done (well, I haven't rebooted the system yet).

#26 Updated by intrigeri over 3 years ago

  • Target version changed from Tails_2.2 to Tails_2.3

#27 Updated by intrigeri over 3 years ago

  • Related to Bug #11181: Better abstraction of ::local::node added

#28 Updated by intrigeri over 3 years ago

  • Related to Bug #11213: Abstract common shorewall puppet bits added

#29 Updated by intrigeri over 3 years ago

  • Assignee changed from intrigeri to bertagaz

Also, please enable AppArmor there, or justify why we don't want it on that system :)

That was because of #11181, as it required to duplicate some code while this other ticket is not resolved, I didn't take care of enabling apparmor. Now it's done (well, I haven't rebooted the system yet).

Cool, thanks. Code duplication was not needed actually. I did a little bit of refactoring to avoid it.

I think we're good here, feel free to close this ticket as resolved once you've rebooted ecours and confirmed that AppArmor is indeed enabled.

#30 Updated by bertagaz over 3 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • Target version deleted (Tails_2.3)
  • QA Check deleted (Ready for QA)

Rebooted, setup and documented the process. Apparmor is enabled, ipv6 disabled. Everything works fine, case closed!

#31 Updated by intrigeri over 3 years ago

  • Status changed from Resolved to In Progress
  • Assignee set to bertagaz
  • Target version set to Tails_2.3

bertagaz wrote:

Rebooted, setup and documented the process.

Yay! I've looked at the reboot doc & corresponding Puppet code, and I wonder why you switched to dropbear. It feels like it adds risks and complexity compared to the OOB access we already had. Any explanation?

#32 Updated by bertagaz over 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Info Needed

intrigeri wrote:

bertagaz wrote:

Rebooted, setup and documented the process.

Yay! I've looked at the reboot doc & corresponding Puppet code, and I wonder why you switched to dropbear. It feels like it adds risks and complexity compared to the OOB access we already had. Any explanation?

I guess more by habit than anything else. I can remove that if you think its inappropriate.

#33 Updated by intrigeri over 3 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Info Needed to Dev Needed

I guess more by habit than anything else. I can remove that if you think its inappropriate.

Keep it if, and only if, there is a good reason for us to keep and maintain it. (The less stuff to maintain, the better. The less open doors, the better.)

#34 Updated by bertagaz over 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:

I guess more by habit than anything else. I can remove that if you think its inappropriate.

Keep it if, and only if, there is a good reason for us to keep and maintain it. (The less stuff to maintain, the better. The less open doors, the better.)

Ok, purged the package and rebuild the initramfs. Removed the instructions from our git repo too. Feel free to close this ticket if happy.

#35 Updated by intrigeri over 3 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Ready for QA to Info Needed

Ok, purged the package and rebuild the initramfs. Removed the instructions from our git repo too.

Good! Any reason to keep the network configuration (ip=...) on GRUB_CMDLINE_LINUX, then?

#36 Updated by bertagaz over 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Info Needed to Ready for QA

intrigeri wrote:

Ok, purged the package and rebuild the initramfs. Removed the instructions from our git repo too.

Good! Any reason to keep the network configuration (ip=...) on GRUB_CMDLINE_LINUX, then?

Not really, apart I've been too fast there. Good catch! Fixed in puppet-tails:ca15474

#37 Updated by intrigeri over 3 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • QA Check changed from Ready for QA to Pass

Also available in: Atom PDF