Project

General

Profile

Feature #7763

Move our email reminders (e.g. for meetings) to Puppet

Added by sajolida over 5 years ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
-
Start date:
03/03/2015
Due date:
% Done:

100%

Feature Branch:
https://0xacab.org/muri/puppet-tails/tree/7763-meeting-reminder
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

Our meeting reminder setup recipes should be generic, and live in the
(public) "tails" Puppet module, as a tails::meeting::reminder class,
I guess, that will install whatever is needed from Debian and from
your upstream Git repo (no, I won't ask for a Debian package ;)

Then, most of the config can be public as well, and can live in the
very same module. If there are additional private config bits, then
we'll put it into a dedicated private Puppet module.

See https://tails.boum.org/contribute/how/sysadmin/#index4h1 and
especially the "If you don't know Puppet" subsection. Focus on "1. Prepare configuration, scripts and whatever is needed".

Then, once we have this setup description info, intrigeri can translate it into Puppet.

lhf.yaml View (1.42 KB) sajolida, 03/05/2015 03:00 PM

monthly.yaml View (1.51 KB) sajolida, 03/05/2015 03:00 PM

code-of-conduct.eml (4.29 KB) sajolida, 11/12/2019 10:50 AM

monthly-report.eml (989 Bytes) sajolida, 11/12/2019 10:50 AM

reimbursements.eml (512 Bytes) sajolida, 11/12/2019 10:50 AM


Subtasks

Feature #8997: Write YAML files to describe the cronjob needed for each reminderResolved


Related issues

Related to Tails - Feature #7899: Automatically announce low-hanging fruit sessions Resolved 09/14/2014
Related to Tails - Feature #14725: Quarterly reminder to tails-project@ about sponsorship (from [misc]) Resolved 09/26/2017
Related to Tails - Feature #14507: Have the FT meeting reminder automatically avoid Fridays, Saturdays, and Sundays Resolved 08/30/2017

History

#1 Updated by intrigeri over 5 years ago

  • Description updated (diff)

(Publishing private email without seeking the author's consent first feels wrong enough, no need to keep the meta part of it..)

#2 Updated by sajolida over 5 years ago

  • Priority changed from Normal to Elevated
  • Parent task deleted (#7523)

#3 Updated by sajolida almost 5 years ago

  • Assignee deleted (sajolida)

I won't learn Puppet and do that I'm sorry...

#4 Updated by intrigeri almost 5 years ago

  • Assignee set to sajolida
  • QA Check set to Info Needed

No problem! May you please share your existing code and configuration somewhere, so that we don't start from scratch?

#5 Updated by sajolida almost 5 years ago

  • Assignee deleted (sajolida)
  • QA Check deleted (Info Needed)

Cron configuration:

MAILTO=sajolida@pimienta.org
LANG=en_US.UTF-8
0 0 15 * *      MONTHLY=$(date --date="$(date '+\%Y-\%m-03 +1 month')" +\%F) ; ruby $HOME/meeting/meeting.rb --locale en_US.UTF-8 --template $HOME/monthly.eml --date $MONTHLY --subject "Tails contributors meeting: $(date --date=$MONTHLY '+\%A \%B \%d')" --from sajolida@pimienta.org --to "tails-dev@boum.org tails-project@boum.org tails-ux@boum.org tails-l10n@boum.org" 
0 0 1 * *       MONTHLY=$(date --date="$(date '+\%Y-\%m-03')" +\%F) ; ruby $HOME/meeting/meeting.rb --locale en_US.UTF-8 --template $HOME/monthly.eml --date $MONTHLY --subject "Tails contributors meeting: $(date --date=$MONTHLY '+\%A \%B \%d')" --from sajolida@pimienta.org --to "tails-dev@boum.org tails-project@boum.org tails-ux@boum.org tails-l10n@boum.org" 

Needed packages:

  • ruby-liquid

Upstream repo:

https://un.poivron.org/~sajolida/meeting.git

Email template:

The next Tails contributors meeting is scheduled for:

    {{ date }}
        #tails-dev on irc.oftc.net
         9 pm in Paris
         8 pm in London
         3 pm in New-York
        12 pm in San Francisco

Every one interested in contributing to Tails is welcome.

Feel free to propose and prepare discussion topics. Either:

  - Raise them in this thread so that others can ask details and prepare the
    discussion too.

  - Update the blueprint of the agenda:

    https://tails.boum.org/blueprint/monthly_meeting/

  - Make sure that the discussion tickets that you want to treat during the
    meeting are well detailed on Redmine.

If you want to get involved but don't know yet how, please introduce
yourself during the meeting, and be sure to tell us what you are
interested in.

The meeting might not be the most adequate time and place to properly
introduce newcomers to the development process, but at least it should
be a fine place to know each others, and schedule a better
suited event.

#6 Updated by sajolida almost 5 years ago

  • Related to Feature #7899: Automatically announce low-hanging fruit sessions added

#7 Updated by Dr_Whax almost 5 years ago

so it isn't forgotten:

should be mostly a generic class, a config class, and a define. the generic class does vcsrepo. the define does cron. the config class uses the define as many times as needed (until we put that into hiera, hopefully). or something.

#8 Updated by Dr_Whax almost 5 years ago

  • Assignee set to Dr_Whax

#9 Updated by intrigeri almost 5 years ago

  • Target version set to Tails_1.4

Let's set a deadline, and if not done by then, I think bertagaz or I should take over (it could be considered as a part of the daily sysadmin duty shifts).

#10 Updated by sajolida almost 5 years ago

We could use YAML files (see attachments) to have Puppet generate the corresponding cron jobs. Ideally such files would be hosted in a dedicated public Git repo.

Notes:

  • $DATE in the subject must be replaced by the "date" attribute.
  • In the "date" attribute I left the \ escaping for cron. So the command don't run fine if copy-pasted from there.

#11 Updated by intrigeri almost 5 years ago

Note to the implementer: these YAML files are here as examples of what the Hiera configuration could be, if we decide to go this way to configure these cronjobs, as opposed to Puppet code in our manifests. This doesn't block the initial implementation (only what I called the "config class" might be replaced by bits of Hiera, I think).

#12 Updated by intrigeri over 4 years ago

  • Target version changed from Tails_1.4 to Tails_1.4.1

Postponing.

DrWhax: IIRC you took this ticket as a potentially nice way to get deeper into Puppet, right? If you need help from me, feel free to ask. I've no idea what Puppet experience you had time to grab in the last 6 months, so please point me to whatever Puppet bits you wrote or modified, and to whatever bits of Puppet documentation / books / tutorial you've read, so that I can adjust the help I'll provide accordingly. IIRC your other, more involved Puppet tasks are approaching, so the earlier you're up to speed, the better :)

#13 Updated by BitingBird over 4 years ago

  • Target version changed from Tails_1.4.1 to Tails_1.5

#14 Updated by intrigeri over 4 years ago

  • Target version changed from Tails_1.5 to Tails_1.6

Postponing one last time. If not done in the next release cycle, I guess that someone else (e.g. bertagaz or I as part of our sysadmin shifts) will need to take over.

#15 Updated by bertagaz about 4 years ago

  • Target version changed from Tails_1.6 to Tails_1.7

#16 Updated by Dr_Whax about 4 years ago

  • Assignee deleted (Dr_Whax)
  • Target version changed from Tails_1.7 to 2016

#17 Updated by intrigeri over 3 years ago

  • Target version deleted (2016)

#18 Updated by sajolida about 2 years ago

  • Related to Feature #14725: Quarterly reminder to tails-project@ about sponsorship (from [misc]) added

#19 Updated by u almost 2 years ago

  • Related to Feature #14507: Have the FT meeting reminder automatically avoid Fridays, Saturdays, and Sundays added

#20 Updated by muri 12 months ago

  • Assignee set to muri

#21 Updated by muri 9 months ago

  • Assignee changed from muri to intrigeri
  • Feature Branch set to https://0xacab.org/muri/puppet-tails/tree/7763-meeting-reminder

Ok, so... I feel like I've forgotten everything I knew about puppet. Didn't even know there were data types! I tried my best to create something on https://0xacab.org/muri/puppet-tails/tree/7763-meeting-reminder. I have created on class that should install the script and added one definition for the cronjob, which requires the class. I'm not sure if it makes sense to review it or if it would be better to be written by someone with more puppet skillset. @intrigeri i've set you as assignee, maybe you can take a short look (but no hurry whatsoever...)

#22 Updated by intrigeri 8 months ago

  • Assignee changed from intrigeri to muri

Thanks!

In the class:

  • I'll skip everything that's about managing a clone of the script's repo until we reach a conclusion on #14507#note-41. Moving the code to puppet-tails.git would allow us to get rid of big parts of the code here (and incidentally, the parts on which I would have more comments).
  • I think $user should not default to $title, which does not make sense for a class. I propose tails_meeting_reminder.

In the defined resource:

  • Please use long option names (e.g. --from) in the command line, for better readability.
  • We need to deploy the template, likely via the Puppet file server.
  • We should validate $fromemailaddress and items in $addresses.
  • The command should find the path of the script and the value of user via ${tails::meeting::...} instead of hard-coding it.
  • I think the cron resource can simply require => Class['::tails::meeting']. I'm not sure what require => [Tails::Meeting['tails_meetingscript']] means (maybe nothing Puppet would understand? :)

#23 Updated by muri 6 months ago

  • Assignee changed from muri to intrigeri

Hi,

intrigeri wrote:

Thanks!

In the class:

  • I'll skip everything that's about managing a clone of the script's repo until we reach a conclusion on #14507#note-41. Moving the code to puppet-tails.git would allow us to get rid of big parts of the code here (and incidentally, the parts on which I would have more comments).

Oke, as mentioned in #14507 I've now moved the code into the files directory of the puppet module and in turn removed all the stuff related to the vcs module

  • I think $user should not default to $title, which does not make sense for a class. I propose tails_meeting_reminder.

done

In the defined resource:

  • Please use long option names (e.g. --from) in the command line, for better readability.

done

  • We need to deploy the template, likely via the Puppet file server.

done

  • We should validate $fromemailaddress and items in $addresses.

done

  • The command should find the path of the script and the value of user via ${tails::meeting::...} instead of hard-coding it.

I'm not sure how to access variables from the class above- should I create arguments user and path in tails::meeting for that?

  • I think the cron resource can simply require => Class['::tails::meeting']. I'm not sure what require => [Tails::Meeting['tails_meetingscript']] means (maybe nothing Puppet would understand? :)

done

I tried to find out how to test the module locally, but I failed. I guess it should somehow work with puppet apply --modulepath=/path/to/tails/modules, which works basically, but it complains that it doesn't find Stdlib::Absolutepath although I have installed puppet-module-puppetlabs-stdlib. Setting --modulepath=/usr/share/puppet/modules.available/puppetlabs-stdlib:/path/to/tails/modules doesn't help.

#24 Updated by intrigeri about 1 month ago

  • Assignee changed from intrigeri to muri

Hi @muri,

this fell through the cracks… sorry for the delay!

All the changes you made look good to me, yeah! Except a few minor things that I'm happy to fix myself when deploying, if you'd like (I'm documenting them below mostly as a way to share knowledge):

  • We need to deploy the template, likely via the Puppet file server.

done

I see you did this in tails::meeting::reminder, which means that if we use this defined resource several times with the default value for the $template parameter, Puppet will complain that the File["${tails::meeting::homedir}/${template}"] resource is defined multiple times. One could argue it's a feature, that will remind us that the default template is only suitable for one type of meetings, and when we reuse this for a 2nd kind of meeting, we'll need to add a new template; so I'm fine with leaving it as-is. What do you think?

Note that '${path}' won't be expanded due to the single quotes. In this case you can simply write $path (puppet-lint would suggest it).

I would use validate_email_address instead of a regexp to validate email addresses. YMMV.

  • The command should find the path of the script and the value of user via ${tails::meeting::...} instead of hard-coding it.

I'm not sure how to access variables from the class above- should I create arguments user and path in tails::meeting for that?

You can write $tails::meeting::user and $tails::meeting::path. I don't remember if they need to be class parameters (user is already OK, path is not) or if any variable is accessible this way. I would say let's try without making path a class parameter and I'll fix it if it breaks :)

I tried to find out how to test the module locally, but I failed. I guess it should somehow work with puppet apply --modulepath=/path/to/tails/modules, which works basically, but it complains that it doesn't find Stdlib::Absolutepath although I have installed puppet-module-puppetlabs-stdlib. Setting --modulepath=/usr/share/puppet/modules.available/puppetlabs-stdlib:/path/to/tails/modules doesn't help.

What about --modulepath=/usr/share/puppet/modules.available:/path/to/tails/modules?

Please let me know if I should just deploy this and fix whatever breaks, or if you want to act on my above review yourself :)

#25 Updated by muri about 1 month ago

  • Assignee changed from muri to intrigeri

Hi intrigeri,

Please let me know if I should just deploy this and fix whatever breaks, or if you want to act on my above review yourself :)

Thanks for the valuable and meaningful review(s)! Please just go ahead and, I don't think we need another roundtrip ;)

#26 Updated by intrigeri about 1 month ago

  • Status changed from Confirmed to In Progress
  • Target version set to Tails_4.1

OK!

My plan is to do this:

  1. Fixup the few things mentioned in my last review
  2. Merge
  3. Use this new code to set up one reminder (FT meeting)
  4. Fix stuff as needed
  5. Once it works, propose migrating the reminders that sajolida manages to our infra
  6. If we reach an agreement, do it.

I hope I'll find time to do the first 3 steps by the end of the month, as opportunities for structured procrastination. I can't promise it'll work, though.

#27 Updated by intrigeri about 1 month ago

Deployed! I've configured a test reminder, sent to me, that should fire tomorrow. Let's see how it goes. Thanks again muri :)

#28 Updated by intrigeri about 1 month ago

The FT meeting reminder is now in production.

I've documented this new facility in https://tails.boum.org/contribute/working_together/roles/sysadmins/#meeting-reminder and will let the community know about it.

#29 Updated by intrigeri about 1 month ago

@sajolida, I'd like to try migrating at least some of the reminder cronjobs, that currently run from your crontab on misc.lizard, to this new facility. The idea is that the code and configuration would then live in a public Git repo, so more than one person can contribute to them. What do you think?

#30 Updated by sajolida about 1 month ago

That would be sweeeet! Here is all the data I have about it:

# m h  dom mon dow   command
0 0 1 * *    ruby $HOME/meeting/meeting.rb --locale en_US.UTF-8 --template $HOME/monthly-report.eml --subject "Preparing the next monthly report" --from "Tails <tails@boum.org>" --to "tails-project@boum.org" 
0 0 1 1,4,7,10 *    ruby $HOME/meeting/meeting.rb --locale en_US.UTF-8 --template $HOME/reimbursements.eml --subject "Reminder about reimbursements and sponsorship to attend events on behalf of Tails" --from "Tails <tails@boum.org>" --to "tails-project@boum.org" 
0 0 1 3,6,9,12 *    bash $HOME/assemble-code-of-conduct.sh ; ruby $HOME/meeting/meeting.rb --locale en_US.UTF-8 --template $HOME/code-of-conduct.eml --subject "Reminder about our Code of Conduct" --from "Tails <tails@boum.org>" --to "tails-project@boum.org tails-dev@boum.org tails-testers@boum.org tails-ux@boum.org tails-l10n@boum.org" 

And template emails in attachment.

#31 Updated by intrigeri about 1 month ago

That would be sweeeet! Here is all the data I have about it:

@sajolida, may I share these scripts with muri?

@muri, would you be interested to adjust the script so it can meet the needs of these other, non-meeting, reminders?
I'm happy to help figuring out what's needed, if you want.

#32 Updated by sajolida about 1 month ago

@sajolida, may I share these scripts with muri?

Sure.

#33 Updated by muri about 1 month ago

intrigeri wrote:

@muri, would you be interested to adjust the script so it can meet the needs of these other, non-meeting, reminders?
I'm happy to help figuring out what's needed, if you want.

Sure! Just send me the the scripts and I'll have look!

#34 Updated by intrigeri about 1 month ago

muri wrote:

intrigeri wrote:

muri, would you be interested to adjust the script so it can meet the needs of these other, non-meeting, reminders?
I'm happy to help figuring out what's needed, if you want.

Sure! Just send me the the scripts and I'll have look!

Done!

#35 Updated by intrigeri about 1 month ago

  • Assignee changed from intrigeri to muri
  • Target version deleted (Tails_4.1)

@muri, please make sure you base this follow-up work on the current puppet-tails.git: I had to change a few things while/after deploying.

#36 Updated by intrigeri about 1 month ago

  • Subject changed from Move our meeting reminder to Puppet to Move our email reminders (e.g. for meetings) to Puppet

Broadening the scope since the next step is to migrate non-meeting reminders :)

Also available in: Atom PDF