Project

General

Profile

Feature #11806

Bug #11680: Upgrade server hardware (2017-2019 edition)

Update server storage planning needs for at least 2017

Added by intrigeri over 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
09/19/2016
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

This is basically #10851, 2017 edition. See parent ticket for other things that need to be taken into account.


Related issues

Related to Tails - Feature #10851: Give lizard enough free storage to host our freezable APT repository Resolved 01/04/2016
Related to Tails - Bug #12008: torbrowser archive disks are running out of disk space Resolved 12/01/2016
Related to Tails - Bug #12119: Shrink lizard's jenkins-data LV Resolved 01/07/2017
Related to Tails - Bug #12469: Resize ISO history and Tor Browser archive LVs Resolved 04/22/2017
Related to Tails - Feature #15780: Update server storage planning needs for 2019-2020 Rejected 08/09/2018
Blocked by Tails - Feature #12002: Estimate hardware cost of reproducible builds in Jenkins Resolved 11/28/2016
Blocks Tails - Feature #12111: APT snapshots: add arm64 architecture In Progress 01/03/2017
Blocks Tails - Feature #13284: Core work: Sysadmin (Adapt our infrastructure) Confirmed 06/30/2017
Blocks Tails - Bug #13425: Upgrade lizard's storage (2017 edition) Resolved 07/05/2017

History

#1 Updated by intrigeri over 2 years ago

  • Related to Feature #10851: Give lizard enough free storage to host our freezable APT repository added

#2 Updated by intrigeri over 2 years ago

  • Description updated (diff)

#3 Updated by anonym over 2 years ago

  • Target version changed from 284 to Tails 2.10

#4 Updated by intrigeri over 2 years ago

  • Related to Bug #12008: torbrowser archive disks are running out of disk space added

#5 Updated by intrigeri over 2 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 0 to 20
  • QA Check set to Ready for QA

Added a new, updated spreadsheet in sysadmin.git (storage-2017-ticket_11806.ods). My conclusion is that we should be good until the end of 2018, without purchasing any additional storage, if:

  • we reclaim unused space (#12119)
  • we make use of 2/3 of the space currently unallocated in our PVs
  • my estimates are mostly correct; in particular the ones about space needed for reproducible builds (Vagrant baseboxes) are rough guesses, because the current design is still stored on your hard drive only and I can't read it (#11988); 5GB should be enough to store a dozen baseboxes (and more likely 20); I don't know if we want to create & keep more baseboxes than that in 2 years

Please review and reassign to me. We can finalize this once #11988 is done.

#6 Updated by intrigeri over 2 years ago

  • Related to Bug #12119: Shrink lizard's jenkins-data LV added

#7 Updated by anonym over 2 years ago

  • Target version changed from Tails 2.10 to Tails_2.11

#8 Updated by bertagaz about 2 years ago

  • Target version changed from Tails_2.11 to Tails_2.12

#9 Updated by bertagaz about 2 years ago

  • Priority changed from Normal to Elevated

#10 Updated by intrigeri about 2 years ago

  • Related to Feature #12111: APT snapshots: add arm64 architecture added

#11 Updated by intrigeri about 2 years ago

  • Target version changed from Tails_2.12 to Tails_3.0~rc1

So, 2.12 wasn't realistic, that's fine. Do you think you can handle this during your next sysadmin shift (weeks 17-19)? Let's make sure we're not going to run out of disk space and avoid having to deal with such problems in a hurry :)

#12 Updated by intrigeri about 2 years ago

  • Related to Bug #12469: Resize ISO history and Tor Browser archive LVs added

#13 Updated by bertagaz about 2 years ago

  • Target version changed from Tails_3.0~rc1 to Tails_3.0

#14 Updated by intrigeri about 2 years ago

  • Blocked by Feature #12002: Estimate hardware cost of reproducible builds in Jenkins added

#15 Updated by intrigeri about 2 years ago

Once reviewed + #12002 is done, please reassign to me as "Dev Needed" so I update the plan taking into account the Vagrant thing :)

#16 Updated by intrigeri about 2 years ago

I've updated the spreadsheet in Git to take #12111 into account, so please make sure you review the latest version :)

#17 Updated by intrigeri about 2 years ago

  • Related to deleted (Feature #12111: APT snapshots: add arm64 architecture)

#18 Updated by intrigeri about 2 years ago

#19 Updated by intrigeri almost 2 years ago

  • Target version changed from Tails_3.0 to Tails_3.1

Blocked by #12002 (that has no target version other than "2017") and then in turn by some upcoming changes in our CI system that are in the early research phase, so postponing.

#20 Updated by intrigeri almost 2 years ago

  • Priority changed from Elevated to High

We're short on disk space on several partitions, and can't grow them as much as we would like as we need more storage and we're blocking on #12002 and then on #11806 => bumping priority.

#21 Updated by intrigeri almost 2 years ago

  • Blocks Feature #13284: Core work: Sysadmin (Adapt our infrastructure) added

#22 Updated by bertagaz almost 2 years ago

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 20 to 30
  • QA Check changed from Ready for QA to Info Needed

intrigeri wrote:

We're short on disk space on several partitions, and can't grow them as much as we would like as we need more storage and we're blocking on #12002 and then on #11806 => bumping priority.

Your estimates sound good so far for me. I've pushed a newer revision that:

  • Add the growth with 4 new isobuilders root partitions to the equation. Not so much to count in the end, but I wanted to add this root partitions line.
  • Add the isobuilders libvirt partitions (but that's only 80G)
  • Update from 200G to 400G to include the growth of the Jenkins partitions for 8 isobuilders in the "reproducible builds" line.
  • Did not touch the "Jenkins data" line (artifacts), as we are currently using half only of it, and the 20% growth ratio we agreed on should not fill this partition that much.

Still missing the APT snapshots updates from #12002, but worth having a look already if you wish so.

#23 Updated by bertagaz almost 2 years ago

bertagaz wrote:

intrigeri wrote:

We're short on disk space on several partitions, and can't grow them as much as we would like as we need more storage and we're blocking on #12002 and then on #11806 => bumping priority.

Your estimates sound good so far for me. I've pushed a newer revision that:

  • Add the growth with 4 new isobuilders root partitions to the equation. Not so much to count in the end, but I wanted to add this root partitions line.
  • Add the isobuilders libvirt partitions (but that's only 80G)
  • Update from 200G to 400G to include the growth of the Jenkins partitions for 8 isobuilders in the "reproducible builds" line.
  • Did not touch the "Jenkins data" line (artifacts), as we are currently using half only of it, and the 20% growth ratio we agreed on should not fill this partition that much.

Still missing the APT snapshots updates from #12002, but worth having a look already if you wish so.

Added storage-2017-ticket_11806_2017Q2.ods in the same Git repo (different file so that you can easily compare both). It adds numbers for the first half of 2017, which leads to refine a bit the estimates for 2018. Also added an estimate for APT snapshots growth with a pestimistic approach.

IIRC the plan was to use the last 2 HDD slots on lizard's case to add 2x2T disks. According to this estimate it would work and leave a bit of disk space.

#24 Updated by intrigeri almost 2 years ago

  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 30 to 40
  • QA Check deleted (Info Needed)

Reviewed, fixed/update/improved some stuff. Comments & questions:

  • what's "VM's root" exactly? Is that only isobuilders or all VMs? Please clarify on the spreadsheet.
  • in some places I was confused as the estimates for isobuilders there were not the same as what you wrote on the blueprint for #12002, so please pick one set of estimates and apply it consistently to both that blueprint and the spreadsheet (probably the one that takes into account future growth as that is part of the cost of reproducible builds);
  • I've added 100G of slack to take into account Piwik (see the corresponding thread).
  • I've bumped a bit the APT snapshots estimate to better take #12111 into account.

So indeed, at first glance we're looking at adding about 2TB of storage. I don't know if we can add 2 more drives though, and I would like to retire our two 5-yeard-old rotating hard-drives anyway, but let's not bother speculating until we have the final number.

#25 Updated by bertagaz almost 2 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check set to Info Needed

intrigeri wrote:

Reviewed, fixed/update/improved some stuff. Comments & questions:

  • what's "VM's root" exactly? Is that only isobuilders or all VMs? Please clarify on the spreadsheet.

I meant all the VMs, including the possibly 4 new isobuilders. Fixed in the file.

  • in some places I was confused as the estimates for isobuilders there were not the same as what you wrote on the blueprint for #12002, so please pick one set of estimates and apply it consistently to both that blueprint and the spreadsheet (probably the one that takes into account future growth as that is part of the cost of reproducible builds);

Not sure which place you're talking about in this file. Maybe the difference you see is because there I've counted 8 isobuilders in the estimates for 2018?

  • I've added 100G of slack to take into account Piwik (see the corresponding thread).
  • I've bumped a bit the APT snapshots estimate to better take #12111 into account.

Good catch!

So indeed, at first glance we're looking at adding about 2TB of storage. I don't know if we can add 2 more drives though, and I would like to retire our two 5-yeard-old rotating hard-drives anyway, but let's not bother speculating until we have the final number.

The case we have is supposed to be able to handle two more disks, but we should ask. It looks like that in the end of 2018, we'll have (for now) only 400G left, so I wonder if we shouldn't just add two disk, and then think later about upgrading the old rotating one to get more disk space.

#26 Updated by intrigeri almost 2 years ago

  • Blocks Bug #13425: Upgrade lizard's storage (2017 edition) added

#27 Updated by intrigeri almost 2 years ago

  • Status changed from In Progress to Resolved
  • what's "VM's root" exactly? Is that only isobuilders or all VMs? Please clarify on the spreadsheet.

I meant all the VMs, including the possibly 4 new isobuilders. Fixed in the file.

Thank you :)

  • in some places I was confused as the estimates […]

Not sure which place you're talking about in this file. Maybe the difference you see is because there I've counted 8 isobuilders in the estimates for 2018?

Yes, probably. I think I might have written my last comment before having fully understood this.

The case we have is supposed to be able to handle two more disks, but we should ask. It looks like that in the end of 2018, we'll have (for now) only 400G left, so I wonder if we shouldn't just add two disk, and then think later about upgrading the old rotating one to get more disk space.

Let's discuss that on #13425, which is about addressing the problem this ticket defines.

So I think we're done here, woohoo!

Conclusion: adding 1.7 TB of storage should be enough to satisfy our needs until the end of 2018.

#28 Updated by intrigeri almost 2 years ago

  • Assignee deleted (intrigeri)
  • % Done changed from 40 to 100
  • QA Check changed from Info Needed to Pass

#29 Updated by intrigeri 10 months ago

  • Related to Feature #15780: Update server storage planning needs for 2019-2020 added

Also available in: Atom PDF