Project

General

Profile

Feature #9400

Feature #9399: Extend lizard's storage capacity

Evaluate future storage needs on lizard

Added by intrigeri over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
05/14/2015
Due date:
% Done:

100%

Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

  • isotesterN VMs: at least 45GB/VM, and we want to set up 3 more such VMs; note that storage needs will increase once snapshots are used more intensively in the test suite, which is WIP (#5571), so we should account for that
  • autobuilt ISOs: now that we have better estimates of how many branches we're going to build, and how long we're going to keep them (see proposed algorithm on #6439), we should redo our disk storage needs estimates
  • Tor Browser archive: will current allocation be sufficient for, say, a year?
  • Tails ISO archive: will current allocation be sufficient for, say, a year?
  • freezable APT repo
  • bitcoind

Subtasks

Feature #9401: Evaluate peak working directory size during a full test suite runResolved

Feature #9508: Evaluate freezable APT repo's storage needsResolved


Related issues

Related to Tails - Feature #10243: Update system requirements for lizard failover Resolved 09/25/2015
Related to Tails - Bug #16025: jenkins-data-disk is running out of diskspace again Resolved 10/03/2018

History

#1 Updated by intrigeri over 4 years ago

  • Description updated (diff)

#2 Updated by intrigeri over 4 years ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress
  • Tor Browser archive:
    • we store 2 copies
    • current = 10GB (*2) allocated, 6GB (*2) used
    • growth = 2GB/month (we've imported 3 alphas of 4.5, so that's perhaps an extreme case, but let's plan for the worst)
    • in a year = 6+12*2 = 30GB; but we store two copies => 60GB i.e. +40GB
    • performance: can be slow (the subset that's used for ISO builds is served by apt-cacher-ng)
  • ISO archive
    • we store 2 copies
    • current = 60GB (*2) allocated, 40GB (*2) used
    • growth = 1.2GB/month
    • in a year = 40+12*1.2 = 55GB; but we store two copies => 110GB, that fits into the currently allocated space
    • performance: can be slow (not public)
  • autobuilt ISOs
    • current = 100GB allocated
    • peak = (25 not-merged-yet topic branches + 5 main branches) * 15 builds of each = 450 ISOs = 500GB i.e. +400GB
    • performance: can be slow (limiting factor is NFS speed)

#3 Updated by intrigeri over 4 years ago

  • bitcoind
    • current = 52GB allocated, 43GB used
    • growth = roughly doubles each year
    • in a year = 90GB i.e. +40GB
    • performance: should be fast or on separate storage from services that must be fast themselves

#4 Updated by intrigeri over 4 years ago

  • freezable APT repo:
    • current = 0
    • in a year = 425GB, i.e. +425GB
    • performance: can be slow (the subset that's used for ISO builds is served by apt-cacher-ng)

#5 Updated by intrigeri over 4 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check set to Ready for QA

Only the info from #9401 is now missing.

I think we should also move to faster storage:

  • apt-cacher-ng's data (it's hit by more and more systems, and we can't allocate enough RAM to it so that e.g. data used by ISO builds would be in memory cache): 20GB
  • isobuilderN's Jenkins partition (it contains the workspaces): 2*6GB
  • the puppetmaster's data partition (it is hit by more and more systems): 5GB

So, summing up, in a year we'll need, in addition to what we already have:

  • slow storage: (Tor Browser archive + ISO archive + autobuilt ISOs + freezable APT repo - bitcoind - acng data - isobuilderN's Jenkins data - puppetmaster's data) = 40 + 0 + 400 + 425 - 52 - 20 - 12 - 5 = 776GB => a couple 1TB disks would be fine; perhaps 2TB would be more future-proof if not crazy expensive
  • fast storage: (isotesterN * 3 + bitcoind + acng data + isobuilderN's Jenkins data + puppetmaster's data) = XXX * 3 + 90 + 20 + 12 + 5 = XXX * 3 + 127GB => might fit into a couple 256GB SSDs, if we compromise a bit here and there (e.g. isotesterN's system partition can be on slow storage)

bertagaz, can you please have a look, even if we're still lacking a value for XXX? As said elsewhere already, I think this blocks #6196, so we'd better move quickly on this one.

#6 Updated by intrigeri over 4 years ago

So, we now know XXX: 4GB for the system + 2GB for Git checkout and ISO + 14GB for /tmp/TailsToaster = 20GB.

=> in a year our additional fast storage needs will be: (isotesterN * 3 + bitcoind + acng data + isobuilderN's Jenkins data + puppetmaster's data) = 20 * 3 + 90 + 20 + 12 + 5 = 187GB => will fit into a couple 256GB SSDs.

So, bertagaz: I'm proposing we by a couple 1 or 2TB rotating drives (depending on the price delta, and also perhaps some 2TB drives are faster due to higher density), and a couple 200-320GB SSDs (focussing on performance + reliability more than on size).

And when we get ourselves another box dedicated to running the test suite, we'll see if we migrate the SSDs there or buy new ones. I'll ask our beloved hardware expert what exact gear he recommends these days, which should give you a bit of time to comment before the order is passed.

#7 Updated by intrigeri over 4 years ago

intrigeri wrote:

I'll ask our beloved hardware expert what exact gear he recommends these days [...]

Done.

#8 Updated by intrigeri over 4 years ago

Current best option seems to be 2*1TB Samsung 850EVO SSDs.

#9 Updated by intrigeri over 4 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • QA Check changed from Ready for QA to Pass

bertagaz privately ack'ed the current proposal. Next steps will be tracked in new, dedicated tickets.

#10 Updated by intrigeri over 4 years ago

  • Blocks Feature #9551: Have more storage plugged into lizard added

#11 Updated by intrigeri over 4 years ago

  • Blocks deleted (Feature #9551: Have more storage plugged into lizard)

#12 Updated by intrigeri almost 4 years ago

  • Related to Feature #10243: Update system requirements for lizard failover added

#13 Updated by bertagaz 12 months ago

  • Related to Bug #16025: jenkins-data-disk is running out of diskspace again added

Also available in: Atom PDF