Feature #9399: Extend lizard's storage capacity
Evaluate future storage needs on lizard
- isotesterN VMs: at least 45GB/VM, and we want to set up 3 more such VMs; note that storage needs will increase once snapshots are used more intensively in the test suite, which is WIP (#5571), so we should account for that
- autobuilt ISOs: now that we have better estimates of how many branches we're going to build, and how long we're going to keep them (see proposed algorithm on #6439), we should redo our disk storage needs estimates
- Tor Browser archive: will current allocation be sufficient for, say, a year?
- Tails ISO archive: will current allocation be sufficient for, say, a year?
- freezable APT repo
#2 Updated by intrigeri over 4 years ago
- Description updated (diff)
- Status changed from Confirmed to In Progress
- Tor Browser archive:
- we store 2 copies
- current = 10GB (*2) allocated, 6GB (*2) used
- growth = 2GB/month (we've imported 3 alphas of 4.5, so that's perhaps an extreme case, but let's plan for the worst)
- in a year = 6+12*2 = 30GB; but we store two copies => 60GB i.e. +40GB
- performance: can be slow (the subset that's used for ISO builds is served by apt-cacher-ng)
- ISO archive
- we store 2 copies
- current = 60GB (*2) allocated, 40GB (*2) used
- growth = 1.2GB/month
- in a year = 40+12*1.2 = 55GB; but we store two copies => 110GB, that fits into the currently allocated space
- performance: can be slow (not public)
- autobuilt ISOs
- current = 100GB allocated
- peak = (25 not-merged-yet topic branches + 5 main branches) * 15 builds of each = 450 ISOs = 500GB i.e. +400GB
- performance: can be slow (limiting factor is NFS speed)
#5 Updated by intrigeri over 4 years ago
- Assignee changed from intrigeri to bertagaz
- QA Check set to Ready for QA
Only the info from #9401 is now missing.
I think we should also move to faster storage:
apt-cacher-ng's data (it's hit by more and more systems, and we can't allocate enough RAM to it so that e.g. data used by ISO builds would be in memory cache): 20GB
- isobuilderN's Jenkins partition (it contains the workspaces): 2*6GB
- the puppetmaster's data partition (it is hit by more and more systems): 5GB
So, summing up, in a year we'll need, in addition to what we already have:
- slow storage: (Tor Browser archive + ISO archive + autobuilt ISOs + freezable APT repo - bitcoind - acng data - isobuilderN's Jenkins data - puppetmaster's data) = 40 + 0 + 400 + 425 - 52 - 20 - 12 - 5 = 776GB => a couple 1TB disks would be fine; perhaps 2TB would be more future-proof if not crazy expensive
- fast storage: (isotesterN * 3 + bitcoind + acng data + isobuilderN's Jenkins data + puppetmaster's data) = XXX * 3 + 90 + 20 + 12 + 5 = XXX * 3 + 127GB => might fit into a couple 256GB SSDs, if we compromise a bit here and there (e.g. isotesterN's system partition can be on slow storage)
bertagaz, can you please have a look, even if we're still lacking a value for XXX? As said elsewhere already, I think this blocks #6196, so we'd better move quickly on this one.
#6 Updated by intrigeri over 4 years ago
So, we now know XXX: 4GB for the system + 2GB for Git checkout and ISO + 14GB for
/tmp/TailsToaster = 20GB.
=> in a year our additional fast storage needs will be: (isotesterN * 3 + bitcoind + acng data + isobuilderN's Jenkins data + puppetmaster's data) = 20 * 3 + 90 + 20 + 12 + 5 = 187GB => will fit into a couple 256GB SSDs.
So, bertagaz: I'm proposing we by a couple 1 or 2TB rotating drives (depending on the price delta, and also perhaps some 2TB drives are faster due to higher density), and a couple 200-320GB SSDs (focussing on performance + reliability more than on size).
And when we get ourselves another box dedicated to running the test suite, we'll see if we migrate the SSDs there or buy new ones. I'll ask our beloved hardware expert what exact gear he recommends these days, which should give you a bit of time to comment before the order is passed.