Project

General

Profile

Bug #17288

ENOSPC during build, while upgrading the Vagrant box

Added by CyrilBrulebois 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
feature/15342-cache-wiki
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Just got ENOSPC on the apt-cacher-ng space:

==> default: ================
==> default: Machine id: 2fa0c2c5-1287-428c-9dc7-c00ddbd6084e
==> default: Should be mounting folders
==> default:  /amnesia.git, opts: {:type=>:"9p", :readonly=>true, :guestpath=>"/amnesia.git", :hostpath=>"/home/kibi/work/clients/tails/tails.git", :disabled=>false, :__vagrantfile=>true, :target=>"/amnesia.git", :accessmode=>"passthrough", :mount=>true, :mount_tag=>"c751b9959246f02ca2b70c18120244c"}
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Configuring and enabling network interfaces...
==> default: mounting p9 share in guest
==> default: Running provisioner: shell...
    default: Running: inline script
==> default: I: Configuring HTTP proxy...
==> default: /dev/vdb1: 20612/983040 files (2.6% non-contiguous), 3923212/3932159 blocks
==> default: I: Updating debian-security APT source...
==> default: I: Current APT sources are:
==> default: deb http://time-based.snapshots.deb.tails.boum.org/debian/2019100904 buster main
==> default: deb http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201/ buster/updates main
==> default: deb http://time-based.snapshots.deb.tails.boum.org/debian/2019100904 buster-updates main
==> default: deb http://time-based.snapshots.deb.tails.boum.org/tails/2019102001/ builder-jessie main
==> default: I: Upgrading system...
==> default: Warning: apt-key output should not be parsed (stdout is not a terminal)
==> default: OK
==> default: Get:1 http://time-based.snapshots.deb.tails.boum.org/debian/2019100904 buster InRelease [5444 B]
==> default: Get:2 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates InRelease [5404 B]
==> default: Get:3 http://time-based.snapshots.deb.tails.boum.org/debian/2019100904 buster-updates InRelease [5313 B]
==> default: Get:4 http://time-based.snapshots.deb.tails.boum.org/tails/2019102001 builder-jessie InRelease [5308 B]
==> default: Get:5 http://time-based.snapshots.deb.tails.boum.org/debian/2019100904 buster/main amd64 Packages [16.9 MB]
==> default: Get:6 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates/main amd64 Packages [262 kB]
==> default: Get:7 http://time-based.snapshots.deb.tails.boum.org/debian/2019100904 buster-updates/main amd64 Packages [9432 B]
==> default: Get:8 http://time-based.snapshots.deb.tails.boum.org/tails/2019102001 builder-jessie/main amd64 Packages [4258 B]
==> default: Fetched 17.2 MB in 5s (3532 kB/s)
==> default: Reading package lists...
==> default: Reading package lists...
==> default: Building dependency tree...
==> default: 
==> default: Reading state information...
==> default: Calculating upgrade...
==> default: The following packages will be upgraded:
==> default:   file libmagic-mgc libmagic1 libssl1.1 linux-image-4.19.0-6-amd64 openssl
==> default:   sudo
==> default: 7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
==> default: Need to get 52.0 MB of archives.
==> default: After this operation, 46.1 kB of additional disk space will be used.
==> default: Get:1 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates/main amd64 libssl1.1 amd64 1.1.1d-0+deb10u2 [1538 kB]
==> default: Get:2 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates/main amd64 file amd64 1:5.35-4+deb10u1 [66.4 kB]
==> default: Get:3 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates/main amd64 libmagic1 amd64 1:5.35-4+deb10u1 [117 kB]
==> default: Get:4 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates/main amd64 libmagic-mgc amd64 1:5.35-4+deb10u1 [242 kB]
==> default: Err:5 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates/main amd64 linux-image-4.19.0-6-amd64 amd64 4.19.67-2+deb10u2
==> default:   503  OUT OF DISK SPACE [IP: 127.0.0.1 3142]
==> default: Err:6 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates/main amd64 openssl amd64 1.1.1d-0+deb10u2
==> default:   503  OUT OF DISK SPACE [IP: 127.0.0.1 3142]
==> default: Err:7 http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201 buster/updates/main amd64 sudo amd64 1.8.27-1+deb10u1
==> default:   503  OUT OF DISK SPACE [IP: 127.0.0.1 3142]
==> default: Fetched 1965 kB in 4s (448 kB/s)
==> default: E: Failed to fetch http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201/pool/updates/main/l/linux-signed-amd64/linux-image-4.19.0-6-amd64_4.19.67-2+deb10u2_amd64.deb  503  OUT OF DISK SPACE [IP: 127.0.0.1 3142]
==> default: E: Failed to fetch http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201/pool/updates/main/o/openssl/openssl_1.1.1d-0+deb10u2_amd64.deb  503  OUT OF DISK SPACE [IP: 127.0.0.1 3142]
==> default: E: Failed to fetch http://time-based.snapshots.deb.tails.boum.org/debian-security/2019120201/pool/updates/main/s/sudo/sudo_1.8.27-1+deb10u1_amd64.deb  503  OUT OF DISK SPACE [IP: 127.0.0.1 3142]
==> default: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.
rake aborted!
VagrantCommandError: 'vagrant ["up"]' command failed with exit status 1
/home/kibi/work/clients/tails/tails.git/Rakefile:116:in `rescue in run_vagrant'
/home/kibi/work/clients/tails/tails.git/Rakefile:114:in `run_vagrant'
/home/kibi/work/clients/tails/tails.git/Rakefile:574:in `block (2 levels) in <top (required)>'
CommandError: command ["vagrant", "up", {:chdir=>"./vagrant"}] failed with exit status 1
/home/kibi/work/clients/tails/tails.git/Rakefile:76:in `run_command'
/home/kibi/work/clients/tails/tails.git/Rakefile:114:in `run_vagrant'
/home/kibi/work/clients/tails/tails.git/Rakefile:574:in `block (2 levels) in <top (required)>'
Tasks: TOP => build => vm:up
(See full trace by running task with --trace)

Running either of those trimmed down space to something more reasonable (from 100%):

# /usr/lib/apt-cacher-ng/acngtool shrink 10G -f
/dev/vdb1                         15G   12G  2.9G  81% /var/cache/apt-cacher-ng

# /usr/lib/apt-cacher-ng/acngtool shrink 8G -f
/dev/vdb1                         15G  9.9G  4.9G  68% /var/cache/apt-cacher-ng

I don't think I should have to do that on my own…


Related issues

Related to Tails - Bug #16020: acngtool shrink is insufficient to maintain acng cache size Resolved 10/02/2018
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision 1fb17215 (diff)
Added by intrigeri 3 months ago

Build system: shrink the apt-cacher-ng cache after a successful build too (refs: #17288)

7f0a1f0e42792cc715f28f8566c98fa1e6bacda3 moved this cache cleanup operation from
the provisioning script to build-tails, so it's run before each build. This is
great and improves things in most cases. But there's one situation in which
this change makes things worse:

1. Before build N, the cache was approaching the 10G limit.
2. Build N has filled the cache.
3. While we provision the VM for build N+1, APT operations fail:
apt-cacher-ng returns a "503 OUT OF DISK SPACE" error.

To mitigate this, let's do garbage collection both before and after
a build.

Revision f5c75d4e
Added by intrigeri 3 months ago

Merge branch 'feature/15342-cache-wiki' into stable (Closes: #15342, #17288, #16607, #17289, #11411)

History

#1 Updated by intrigeri 4 months ago

  • Related to Bug #16020: acngtool shrink is insufficient to maintain acng cache size added

#2 Updated by intrigeri 4 months ago

#3 Updated by intrigeri 4 months ago

OK, so this time (compared to #16020) the failure happens while upgrading the build VM, in vagrant/provision/setup-tails-builder, instead of later on in vagrant/provision/assets/build-tails.

What about moving the acng cache trimming from vagrant/provision/assets/build-tails to somewhere in vagrant/provision/setup-tails-builder before I: Upgrading system...?

I don't use the vmproxy myself so I'm not well placed to work on this.

#4 Updated by intrigeri 3 months ago

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Feature Branch set to feature/15342-cache-wiki

#5 Updated by intrigeri 3 months ago

  • Subject changed from ENOSPC while building to ENOSPC during build, while upgrading the Vagrant box

#6 Updated by intrigeri 3 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)
  • Target version set to Tails_4.2

#7 Updated by intrigeri 3 months ago

  • Target version changed from Tails_4.2 to Tails_4.3

#8 Updated by segfault 3 months ago

  • Assignee set to segfault

#9 Updated by intrigeri 3 months ago

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF