Project

General

Profile

Feature #11817

Optimize I/O settings on lizard

Added by intrigeri almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
09/20/2016
Due date:
% Done:

100%

Feature Branch:
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description


Related issues

Blocks Tails - Feature #11179: Enable automatic NUMA balancing on lizard Resolved 02/29/2016

History

#1 Updated by intrigeri almost 3 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

intrigeri wrote:

  • I/O scheduler = noop in KVM guests

Done.

#3 Updated by intrigeri almost 3 years ago

This will require a Puppet module to manage sysfs (https://github.com/github/puppet-sysfs or https://github.com/luxflux/puppet-sysfs), or some tiny systemd unit file that runs a few echo commands.

#4 Updated by intrigeri almost 3 years ago

  • % Done changed from 10 to 20

intrigeri wrote:

This will require a Puppet module to manage sysfs (https://github.com/github/puppet-sysfs or https://github.com/luxflux/puppet-sysfs), or some tiny systemd unit file that runs a few echo commands.

Done as well.

I'll come back to it in a few weeks to check the munin graphs.

#5 Updated by intrigeri almost 3 years ago

  • Description updated (diff)

#6 Updated by intrigeri almost 3 years ago

  • Description updated (diff)

#7 Updated by intrigeri almost 3 years ago

Left to do (before benchmarking): set io=native for all VM disks.

#8 Updated by intrigeri almost 3 years ago

  • % Done changed from 20 to 30

Left to do (before benchmarking): set io=native for all VM disks.

Done as well. I'll come back to it in a few weeks to check the munin graphs.

#9 Updated by intrigeri almost 3 years ago

  • Blocks Feature #11179: Enable automatic NUMA balancing on lizard added

#10 Updated by intrigeri almost 3 years ago

We might also want:

  • virtio-blk "data plane", that is supported in libvirt 1.2.8+ (see new syntax in http://wiki.qemu.org/ChangeLog/2.1) via the iothread disk attribute and the iothreads domain attribute; see https://wiki.mikejung.biz/KVM_/_Xen#virtio-blk_iothreads_.28x-data-plane.29; this apparently implies using cache=directsync
  • guest clocksource: TSC (found one single reference, not backed by any explanation)
  • disable delayed accounting (nodelayacct) (found one single reference, and others that say it has very little impact)
  • modprobe virtio_blk use_bio=1 on SSD-backed guests (found one single reference, and parameter not available anyway at least on Debian)

#12 Updated by intrigeri almost 3 years ago

Once all this is done I'll update our VM creation doc accordingly.

#13 Updated by intrigeri almost 3 years ago

Tried to enable iothreads (lizard's etckeeper ae7bbf3) on apt.lizard and bitcoin.lizard. The resulting QEMU command line looks good, but virsh qemu-monitor-command --hmp bitcoin 'info qtree' says that x-data-plane is off, which might or might not be expected with iothreads enabled (perhaps that's just because the interface to these options changed in QEMU 2.1). I'll check Munin graphs in a week or three and see if this impacts anything.

#14 Updated by intrigeri almost 3 years ago

  • % Done changed from 40 to 50

intrigeri wrote:

Tried to enable iothreads (lizard's etckeeper ae7bbf3) on apt.lizard and bitcoin.lizard. The resulting QEMU command line looks good, but virsh qemu-monitor-command --hmp bitcoin 'info qtree' says that x-data-plane is off, which might or might not be expected with iothreads enabled (perhaps that's just because the interface to these options changed in QEMU 2.1). I'll check Munin graphs in a week or three and see if this impacts anything.

So iothreads seem to help. Next steps:

  • enable iothreads on all VMs
  • document all the tweaks done on this ticket in our VM creation doc

#15 Updated by intrigeri almost 3 years ago

  • % Done changed from 50 to 60

Next steps are still:

  • enable iothreads on all VMs
  • document all the tweaks done on this ticket in our VM creation doc

#16 Updated by intrigeri almost 3 years ago

  • enable iothreads on all VMs

Done (commit 2217164 in lizard's etckeeper). Left to do:

  • shutdown + start jenkins.lizard + all isobuilders so that they get the new settings
  • document all the tweaks done on this ticket in our VM creation doc

#17 Updated by intrigeri almost 3 years ago

  • shutdown + start jenkins.lizard + all isobuilders so that they get the new settings

Still the case. And then:

  • Wait for a week and check Munin.
  • document all the tweaks done on this ticket in our VM creation doc

Done. I'll ask bertagaz to review once the rest is done as well.

#18 Updated by intrigeri almost 3 years ago

  • % Done changed from 60 to 70

#19 Updated by intrigeri almost 3 years ago

  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 70 to 80
  • QA Check set to Ready for QA
  • shutdown + start jenkins.lizard + all isobuilders so that they get the new settings

Done. Time to review!

#22 Updated by bertagaz almost 3 years ago

  • Target version changed from Tails_2.7 to Tails_2.9.1

#23 Updated by anonym almost 3 years ago

  • Target version changed from Tails_2.9.1 to Tails 2.10

#24 Updated by anonym over 2 years ago

  • Target version changed from Tails 2.10 to Tails_2.11

#25 Updated by bertagaz over 2 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (bertagaz)
  • % Done changed from 80 to 100
  • QA Check deleted (Ready for QA)

intrigeri wrote:

  • shutdown + start jenkins.lizard + all isobuilders so that they get the new settings

Done. Time to review!

Munin shows better stats indeed, nice work!

Also available in: Atom PDF