Project

General

Profile

Feature #6729

Bump the number of CPU cores the testing VM has

Added by intrigeri over 5 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
Start date:
02/19/2014
Due date:
% Done:

100%

Feature Branch:
test/10971-more-CPUs-for-TailsToaster
Type of work:
Code
Starter:
No
Affected tool:

Description

This would allow running it faster, on hardware that can take giving >1 CPU cores to TailsToaster. Also, >1 CPU cores is closer to most hardware users run Tails on, and we've already seen situation where this matters (e.g. #11507).


Related issues

Related to Tails - Feature #10971: Try giving 6 vcpus to each isotester on lizard Resolved 01/18/2016
Related to Tails - Feature #11113: Run more ISO testers Resolved 02/12/2016
Related to Tails - Feature #11011: Design a 2nd ISO testing machine Rejected 01/27/2016
Blocks Tails - Bug #10442: Totem "Watching a WebM video over HTTPS" test never passes on Jenkins Resolved 10/28/2015

Associated revisions

Revision 08c08e67 (diff)
Added by anonym over 3 years ago

Disable WebM test since the feature is broken in the Tor Browser.

... but only on a single core system, which is what we happen to use
in our automated test suite VM at the moment.

Refs: #11507, #6729

Revision 11a7a9a4 (diff)
Added by anonym over 3 years ago

Revert "Disable WebM test since the feature is broken in the Tor Browser."

This reverts commit 08c08e67937f3d6fadd266dc2ab43c0877b2963e.

With more than 1 core in the tester VM, WebM is no longer broken in
Tails, so the test can be enabled again.

Refs: #6729, #11507

Revision ce14f658 (diff)
Added by intrigeri about 3 years ago

Revert "Revert "Test suite: give the system under testing 2 vCPUs.""

This reverts commit 110557baa6bcee48a02f24cb466ae35ae929f249.

refs: #6729

Revision bd961bd8
Added by anonym about 3 years ago

Merge remote-tracking branch 'origin/feature/from-intrigeri-for-2.6' into devel

Fix-committed: #5650, #6729, #6850, #8485, #10190, #10298, #10733, #10733, #11281, #11588, #11582, #11590

History

#1 Updated by intrigeri about 4 years ago

On my dev system, testing an ISO built from current stable, test suite from current stable branch as well:

  • evince.feature, --view, no video capture:
    • 1 vcpu: 8:54, 9:10, 8:55
    • 2 vcpus: 8:21, 8:01, 8:02
  • evince.feature, no --view, --capture enabled:
    • 1 vcpu: 9:26, 9:47, 10:18
    • 2 vcpus: 9:04, 9:26, 8:41

No idea if this would be useful for anyone else (e.g. we would have to benchmark this on isotesterN.lizard), but at least for me it would save quite some test suite runtime.

#2 Updated by intrigeri about 4 years ago

Implementation hints: see set_ram_size for how to dynamically patch the VM's config. But in this case it should probably be done in the "a computer" step, depending on the configuration.

#3 Updated by kytv about 4 years ago

This probably isn't "good" but it seems to work.

--- a/features/step_definitions/common_steps.rb
+++ b/features/step_definitions/common_steps.rb
@@ -77,6 +77,7 @@ end
 Given /^a computer$/ do
   @vm.destroy_and_undefine if @vm
   @vm = VM.new($virt, VM_XML_PATH, $vmnet, $vmstorage, DISPLAY)
+  @vm.set_number_of_cpus($config['NUMBER_OF_CPUS']) if $config['NUMBER_OF_CPUS']
 end

 Given /^the computer has (\d+) ([[:alpha:]]+) of RAM$/ do |size, unit|
diff --git a/features/support/helpers/vm_helper.rb b/features/support/helpers/vm_helper.rb
index d838130..0c219a5 100644
--- a/features/support/helpers/vm_helper.rb
+++ b/features/support/helpers/vm_helper.rb
@@ -293,6 +293,18 @@ class VM
     return list
   end

+  def set_number_of_cpus(num_of_cpus)
+    raise "The number of CPUs can only be set for inactive vms" if is_running?
+
+    cpus_on_host = File.read('/proc/cpuinfo').scan(/^processor\s*:/).size.to_i
+    num_of_cpus <= cpus_on_host ? use_cpus = num_of_cpus : use_cpus = cpus_on_host
+
+    domain_xml = REXML::Document.new(@domain.xml_desc)
+    domain_xml.elements['domain/vcpu'].text = use_cpus
+    domain_xml.elements['domain/vcpu'].attributes['placement'] = 'static'
+    update(domain_xml.to_s)
+  end
+
   def set_ram_size(size, unit = "KiB")
     raise "System memory can only be added to inactice vms" if is_running?
     domain_xml = REXML::Document.new(@domain.xml_desc)

#4 Updated by intrigeri about 4 years ago

  • Status changed from Confirmed to In Progress

#5 Updated by intrigeri over 3 years ago

test/10971-more-CPUs-for-TailsToaster gives 2 vcpus to TailsToaster. This will allow us to measure if it makes any difference on Jenkins.

#6 Updated by intrigeri over 3 years ago

  • Related to Feature #10971: Try giving 6 vcpus to each isotester on lizard added

#8 Updated by intrigeri over 3 years ago

The last 10 successful test suite runs (between Feb 14 and 26) on the devel branch took 56.6 minutes on average. The last 10 successful test suite runs (between Feb 16 and 26) on that branch took 48.2 minutes on average. That's almost 15% less. Of course, this doesn't tell us that our infrastructure would be able to run more tests, or run tests faster, if we gave all TailsToaster's a 2nd CPU, but still: this encourages me to take this idea into account while working on #11011 and #11113.

#9 Updated by intrigeri over 3 years ago

#10 Updated by intrigeri over 3 years ago

#11 Updated by intrigeri over 3 years ago

I've merged this branch into experimental, and created https://jenkins.tails.boum.org/job/manual_test_Tails_ISO_experimental/ for easier benchmarking.

#12 Updated by intrigeri over 3 years ago

  • Blueprint set to https://tails.boum.org/blueprint/hardware_for_automated_tests_take2/#benchmarks

#13 Updated by intrigeri over 3 years ago

  • Assignee set to intrigeri
  • Target version set to Tails_2.9.1

According to my benchmarks, giving TailsToaster a 2nd vcpu decreases the worst case test suite run time (when running the test suite on all isotesters in parallel while building ISOs on all isobuilders) by about 17% on lizard. However, when doing so I see a weird test suite failure here and there, that I would not see without a 2nd vcpu, which is not very confidence-inspiring. So I say let's come back to it later, and see if we need to squeeze some more performance out of our current setup (after the great improvements we already got on #11009).

#14 Updated by intrigeri over 3 years ago

  • % Done changed from 0 to 10

#15 Updated by intrigeri over 3 years ago

  • Related to Bug #10442: Totem "Watching a WebM video over HTTPS" test never passes on Jenkins added

#16 Updated by intrigeri over 3 years ago

anonym, given one step of the test suite became fragile with Tor Browser 6.x unless you give 2 vcpus to TailsToaster, maybe you want to take this one (or the cheap version that only bumps the number of vcpus to 2) for 2.4?

#17 Updated by intrigeri over 3 years ago

  • Related to deleted (Bug #10442: Totem "Watching a WebM video over HTTPS" test never passes on Jenkins)

#18 Updated by intrigeri over 3 years ago

  • Blocks Bug #10442: Totem "Watching a WebM video over HTTPS" test never passes on Jenkins added

#19 Updated by anonym over 3 years ago

  • Subject changed from Allow configuring how many CPU cores the testing VM gets to Bump the numbers of CPU cores the testing VM has
  • Description updated (diff)
  • Assignee changed from intrigeri to anonym
  • Target version changed from Tails_2.9.1 to Tails_2.6

intrigeri wrote:

anonym, given one step of the test suite became fragile with Tor Browser 6.x unless you give 2 vcpus to TailsToaster, maybe you want to take this one (or the cheap version that only bumps the number of vcpus to 2) for 2.4?

I'll take it, but with a less optimistic schedule. I want more data!

Also, I think it's clear to us that we shouldn't allow configuring the number of cores, since it clearly has an effect on tests. Instead, we should fix on some N number of cores, and N = 2 seems reasonable.

#20 Updated by intrigeri over 3 years ago

Also, I think it's clear to us that we shouldn't allow configuring the number of cores, since it clearly has an effect on tests. Instead, we should fix on some N number of cores, and N = 2 seems reasonable.

ACK, please retitle accordingly :)

#21 Updated by anonym over 3 years ago

  • Subject changed from Bump the numbers of CPU cores the testing VM has to Bump the number of CPU cores the testing VM has

intrigeri wrote:

Also, I think it's clear to us that we shouldn't allow configuring the number of cores, since it clearly has an effect on tests. Instead, we should fix on some N number of cores, and N = 2 seems reasonable.

ACK, please retitle accordingly :)

Actually I did. :) Noticed a typo, though.

#23 Updated by intrigeri about 3 years ago

The test/10971-more-CPUs-for-TailsToaster+11588+10733 branch seems to fix the "timeout after clicking Login in the Greeter" issue that I see on my branches that re-enable USB tests:

So I'm now starting to thing that instead of trying to fix that robustness issue in the "1 CPU for TailsToaster" case, we should instead work on stabilizing the "2 CPU for TailsToaster" branch, that additionally saves 10% of the test suite runtime (with USB tests included) on Jenkins, and fixes weird stuff like #10442.

#24 Updated by anonym about 3 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 10 to 100
  • QA Check set to Pass

#25 Updated by anonym almost 3 years ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF