Project

General

Profile

Bug #9097

Bug #10250: Eliminate manual test suite

Automatic test: measure boot time

Added by BitingBird about 4 years ago. Updated 3 months ago.

Status:
Confirmed
Priority:
Low
Assignee:
-
Category:
Test suite
Target version:
-
Start date:
03/23/2015
Due date:
% Done:

0%

QA Check:
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

From manual test suite:

Measure boot time (from syslinux menu the GNOME dektop ready - quickly press enter in the greeter), then on some reference bare metal hardware, and compare with previous version. The new one should not be significantly slower to start.

The test suite needs to have access to data for previous releases, and should be allowed to increase a certain pourcentage (20-30 % ?).

History

#1 Updated by intrigeri about 4 years ago

The test suite needs to have access to data for previous releases, and should be allowed to increase a certain pourcentage (20-30 % ?).

Thanks for filing this ticket while I was too tired to do it!

After thinking a bit more about it, my initial idea was:

  • on a given system, the test suite could store boot time data (on a per-branch basis) somehow;
  • when running the test suite, previous boot time could be compared to current one, and raise an error if it's much longer.

Both could be done either purely via cucumber, or via Jenkins (there are Jenkins plugins to let it understand cucumber concepts, and IIRC there also are plugins to let it monitor job runtime and raise errors if it takes too long).

But now I'm less sure that it's worth doing it specifically for boot time: whatever operation we automatically test should generally not take way longer in Tails N+1 than in Tails N. If it does, maybe it's a bug. So perhaps leveraging Jenkins' cucumber support to monitor per-scenario runtime would be a nice generalization of my initial idea. And then, it becomes a sysadmin task rather than a Test suite one.

Thoughts?

#2 Updated by intrigeri almost 3 years ago

  • Parent task set to #10250

#3 Updated by spriver almost 2 years ago

  • Assignee set to anonym
  • QA Check set to Info Needed

Would the output of systemd-analyze or systemd-analyze blame be useful here? At least it'd be possible to detect services that take unusually long to start (and could be compared to a previous version).

#4 Updated by intrigeri 3 months ago

  • Assignee deleted (anonym)
  • QA Check deleted (Info Needed)

spriver wrote:

Would the output of systemd-analyze or systemd-analyze blame be useful here?

@spriver, yes.

Also available in: Atom PDF