Project

General

Profile

Feature #10507

Feature #5288: Run the test suite automatically on autobuilt ISOs

Document how to get the statistics we need out of Jenkins

Added by intrigeri over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
Start date:
11/07/2015
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
Type of work:
Contributors documentation
Blueprint:
Starter:
Affected tool:

Description

For SponsorS reports we need:

  • average daily number of topic branches we've built at least one ISO image from (note that it's not exactly what I asked for #9939, sorry);
  • number of test suite runs per release cycle.

For #9939, bertagaz compiled close enough stats "According to the GolbalBuildStat Jenkins plugin", which is great, but it is not documented how someone else would do it themselves. A very rough doc draft in the sysadmin team Git repo would be good enough IMO, as long as at least I am able to follow it to successfully compile such stats.

This process will also allow us to identify any manual steps in this process that should be automated, if any (but automating them is not part of this ticket, of course).

History

#2 Updated by bertagaz over 3 years ago

  • Assignee changed from bertagaz to intrigeri
  • % Done changed from 0 to 80
  • QA Check set to Ready for QA

Added a page in the sysadmins repo where I explain the method I used to get this statistics.

I think there's all that's needed to automate for the build jobs, but as explained for the test jobs it's a bit different for the moment.

#3 Updated by intrigeri over 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 80 to 90

For successful ISO builds: thanks for the draft! For many reasons that you can guess, I disliked using grep to "parse" XML so I took a couple dozen minutes to rewrite it using an actual XML parser: https://git-tails.immerda.ch/puppet-tails/tree/files/jenkins/master/successful-ISO-builds. For October it gives me the same result (935) as what you reported, so it seems that it works fine, but still: please review it.

For test suite runs: thanks for the explanation. You wrote you "modified the URL of this page" but I don't know what page you're referring to. Anyway, the details are mostly specific to the October special case, so I've ignored them and instead focussed on the general case, starting with November. I really have a hard time making any sense of the Global Build Stats interface, even with your explanations; I guess I would need step-by-step instructions; but wait! To avoid the web UI I instead wrote a script that counts runs of the test suite that were not aborted: https://git-tails.immerda.ch/puppet-tails/tree/files/jenkins/master/ISO-test-suite-runs. Please review it too, and if that's not exactly what we should could, please clarify and I can adjust the code.

Now I want to add timer units (systemd.timer(5)) to run these scripts at the beginning of each month, against the results file of past month, and publish their results. It would allow non-sysadmins to check metrics and write reports. This is why I put the script there instead of in the sysadmins private repo. We'll see :)

(Off-topic starts here. In November I see an amount of aborted test suite runs (209) that seems pretty large compared to the number of non-aborted runs (314). I guess I am missing some info to make sense of this, but from my standpoint it seems concerning. If this problem is tracked by a ticket, please point me to it. Otherwise, please ensure it's tracked, or explain me why it's not a problem. Thanks! :)

#4 Updated by bertagaz over 3 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (bertagaz)
  • QA Check changed from Ready for QA to Pass

intrigeri wrote:

For successful ISO builds: thanks for the draft! For many reasons that you can guess, I disliked using grep to "parse" XML so I took a couple dozen minutes to rewrite it using an actual XML parser: https://git-tails.immerda.ch/puppet-tails/tree/files/jenkins/master/successful-ISO-builds. For October it gives me the same result (935) as what you reported, so it seems that it works fine, but still: please review it.

That was a draft, nice script!

For test suite runs: thanks for the explanation. You wrote you "modified the URL of this page" but I don't know what page you're referring to. Anyway, the details are mostly specific to the October special case, so I've ignored them and instead focussed on the general case, starting with November. I really have a hard time making any sense of the Global Build Stats interface, even with your explanations; I guess I would need step-by-step instructions; but wait! To avoid the web UI I instead wrote a script that counts runs of the test suite that were not aborted: https://git-tails.immerda.ch/puppet-tails/tree/files/jenkins/master/ISO-test-suite-runs. Please review it too, and if that's not exactly what we should could, please clarify and I can adjust the code.

Can't say much about the perl aspect, but considering the previous script, it makes sense.

Now I want to add timer units (systemd.timer(5)) to run these scripts at the beginning of each month, against the results file of past month, and publish their results. It would allow non-sysadmins to check metrics and write reports. This is why I put the script there instead of in the sysadmins private repo. We'll see :)

I'll let you manage the way you'll encode this publication step in Redmine ticket/your todo list.

(Off-topic starts here. In November I see an amount of aborted test suite runs (209) that seems pretty large compared to the number of non-aborted runs (314). I guess I am missing some info to make sense of this, but from my standpoint it seems concerning. If this problem is tracked by a ticket, please point me to it. Otherwise, please ensure it's tracked, or explain me why it's not a problem.

Thanks for ticket #10717. I was aware of this high umber of aborted job, my priority was to fix the test suite robustness first, then have a look at it.

#5 Updated by intrigeri over 3 years ago

  • Status changed from Fix committed to Resolved

(We're not going to wait for a release.)

#6 Updated by bertagaz over 3 years ago

  • % Done changed from 90 to 100

Also available in: Atom PDF