Project

General

Profile

Feature #15330

Run the doctests of the new Python scripts

Added by segfault over 1 year ago. Updated about 1 year ago.

Status:
Confirmed
Priority:
Low
Assignee:
-
Category:
Continuous Integration
Target version:
-
Start date:
02/19/2018
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

The Python scripts that were ported from bash as part of #11198 have doctests. We should run them on our CI.

History

#1 Updated by intrigeri over 1 year ago

  • Assignee set to segfault
  • QA Check set to Info Needed

Three questions that will help evaluating how much work this will require:

  • In which environment are these tests meant to be run? I've tried running config/chroot_local-includes/usr/local/bin/tails-get-bootinfo doctest on my Debian sid and it fails. Do these tests have to run inside Tails? If yes, can they run outside of the desktop session?
  • Is there a good way to automatically list scripts that have such doctests, or do we need to start maintaining a list in a file?
  • Do these tests work reliably for you?

#2 Updated by intrigeri over 1 year ago

Three questions that will help evaluating how much work this will require:

And another one: do these tests have noticeable side effects? (I wonder if running each of them invalidates the test environment; if running in the context of the Tails automated test suite, that would imply restoring a fresh snapshot between each individual test run, which is fine but we need to know if we have to do that :)

#3 Updated by segfault over 1 year ago

intrigeri wrote:

Three questions that will help evaluating how much work this will require:

  • In which environment are these tests meant to be run? I've tried running config/chroot_local-includes/usr/local/bin/tails-get-bootinfo doctest on my Debian sid and it fails. Do these tests have to run inside Tails?

Yes, they are meant to be run inside Tails.

If yes, can they run outside of the desktop session?

No, some of tests start graphical applications, so they cannot be run outside a desktop session. Those tests also require that the started application is being closed before the test can continue.

  • Is there a good way to automatically list scripts that have such doctests, or do we need to start maintaining a list in a file?

I don't know of a good way to list scripts with doctests automatically, I think we need to maintain a list.

  • Do these tests work reliably for you?

No, but it shouldn't require a lot of work to make them work reliably (the tails-get-bootinfo test seems to be broken, and the tails-upgrade-frontend-wrapper test assumes that there is enough free memory for an upgrade, else the test fails).

And another one: do these tests have noticeable side effects? (I wonder if running each of them invalidates the test environment; if running in the context of the Tails automated test suite, that would imply restoring a fresh snapshot between each individual test run, which is fine but we need to know if we have to do that :)

None of the 4 scripts we shipped in Tails 3.6 have noticeable side effects, but I didn't check the other scripts that were already ported and not shipped yet.

#4 Updated by segfault over 1 year ago

  • QA Check deleted (Info Needed)

#5 Updated by intrigeri about 1 year ago

  • Assignee deleted (segfault)
  • Priority changed from Normal to Low

Thanks for these clarifications. Running these tests on our CI requires vastly more work than I was initially expecting => setting this to low priority and unassigning from me.

Dear ticket triager: feel free to close as rejected if nobody has picked this up during your next triaging round.

Also available in: Atom PDF