Project

General

Profile

Feature #10270

Bug #10250: Eliminate manual test suite

Automate WhisperBack tests

Added by anonym about 4 years ago. Updated 3 months ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
-
Start date:
09/26/2015
Due date:
% Done:

0%

Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:
WhisperBack

Description

We may not want to automate this test because it would spam tails-bugs@, and parts of it requires manual verification from someone there. Still, it would be relevant to test that WhisperBack doesn't itself detects that it fails to send the email, because that indicates a definite failure (i.e. that our hidden service is down, but perhaps that's better tested by some external monitoring tool?).

Alternatively, if we want such a test, we could tag it @release-testing, and somehow make it so that scenarios tagged @release-testing are only run when the test suite is invoked with a --release-testing option (or ... -- --tag @release-testing), intended to only be used when testing a release (this one could also ensure that --iso != --old-iso, etc.).

History

#1 Updated by anonym about 4 years ago

  • Parent task set to #10250

#2 Updated by kytv about 4 years ago

  • Tracker changed from Bug to Feature

#3 Updated by intrigeri about 4 years ago

Whether WhisperBack's SMTP server is up is one of the services that we'll setup monitoring for with high priority (https://tails.boum.org/blueprint/monitor_servers/#services). I'd rather not notify all developers when it's down, which would be basically what happens if that service is down and we rely on it in our test suite. It's bad enough that our test suite relies on so much of our infra already, let's not add to it when possible.

A way to test the Tails ISO side of things would be to fire up a tiny/mock SMTP server behind a onion service as part of the test suite, live-patch WhisperBack config to use that onion service and its TLS certificate, and ensure the email is sent to that SMTP server. This would be the ideal solution but I doubt the benefits are worth the effort, especially as long as we've no canonical, tested and robust way to fire up and manage such temporary services in our test suite framework.

Alternatively, if we want such a test, we could tag it @release-testing, and somehow make it so that scenarios tagged @release-testing are only run when the test suite is invoked with a --release-testing option (or ... -- --tag @release-testing), intended to only be used when testing a release (this one could also ensure that --iso != --old-iso, etc.).

This sounds like a fine way to drop the part of the manual test that can be automated. I'm a bit wary of twisting the test suite semantics, though: we'll see "all tests are green", while one is only green once frontdesk have confirmed they've received the email and it was encrypted. I guess we can live with it, but I'd love it if anonym gave it a few more thinking minutes of his amazing brain.

#4 Updated by intrigeri about 4 years ago

Note that these tests most likely will need the doc Cucumber tag.

#5 Updated by intrigeri over 3 years ago

  • Type of work changed from Discuss to Research

#6 Updated by intrigeri 3 months ago

  • Affected tool set to WhisperBack

Also available in: Atom PDF