Failing scenarios in After scenario hooks can lead to inconsistency
I just noticed a discrepancy between this run's debug.log's summary:
Failing Scenarios: cucumber features/additional_software_packages.feature:69 # Scenario: Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when online cucumber features/electrum.feature:15 # Scenario: Using a persistent Electrum configuration cucumber features/tor_stream_isolation.feature:9 # Scenario: tails-security-check is using the Tails-specific SocksPort cucumber features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS 219 scenarios (4 failed, 215 passed) 1658 steps (3 failed, 1655 passed) 346m16.087s
and Jenkins' cucumber plugin's summary:
All Failed Scenarios ==================== Test Name Duration Age --------- -------- --- Electrum Bitcoin client ǂ Using a persistent Electrum configuration 7 min 53 sec 1 Tor stream isolation is effective ǂ tails-security-check is using the Tails-specific SocksPort 20 sec 1 Using Totem ǂ Watching a WebM video over HTTPS 26 min 1
Notice how the
Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when online scenario is present in the former but absent from the latter.
In cucumber.json I can see that this scenario's result has status "passed". My theory is that the result is recorded before we run our After Scenario hook, so if we fail the scenario there, as happened in this scenario's case thanks to the @check_tor_leaks tag and the firewall leak detector hitting #16148.
I have attached the run's logs for future investigation.