Project

General

Profile

Bug #16150

FirewallAssertionFailedError should mark the scenario as failed in Jenkins Cucumber report

Added by bertagaz 6 months ago. Updated 3 months ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
-
Start date:
11/23/2018
Due date:
% Done:

0%

QA Check:
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

While noticing #16148, it appeared that a failing @check_tor_leaks hook does not mark the concerned scenario as failed in Cucumber's JSON output. This is only about the JSON output: on the stdout/stderr and exit code front, the test suite and Cucumber bits behave as intended and the scenario is correctly considered as failed.

The Cucumber JSON output only records per-step status, not per-scenario status, and it does not take into account the After('@product', '@check_tor_leaks') cucumber hook that runs our code. This leads to bad and misleading reporting by the Jenkins cucumber report plugin, that relies on Cucumber's JSON output. We should make it so that a failing check_tor_leaks hook ensure the scenario is recorded as failed in Cucumber's JSON output.

cucumber.json (195 Bytes) intrigeri, 11/24/2018 05:14 AM


Related issues

Related to Tails - Feature #9424: Support all cucumber formatters Resolved 05/18/2015

History

#1 Updated by intrigeri 6 months ago

  • Assignee set to bertagaz
  • QA Check set to Info Needed

bertagaz wrote:

We should make it so that a failing @check_tor_leaks hook ensure the scenario is considered as failed by cucumber.

Well, at least on the stdout/stderr and exit code front, the test suite and Cucumber bits behave as intended and the scenario is considered as failed. On https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_feature-14596-automated-tests-for-asp-gui-on-stable/70/console I see:

17:56:19 Failing Scenarios:
17:56:19 cucumber features/additional_software_packages.feature:87 # Scenario: Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when online
17:56:19 
17:56:19 175 scenarios (1 failed, 174 passed)
17:56:19 1266 steps (1266 passed)
17:56:19 258m38.495s
17:56:23 Build step 'Execute shell' marked build as failure

So the way we do things in a hook seems to be at least somewhat supported by Cucumber. That's good news :)

Then, to tell whether this is a Cucumber or Jenkins bug, the question is: is the JSON output buggy or is the Jenkins plugin failing to understanding it?

#2 Updated by intrigeri 6 months ago

  • Description updated (diff)

#3 Updated by bertagaz 6 months ago

intrigeri wrote:

bertagaz wrote:

We should make it so that a failing @check_tor_leaks hook ensure the scenario is considered as failed by cucumber.

Well, at least on the stdout/stderr and exit code front, the test suite and Cucumber bits behave as intended and the scenario is considered as failed. On https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_feature-14596-automated-tests-for-asp-gui-on-stable/70/console I see:

[...]

So the way we do things in a hook seems to be at least somewhat supported by Cucumber. That's good news :)

Then, to tell whether this is a Cucumber or Jenkins bug, the question is: is the JSON output buggy or is the Jenkins plugin failing to understanding it?

Sorry if I was sloppy, I've checked that before reporting and indeed the cucumber json is buggy :

        "keyword":"Scenario",
        "name":"Recovering in offline mode after Additional Software previously failed to upgrade \
   and then succeed to upgrade when online",
        "line":65,
        "description":"",
        "type":"scenario",
[...]
            "result":{
              "status":"passed",
              "duration":814261910
            },
            "output":[
              "Scenario failed at time 02:56:48",
              "",[...]
            ]
          }

#4 Updated by intrigeri 6 months ago

  • Related to Feature #9424: Support all cucumber formatters added

#5 Updated by intrigeri 6 months ago

  • File cucumber.json added
  • Subject changed from FirewallAssertionFailedError should mark the scenario as failing to FirewallAssertionFailedError should mark the scenario as failed in Jenkins Cucumber report
  • Description updated (diff)
  • Category changed from Test suite to Continuous Integration
  • Assignee deleted (bertagaz)
  • Target version deleted (Tails_3.11)
  • QA Check deleted (Info Needed)

To be clear, the Cucumber JSON output does not include per-scenario status: as the indentation suggests and a closer look at the JSON data demonstrates, this "result":{ "status":"passed" is about the last step of the scenario, not about the scenario as a whole; as you said, the code run in the After hook is not a step and not part of the scenario, so indeed that info is missing in the JSON and that does not qualify as a Cucumber bug :/ Ouch! It's unfortunate that we did not notice that limitation when we started relying on this JSON output while using After hooks to perform additional tests; but this is totally understandable: the bug only happens in corner cases and it took us 3.5 years to notice the problem at all.

Adjusting metadata:

  • CI because:
    • Only CI is affected.
    • The After hook predates the choice of using the JSON output.
    • The reliance on the JSON output is 100% prompted by Jenkins plugin ecosystem. Were we using the junit or html output, we would not be affected. BTW, if that HTML is good enough, we could use that and ditch the Cucumber Jenkins plugin. Note that the Cucumber output landscape is going to change drastically (https://github.com/cucumber/cucumber/issues/415, https://github.com/cucumber/cucumber/issues/524) at some point.
    • AFAIK Cucumber provides no other way to do what we need here, so I don't think we can workaround this problem in the test suite itself, short of explicitly adding a step to every scenario that needs check_tor_leaks, which is too noisy and error-prone (it will be forgotten now and then).
  • No target version because thankfully, the entire test suite run is marked as failed anyway, so there's a chance that one notices the issue (if that's the only failure: by looking at the console output; if there are more failures: because of the extra build artifacts).

Also available in: Atom PDF