Project

General

Profile

Bug #16150

After hooks should be able to mark the scenario as failed in Jenkins Cucumber report

Added by bertagaz 8 months ago. Updated 22 days ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Continuous Integration
Target version:
-
Start date:
11/23/2018
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

While noticing #16148, it appeared that a failing @check_tor_leaks hook does not mark the concerned scenario as failed in Cucumber's JSON output. This is only about the JSON output: on the stdout/stderr and exit code front, the test suite and Cucumber bits behave as intended and the scenario is correctly considered as failed.

The Cucumber JSON output only records per-step status, not per-scenario status, and it does not take into account the After('@product', '@check_tor_leaks') cucumber hook that runs our code. This leads to bad and misleading reporting by the Jenkins cucumber report plugin, that relies on Cucumber's JSON output. We should make it so that a failing check_tor_leaks hook ensure the scenario is recorded as failed in Cucumber's JSON output.

cucumber.json (195 Bytes) intrigeri, 11/24/2018 05:14 AM


Related issues

Related to Tails - Feature #9424: Support all cucumber formatters Resolved 05/18/2015
Duplicated by Tails - Bug #16768: Failing scenarios in After scenario hooks can lead to inconsistency Duplicate

History

#1 Updated by intrigeri 8 months ago

  • Assignee set to bertagaz
  • QA Check set to Info Needed

bertagaz wrote:

We should make it so that a failing @check_tor_leaks hook ensure the scenario is considered as failed by cucumber.

Well, at least on the stdout/stderr and exit code front, the test suite and Cucumber bits behave as intended and the scenario is considered as failed. On https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_feature-14596-automated-tests-for-asp-gui-on-stable/70/console I see:

17:56:19 Failing Scenarios:
17:56:19 cucumber features/additional_software_packages.feature:87 # Scenario: Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when online
17:56:19 
17:56:19 175 scenarios (1 failed, 174 passed)
17:56:19 1266 steps (1266 passed)
17:56:19 258m38.495s
17:56:23 Build step 'Execute shell' marked build as failure

So the way we do things in a hook seems to be at least somewhat supported by Cucumber. That's good news :)

Then, to tell whether this is a Cucumber or Jenkins bug, the question is: is the JSON output buggy or is the Jenkins plugin failing to understanding it?

#2 Updated by intrigeri 8 months ago

  • Description updated (diff)

#3 Updated by bertagaz 8 months ago

intrigeri wrote:

bertagaz wrote:

We should make it so that a failing @check_tor_leaks hook ensure the scenario is considered as failed by cucumber.

Well, at least on the stdout/stderr and exit code front, the test suite and Cucumber bits behave as intended and the scenario is considered as failed. On https://jenkins.tails.boum.org/view/Tails_ISO/job/test_Tails_ISO_feature-14596-automated-tests-for-asp-gui-on-stable/70/console I see:

[...]

So the way we do things in a hook seems to be at least somewhat supported by Cucumber. That's good news :)

Then, to tell whether this is a Cucumber or Jenkins bug, the question is: is the JSON output buggy or is the Jenkins plugin failing to understanding it?

Sorry if I was sloppy, I've checked that before reporting and indeed the cucumber json is buggy :

        "keyword":"Scenario",
        "name":"Recovering in offline mode after Additional Software previously failed to upgrade \
   and then succeed to upgrade when online",
        "line":65,
        "description":"",
        "type":"scenario",
[...]
            "result":{
              "status":"passed",
              "duration":814261910
            },
            "output":[
              "Scenario failed at time 02:56:48",
              "",[...]
            ]
          }

#4 Updated by intrigeri 8 months ago

  • Related to Feature #9424: Support all cucumber formatters added

#5 Updated by intrigeri 8 months ago

  • File cucumber.json added
  • Subject changed from FirewallAssertionFailedError should mark the scenario as failing to FirewallAssertionFailedError should mark the scenario as failed in Jenkins Cucumber report
  • Description updated (diff)
  • Category changed from Test suite to Continuous Integration
  • Assignee deleted (bertagaz)
  • Target version deleted (Tails_3.11)
  • QA Check deleted (Info Needed)

To be clear, the Cucumber JSON output does not include per-scenario status: as the indentation suggests and a closer look at the JSON data demonstrates, this "result":{ "status":"passed" is about the last step of the scenario, not about the scenario as a whole; as you said, the code run in the After hook is not a step and not part of the scenario, so indeed that info is missing in the JSON and that does not qualify as a Cucumber bug :/ Ouch! It's unfortunate that we did not notice that limitation when we started relying on this JSON output while using After hooks to perform additional tests; but this is totally understandable: the bug only happens in corner cases and it took us 3.5 years to notice the problem at all.

Adjusting metadata:

  • CI because:
    • Only CI is affected.
    • The After hook predates the choice of using the JSON output.
    • The reliance on the JSON output is 100% prompted by Jenkins plugin ecosystem. Were we using the junit or html output, we would not be affected. BTW, if that HTML is good enough, we could use that and ditch the Cucumber Jenkins plugin. Note that the Cucumber output landscape is going to change drastically (https://github.com/cucumber/cucumber/issues/415, https://github.com/cucumber/cucumber/issues/524) at some point.
    • AFAIK Cucumber provides no other way to do what we need here, so I don't think we can workaround this problem in the test suite itself, short of explicitly adding a step to every scenario that needs check_tor_leaks, which is too noisy and error-prone (it will be forgotten now and then).
  • No target version because thankfully, the entire test suite run is marked as failed anyway, so there's a chance that one notices the issue (if that's the only failure: by looking at the console output; if there are more failures: because of the extra build artifacts).

#7 Updated by intrigeri about 2 months ago

  • Duplicated by Bug #16768: Failing scenarios in After scenario hooks can lead to inconsistency added

#8 Updated by anonym about 1 month ago

  • Subject changed from FirewallAssertionFailedError should mark the scenario as failed in Jenkins Cucumber report to After hooks should be able to mark the scenario as failed in Jenkins Cucumber report

#9 Updated by anonym 22 days ago

anonym wrote:

Reported upstream: https://github.com/cucumber/cucumber/issues/630

For some reason it was moved to cucumber-jvm and someone tested but couldn't reproduce. So I tried to make my own minimal test case for cucumber-ruby:

mkdir cucumber-test
cd cucumber-test
cucumber --init
cat > features/test.feature <<EOF
Feature: Test feature
  Scenario: Test scenario
    Given something
    Then whatever
EOF
cat > features/step_definitions/test.rb <<EOF
Given /^.*$/ do
  next
end
EOF
cat > features/support/hooks.rb <<EOF
# Note: Cucumber After hooks are executed in the *reverse* order they are
# listed
After do |scenario|
  # Check stderr for this to know the final status
  STDERR.puts "Scenario status: #{scenario.failed? ? "failed" : "passed"}" 
end
After do
  raise "fail" 
end
EOF
cucumber --format json features/test.feature

This results in:

Scenario status: failed
[
  {
    "uri": "features/test.feature",
    "id": "test-feature",
    "keyword": "Feature",
    "name": "Test feature",
    "description": "",
    "line": 1,
    "elements": [ 
      {
        "id": "test-feature;test-scenario",
        "keyword": "Scenario",
        "name": "Test scenario",
        "description": "",
        "line": 2,
        "type": "scenario",
        "before": [
          {
            "match": {
              "location": "/usr/lib/ruby/vendor_ruby/cucumber/filters/prepare_world.rb:27" 
            },
            "result": {
              "status": "passed",
              "duration": 1598
            }
          }
        ],
        "steps": [
          {
            "keyword": "Given ",
            "name": "something",
            "line": 3,
            "match": {
              "location": "features/step_definitions/test.rb:1" 
            },
            "result": {
              "status": "passed",
              "duration": 21578
            }
          },
          {
            "keyword": "Then ",
            "name": "whatever",
            "line": 4,
            "match": {
              "location": "features/step_definitions/test.rb:1" 
            },
            "result": {
              "status": "passed",
              "duration": 7378
            }
          }
        ],
        "after": [
          {
            "match": {
              "location": "features/support/hooks.rb:7" 
            },
            "result": {
              "status": "failed",
              "error_message": "fail (RuntimeError)\n./features/support/hooks.rb:8:in `After'",
              "duration": 80566
            }
          },
          {
            "match": {
              "location": "features/support/hooks.rb:3" 
            },
            "result": {
              "status": "passed",
              "duration": 115039
            }
          }
        ]
      }
    ]
  }
]

Note the "after" field, which records the error. So it seems something is wrong with our particular test suite, after all. I quickly tried dropping in our extra_hooks.rb since it monkey-patches cucumber and certainly could be the culprit, but I still couldn't reproduce the absence of the "after" field.

Next step is to run our test suite and progressively rip out parts and see when the "after" field reappears.

Also available in: Atom PDF