Project

General

Profile

Bug #17080

Upgrade Cucumber on Jenkins isotesters

Added by intrigeri 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
Start date:
Due date:
% Done:

0%

Feature Branch:
puppet-tails:cucumber2, jenkins-jobs:cucumber2
Type of work:
Sysadmin
Blueprint:
Starter:
Affected tool:

Description

Once we've done #10068 we might be able to upgrade to Cucumber 2.x from Stretch, while we're currently running 1.3 from Jessie.

Goals:

  • stop running Jessie-era Cucumber on our CI
  • avoid confusion between what developers see locally and what our CI does (e.g. #16150#note-9)
  • fix #16150
  • unblock #10328

See #10328 for things that might need to happen at the same time.

Finally, as part of this ticket, we should revert eccc25460099522ad26d33c93f3d70601ae63ee9.


Related issues

Related to Tails - Feature #11739: Upgrade our isotesters to Stretch Resolved 08/28/2016
Blocked by Tails - Bug #10068: Upgrade to Jenkins 2.x, using upstream packages Resolved 01/08/2018
Blocks Tails - Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report Confirmed 11/23/2018
Blocks Tails - Feature #10328: Clean up features with Scenario Outlines Confirmed 10/03/2015
Blocks Tails - Feature #17083: Remove compatibility code for Cucumber << 2.4.0 from our test suite Resolved

History

#1 Updated by intrigeri 3 months ago

  • Related to Feature #10328: Clean up features with Scenario Outlines added

#2 Updated by intrigeri 3 months ago

  • Related to Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report added

#3 Updated by intrigeri 3 months ago

  • Blocked by Bug #10068: Upgrade to Jenkins 2.x, using upstream packages added

#4 Updated by intrigeri 3 months ago

  • Description updated (diff)

#5 Updated by intrigeri 3 months ago

#6 Updated by intrigeri 3 months ago

  • Related to deleted (Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report)

#7 Updated by intrigeri 3 months ago

  • Blocks Bug #16150: After hooks should be able to mark the scenario as failed in Jenkins Cucumber report added

#8 Updated by intrigeri 3 months ago

  • Related to deleted (Feature #10328: Clean up features with Scenario Outlines)

#9 Updated by intrigeri 3 months ago

  • Blocks Feature #10328: Clean up features with Scenario Outlines added

#10 Updated by intrigeri 3 months ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Feature Branch set to puppet-tails:cucumber2

The problem that made us stick to Cucumber 1.x last time (https://issues.jenkins-ci.org/browse/JENKINS-29328) still occurs. Three years later, the corresponding PR (https://github.com/jenkinsci/cucumber-testresult-plugin/pull/9) still hasn't been finalized nor merged. Note that this PR workarounds the crash bug but does not give us correct reporting of failure in after/before that are part of Background (#16150), which is one of the main incentives of upgrading Cucumber on our CI.

We could post-process the JSON file to drop the new before/after arrays before we pass it to the Cucumber Test Result Jenkins plugin. This would allow us to upgrade Cucumber, which is a first step. But we'll still not be in a position to fix #16150 and I suspect this will create even more confusion for developers.

Given the Cucumber Test Result Jenkins plugin is up for adoption and mostly unmaintained, I think our options are:

  • Give up on the goals of this ticket and revisit in a year or so, when we'll rethink the future of our CI and how Jenkins fits into it.
  • Switch to another Cucumber plugin for Jenkins, namely Cucumber Reports (GitHub), which is actively maintained and is supported by JJB. I'll give it a quick try.
  • Have Cucumber output JUnit XML and use whatever Jenkins has to display the results from such a data source, without particular knowledge of Cucumber concepts. Possible drawbacks:
    • I don't know if the test results will be presented in a usable way. I have my doubts.
    • I don't know if the JUnit XML output provides all the info we need to fix #16150 and unblock #10328.

#11 Updated by intrigeri 3 months ago

  • Feature Branch changed from puppet-tails:cucumber2 to puppet-tails:cucumber2, jenkins-jobs:cucumber2

#12 Updated by intrigeri 3 months ago

  • Status changed from In Progress to Needs Validation
  • Target version set to Tails_4.0

This was deployed: upgraded to Cucumber 2.4.0 from Stretch, switched to Cucumber Reports plugin.

Next steps:

  • check that it works as intended on lizard (worked fine on sib so far)
  • tell users of Jenkins about the web UI change
  • tell users of Jenkins that they can upgrade Cucumber to Stretch's in their test systems
  • prepare a branch that reverts eccc25460099522ad26d33c93f3d70601ae63ee9

#13 Updated by intrigeri 2 months ago

  • Blocks Feature #17083: Remove compatibility code for Cucumber << 2.4.0 from our test suite added

#14 Updated by intrigeri 2 months ago

  • Status changed from Needs Validation to Resolved

intrigeri wrote:

Next steps:

  • check that it works as intended on lizard (worked fine on sib so far)
  • tell users of Jenkins about the web UI change
  • tell users of Jenkins that they can upgrade Cucumber to Stretch's in their test systems

Done.

See you on #17083 :)

Also available in: Atom PDF