Project

General

Profile

Bug #17472

Jenkins does not run the full test suite for tags

Added by anonym about 2 months ago. Updated 3 days ago.

Status:
Needs Validation
Priority:
Normal
Assignee:
Category:
Continuous Integration
Target version:
-
Start date:
Due date:
% Done:

0%

Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

When Jenkins tested the Tails 4.3 tag, it didn't run @fragile scenarios, which is a bug. See e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2205/

In the console output I see this trace:

14:17:21 + echo origin/stable
14:17:21 + grep -q -E \+force-all-tests$
14:17:21 + [ stable = testing ]
14:17:21 + [ stable = feature/tor-nightly-master ]
14:17:21 + [ stable = devel ]
14:17:21 + git describe --tags --exact-match 4904a1fde8887488167a8a6c34b34c45b51aa112
14:17:21 + [ 4904a1fde8887488167a8a6c34b34c45b51aa112 != 4904a1fde8887488167a8a6c34b34c45b51aa112 ]
14:17:21 + TAGS_ARGS=--tag ~@fragile

for this part of wrap_test_suite:
if echo "${UPSTREAMJOB_GIT_BRANCH}" | grep -q -E '\+force-all-tests$' \
   || [ "${GIT_BRANCH#origin/}" = testing ] \
   || [ "${GIT_BRANCH#origin/}" = feature/tor-nightly-master ] \
   || [ "${GIT_BRANCH#origin/}" = devel ] ; then
    TAGS_ARGS="" 
elif git describe --tags --exact-match "${UPSTREAMJOB_GIT_COMMIT}" >/dev/null 2>&1; then
    TAGS_ARGS="" 
elif [ "${UPSTREAMJOB_GIT_COMMIT}" != "${UPSTREAMJOB_GIT_BASE_BRANCH_HEAD}" ] && \
       only_changes_doc "$UPSTREAMJOB_GIT_BASE_BRANCH_HEAD"; then
    TAGS_ARGS="--tag ~@fragile --tag @doc" 
else
    TAGS_ARGS="--tag ~@fragile" 
fi

So $UPSTREAMJOB_GIT_COMMIT is 4904a1f, which is not the 4.3 tag (it is 2797053), but just one of the commits I made yesterday, so the correct if-case for releases was not hit. Note that 4904a1f is not the commit that was initially tagged 4.3 (FWIW, that was 4172c53), so there was no mix up with the tag update. I have no clue how 4904a1f entered the picture since it doesn't seem very special. Curious!

Associated revisions

Revision 7d39c7fb (diff)
Added by intrigeri 3 days ago

CI: document more known usability issues (refs: #17472)

History

#1 Updated by intrigeri about 1 month ago

  • Status changed from Confirmed to New
  • Assignee changed from intrigeri to anonym

When Jenkins tested the Tails 4.3 tag, it didn't run @fragile scenarios, which is a bug. See e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2205/

What makes you think that this specific run was supposed to test the 4.3 tag?

That test suite run has been triggered by https://jenkins.tails.boum.org/job/build_Tails_ISO_stable/3140/, which was about 4904a1f, which I believed was triggered by the Git push at the end of the "Update other base branches" section of the release process (the timing matches: commit done on Feb 10 10:02:43 2020, build started at Feb 10, 2020 10:17:36 AM). So to me, it seems that not running fragile scenarios in this case was the expected behavior.

The first build from the tag was https://jenkins.tails.boum.org/job/build_Tails_ISO_stable/3142/, started a few hours later.
It triggered https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2207/, which did run the full test suite, which is what we want.

#2 Updated by anonym about 1 month ago

  • Assignee changed from anonym to intrigeri

intrigeri wrote:

When Jenkins tested the Tails 4.3 tag, it didn't run @fragile scenarios, which is a bug. See e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2205/

What makes you think that this specific run was supposed to test the 4.3 tag?

That test_Tails_ISO job says "Revision: 2797053a53225134a74e0bcc80682e05bfcb7edd" which is the commit for the real 4.3 tag. I have always assumed that build_Tails_ISO will trigger a test_Tails_ISO from the exact same commit; I never thought revisions could be mixed like that, and it in fact seems like a source of bad test results. What is going on here? What am I missing?

That test suite run has been triggered by https://jenkins.tails.boum.org/job/build_Tails_ISO_stable/3140/, which was about 4904a1f, which I believed was triggered by the Git push at the end of the "Update other base branches" section of the release process (the timing matches: commit done on Feb 10 10:02:43 2020, build started at Feb 10, 2020 10:17:36 AM). So to me, it seems that not running fragile scenarios in this case was the expected behavior.

The first build from the tag was https://jenkins.tails.boum.org/job/build_Tails_ISO_stable/3142/, started a few hours later.
It triggered https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2207/, which did run the full test suite, which is what we want.

Ah, I see! Cased closed, but I'll wait for an answer to the above before closing the ticket.

#3 Updated by intrigeri about 1 month ago

  • Assignee changed from intrigeri to anonym

Hi,

anonym wrote:

intrigeri wrote:

When Jenkins tested the Tails 4.3 tag, it didn't run @fragile scenarios, which is a bug. See e.g. https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/2205/

What makes you think that this specific run was supposed to test the 4.3 tag?

That test_Tails_ISO job says "Revision: 2797053a53225134a74e0bcc80682e05bfcb7edd" which is the commit for the real 4.3 tag. I have always assumed that build_Tails_ISO will trigger a test_Tails_ISO from the exact same commit; I never thought revisions could be mixed like that, and it in fact seems like a source of bad test results. What is going on here? What am I missing?

OK, I think I now understand the source of the misunderstanding!

I think the confusion comes from the mismatch between:

  • Your correct understanding of the intended high-level behavior: build_Tails_ISO does trigger a test_Tails_ISO from the exact same commit. In the test job run log you'll see git reset --hard 4904a1fde8887488167a8a6c34b34c45b51aa112.
  • Low-level implementation "details", i.e.:
    • When it starts a run of test_Tails_ISO, Jenkins itself does not know which commit we want to test. Hence the confusing "Revision: 2797053a53225134a74e0bcc80682e05bfcb7edd" information it displays.
    • The code that implements that job is currently responsible for resetting the working copy to the desired state.

Unfortunately, I don't think there's any way to hide the confusing "Revision:" info via configuration, short of patching the corresponding Jenkins plugin; and the awful HTML+CSS implementation (OMG!) makes it hard to hide it via CSS :/

Would it help if I documented this in our doc aimed at developers?
Or are we bound to forget occasionally about it and not refer to the doc anyway?

#4 Updated by intrigeri 3 days ago

  • Status changed from New to In Progress

#5 Updated by intrigeri 3 days ago

  • Status changed from In Progress to Needs Validation

Hi,

Would it help if I documented this in our doc aimed at developers?

I went ahead and did it. Please review 7d39c7fb6c1ec4db5ed8f4fb468034b9cc72a445.

Also available in: Atom PDF