Project

General

Profile

Bug #10442

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

Totem "Watching a WebM video over HTTPS" test (almost?) never passes on Jenkins

Added by kytv over 3 years ago. Updated 3 days ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
-
Start date:
10/28/2015
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Related issues

Related to Tails - Feature #11403: Migrate to Tor Browser 6.0.x based on Firefox 45.2 Resolved 05/09/2016
Blocked by Tails - Feature #9521: Use the chutney Tor network simulator in our test suite Resolved 04/15/2016
Blocked by Tails - Bug #10497: wait_until_tor_is_working helper is fragile Resolved 11/06/2015
Blocked by Tails - Bug #10381: The "I open the address" steps are fragile Resolved 10/15/2015
Blocked by Tails - Feature #6729: Bump the number of CPU cores the testing VM has Resolved 02/19/2014
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed 03/22/2019

Associated revisions

Revision 3ff6c20e (diff)
Added by anonym about 3 years ago

Watch the WebM used for our test from our website.

This might improve reliability somewhat. At least we depend on one
less host being up (we already depend on our website).

Refs: #10442

Revision 927012c8 (diff)
Added by intrigeri about 2 years ago

Test suite: mark "Scenario: Watching a WebM video" as fragile (refs: #10442).

I've seen it fail more than 50% of the time on the testing and devel
branches recently.

Revision 8a978562 (diff)
Added by intrigeri 3 months ago

Test suite: clarify what WebM scenarios are fragile (refs: #10442)

There's been some confusion going on wrt. #10442 and WebM test cases.
AFAICT, currently:

1. Browsing the web using the Tor Browser → Watching a WebM video: I don't think
it's fragile anymore. At least it hasn't failed in the last 5 full test suite
runs on the stable branch. Let's re-enable it and if it proves to still be
fragile, mark it as such and reference a new, dedicated ticket.

2. Using Totem → Watching a WebM video over HTTPS: well, it's not just
"fragile", it fails (almost?) every single time on lizard these days.
That's what #10442 is about, let's make this clear in the test suite.

History

#1 Updated by kytv over 3 years ago

  • Parent task set to #10288

#2 Updated by kytv over 3 years ago

  • Feature Branch set to kytv/test/10442-watching-webm-over-https-is-fragile

#4 Updated by intrigeri over 3 years ago

  • Assignee set to kytv

#5 Updated by anonym over 3 years ago

  • Assignee changed from kytv to anonym
  • Target version set to Tails_1.8

There might be something smart to do that will fix all of {#10442, #10381, #10376} at the same time, and increase browser page loading throughout the test suite.

#6 Updated by intrigeri over 3 years ago

  • Target version changed from Tails_1.8 to Tails_2.0

(We're going to mark as fragile all tests that depend on Tor to have bootstrapped for the moment => not so urgent.)

#8 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.0 to Tails_2.2

#9 Updated by intrigeri over 3 years ago

I've seen this one fail immediately after loading our webpage worked. IIRC everything online breaks anyway in our test suite if our website is not up, so I suggest we add an ikiwiki underlay to our website, that ships the smallest WebM video that can fit out test suite's needs, and then we use it from there.

#10 Updated by anonym over 3 years ago

  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.2 to Tails_2.4

#11 Updated by anonym over 3 years ago

  • Priority changed from Elevated to Normal
  • Target version deleted (Tails_2.4)

I'm gonna gamble and focus on Chutney (#9521) hoping it will fix this issue.

#12 Updated by anonym over 3 years ago

  • Blocked by Feature #9521: Use the chutney Tor network simulator in our test suite added

#13 Updated by intrigeri about 3 years ago

  • Feature Branch changed from kytv/test/10442-watching-webm-over-https-is-fragile to test/10442-watching-webm-over-https-is-fragile

#14 Updated by intrigeri about 3 years ago

  • Assignee changed from anonym to intrigeri
  • Target version set to Tails_2.4

Will give it a try.

#16 Updated by intrigeri about 3 years ago

  • Blocked by Bug #10497: wait_until_tor_is_working helper is fragile added

#17 Updated by intrigeri about 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA

(Or should it be for bertagaz?)

Seems to be fixed by chutney.

#18 Updated by intrigeri about 3 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

Same as a few similar tickets: too many false positives so far.

#19 Updated by intrigeri about 3 years ago

  • % Done changed from 50 to 60

This now seems to be pretty robust (after flagging more tests as fragile), so here we're only blocking on #10497 before this can be reviewed'n'merged.

#20 Updated by intrigeri about 3 years ago

  • Related to Feature #6729: Bump the number of CPU cores the testing VM has added

#21 Updated by intrigeri about 3 years ago

So, with Tor Browser 6.x it seems that this ticket is blocked by the cheap version of #6729 (i.e. just bumping the number of vcpus to 2).

#22 Updated by intrigeri about 3 years ago

Actually, might be blocked: I haven't see "file is corrupted" issue on Jenkins yet.

#23 Updated by intrigeri about 3 years ago

  • Blocked by Bug #10381: The "I open the address" steps are fragile added

#24 Updated by intrigeri about 3 years ago

intrigeri wrote:

Actually, might be blocked: I haven't see "file is corrupted" issue on Jenkins yet.

Seen it: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10381-fix-i-open-the-address-test-is-fragile/27/artifact/build-artifacts/01%3A32%3A29_Watching_a_WebM_video.png

Of course I didn't happen on this branch: it can only happen when we run these scenarios, that were flagged fragile due to #10381. So I've merged the branch for #10381 into the one for #10442, and added a blocking relationship.

#25 Updated by intrigeri about 3 years ago

  • Related to deleted (Feature #6729: Bump the number of CPU cores the testing VM has)

#26 Updated by intrigeri about 3 years ago

  • Blocked by Feature #6729: Bump the number of CPU cores the testing VM has added

#27 Updated by intrigeri about 3 years ago

  • Related to Feature #11403: Migrate to Tor Browser 6.0.x based on Firefox 45.2 added

#28 Updated by intrigeri about 3 years ago

  • Assignee changed from intrigeri to anonym

Moving to anonym's plate, since #11403 broke this for real this time.

#29 Updated by anonym about 3 years ago

  • Target version changed from Tails_2.4 to Tails_2.5

#30 Updated by intrigeri almost 3 years ago

  • Target version changed from Tails_2.5 to Tails_2.6

#31 Updated by intrigeri almost 3 years ago

  • Target version changed from Tails_2.6 to Tails_2.7

(And feel free to drop the target version for this one, and instead tackle #10381, since there's no way to have this one done before #10381.)

#32 Updated by bertagaz over 2 years ago

  • Target version changed from Tails_2.7 to Tails_2.9.1

#33 Updated by anonym over 2 years ago

  • Assignee deleted (anonym)
  • Target version deleted (Tails_2.9.1)

#34 Updated by u almost 2 years ago

  • Assignee set to anonym

#35 Updated by u almost 2 years ago

  • Assignee deleted (anonym)

#36 Updated by intrigeri 4 months ago

  • Status changed from In Progress to Confirmed

#37 Updated by intrigeri 4 months ago

#38 Updated by intrigeri 3 months ago

#39 Updated by intrigeri 3 months ago

#40 Updated by intrigeri 3 months ago

  • Subject changed from Watching a WebM video over HTTPS is fragile to Totem "Watching a WebM video over HTTPS" test is fragile

#41 Updated by intrigeri 3 months ago

  • Status changed from Confirmed to In Progress

#42 Updated by intrigeri 3 months ago

  • Assignee set to CyrilBrulebois
  • Target version set to Tails_3.14
  • % Done changed from 60 to 0
  • QA Check changed from Dev Needed to Ready for QA
  • Feature Branch changed from test/10442-watching-webm-over-https-is-fragile to test/10442-webm-video-update-fragile-status

@CyrilBrulebois, here as well, the branch I'm pushing merely updates the test suite fragile status and comments, in order to avoid further confusion. Please review, merge if happy, and reset Status to Confirmed. Then we shall investigate why the Totem test fails, but that'll be for another day (and actually, I'm curious to see what happens on Buster).

#43 Updated by anonym 3 months ago

  • Assignee changed from CyrilBrulebois to anonym

#44 Updated by anonym 3 months ago

  • Status changed from In Progress to Confirmed
  • Assignee deleted (anonym)
  • QA Check deleted (Ready for QA)

#45 Updated by intrigeri 3 months ago

  • Target version deleted (Tails_3.14)

#46 Updated by hefee 7 days ago

I checked many different build on Jenkins and could not found any successful build for cucumber features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS. In detail I checked the range at the end are the buildlogs I looked at:

test_Tails_ISO_bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(1, 12) 
test_Tails_ISO_feature-16356-tor-browser-9.0-force-all-tests range(1, 15) 
test_Tails_ISO_feature-16792-update-chutney-force-all-tests range(1, 5) 
test_Tails_ISO_feature-buster-force-all-tests range(40, 55) 
test_Tails_ISO_hefee-bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(19, 34) 
test_Tails_ISO_test-16820-uefi-force-all-tests range(1, 1) 
test_Tails_ISO_test-anonym-force-all-tests range(1, 4)

#47 Updated by intrigeri 3 days ago

  • Feature Branch deleted (test/10442-webm-video-update-fragile-status)

#48 Updated by intrigeri 3 days ago

  • Subject changed from Totem "Watching a WebM video over HTTPS" test is fragile to Totem "Watching a WebM video over HTTPS" test (almost?) never passes on Jenkins

Also available in: Atom PDF