Bug #10288: Fix newly identified issues to make our test suite more robust and faster
Totem "Watching a WebM video over HTTPS" test (almost?) never passes on Jenkins
Watch the WebM used for our test from our website.
This might improve reliability somewhat. At least we depend on one
less host being up (we already depend on our website).
Test suite: mark "Scenario: Watching a WebM video" as fragile (refs: #10442).
I've seen it fail more than 50% of the time on the testing and devel
Test suite: clarify what WebM scenarios are fragile (refs: #10442)
There's been some confusion going on wrt. #10442 and WebM test cases.
1. Browsing the web using the Tor Browser → Watching a WebM video: I don't think
it's fragile anymore. At least it hasn't failed in the last 5 full test suite
runs on the stable branch. Let's re-enable it and if it proves to still be
fragile, mark it as such and reference a new, dedicated ticket.
2. Using Totem → Watching a WebM video over HTTPS: well, it's not just
"fragile", it fails (almost?) every single time on lizard these days.
That's what #10442 is about, let's make this clear in the test suite.
#9 Updated by intrigeri over 3 years ago
I've seen this one fail immediately after loading our webpage worked. IIRC everything online breaks anyway in our test suite if our website is not up, so I suggest we add an ikiwiki underlay to our website, that ships the smallest WebM video that can fit out test suite's needs, and then we use it from there.
#24 Updated by intrigeri about 3 years ago
Actually, might be blocked: I haven't see "file is corrupted" issue on Jenkins yet.
Of course I didn't happen on this branch: it can only happen when we run these scenarios, that were flagged fragile due to #10381. So I've merged the branch for #10381 into the one for #10442, and added a blocking relationship.
- Assignee set to CyrilBrulebois
- Target version set to Tails_3.14
- % Done changed from 60 to 0
- QA Check changed from Dev Needed to Ready for QA
- Feature Branch changed from test/10442-watching-webm-over-https-is-fragile to test/10442-webm-video-update-fragile-status
@CyrilBrulebois, here as well, the branch I'm pushing merely updates the test suite fragile status and comments, in order to avoid further confusion. Please review, merge if happy, and reset Status to Confirmed. Then we shall investigate why the Totem test fails, but that'll be for another day (and actually, I'm curious to see what happens on Buster).
I checked many different build on Jenkins and could not found any successful build for cucumber
features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS. In detail I checked the range at the end are the buildlogs I looked at:
test_Tails_ISO_bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(1, 12) test_Tails_ISO_feature-16356-tor-browser-9.0-force-all-tests range(1, 15) test_Tails_ISO_feature-16792-update-chutney-force-all-tests range(1, 5) test_Tails_ISO_feature-buster-force-all-tests range(40, 55) test_Tails_ISO_hefee-bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(19, 34) test_Tails_ISO_test-16820-uefi-force-all-tests range(1, 1) test_Tails_ISO_test-anonym-force-all-tests range(1, 4)