Project

General

Profile

Bug #10442

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

Totem "Watching a WebM video over HTTPS" test never passes on Jenkins

Added by kytv about 4 years ago. Updated 3 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Test suite
Target version:
Start date:
10/28/2015
Due date:
% Done:

100%

Feature Branch:
test/10442-totem-watching-webm-over-https
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

An error dialog is displayed:

  • title: "An error occurred"
  • text: "The movie could not be read."

Nothing obvious in the Journal (e.g. no AppArmor error) except that for each attempt, I see this: org.gnome.Shell.desktop[7280]: Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x3200013 ().

Running torsocks totem https://tails.boum.org/lib/test_suite/test.webm works fine in a local VM (2 vCPUs, 2GB RAM) running Tails 4.0~beta1 on my sid system:

  • virtio GPU with 3d acceleration enabled
  • virtio GPU without 3d accel
  • same GPU as the one we use on Jenkins: type='qxl' ram='65536' vram='131072' vgamem='16384'

I can't reproduce either when I run the test suite manually in a Stretch VM on my system; so nested virtualization is not the only explanation.

But I can reproduce this problem on my local Jenkins (worker1.ant01, nested virtualization), which shows that the problem is not specific to lizard.

wget https://tails.boum.org/lib/test_suite/test.webm works just fine when run by hand on worker1.ant01 and isotesterN.lizard. The former uses our VPN to connect to tails.b.o while the latter uses direct inter-lizard-VM IP connectivity to do so. Both resolve tails.b.o to a RFC1918 address.


Related issues

Related to Tails - Feature #11403: Migrate to Tor Browser 6.0.x based on Firefox 45.2 Resolved 05/09/2016
Related to Tails - Feature #14588: Self-host our website Resolved 10/03/2018
Blocked by Tails - Feature #9521: Use the chutney Tor network simulator in our test suite Resolved 04/15/2016
Blocked by Tails - Bug #10497: wait_until_tor_is_working helper is fragile Resolved 11/06/2015
Blocked by Tails - Bug #10381: The "I open the address" steps are fragile Resolved 10/15/2015
Blocked by Tails - Feature #6729: Bump the number of CPU cores the testing VM has Resolved 02/19/2014
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision 3ff6c20e (diff)
Added by anonym over 3 years ago

Watch the WebM used for our test from our website.

This might improve reliability somewhat. At least we depend on one
less host being up (we already depend on our website).

Refs: #10442

Revision 927012c8 (diff)
Added by intrigeri over 2 years ago

Test suite: mark "Scenario: Watching a WebM video" as fragile (refs: #10442).

I've seen it fail more than 50% of the time on the testing and devel
branches recently.

Revision 8a978562 (diff)
Added by intrigeri 9 months ago

Test suite: clarify what WebM scenarios are fragile (refs: #10442)

There's been some confusion going on wrt. #10442 and WebM test cases.
AFAICT, currently:

1. Browsing the web using the Tor Browser → Watching a WebM video: I don't think
it's fragile anymore. At least it hasn't failed in the last 5 full test suite
runs on the stable branch. Let's re-enable it and if it proves to still be
fragile, mark it as such and reference a new, dedicated ticket.

2. Using Totem → Watching a WebM video over HTTPS: well, it's not just
"fragile", it fails (almost?) every single time on lizard these days.
That's what #10442 is about, let's make this clear in the test suite.

Revision 2ade96d1 (diff)
Added by intrigeri 4 months ago

Test suite: fix "Watching a WebM video over HTTPS" scenario on Jenkins (refs: #10442)

Revision b7f6cd84 (diff)
Added by intrigeri 4 months ago

Have @check_tor_leaks let us connect to the host:port this test is about (refs: #10442)

Otherwise we get a FirewallAssertionFailedError when running this step
on Jenkins.

Revision b0c35105 (diff)
Added by intrigeri 4 months ago

Rename function and variable to better match their current usage (refs: #10442)

Revision 3ec7305b
Added by anonym 4 months ago

Merge remote-tracking branch 'origin/test/10442-totem-watching-webm-over-https' into stable

Fix-committed: #10442

History

#1 Updated by kytv about 4 years ago

  • Parent task set to #10288

#2 Updated by kytv about 4 years ago

  • Feature Branch set to kytv/test/10442-watching-webm-over-https-is-fragile

#4 Updated by intrigeri about 4 years ago

  • Assignee set to kytv

#5 Updated by anonym about 4 years ago

  • Assignee changed from kytv to anonym
  • Target version set to Tails_1.8

There might be something smart to do that will fix all of {#10442, #10381, #10376} at the same time, and increase browser page loading throughout the test suite.

#6 Updated by intrigeri about 4 years ago

  • Target version changed from Tails_1.8 to Tails_2.0

(We're going to mark as fragile all tests that depend on Tor to have bootstrapped for the moment => not so urgent.)

#8 Updated by anonym almost 4 years ago

  • Target version changed from Tails_2.0 to Tails_2.2

#9 Updated by intrigeri almost 4 years ago

I've seen this one fail immediately after loading our webpage worked. IIRC everything online breaks anyway in our test suite if our website is not up, so I suggest we add an ikiwiki underlay to our website, that ships the smallest WebM video that can fit out test suite's needs, and then we use it from there.

#10 Updated by anonym almost 4 years ago

  • Priority changed from Normal to Elevated
  • Target version changed from Tails_2.2 to Tails_2.4

#11 Updated by anonym almost 4 years ago

  • Priority changed from Elevated to Normal
  • Target version deleted (Tails_2.4)

I'm gonna gamble and focus on Chutney (#9521) hoping it will fix this issue.

#12 Updated by anonym almost 4 years ago

  • Blocked by Feature #9521: Use the chutney Tor network simulator in our test suite added

#13 Updated by intrigeri over 3 years ago

  • Feature Branch changed from kytv/test/10442-watching-webm-over-https-is-fragile to test/10442-watching-webm-over-https-is-fragile

#14 Updated by intrigeri over 3 years ago

  • Assignee changed from anonym to intrigeri
  • Target version set to Tails_2.4

Will give it a try.

#16 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10497: wait_until_tor_is_working helper is fragile added

#17 Updated by intrigeri over 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA

(Or should it be for bertagaz?)

Seems to be fixed by chutney.

#18 Updated by intrigeri over 3 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

Same as a few similar tickets: too many false positives so far.

#19 Updated by intrigeri over 3 years ago

  • % Done changed from 50 to 60

This now seems to be pretty robust (after flagging more tests as fragile), so here we're only blocking on #10497 before this can be reviewed'n'merged.

#20 Updated by intrigeri over 3 years ago

  • Related to Feature #6729: Bump the number of CPU cores the testing VM has added

#21 Updated by intrigeri over 3 years ago

So, with Tor Browser 6.x it seems that this ticket is blocked by the cheap version of #6729 (i.e. just bumping the number of vcpus to 2).

#22 Updated by intrigeri over 3 years ago

Actually, might be blocked: I haven't see "file is corrupted" issue on Jenkins yet.

#23 Updated by intrigeri over 3 years ago

  • Blocked by Bug #10381: The "I open the address" steps are fragile added

#24 Updated by intrigeri over 3 years ago

intrigeri wrote:

Actually, might be blocked: I haven't see "file is corrupted" issue on Jenkins yet.

Seen it: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10381-fix-i-open-the-address-test-is-fragile/27/artifact/build-artifacts/01%3A32%3A29_Watching_a_WebM_video.png

Of course I didn't happen on this branch: it can only happen when we run these scenarios, that were flagged fragile due to #10381. So I've merged the branch for #10381 into the one for #10442, and added a blocking relationship.

#25 Updated by intrigeri over 3 years ago

  • Related to deleted (Feature #6729: Bump the number of CPU cores the testing VM has)

#26 Updated by intrigeri over 3 years ago

  • Blocked by Feature #6729: Bump the number of CPU cores the testing VM has added

#27 Updated by intrigeri over 3 years ago

  • Related to Feature #11403: Migrate to Tor Browser 6.0.x based on Firefox 45.2 added

#28 Updated by intrigeri over 3 years ago

  • Assignee changed from intrigeri to anonym

Moving to anonym's plate, since #11403 broke this for real this time.

#29 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.4 to Tails_2.5

#30 Updated by intrigeri over 3 years ago

  • Target version changed from Tails_2.5 to Tails_2.6

#31 Updated by intrigeri over 3 years ago

  • Target version changed from Tails_2.6 to Tails_2.7

(And feel free to drop the target version for this one, and instead tackle #10381, since there's no way to have this one done before #10381.)

#32 Updated by bertagaz about 3 years ago

  • Target version changed from Tails_2.7 to Tails_2.9.1

#33 Updated by anonym about 3 years ago

  • Assignee deleted (anonym)
  • Target version deleted (Tails_2.9.1)

#34 Updated by u over 2 years ago

  • Assignee set to anonym

#35 Updated by u over 2 years ago

  • Assignee deleted (anonym)

#36 Updated by intrigeri 9 months ago

  • Status changed from In Progress to Confirmed

#37 Updated by intrigeri 9 months ago

#38 Updated by intrigeri 9 months ago

#39 Updated by intrigeri 9 months ago

#40 Updated by intrigeri 9 months ago

  • Subject changed from Watching a WebM video over HTTPS is fragile to Totem "Watching a WebM video over HTTPS" test is fragile

#41 Updated by intrigeri 9 months ago

  • Status changed from Confirmed to In Progress

#42 Updated by intrigeri 9 months ago

  • Assignee set to CyrilBrulebois
  • Target version set to Tails_3.14
  • % Done changed from 60 to 0
  • QA Check changed from Dev Needed to Ready for QA
  • Feature Branch changed from test/10442-watching-webm-over-https-is-fragile to test/10442-webm-video-update-fragile-status

@CyrilBrulebois, here as well, the branch I'm pushing merely updates the test suite fragile status and comments, in order to avoid further confusion. Please review, merge if happy, and reset Status to Confirmed. Then we shall investigate why the Totem test fails, but that'll be for another day (and actually, I'm curious to see what happens on Buster).

#43 Updated by anonym 9 months ago

  • Assignee changed from CyrilBrulebois to anonym

#44 Updated by anonym 9 months ago

  • Status changed from In Progress to Confirmed
  • Assignee deleted (anonym)
  • QA Check deleted (Ready for QA)

#45 Updated by intrigeri 8 months ago

  • Target version deleted (Tails_3.14)

#46 Updated by hefee 6 months ago

I checked many different build on Jenkins and could not found any successful build for cucumber features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS. In detail I checked the range at the end are the buildlogs I looked at:

test_Tails_ISO_bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(1, 12) 
test_Tails_ISO_feature-16356-tor-browser-9.0-force-all-tests range(1, 15) 
test_Tails_ISO_feature-16792-update-chutney-force-all-tests range(1, 5) 
test_Tails_ISO_feature-buster-force-all-tests range(40, 55) 
test_Tails_ISO_hefee-bugfix-16471-drop-time-synchronization-hacks-force-all-tests range(19, 34) 
test_Tails_ISO_test-16820-uefi-force-all-tests range(1, 1) 
test_Tails_ISO_test-anonym-force-all-tests range(1, 4)

#47 Updated by intrigeri 6 months ago

  • Feature Branch deleted (test/10442-webm-video-update-fragile-status)

#48 Updated by intrigeri 6 months ago

  • Subject changed from Totem "Watching a WebM video over HTTPS" test is fragile to Totem "Watching a WebM video over HTTPS" test (almost?) never passes on Jenkins

#49 Updated by intrigeri 4 months ago

  • Description updated (diff)

#50 Updated by intrigeri 4 months ago

  • Description updated (diff)
  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Target version set to Tails_3.16

Reporting about a bunch of situations in which I can, and cannot, reproduce this.

The problem happens on all systems that resolve tails.b.o to a RFC1918 address, and I can't reproduce it anywhere else, so I think it is caused by:

       ClientDNSRejectInternalAddresses 0|1
           If true, Tor does not believe any anonymously retrieved DNS answer that tells
           it that an address resolves to an internal address (like 127.0.0.1 or
           192.168.0.1). This option prevents certain browser-based attacks; it is not
           allowed to be set on the default network. (Default: 1)

       ClientRejectInternalAddresses 0|1
           If true, Tor does not try to fulfill requests to connect to an internal
           address (like 127.0.0.1 or 192.168.0.1) unless an exit node is specifically
           requested (for example, via a .exit hostname, or a controller request). If
           true, multicast DNS hostnames for machines on the local network (of the form
           *.local) are also rejected. (Default: 1)

I think we need to disable these options at least for the affected scenario.

#51 Updated by intrigeri 4 months ago

We use Chutney and TestingTorNetwork, which disables the 2 aforementioned options, but in features/step_definitions/chutney.rb we re-enable ClientRejectInternalAddresses (that we need enabled for at least one scenario).

#52 Updated by intrigeri 4 months ago

  • Priority changed from Normal to Elevated
  • Feature Branch set to test/10442-totem-watching-webm-over-https

(We now run fragile tests on Jenkins for our main branches, so having a scenario that always fails there makes the output of our CI more painful to use.)

#53 Updated by intrigeri 4 months ago

#54 Updated by intrigeri 4 months ago

This fixes this bug on my local Jenkins (the video is played in Totem), except the check for Tor leaks of course makes a fuss out of it:

Unexpected connections were made:
  #<OpenStruct mac_saddr="50:54:00:ae:aa:8a", mac_daddr="52:54:00:d7:c9:e3", protocol="tcp", sport=48124, dport=443, saddr="10.2.1.247", daddr="192.168.122.6"> (FirewallAssertionFailedError)
  /var/lib/jenkins/workspace/test_Tails_ISO_test-10442-totem-watching-webm-over-https/features/support/helpers/firewall_helper.rb:97:in `After'

#55 Updated by intrigeri 4 months ago

In passing, I'm surprised that the Tor Browser's "Watching a WebM video" test does not fail in the same way; ditto for checking for upgrades: they all connect to tails.boum.org:443 as well. I suspect that the way they resolve this hostname (using Tor's SOCKS support and presumably proper "the SOCKS proxy resolves DNS") yields different results than torsocks totem, which will do DNS resolution separately i.e. go through tor's DNS port. Still, I'm surprised: regardless of how exactly DNS resolution is requested, at the end of the day it should be performed by some exit node in our Chutney network, so I would assume it should yield the same results. Anyways.

#56 Updated by intrigeri 4 months ago

  • Subject changed from Totem "Watching a WebM video over HTTPS" test (almost?) never passes on Jenkins to Totem "Watching a WebM video over HTTPS" test never passes on Jenkins

#57 Updated by intrigeri 4 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee changed from intrigeri to anonym

@anonym, I came up with something that works on my local Jenkins (which as explained above & in the description, is affected by the bug for the same reasons as lizard's isotesters). I'm not super happy with it but I could not find a better fix. This being said, given this bug makes all base branch tests fail on Jenkins, I'm in favor of a quick resolution, even if slightly suboptimal.

I'll verify today that this works on lizard too but if you happen to pop up here on a Sunday (which I don't expect), feel free to beat me to it.

#58 Updated by intrigeri 4 months ago

Worked 2/2 times on lizard :)

#59 Updated by anonym 4 months ago

  • Status changed from Needs Validation to 11
  • Assignee deleted (anonym)
  • % Done changed from 0 to 100

intrigeri wrote:

In passing, I'm surprised that the Tor Browser's "Watching a WebM video" test does not fail in the same way; ditto for checking for upgrades: they all connect to tails.boum.org:443 as well. I suspect that the way they resolve this hostname (using Tor's SOCKS support and presumably proper "the SOCKS proxy resolves DNS") yields different results than torsocks totem, which will do DNS resolution separately i.e. go through tor's DNS port. Still, I'm surprised: regardless of how exactly DNS resolution is requested, at the end of the day it should be performed by some exit node in our Chutney network, so I would assume it should yield the same results.

Frankly, this sounds slightly concerning, almost like there's a DNS leak or similar. Or do you see anything redeeming?


Other than that, code looks good. My tests do not look good, however, as I very often see @check_tor_leaks:

      Unexpected connections were made:
        #<OpenStruct mac_saddr="00:00:00:00:00:00", mac_daddr="00:00:00:00:00:00", protocol="tcp", sport=5015, dport=38432, saddr="10.2.1.1", daddr="10.2.1.202"> (FirewallAssertionFailedError)

but it turns out that is not caused by your branch. I've at least seen the scenario pass once, so I'm happy. Merged!

#60 Updated by intrigeri 4 months ago

Other than that, code looks good. My tests do not look good, however, as I very often see @check_tor_leaks:

>       Unexpected connections were made:
>         #<OpenStruct mac_saddr="00:00:00:00:00:00", mac_daddr="00:00:00:00:00:00", protocol="tcp", sport=5015, dport=38432, saddr="10.2.1.1", daddr="10.2.1.202"> (FirewallAssertionFailedError)
> 

but it turns out that is not caused by your branch.

@anonym, I'm curious now :) What is this caused by?

#61 Updated by intrigeri 4 months ago

Hi @anonym !

Frankly, this sounds slightly concerning, almost like there's a DNS leak or similar. Or do you see anything redeeming?

I am also slightly concerned by the fact I don't understand what's going on.

But given this test passed just fine (without the firewall leak checker raising hell) on systems without a "special" name resolution setup, I'm not too worried: this shows that Tails behaves as expected. Note that "as expected" here can plausibly include "Totem's DNS resolution mechanism escapes torsocks and ends up querying the Tor DNS port itself", which would not be a big surprise (torsocks tries it best but it gives no guarantee and there are well-known ways to escape it, be it maliciously or simply by using techniques that torsocks can't catch).

#62 Updated by CyrilBrulebois 3 months ago

  • Status changed from 11 to Resolved

Also available in: Atom PDF