Project

General

Profile

Feature #7066

Investigate if we could reuse some of the TBB automated test suite

Added by intrigeri about 5 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
Start date:
04/11/2014
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
test/7066-run-tbb-tests
Type of work:
Research
Blueprint:
Starter:
No
Affected tool:
Browser

Description

It might be that the new (and quickly growing) TBB test suite has some gems we could reuse:

It's probably worth talking with Nicolas Vigier (boklm) about it, as he has had a look to our own test suite at some point, and could probably provide good insight regarding the reusability of his work.


Related issues

Blocks Tails - Feature #9561: Run the Tor Browser's own test suite inside our own Confirmed 06/11/2015

History

#1 Updated by boklm about 5 years ago

This looks like a good idea. The TBB test suite currently expects to find a TBB tarball that it extracts before running the tests on it. I can add an option so that it can skip the tarball extraction and run the tests on the browser that is installed on the system. I will let you know when it's done.

#2 Updated by BitingBird over 4 years ago

Could somebody please add boklm to the project, so I can assign this ticket to him ?

#3 Updated by intrigeri over 4 years ago

  • Assignee set to boklm

#4 Updated by BitingBird about 4 years ago

boklm, any news on this?

#5 Updated by intrigeri about 4 years ago

  • Assignee changed from boklm to intrigeri
  • Target version set to Tails_1.4.1

I'll meet boklm next month to look at this problem and work on it if doable.

#6 Updated by intrigeri about 4 years ago

  • Feature Branch set to test/7066-run-tbb-tests

#7 Updated by intrigeri almost 4 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from intrigeri to anonym
  • % Done changed from 0 to 50
  • QA Check set to Ready for QA

So, here's a quick summary of the current state of things in the test/7066-run-tbb-tests branch:

  • that's a quite ugly proof-of-concept only, in no way ready to be merged;
  • at test suite runtime, we clone (our own clone of) the Tor Browser's test suite Git repos and run its install-deps and setup-virtualenv scripts; this is not very nice, and it's also quite slow, but I'm not sure how we can do better without bloating the Tails ISO;
  • the Tails firewall is disabled, as a shortcut to allow Tor Browser to talk to whatever local web server is set up by Mozmill and Selenium; one should check if they can use a fixed port, and if so, instead we should add a test suite step that inserts the iptables/netfilter rules needed to allow the test suite to run, instead of entirely disabling the firewall;
  • Selenium and Mozmill tests are run in a way that bypasses the tbb-testsuite wrapper, IIRC because back then there was no way to enable only the tests we want; once this is fixed (which might be the case already, see the new --testsuite= option) we should run something like ./tbb-testsuite --tor-control-port=9052 --tor-socks-port=9150 --no-starttor --enable-tests=...;
  • we run all Selenium tests;
  • we run some Mozmill tests, excluding those that depend on tbb-testsuite to have prepared things ahead.

Now zooming out. The Tor Browser's and Tails' test suites have a very different general approach:

  • the Tails test suite (almost) only interacts with the browser in the very same way as a user would; and it checks success/failure either from the outside (blackbox style), or by looking at things a user would see;
  • the Tor Browser's test suite interacts with the browser using JS that Mozmill/Selenium have injected into the browser; and it checks success/failure the same way, asking the browser about its internal state.

As a result, on the one hand many kinds of tests seem to be way easier to develop within the Tor Browser's test suite framework. On the other hand, of course the browser under testing is modified (which we generally avoid) and whatever it says about success/failure is blindly trusted (while we generally try to check things from the outside). IMO we should integrate the TB's test suite into our own (as the PoC branch does), and whenever we want to add browser tests, we should decide, on a case-by-case basis, which approach is more appropriate (in terms of "cost/benefit", as they would say).

To end with, I see a few major reasons why we should run the TB test suite inside our own:

  1. it cheaply increases test coverage of the Tor Browser as shipped in Tails;
  2. it allows boklm to easily and continuously check that TB changes don't break Tails use case;
  3. if we combined this with automatically building ISO images that include dev snapshots of the Tor Browser, then we could identify problems earlier in the dev process than when we import a new TB;
  4. it widens the collaboration area between Tails and TB teams, by providing more reasons to communicate and keep each other informed of what we're up to.

Now, if we agree that it's the way to go, then I'll close this ticket and create one (or more) new tickets about making this real and bringing it to a mergeable state.

Thoughts? (this research is "Ready for QA", not the branch itself :)

#8 Updated by anonym almost 4 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

intrigeri wrote:

  • the Tails firewall is disabled, as a shortcut to allow Tor Browser to talk to whatever local web server is set up by Mozmill and Selenium; one should check if they can use a fixed port, and if so, instead we should add a test suite step that inserts the iptables/netfilter rules needed to allow the test suite to run, instead of entirely disabling the firewall;

Doesn't the TB test suite at all times capture the network and look for leaks? Any way we should be good if the scenario in our test suite that run's the TB test suite is tagged @check_tor_leaks unless we have leaks that the firewall blocks, which then will be discovered. Right?

Now zooming out. The Tor Browser's and Tails' test suites have a very different general approach:

  • the Tails test suite (almost) only interacts with the browser in the very same way as a user would; and it checks success/failure either from the outside (blackbox style), or by looking at things a user would see;
  • the Tor Browser's test suite interacts with the browser using JS that Mozmill/Selenium have injected into the browser; and it checks success/failure the same way, asking the browser about its internal state.

As a result, on the one hand many kinds of tests seem to be way easier to develop within the Tor Browser's test suite framework. On the other hand, of course the browser under testing is modified (which we generally avoid) and whatever it says about success/failure is blindly trusted (while we generally try to check things from the outside). IMO we should integrate the TB's test suite into our own (as the PoC branch does), and whenever we want to add browser tests, we should decide, on a case-by-case basis, which approach is more appropriate (in terms of "cost/benefit", as they would say).

I agree.

To end with, I see a few major reasons why we should run the TB test suite inside our own:

  1. it cheaply increases test coverage of the Tor Browser as shipped in Tails;
  2. it allows boklm to easily and continuously check that TB changes don't break Tails use case;
  3. if we combined this with automatically building ISO images that include dev snapshots of the Tor Browser, then we could identify problems earlier in the dev process than when we import a new TB;
  4. it widens the collaboration area between Tails and TB teams, by providing more reasons to communicate and keep each other informed of what we're up to.

Awesome!

Now, if we agree that it's the way to go, then I'll close this ticket and create one (or more) new tickets about making this real and bringing it to a mergeable state.

Thoughts? (this research is "Ready for QA", not the branch itself :)

Let's do it!

#9 Updated by intrigeri almost 4 years ago

intrigeri wrote:

  • the Tails firewall is disabled, as a shortcut to allow Tor Browser to talk to whatever local web server is set up by Mozmill and Selenium; one should check if they can use a fixed port, and if so, instead we should add a test suite step that inserts the iptables/netfilter rules needed to allow the test suite to run, instead of entirely disabling the firewall;

Doesn't the TB test suite at all times capture the network and look for leaks?

I think it does that if, and only if, mbox is installed. I doubt we can do it since 1. mbox is not in Debian AFAIK; 2. mbox works "by providing a layered sandbox filesystem and by interposing on system calls with ptrace and seccomp/BPF", which may not play well with our AppArmor sandboxing.

Any way we should be good if the scenario in our test suite that run's the TB test suite is tagged @check_tor_leaks unless we have leaks that the firewall blocks, which then will be discovered. Right?

Absolutely.

Now zooming out. The Tor Browser's and Tails' test suites have a very different general approach: [...] IMO we should integrate the TB's test suite into our own (as the PoC branch does), and whenever we want to add browser tests, we should decide, on a case-by-case basis, which approach is more appropriate (in terms of "cost/benefit", as they would say).

I agree.

OK, good.

To end with, I see a few major reasons why we should run the TB test suite inside our own: [...]

Awesome!

:)

Now, if we agree that it's the way to go, then I'll close this ticket and create one (or more) new tickets about making this real and bringing it to a mergeable state.

Let's do it!

OK, I'll do that today then. Yay!

#10 Updated by intrigeri almost 4 years ago

  • Blocks Feature #9561: Run the Tor Browser's own test suite inside our own added

#11 Updated by intrigeri almost 4 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 50 to 100
  • QA Check changed from Dev Needed to Pass
  • Affected tool set to Browser

Also available in: Atom PDF