Project

General

Profile

Feature #11356

Add Chutney to our isotesters

Added by anonym about 3 years ago. Updated about 3 years ago.

Status:
Rejected
Priority:
Elevated
Assignee:
-
Category:
Test suite
Target version:
Start date:
04/15/2016
Due date:
% Done:

100%

Feature Branch:
Type of work:
Wait
Blueprint:
Starter:
Affected tool:

Description

Please add, to some directory readable by the user running the automated test suite, a Git checkout at commit commit:5a63784d408e50924f3c38903fc28ae39bf7e0aa with my patches from:

Also add the file features/config/local.d/chutney.yml (I might be wrong about local.d) with the contents:

Chutney:
  src_dir: "/path/to/chutney-src-tree" 

where the path is whatever path you put Chutney into above.

do_not_use_system_torrc.patch View (1.29 KB) anonym, 04/24/2016 01:46 PM


Related issues

Related to Tails - Feature #9521: Use the chutney Tor network simulator in our test suite Resolved 04/15/2016

History

#1 Updated by intrigeri about 3 years ago

  • Assignee changed from bertagaz to anonym
  • QA Check set to Info Needed

a Git checkout at commit commit:5a63784d408e50924f3c38903fc28ae39bf7e0aa

IMO hard-coding the exact commit we want won't scale, and we instead need to track a branch. So, what branch?

#2 Updated by anonym about 3 years ago

  • Assignee changed from anonym to bertagaz

intrigeri wrote:

a Git checkout at commit commit:5a63784d408e50924f3c38903fc28ae39bf7e0aa

IMO hard-coding the exact commit we want won't scale, and we instead need to track a branch. So, what branch?

Ok, IIRC that was what someone suggested. The only branch upstream has is master.

Also, please see ee566389e24a160b79b55590befd4456328c466d that adds instructions for setting this up. Is it getting too complex to have this setup working?

#3 Updated by anonym about 3 years ago

  • Priority changed from Normal to Elevated

I'd like to get tests running as soon as possible so we can start gathering stats about how much of an improvement this is.

#4 Updated by anonym about 3 years ago

I just realized that test/9521-chutney introduces Tor as a dependency for running the automated test suite, so that should be taken into account on the puppet side.

#5 Updated by bertagaz about 3 years ago

  • Assignee changed from bertagaz to anonym

Ok, so I added a chutney checkout on all isotesters, cloning master branch of https://git.torproject.org/chutney.git. I also configured the test suite as explained.

First run failed with a 'KeyError: 'include:bridge-obfs4.tmpl', see https://jenkins.tails.boum.org/job/test_Tails_ISO_test-9521-chutney/10/console

What am I missing?

#6 Updated by bertagaz about 3 years ago

  • Assignee changed from anonym to bertagaz

anonym wrote:

I just realized that test/9521-chutney introduces Tor as a dependency for running the automated test suite, so that should be taken into account on the puppet side.

Tor is already installed everywhere. :)

#7 Updated by bertagaz about 3 years ago

  • Assignee changed from bertagaz to anonym

bertagaz wrote:

What am I missing?

Got it, missing the bridge-obfs4.tmpl file/link in the torrc_template/ directory of the chutney repo checkout. Silly me, RTFM!

That's a bit troubling because there is no permanent checkout of the Tails Git repo on the isotesters. It's cloned fresh for each test suite run in the Jenkins workspace, and then deleted.

I don't think we should provide this file on the isotesters using puppet, or it would have to be upgraded each time it changes in Tails repo.

We could modify the test suite wrapper script on the isotesters so that it copies this file before each run, and delete it afterward to leave the chutney repo clone clean.

But I think this should probably be resolved at the test suite code level. The chutney instance it starts should be able to take any configuration we ship in our Tails repo. I can deploy the wrapper solution while it's not implemented in the test suite code though.

#8 Updated by anonym about 3 years ago

  • Assignee changed from anonym to bertagaz
  • QA Check changed from Info Needed to Dev Needed

bertagaz wrote:

We could modify the test suite wrapper script on the isotesters so that it copies this file before each run, and delete it afterward to leave the chutney repo clone clean.

Could we do that for now, as an ugly/hacky short-term solution given the long-term solution below? Something like:

if [ -d "${TAILS_GIT}"/features/chutney ]; then
    cp "${TAILS_GIT}"/features/chutney/*.tmpl /path/to/chutney-src/torrc_templates/
fi

Just so we can start gathering stats ASAP.

But I think this should probably be resolved at the test suite code level. The chutney instance it starts should be able to take any configuration we ship in our Tails repo. I can deploy the wrapper solution while it's not implemented in the test suite code though.

The long-term solution is #11364 which indeed will allow us to solve this nicely inside the test suite's code. I plan to get to it early May.

#9 Updated by bertagaz about 3 years ago

  • Assignee changed from bertagaz to anonym
  • QA Check changed from Dev Needed to Info Needed

anonym wrote:

bertagaz wrote:

We could modify the test suite wrapper script on the isotesters so that it copies this file before each run, and delete it afterward to leave the chutney repo clone clean.

Could we do that for now, as an ugly/hacky short-term solution given the long-term solution below? Something like:

I deployed something similar but also taking care of removing the files, otherwise the chutney clone repo would have dangling files preventing it to be updated.

Just so we can start gathering stats ASAP.

Sadly it still fail, see this run in Jenkins. I wonder what's happening. We're running the test suite using sudo, maybe some necessary environment variables are removed by it? Did you see hat kind of errors already?

The long-term solution is #11364 which indeed will allow us to solve this nicely inside the test suite's code. I plan to get to it early May.

\o/

#10 Updated by bertagaz about 3 years ago

bertagaz wrote:

Sadly it still fail, see this run in Jenkins. I wonder what's happening. We're running the test suite using sudo, maybe some necessary environment variables are removed by it? Did you see hat kind of errors already?

I've removed the --quiet option used in TorNet.py to start the Tor nodes so that we get a bit more debug output. Result can be seen in this Jenkins run. Still analyzing.

#11 Updated by bertagaz about 3 years ago

Upgraded Tor to Jessie-backports version (0.2.7.6-1~bpo8+1) and it seems to vanish the previous reported troubles away. Still there's now a new error as one can see here. It errors out with:

00:00:53.667 Creating identity key /var/lib/chutney/net/nodes/000auth/keys/authority_identity_key for test000auth with \
tor-gencert --create-identity-key --passphrase-fd 0 -i /var/lib/chutney/net/nodes/000auth/keys/authority_identity_key \
-s /var/lib/chutney/net/nodes/000auth/keys/authority_signing_key -c /var/lib/chutney/net/nodes/000auth/keys/authority_certificate \
-m 12 -a 127.0.0.1:7000
00:00:53.667 
00:00:53.667 Error when calling 'tor --quiet --list-fingerprint --orport 1 --dirserver xyzzy 127.0.0.1:1 \
ffffffffffffffffffffffffffffffffffffffff --datadirectory /var/lib/chutney/net/nodes/000auth'. \
It gave '' as a fingerprint  and None on stderr.

#12 Updated by bertagaz about 3 years ago

With a bit more debug as shown in the last Jenkins test run, it seems the fingerprint command step fails with this error message:

Apr 24 13:56:07.470 [warn] Failed to parse/validate config: Failed to configure rendezvous options. See logs for details.

#13 Updated by bertagaz about 3 years ago

After a bit more investigation, I found something: at the time this --list-fingerprint command is issued, there's no torrc in the net/nodes/000auth subdir, only the authority keys.

I can reproduce this bug by starting chutney by hand. But when I move /etc/tor/torrc to somewhere else (and not even stop the system Tor), then chutney can configure and start correctly. I'll try a start a Jenkins run with this trick to see if it solves the issue.

#14 Updated by bertagaz about 3 years ago

Jenkins run n°17 seems to confirm that with no /etc/tor/torrc, chutney is starting fine. There are a lot of Tor processes running in this session.

I'm not sure now where this torrc sourcing comes from. From Tor itself, or from chutney?

#15 Updated by anonym about 3 years ago

bertagaz wrote:

Jenkins run n°17 seems to confirm that with no /etc/tor/torrc, chutney is starting fine. There are a lot of Tor processes running in this session.

I can confirm that Chutney will use /etc/tor/torrc at one point.

I'm not sure now where this torrc sourcing comes from. From Tor itself, or from chutney?

I managed to track it down, and I'll file an upstream bug. Until then, please try the attached patch to the Chutney's you just deployed.

#16 Updated by anonym about 3 years ago

anonym wrote:

I managed to track it down, and I'll file an upstream bug.

Done: https://trac.torproject.org/projects/tor/ticket/18882

#17 Updated by bertagaz about 3 years ago

  • Assignee changed from bertagaz to anonym
  • QA Check changed from Dev Needed to Info Needed

Ok, I've added a chutney repo at git.tails.boum.org, with a feature/tails_test_suite branch containing your three patches. It's deployed on the isotesters, and I've started a build, which should lead in a new test suite run if everything goes well. Finger crossed!

#18 Updated by bertagaz about 3 years ago

  • % Done changed from 0 to 30
  • QA Check deleted (Info Needed)
  • Type of work changed from Sysadmin to Wait

So that seems to work quite well. Chutney started correctly, is using TMP_DIR/chutney-data and the Tor processes are listening on the correct IP.

I'm putting this ticket on hold with a Wait type of work. Once your patches are merged upstream and #11364 is resolved, please re-assign it to me so that I'll complete this ticket by removing all our workarounds.

#19 Updated by bertagaz about 3 years ago

  • Blocked by Bug #11364: Add option to Chutney for override torrc_templates directory. added

#20 Updated by anonym about 3 years ago

bertagaz wrote:

So that seems to work quite well. Chutney started correctly, is using TMP_DIR/chutney-data and the Tor processes are listening on the correct IP.

Great! I've quikly pushed a branch test/9521-chutney-with-fragile-scenarios that removes the @fragile tags that were added for Tor-related reasons (I did this very quickly, so there may be mistakes). Let's see how it goes.

I'm putting this ticket on hold with a Wait type of work. Once your patches are merged upstream and #11364 is resolved, please re-assign it to me so that I'll complete this ticket by removing all our workarounds.

Ack!

#21 Updated by anonym about 3 years ago

anonym wrote:

I've quikly pushed a branch test/9521-chutney-with-fragile-scenarios that removes the @fragile tags that were added for Tor-related reasons (I did this very quickly, so there may be mistakes). Let's see how it goes.

And I also pushed test/9521-with-fragile-scenarios which is stable (should probably have been devel, whatever) with the same tags removed.

#22 Updated by intrigeri about 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from anonym to bertagaz
  • Type of work changed from Wait to Sysadmin

I see that vcsrepo { $chutney_srcdir is done as root: let's please not do that. I think we consistently avoid it everywhere else, no?

#23 Updated by bertagaz about 3 years ago

  • Assignee changed from bertagaz to anonym
  • Type of work changed from Sysadmin to Wait

intrigeri wrote:

I see that vcsrepo { $chutney_srcdir is done as root: let's please not do that. I think we consistently avoid it everywhere else, no?

Right, done.

#24 Updated by anonym about 3 years ago

bertagaz wrote:

Assignee changed from bertagaz to anonym

I'm not entirely sure what I am supposed to do with this ticket now... any idea? :)

#25 Updated by anonym about 3 years ago

  • Parent task deleted (#9521)

The plan is to merge test/9521-chutney and close #9521 before this ticket is fixed, so unparenting.

#26 Updated by anonym about 3 years ago

  • Related to Feature #9521: Use the chutney Tor network simulator in our test suite added

#27 Updated by intrigeri about 3 years ago

  • Blocked by deleted (Bug #11364: Add option to Chutney for override torrc_templates directory.)

#28 Updated by intrigeri about 3 years ago

  • Status changed from In Progress to Rejected
  • Assignee deleted (anonym)
  • % Done changed from 30 to 100

We use a Git submodule for chutney nowadays, so I've dropped the copy we had added to our isotesters.

Also available in: Atom PDF