Project

General

Profile

Feature #16348

Upgrade to tor 0.3.5

Added by intrigeri 4 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
-
Target version:
Start date:
01/12/2019
Due date:
% Done:

100%

Estimated time:
2.50 h
QA Check:
Pass
Feature Branch:
bugfix/16348-tor-0.3.5+force-all-tests
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

pidgin.feature started failing with the upgrade to Tor 0.3.5: fails connecting to the XMPP server when tested by hand on this branch, unless I manually configure the proxy settings to use SOCKS5 127.0.0.1:9050, i.e. the global Pidgin proxy settings we set in config/chroot_local-includes/etc/skel/.purple/prefs.xml seem to be ignored.

This upgrade popped up too late in the 3.12 cycle to deal with that, so let's upgrade for 3.13.


Related issues

Related to Tails - Bug #16349: Stick to Tor 0.3.4 in Tails 3.12 Resolved 01/12/2019
Related to Tails - Bug #16471: Drop time synchronization hacks that tor 0.3.5 and 0.4.x made obsolete (if any) In Progress 02/17/2019
Blocks Tails - Feature #15507: Core work 2019Q1: Foundations Team Resolved 04/08/2018

Associated revisions

Revision 3d88e030 (diff)
Added by intrigeri 4 months ago

Install tor 0.3.4 from out custom APT repo (refs: #16348, #16349).

Revision b15b0dde (diff)
Added by intrigeri 4 months ago

Rename APT overlay to match new branch name (refs: #16348, #16349).

Revision e3190de5 (diff)
Added by Sandro Knauß 3 months ago

Revert "Install tor 0.3.4 from out custom APT repo (refs: #16348, #16349)."

This reverts commit 3d88e030ca9aac8348b46d08c4994c19b2bb607e.

Revision 8f84aacc (diff)
Added by Sandro Knauß 3 months ago

Fix pidgin to connect via tor to services (refs: #16348)

With SOCKS5 pidgin is not able to connect via tor. If we change this to
SOCKS4, we are able to connect to XMPP Server and to IRC.

Revision c66b89db (diff)
Added by Sandro Knauß 3 months ago

Revert "Fix pidgin to connect via tor to services (refs: #16348)"

This reverts commit 8f84aacca536d59e3ec1f6044a8804f5e8e1973d.

With tor 0.3.5.8 this patch is not needed anymore.

Revision 0e8a748f (diff)
Added by Sandro Knauß 2 months ago

Revert "Install tor 0.3.4 from out custom APT repo (refs: #16348, #16349)."

This reverts commit 3d88e030ca9aac8348b46d08c4994c19b2bb607e.

Revision 7168d2ff (diff)
Added by intrigeri 2 months ago

Bump APT snapshot of the torproject archive to 2019031301 (refs: #16348).

The 2019022601 snapshot has expired and was garbage-collected.

Revision b7a86ad3
Added by intrigeri 2 months ago

Merge remote-tracking branch 'origin/bugfix/16348-tor-0.3.5+force-all-tests' into stable (Fix-committed: #16348)

History

#1 Updated by intrigeri 4 months ago

  • Related to Bug #16349: Stick to Tor 0.3.4 in Tails 3.12 added

#2 Updated by intrigeri 4 months ago

#3 Updated by intrigeri 4 months ago

  • Status changed from Confirmed to In Progress

#4 Updated by intrigeri 4 months ago

  • Subject changed from Upgrade to Tor 0.3.5 to Upgrade to tor 0.3.5
  • Description updated (diff)

#5 Updated by hefee 3 months ago

  • Assignee set to hefee

#6 Updated by hefee 3 months ago

  • Feature Branch set to bugfix/16349-tor-0.3.4+force-all-tests

#7 Updated by intrigeri 3 months ago

  • Feature Branch deleted (bugfix/16349-tor-0.3.4+force-all-tests)

#8 Updated by hefee 3 months ago

  • Feature Branch set to hefee/bugfix/16349-tor-0.3.5+force-all-tests

tested with tor 0.3.5.7-1~d90.stretch+1 by hand successfully:
[x] onionshare
[x] gobby
[x] onion circuits
[x] Tor Browser
[x] Thunderbird (only connection via IMAP, not the whole sending, PGP etc.)

#9 Updated by hefee 3 months ago

pidgin is able to connect via tor, if we select "SOCKS4" instead of "SOCKS5".

(13:24:20) account: Connecting to account test@netzguerilla.net/.
(13:24:20) connection: Connecting. gc = 0x55e53a4caf30
(13:24:20) dnssrv: querying SRV record for netzguerilla.net: _xmpp-client._tcp.netzguerilla.net
(13:24:20) dnssrv: res_query returned an error
(13:24:20) dnsquery: Performing DNS lookup for 127.0.0.1
(13:24:20) dnsquery: IP resolved for 127.0.0.1
(13:24:20) proxy: Attempting connection to 127.0.0.1
(13:24:20) proxy: Connecting to netzguerilla.net:5222 via 127.0.0.1:9050 using SOCKS4
(13:24:20) proxy: Connection in progress.
(13:24:20) socks4 proxy: Connected.
(13:24:20) socks4 proxy: Attempting to use remote DNS.
(13:24:20) proxy: Connected to netzguerilla.net:5222.
(13:24:20) jabber: Sending (test@netzguerilla.net): <?xml version='1.0' ?>
(13:24:20) jabber: Sending (test@netzguerilla.net): <stream:stream to='netzguerilla.net' xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>
(13:24:20) jabber: Recv (365): <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' from='netzguerilla.net' id='5f8ad072-f676-4cf0-b93b-a0ad134f62dd' xml:lang='en' xmlns='jabber:client'><stream:features><register xmlns='http://jabber.org/features/iq-register'/><starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'><required/></starttls></stream:features>
(13:24:20) jabber: Sending (test@netzguerilla.net): <starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
(13:24:21) jabber: Recv (50): <proceed xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
(13:24:21) nss: SSL version 3.3 using 256-bit AES with 160-bit SHA1 MAC
Server Auth: 2048-bit RSA, Key Exchange: 384-bit ECDHE, Compression: NULL
Cipher Suite Name: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
(13:24:21) nss: subject=CN=conference.netzguerilla.net issuer=CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
(13:24:21) nss: subject=CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US issuer=CN=DST Root CA X3,O=Digital Signature Trust Co.
(13:24:21) nss: subject=CN=DST Root CA X3,O=Digital Signature Trust Co. issuer=CN=DST Root CA X3,O=Digital Signature Trust Co.
(13:24:21) certificate/x509/tls_cached: Starting verify for netzguerilla.net
(13:24:21) certificate/x509/tls_cached: Checking for cached cert...
(13:24:21) certificate/x509/tls_cached: ...Found cached cert

#10 Updated by hefee 3 months ago

With SOCKS5 the pidgin is logging this:

(13:33:35) util: Writing file prefs.xml to directory /home/amnesia/.purple
(13:33:35) util: Writing file /home/amnesia/.purple/prefs.xml
(13:33:37) util: Writing file accounts.xml to directory /home/amnesia/.purple
(13:33:37) util: Writing file /home/amnesia/.purple/accounts.xml
(13:33:46) account: Connecting to account gonda@irc.oftc.net.
(13:33:46) connection: Connecting. gc = 0x55e53aed4fa0
(13:33:46) dnsquery: Performing DNS lookup for 127.0.0.1
(13:33:46) dnsquery: IP resolved for 127.0.0.1
(13:33:46) proxy: Attempting connection to 127.0.0.1
(13:33:46) proxy: Connecting to irc.oftc.net:6697 via 127.0.0.1:9050 using SOCKS5
(13:33:46) socks5 proxy: Connection in progress
(13:33:46) socks5 proxy: Connected.
(13:33:46) socks5 proxy: Able to read.
(13:33:46) socks5 proxy: Got auth response.
(13:33:46) socks5 proxy: general SOCKS server failure
(13:33:46) proxy: Connection attempt failed: general SOCKS server failure

(13:33:46) connection: Connection error on 0x55e53aed4fa0 (reason: 0 description: SSL-Verbindung fehlgeschlagen)
(13:33:46) account: Disconnecting account gonda@irc.oftc.net (0x55e53a0d2460)
(13:33:46) connection: Disconnecting connection 0x55e53aed4fa0

#11 Updated by hefee 3 months ago

okay if someone adds a fake user/password to the proxy dialog, `SOCKS5` also works:

(13:51:40) account: Connecting to account gonda@irc.oftc.net.
(13:51:40) connection: Connecting. gc = 0x55e53a888690
(13:51:40) dnsquery: Performing DNS lookup for 127.0.0.1
(13:51:40) dnsquery: IP resolved for 127.0.0.1
(13:51:40) proxy: Attempting connection to 127.0.0.1
(13:51:40) proxy: Connecting to irc.oftc.net:6697 via 127.0.0.1:9150 using SOCKS5
(13:51:40) socks5 proxy: Connection in progress
(13:51:40) socks5 proxy: Connected.
(13:51:40) socks5 proxy: Able to read.
(13:51:40) socks5 proxy: Got auth response.
(13:51:40) s5: reallocing from 5 to 8
(13:51:40) s5: reallocing from 8 to 10
(13:51:40) proxy: Connected to irc.oftc.net:6697.
(13:51:41) nss: SSL version 3.3 using 128-bit AES-GCM with 128-bit AEAD MAC

#12 Updated by hefee 3 months ago

validate by hand that Pidgin is working with IRC for branch hefee/bugfix/16349-tor-0.3.5+force-all-tests.

#13 Updated by hefee 3 months ago

@intrigeri: should we create a upstream bugreport about the not accepting non user/pass woth SOCKS5? Not sure if that is a Tor or Pidgin issue.

#14 Updated by intrigeri 3 months ago

@intrigeri: should we create a upstream bugreport about the not accepting non user/pass woth SOCKS5? Not sure if that is a Tor or Pidgin issue.

Yes, if you can figure out which one of these is buggy. I would start by looking for related items in the tor's changelog.

#15 Updated by hefee 3 months ago

  • Related to Bug #16471: Drop time synchronization hacks that tor 0.3.5 and 0.4.x made obsolete (if any) added

#16 Updated by hefee 3 months ago

  • Description updated (diff)
  • Assignee deleted (hefee)
  • QA Check set to Ready for QA

pidgin now successfully connects to IRC and XMPP. So this is ready for QA.

One issue so far:

Jenkins fails for "Time syncing ǂ Clock is one day in the future in bridge mode" but only for build #3:

https://jenkins.tails.boum.org/job/test_Tails_ISO_hefee-bugfix-16349-tor-0.3.5-force-all-tests/lastCompletedBuild/cucumberTestReport/time-syncing/clock-is-one-day-in-the-future-in-bridge-mode/

I created an upstream bugreport for pidgin to fix the need for user/pass:

https://developer.pidgin.im/ticket/17380

(as curl is able to use tor as SOCKS5 proxy without user/pass).

#17 Updated by intrigeri 3 months ago

  • Assignee set to intrigeri

#18 Updated by intrigeri 3 months ago

With SOCKS5 the pidgin is logging this:

> [...]
> (13:33:46) socks5 proxy: general SOCKS server failure
> (13:33:46) proxy: Connection attempt failed: general SOCKS server failure

> (13:33:46) connection: Connection error on 0x55e53aed4fa0 (reason: 0 description: SSL-Verbindung fehlgeschlagen)
> (13:33:46) account: Disconnecting account gonda@irc.oftc.net (0x55e53a0d2460)
> (13:33:46) connection: Disconnecting connection 0x55e53aed4fa0
> 

I suggest not using OFTC for testing: very often it blocks connections that come from Tor, so test results won't be 100% reliable. I would suggest instead using a known-working XMPP server.

#19 Updated by intrigeri 3 months ago

  • Assignee changed from intrigeri to hefee
  • QA Check changed from Ready for QA to Dev Needed

pidgin now successfully connects to IRC and XMPP. So this is ready for QA.

Glad to see progress here.

A solution based on modifying /etc/skel/.purple/prefs.xml will only avoid breakage for non-persistent Pidgin config and for persistent Pidgin config created with Tails 3.13. But any pre-existing persistent Pidgin config won't benefit from this change. So our options are:

  1. Ideally, find a solution that does not require editing the Pidgin config at all. For example:
    1. Some network software will honor $SOCKS5_USER and $SOCKS5_PASSWORD. I don't know if that's the case here but if it can be made to work, this would be ideal.
    2. If this is really a bug in Pidgin (that for some reason is triggered by updating tor, while it worked fine before): fix Pidgin. It hasn't been updated in stable since the Stretch release so shipping a patched package does not seem too scary in terms of maintenance cost.
    3. Now that you have this additional constraint in mind, perhaps your unleashed imagination will find another workaround :)
  2. If (1) is not possible, then evaluate how feasible it is to update persistent Pidgin config the first time it's unlocked or used, when booting Tails 3.13+. It's XML which is never trivial to deal with. I know you'd rather not change users' config without prompting but this opinion does not match what we've done in the past in similar cases, so if you really want to prompt users, please check with sajolida before spending time on writing such code.
  3. If (1) is not possible and (2) is too expensive — be it technically or socially — the worst case scenario is that we document in the release notes how affected users must edit their config themselves. That's pretty bad because, well, many users don't read release notes, so it'll inevitably add more work to our help desk's plate. I don't know how to compare this additional work and users pain with the work needed by (2).

One issue so far:
Jenkins fails for "Time syncing ǂ Clock is one day in the future in bridge mode" but only for build #3:

(#11589 i.e. the reason why this test is marked as fragile.)

I created an upstream bugreport for pidgin […]

Great! \o/

#20 Updated by hefee 3 months ago

  • Assignee changed from hefee to intrigeri
  • QA Check changed from Dev Needed to Info Needed

Okay after digging further into pidgin/tor. I stumbled over:
https://github.com/torproject/tor/commit/790150e57a98221fbb4cfdc5c34b3395808416b4#diff-79be88f1aef796da2f29f8e3b7913f26
that leads me to the issue:
https://trac.torproject.org/projects/tor/ticket/29175

So it is actually a tor issue. I can verify, that I see the same logs in the tor journal. Can you tell if, that will be backported to 0.3.5.8?

How to went on from here?

#21 Updated by intrigeri 3 months ago

  • Assignee changed from intrigeri to hefee
  • QA Check changed from Info Needed to Dev Needed

Great research!

Can you tell if, that will be backported to 0.3.5.8?

Reading the ticket you've linked to, it seems the fix has already been backported to the 0.3.5.x branch.

How to went on from here?

Wait for 0.3.5.8 to be released; I'll send you more info privately.

#22 Updated by intrigeri 3 months ago

Wait for 0.3.5.8 to be released; I'll send you more info privately.

Actually this is public info: https://lists.torproject.org/pipermail/tor-talk/2019-February/044877.html

#23 Updated by hefee 3 months ago

intrigeri wrote:

Wait for 0.3.5.8 to be released; I'll send you more info privately.

Actually this is public info: https://lists.torproject.org/pipermail/tor-talk/2019-February/044877.html

Okay it got released and 0.3.5.8 now successfully can connect to tor without user/pass (tested currently on my sid system). So we don't need to change pidgin prefs.xml anymore.

#24 Updated by intrigeri 3 months ago

Okay it got released and 0.3.5.8 now successfully can connect to tor without user/pass (tested currently on my sid system). So we don't need to change pidgin prefs.xml anymore.

Woohoo! So we probably need to bump config/APT_snapshots.d/torproject/serial (https://tails.boum.org/contribute/APT_repository/time-based_snapshots/ for context & details).

#25 Updated by hefee 3 months ago

updated config/APT_snapshots.d/torproject/serial and have tor 0.3.5.8 on Tails:
  • no user/pass is needed anymore
  • -> IRC works
  • My jabberserver (netzguerilla.net) is using DNS entries
    _xmpp-server._tcp.netzguerilla.net to connect to im.netzguerilla.net
    this is not working with SOCKS5 setting. Switching to SOCKS4 and disable "Remote-DNS over SOCKS4" make it possible to use my jabber account with tails. After initially connects via SOCKS4/without remote-dns, I can switch back to SOCKS5 as the "Session Management" returns the cached data.

#26 Updated by hefee 3 months ago

  • Assignee changed from hefee to intrigeri
  • QA Check changed from Dev Needed to Info Needed

I need input for descide how to go on for here.

  • is the jabber server SRV records a minor issue and we ship the new tor anyways? IMO think, there at least a quite jabber server out there that uses those records, so it is show stopper.
  • does "Remote-DNS over SOCKS4" is relevant for Tails, as far I understood the fundamental setup of Tails DNS is anyways tunneled over tor.

Following solutions:

  1. update prefs.xml also for persistence to use SOCKS 4/ without "Remote-DNS over SOCKS4".
  2. ignore the issue with server SRV records.
  3. make tor aware of this issue and fix it properly in tor for 0.3.5.X.

I also wanted to mentioned, that I spent now 2h on that ticket, so we have raise the boundary.

#27 Updated by intrigeri 3 months ago

I'll look into this on Monday.

#28 Updated by intrigeri 3 months ago

  • Assignee changed from intrigeri to anonym

Actually, anonym will look into this tomorrow.

#29 Updated by anonym 3 months ago

  • Assignee changed from anonym to hefee

hefee wrote:

I need input for descide how to go on for here.

  • is the jabber server SRV records a minor issue and we ship the new tor anyways? IMO think, there at least a quite jabber server out there that uses those records, so it is show stopper.

The only context I have for a "SRV records issue" is what I could find in the log you posted above. If that is all this is about, then there is no regression -- Tails uses Tor as resolver, and it doesn't support SRV queries so Pidgin has never gotten these (well, maybe it did years ago when we used ttdnsd).

  • does "Remote-DNS over SOCKS4" is relevant for Tails, as far I understood the fundamental setup of Tails DNS is anyways tunneled over tor.

A DNS leak in Tails is indeed safe since our system resolver is Torified. IMHO it is still good practice to enable remote DNS when using SOCKS4, but since we are gonna keep on using SOCKS5 (per your comment in #16348#note-23) this settings is irrelevant to us, right?

Following solutions:

1. update prefs.xml also for persistence to use SOCKS 4/ without "Remote-DNS over SOCKS4".

See above! I'm confused!

2. ignore the issue with server SRV records.

We're stuck with 2 for now. I agree it isn't great, e.g. setting up a RiseUp XMPP account in Tails is pretty awkward since you have to manually configure the "Connect server" as xmpp.riseup.net, which the SRV record would have prevented.

3. make tor aware of this issue and fix it properly in tor for 0.3.5.X.

They are aware.

#30 Updated by anonym 3 months ago

  • % Done changed from 0 to 40

anonym wrote:

hefee wrote:

  • does "Remote-DNS over SOCKS4" is relevant for Tails, as far I understood the fundamental setup of Tails DNS is anyways tunneled over tor.

A DNS leak in Tails is indeed safe since our system resolver is Torified. IMHO it is still good practice to enable remote DNS when using SOCKS4, but since we are gonna keep on using SOCKS5 (per your comment in #16348#note-23) this settings is irrelevant to us, right?

Following solutions:

1. update prefs.xml also for persistence to use SOCKS 4/ without "Remote-DNS over SOCKS4".

See above! I'm confused!

Ah, I think I got it now! I had not paid enough attention to what you said in #16348#note-25. So from what I understand, SOCKS4 + remote-dns actually works around the SRV problem. That is cool! And in retrospect it makes a lot of sense! :)

We definitely should consider your proposal, but please open a new ticket for that. Again, the SRV problem is not a regression caused by Tor 0.3.5.x, so solving it is unrelated to this ticket.

So, as far as I can tell, this ticket should now be ready for a review'n'merge, but I'll hand it back to you first for verifying/disproving my conclusions above.

#31 Updated by hefee 3 months ago

  • Assignee changed from hefee to anonym

hefee wrote:

I also wanted to mentioned, that I spent now 2h on that ticket, so we have raise the boundary.

@intrigeri: We made the rule, that for each ticket have approved 2h. Those are execeded. The new limit is not discussed atm. Is @anonym also in charge to raise the the limit?

anonym wrote:

So, as far as I can tell, this ticket should now be ready for a review'n'merge, but I'll hand it back to you first for verifying/disproving my conclusions above.

As the 2h limit is exceeded I havn't merged/rebuilt the branch again. But as it is in the end only about reverting 3d88e030ca9aac8348b46d08c4994c19b2bb607e that should be fine to give it to review'n'merge.

As I don't except a lot of discussions around the review'n'merge. It should not take me more then 30min in addition as upper limit.

#32 Updated by hefee 3 months ago

I forgotten to tell you find my feature branch at the regular tails.git at (git-tails.immerda.ch/tails) the 0xacab.org is only used for other tails repos, as I don't have commit access there.

#33 Updated by hefee 3 months ago

#34 Updated by hefee 3 months ago

  • QA Check changed from Info Needed to Ready for QA

#35 Updated by intrigeri 3 months ago

  • Estimated time set to 2.50 h

#36 Updated by intrigeri 3 months ago

  • Assignee changed from anonym to hefee
  • QA Check changed from Ready for QA to Dev Needed

I also wanted to mentioned, that I spent now 2h on that ticket, so we have raise the boundary.

@intrigeri: We made the rule, that for each ticket have approved 2h. Those are execeded. The new limit is not discussed atm.

@hefee: done. Meta/process: when you write something like "I spent now 2h on that ticket, we have raise the boundary", please tell me how much extra time you're requesting (I can't answer anything conclusive to "we have raise the boundary" itself).

Is @anonym also in charge to raise the the limit?

No, he's not.

The branch FTBFS on Jenkins. Please fix that (should be a mere matter of merging current stable into your branch) and check that the test suite results look good enough. In case you missed or lost the link I've sent you some time ago: https://tails.boum.org/contribute/merge_policy/#submit :)

#37 Updated by intrigeri 2 months ago

A tip in passing: if you encode this ticket number (16348) in the name of your branch, then Jenkins will make some decisions depending on the "QA Check" status of the ticket. For example, if it's "Ready for QA", it will test reproducibility. So in general it's good practice to do so :)

#38 Updated by hefee 2 months ago

  • Assignee changed from hefee to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

Local merge and Jenkins could build my branch, so I ask for review again.

#39 Updated by intrigeri 2 months ago

  • % Done changed from 40 to 50
  • Feature Branch changed from hefee/bugfix/16349-tor-0.3.5+force-all-tests to bugfix/16348-tor-0.3.5+force-all-tests

@hefee, thank you!

I've cherry-picked the two relevant commits in a new branch forked off stable: your branch was based on devel (used to prepare major releases), our next major release is 7 months away, and we want this in 3.13.

Then I've bumped the serial for the torproject APT snapshot to one that has the version of tor we want and still exists (+ bumped its expiration date). See https://tails.boum.org/contribute/APT_repository/time-based_snapshots/ if you're interested in the details.

I'll check Jenkins build & test results before merging.

#40 Updated by intrigeri 2 months ago

  • Status changed from In Progress to Fix committed
  • % Done changed from 50 to 100

#41 Updated by intrigeri 2 months ago

  • Assignee deleted (intrigeri)
  • QA Check changed from Ready for QA to Pass

Everything looks OK on the Jenkins front, except tests that fail almost all the time these days + a new one (#16555, that affects other branches as well, so not tied to tor 0.3.5).

#42 Updated by intrigeri 2 months ago

  • Related to deleted (Feature #16528: Allow Pidgin to use SRV records)

#43 Updated by CyrilBrulebois 2 months ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF