Project

General

Profile

Bug #12023

htpdate: stop sending User-Agent that fakes Tor Browser

Added by intrigeri almost 3 years ago. Updated 1 day ago.

Status:
Fix committed
Priority:
Normal
Assignee:
Category:
Time synchronization
Target version:
Start date:
12/08/2016
Due date:
% Done:

100%

Feature Branch:
bugfix/17143-17214-12023-remove-obsolete-tor-browser-customization+force-all-tests
Type of work:
Code
Blueprint:
Starter:
Yes
Affected tool:

Description

See "[Tails-dev] Faking htpdate user agent worth it?" (December 2016).

  • config/chroot_local-includes/etc/default/htpdate.user-agent must go
  • Corresponding doc updates: git grep -F /etc/default/htpdate.user-agent

Related issues

Related to Tails - Bug #17233: Ensure only one of our HTP pool has hosts handled by Cloudflare Confirmed
Related to Tails - Bug #11562: Monitor servers from the htpdate pools Confirmed 07/14/2016
Blocked by Tails - Bug #17143: Re-enable Tor Browser's "Quantum bar" Fix committed

Associated revisions

Revision 86bbc539 (diff)
Added by intrigeri 7 days ago

htpdate: stop sending User-Agent that fakes Tor Browser (refs: #12023)

As decided on the "[Tails-dev] Faking htpdate user agent worth it?" thread in
December 2016.

Revision c20e109e (diff)
Added by intrigeri 6 days ago

htpdate: stop sending User-Agent that fakes Tor Browser (refs: #12023)

As decided on the "[Tails-dev] Faking htpdate user agent worth it?" thread in
December 2016.

Revision 05db5aaa (diff)
Added by intrigeri 6 days ago

htpdate: stop sending User-Agent that fakes Tor Browser (refs: #12023)

As decided on the "[Tails-dev] Faking htpdate user agent worth it?" thread in
December 2016.

Revision 102750ce
Added by intrigeri 1 day ago

Merge branch 'bugfix/17143-17214-12023-remove-obsolete-tor-browser-customization+force-all-tests' into stable (Fix-committed: #17143, #17214, #12023)

History

#1 Updated by intrigeri over 2 years ago

  • Description updated (diff)
  • Assignee deleted (intrigeri)
  • Starter set to Yes

#2 Updated by u over 2 years ago

  • Description updated (diff)

#3 Updated by intrigeri 3 months ago

  • Description updated (diff)

#4 Updated by intrigeri 7 days ago

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Target version set to Tails_4.1
  • Feature Branch set to bugfix/17143-17214-12023-remove-obsolete-tor-browser-customization+force-all-tests

While upgrading to Tor Browser 9.0.1 (for #17214 and #17143) I got fed up of having to update this variable manually so I'll drop it.

#5 Updated by intrigeri 6 days ago

No time sync' -related test suite failure on this branch so I think we're good here.

#6 Updated by intrigeri 6 days ago

  • Blocked by Bug #17143: Re-enable Tor Browser's "Quantum bar" added

#7 Updated by intrigeri 5 days ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)

#8 Updated by hefee 4 days ago

  • Assignee set to hefee

#9 Updated by hefee 4 days ago

  • Status changed from Needs Validation to In Progress
  • Assignee changed from hefee to intrigeri

See "[Tails-dev] Faking htpdate user agent worth it?" (December 2016).

In it self the patch is fine. But it is on top of other patches, that haven't merged to stable.

But the rationale why we stop shipping the TorBrowser user agent may result in wired results, as a lot of websites use Cloudflare and they may block tor traffic completely for non TorBrowser user. Btw. did we made sure, that only one pool has domains handled by Cloudflare?

https://support.cloudflare.com/hc/en-us/articles/203306930-Does-CloudFlare-block-Tor-
https://www.zdnet.com/article/cloudflare-ends-captcha-challenges-for-tor-users/

That's why I would argue, that we have to add a test to check, that each domain in the HTP pools are reachable and the time is correct. The failure page of Cloudflare should also have correct headers and we don't care, but I would like to make sure that we are still fine.

#10 Updated by segfault 3 days ago

I reviewed #17214 and #17143, so the branch can be merged once this issue is resolved.

#11 Updated by intrigeri 2 days ago

  • Related to Bug #17233: Ensure only one of our HTP pool has hosts handled by Cloudflare added

#12 Updated by intrigeri 2 days ago

Hi @hefee,

But the rationale why we stop shipping the TorBrowser user agent may result in wired results, as a lot of websites use Cloudflare and they may block tor traffic completely for non TorBrowser user.

Very good point and thanks for the pointers!

That's why I would argue, that we have to add a test to check, that each domain in the HTP pools are reachable and the time is correct. The failure page of Cloudflare should also have correct headers and we don't care, but I would like to make sure that we are still fine.

First, I wanted to at least know what the answer to these questions would currently be, so I did this:

. config/chroot_local-includes/etc/default/htpdate.pools  
for host in $(echo "$HTP_POOL_1 $HTP_POOL_2 $HTP_POOL_3" | tr ',' ' ' ) ; do
torsocks curl --silent --head "https://$host/" | grep -iE '^date: '
done

All servers replied and gave the correct date, except https://encrypted.google.com/ that had a Date header ~17 hours in the past. I've retried this one a few times with different circuits (torsocks --isolate), and got various different dates, that were all wrong. OTOH https://www.google.com/ always gives we the correct date so I've switched to this one on the topic branch.

But all this is just one data point i.e. mostly anecdotal evidence, so:

I agree we should have such tests; IMO they should run automatically and regularly (#11562). The only thing I wonder about is whether we should block here on having these tests.

I agree that the changes brought by this ticket & branch may increase the chances that htpdate can't connect to some webservers, in particular those managed by Cloudflare.

I'm not sure how likely that regression is: from the links you've shared, I understand that honoring alt-svc is a key part of Cloudflare's special treatment of Tor Browser; and with or without the change this ticket & branch are about, our htpdate does not use alt-svc (https://tools.ietf.org/html/rfc7838) at all, so it already behaves differently than a real Tor Browser would. I don't know which one among these is the most likely to be blocked by Cloudflare:

  • a client that pretends being Tor Browser, connects over Tor, but does not behave like Tor Browser would (i.e. does not follow alt-svc) ← that's the baseline set by our current implementation
  • a client that honestly says it's curl, connects over Tor, and does not behave like Tor Browser would (i.e. does not follow alt-svc) ← that's what the proposed branch does

Keeping in mind that:

  • the proposed branch saves the RM ~30 minutes during every release process, and we have good reasons to optimize this due to the upcoming 4-weeks release cycle;
  • the data point / anecdotal evidence above suggests that there is no such regression at the moment

… at this point, I'm not sure whether the risk for such regressions is big enough to warrant blocking this change on adding tests (#11562).

What do you think?

Personally, I'd prefer not blocking on #11562. But I'm fine with blocking on it if you want, in which case I would re-prioritize #11562, and merge a version of this branch without the htpdate-related commits. Your call!

Btw. did we made sure, that only one pool has domains handled by Cloudflare?

Oh f****, right, excellent question. I just did and as of now:

  • no server is pool 1 is fronted by Cloudflare (rather expected as these folks tend to like self-hosting better than relying on big 3rd-party companies)
  • 3 servers in pool 2 are fronted by Cloudflare
  • no server is pool 3 is fronted by Cloudflare (rather expected as these companies are big enough to manage their own CDN and attack mitigation)

But I think we should check this regularly. Given this specific problem is not a regression brought by the changes this ticket & branch are about, I've filed #17233 about it and I don't think we should block on this here. But perhaps it'll be cheap to add this check when we implement the other checks that your review suggests; we'll see!

#13 Updated by hefee 2 days ago

May thanks for this detailed answer. I learnt lot new stuff.

Personally, I'd prefer not blocking on #11562. But I'm fine with blocking on it if you want, in which case I would re-prioritize #11562, and merge a version of this branch without the htpdate-related commits. Your call!

I think your short script is enough to tell, that it is not very likely to have a big regression. So I'm fine to accept this patch and not block on #11562.

Btw. did we made sure, that only one pool has domains handled by Cloudflare?

Oh f****, right, excellent question. I just did and as of now:

  • no server is pool 1 is fronted by Cloudflare (rather expected as these folks tend to like self-hosting better than relying on big 3rd-party companies)
  • 3 servers in pool 2 are fronted by Cloudflare
  • no server is pool 3 is fronted by Cloudflare (rather expected as these companies are big enough to manage their own CDN and attack mitigation)

But I think we should check this regularly. Given this specific problem is not a regression brought by the changes this ticket & branch are about, I've filed #17233 about it and I don't think we should block on this here. But perhaps it'll be cheap to add this check when we implement the other checks that your review suggests; we'll see!

ACK.

#14 Updated by hefee 2 days ago

I reviewed 05db5aaa114245af4211a19438e4d442b058d617 and fc643144073f1f3975ab7d3f9f22264f74c32fee and both get accepted. But as they are on top of other patches, so I can't merge.

#15 Updated by intrigeri 1 day ago

Hi,

Thanks!

I reviewed 05db5aaa114245af4211a19438e4d442b058d617 and fc643144073f1f3975ab7d3f9f22264f74c32fee and both get accepted. But as they are on top of other patches, so I can't merge.

segfault already reviewed and ACK'ed these other commits on #17143 and #17214; the only blocker for merging was the discussion on this ticket. So I'm going to merge :)

#16 Updated by intrigeri 1 day ago

  • Related to Bug #11562: Monitor servers from the htpdate pools added

#17 Updated by intrigeri 1 day ago

  • Status changed from In Progress to Fix committed
  • % Done changed from 0 to 100

Also available in: Atom PDF