Project

General

Profile

Bug #15548

Tails can't establish a connection with obfs4 bridges and a hardware clock too far away from UTC

Added by goupille over 1 year ago. Updated 8 days ago.

Status:
Confirmed
Priority:
Normal
Assignee:
Category:
Tor configuration
Target version:
-
Start date:
05/09/2018
Due date:
% Done:

0%

Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Several users reported having issues with obfs4 since a few weeks. I tried with the following bridges


obfs4 76.74.178.195:9443 406A8B5869B72221036291407EC3688C69995F80 cert=FY2R16JOoE2VNCU2gVLWBj6Gg+YBP7mTLU5zl12Fz9iC5TQG6SqE71CFhD3zIuJcEFrcMQ iat-mode=0
obfs4 91.89.79.175:43099 681DD069768F8DF5C94284FFE8CEA600C6EAFB86 cert=Dux9bBro6xfxb7rUr8wdp0TephA7wMV0MjPMAcNqkU/r1Q+56EpHypOFZHRZFDukZNCHYg iat-mode=0
obfs4 34.203.58.40:9443 A3C8B95009DB61BE34B2628C9B9B4E66FCA32FA3 cert=etaJXxltJNQxJlKxVgGXWakop242cHNtzxG/GKroXYPTlzvmsYh818WrcIJ5cZ4DbZREJA iat-mode=0

and ended up with Tor launcher stuck on "Loading Network Consensus" for at least 15 minutes. I saw this behaviour two times on three trials.

I attached a whisperback report and the content of /var/log/tor/log, they are showing a weird jump back in the time :

Apr 20 14:17:56.000 [info] connection_edge_process_inbuf(): data from edge while in 'waiting for connect response' state. Sending it anyway. package_partial=1, buflen=214
Apr 20 14:17:56.000 [info] handle_control_authenticate(): Authenticated control connection (23)
Apr 20 11:30:00.000 [notice] Interrupt: exiting cleanly.
Apr 20 11:30:00.000 [info] or_state_save(): Saved state to "/var/lib/tor/state" 

bug report - obfs4 (554 KB) goupille, 04/20/2018 02:24 PM

journal.txt View (401 KB) mercedes508, 06/15/2018 12:22 PM

torlog.txt View (303 KB) mercedes508, 06/15/2018 12:22 PM


Related issues

Related to Tails - Bug #15743: Too much log when using obfs4 in Tails Resolved 07/19/2018
Related to Tails - Bug #16972: tordate sometimes breaks obfs4 by messing with a correct clock Resolved
Related to Tails - Feature #15599: Improve known issues about clock going backwards Resolved 05/09/2018
Duplicated by Tails - Bug #9268: obfs4 bridges often don't work (maybe MTU?) Duplicate 04/21/2015
Blocked by Tails - Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC Resolved 01/15/2018
Blocked by Tails - Feature #5774: Robust time syncing In Progress 05/17/2015

History

#1 Updated by intrigeri over 1 year ago

  • Subject changed from Tails can't establish a connection with obfs4 bridges to Tails can't establish a connection with obfs4 bridges and a hardware clock >> UTC
  • Category set to Tor configuration
  • Assignee changed from intrigeri to goupille
  • Target version set to Tails_3.7
  • QA Check set to Info Needed
Apr 20 14:17:55 amnesia time[16223]: Waiting for a Tor consensus file to contain a valid time interval
Apr 20 14:17:55 amnesia nm-dispatcher[7795]: /var/lib/tor/ CLOSE_WRITE,CLOSE cached-descriptors.new
Apr 20 14:17:56 amnesia nm-dispatcher[7795]: /var/lib/tor/ CLOSE_WRITE,CLOSE unverified-microdesc-consensus.tmp
Apr 20 14:17:56 amnesia time[16259]: A Tor consensus file now contains a valid time interval.
Apr 20 14:17:56 amnesia time[16262]: We do not have a Tor verified consensus, let's use the unverified one.
Apr 20 14:17:56 amnesia time[16263]: Waiting for the chosen Tor consensus file to contain a valid time interval...
Apr 20 14:17:56 amnesia time[16265]: The chosen Tor consensus now contains a valid time interval, let's use it.
Apr 20 14:17:56 amnesia time[16269]: Tor: valid-after=2018-04-20 10:00:00 | valid-until=2018-04-20 13:00:00
Apr 20 14:17:56 amnesia time[16272]: Current time is 2018-04-20 14:17:56
Apr 20 14:17:56 amnesia time[16277]: Current time is not in valid Tor range, setting to middle of this range: [2018-04-20 11:30:00]
Apr 20 11:30:00 amnesia systemd[1]: Stopping Anonymizing overlay network for TCP...
-- Subject: Unit tor@default.service has begun shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has begun shutting down.
Apr 20 11:30:00 amnesia systemd[1]: Stopped Anonymizing overlay network for TCP.
-- Subject: Unit tor@default.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has finished shutting down.
Apr 20 11:30:00 amnesia systemd[1]: Starting Anonymizing overlay network for TCP...
-- Subject: Unit tor@default.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has begun starting up.

So there was a mismatch between the system clock (that we initialize to whatever the hardware clock says) and the Tor consensus we got.
To me this looks like "Problems when the system clock goes backwards" from https://tails.boum.org/support/known_issues/.

Could you please ensure the hardware clock is set to UTC (not local time) and retry?

#2 Updated by intrigeri over 1 year ago

  • Related to Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC added

#3 Updated by intrigeri over 1 year ago

Assuming goupille confirms this was caused by a RTC set to localtime >> UTC: if we get something done on #15168 soon, great, this will fix this problem; otherwise, I'm starting to think we should notice when we would like to set the clock backwards and point the user to our doc that explains how they should set their RTC to UTC.

#4 Updated by goupille over 1 year ago

  • Assignee changed from goupille to intrigeri

those logs are coming from a VM, I rebooted the VM three times and could reproduce the error the first and the third times. the lgos are coming from the last session. libvirt says that the VM clock is set to UTC.

jumping to 11:30:00 exactly seems really weird, iirc, gnome's time was set to paris time when the error occured, so UTC should have been 12:17:56, no ?

#5 Updated by intrigeri over 1 year ago

  • Assignee changed from intrigeri to goupille

goupille wrote:

jumping to 11:30:00 exactly seems really weird,

That's not random, it's our time sync system in play: Current time is not in valid Tor range, setting to middle of this range: [2018-04-20 11:30:00]. This happens because "Current time is not in valid Tor range", which should never happen if the RTC is in UTC and the host system's clock is correct.

iirc, gnome's time was set to paris time when the error occured

My Tails VMs (with RTC in UTC) display UTC time before the Tails time sync' is done, as expected. So if you're seeing localtime instead, then it strongly suggests that either your VM actually has a RTC set to Paris time (and not to UTC) — which could be a configuration error or a libvirt/QEMU bug — or the host system has a wrong clock or timezone settings. In any case, the result is the same: Tails is given Paris time and believes it's UTC, which results in "Current time is not in valid Tor range".

Please share:

  • the content of /etc/timezone on the host system
  • the output of date && sudo hwclock on the host system; and check that the time is correct using whatever online clock service you want
  • the full clock section of your test VM's definition (virsh dumpxml $VM)

#6 Updated by intrigeri over 1 year ago

Also, if you want to double check, start Tails, enable offline mode in the Greeter, and check the system time after login: it should be the correct UTC time; if it's not, then the underlying virtualization stack or host system is either buggy or badly configured.

#7 Updated by goupille over 1 year ago

intrigeri wrote:

My Tails VMs (with RTC in UTC) display UTC time before the Tails time sync' is done, as expected.

my mistake, my VMs are displaying UTC time even before the time sync...

  • the content of /etc/timezone on the host system
  • the output of date && sudo hwclock on the host system; and check that the time is correct using whatever online clock service you want
  • the full clock section of your test VM's definition (virsh dumpxml $VM)

I sent you an email with those informations (ticket number in the subject)

#8 Updated by intrigeri over 1 year ago

my mistake, my VMs are displaying UTC time even before the time sync...

OK, then please retry and upon failure, send me:

  • the content of the Journal
  • the Tor log
  • the accurate time (with timezone) from outside of Tails

Rationale: I want to check if the Tor consensus is OK of not.

#9 Updated by Mackpetrova over 1 year ago

Set network proxy to manual and use those settings

#10 Updated by Anonymous over 1 year ago

this has been a problem for soooo long.
time syncing has been failing for months if not years, whenever a bridge (obfs4) is selected right at the startup (non persistent mode).
would be great if this would fixed for good.

it is hilariously stupid to need to select "use a bridge" at startup, connected to a regular tor node until time syncing is done, disconnect again, open up the tor settings fast enough and THEN to insert the bridge info... (_)

#11 Updated by bertagaz over 1 year ago

  • Target version changed from Tails_3.7 to Tails_3.8

#12 Updated by mercedes508 over 1 year ago

intrigeri wrote:

my mistake, my VMs are displaying UTC time even before the time sync...

OK, then please retry and upon failure, send me:

  • the content of the Journal
  • the Tor log
  • the accurate time (with timezone) from outside of Tails

Rationale: I want to check if the Tor consensus is OK of not.

I'm attaching these logs from a user experiencing this issue.
The timezone is UTC.

#13 Updated by intrigeri over 1 year ago

I'm attaching these logs from a user experiencing this issue.

Could you reproduce it?

The timezone is UTC.

You mean the hardware clock?

#14 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.8 to Tails_3.9

#15 Updated by u about 1 year ago

Anonymous wrote:

it is hilariously stupid to need to select "use a bridge" at startup, connected to a regular tor node until time syncing is done, disconnect again, open up the tor settings fast enough and THEN to insert the bridge info... (_)

Please read our code of conduct: https://tails.boum.org/contribute/working_together/code_of_conduct/
We very much appreciate being excellent to each other, thank you.

#16 Updated by u about 1 year ago

hi goupille, is this still a thing?

#17 Updated by u about 1 year ago

  • Related to Bug #9268: obfs4 bridges often don't work (maybe MTU?) added

#18 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.9 to Tails_3.10.1

#19 Updated by intrigeri 11 months ago

  • Target version changed from Tails_3.10.1 to Tails_3.11

#20 Updated by intrigeri 10 months ago

Dear help desk, it would be nice to have conclusive, actionable input here: we've been trying to confirm a correlation between the hardware clock timezone and this issue (and #9268) for more than a year, they regularly make it into the help desk's hot topics, and it's still not clear to me what I should do about it.

Let's focus on this simple question: are there users for whom obfs4 never works, despite having a hardware clock set to the correct UTC time?

If the answer is "yes", then we need to investigate further here.

Else, if the answer is "no" or "we don't know" or calming silence, let's mark this ticket as blocked by #15168 (or even reject it) and prioritize that one higher: at least when we'll have fixed #15168 we'll see whether users still have issues with obfs4 :)

Thanks in advance!

#21 Updated by intrigeri 10 months ago

  • Subject changed from Tails can't establish a connection with obfs4 bridges and a hardware clock >> UTC to Tails can't establish a connection with obfs4 bridges and a hardware clock too far away from UTC

Let's focus on this simple question: are there users for whom obfs4 never works, despite having a hardware clock set to the correct UTC time?
If the answer is "yes", then we need to investigate further here.
Else, if the answer is "no" or "we don't know" or calming silence, let's mark this ticket as blocked by #15168 (or even reject it) and prioritize that one higher: at least when we'll have fixed #15168 we'll see whether users still have issues with obfs4 :)

Actually, let's do this instead: if the answer is "yes", file a new ticket. If the answer is "no", then I won't need more input and I'll keep working on it here and on #15168. Thanks!

While looking at #15743 I worked a bit on this today. I got a fresh set of obfs4 bridges that work fine with a RTC in UTC.

If I set the RTC to UTC+1 (<clock offset='variable' adjustment='3600' basis='utc'/> in libvirt), tor never manages to bootstrap. I see:

  • either connection_or_note_state_when_broken(): Connection died in state 'handshaking (proxy) with SSL state (No SSL object)'
  • or clock_skew_warning(): Received directory with skewed time (DIRSERV:148.163.90.126:443): It seems that our clock is ahead by 59 minutes, 58 seconds, or that theirs is behind, or they are sending us the wrong time. Tor requires an accurate clock to work: please check your time, timezone, and date settings

If I set the RTC to UTC-1, tor does bootstrap as I was lucky enough to still be within the consensus validity bounds, which is not guaranteed with a 1-hour incorrect system time, and I see: [info] clock_skew_warning(): Received NETINFO cell with skewed time (OR:148.163.90.126:443): It seems that our clock is behind by 1 hours, 0 minutes, or that theirs is ahead, or they are sending us the wrong time. Tor requires an accurate clock to work: please check your time, timezone, and date settings.

So at least this confirms that we have a problem for obfs4 if the RTC is incorrect or set to a timezone too far away from UTC. What's interesting is that in some cases, we don't even manage to establish a SSL connection to the bridge, so we don't get any time info from it at all, so that's not a failure mode we can recover from even with our dirty "tordate" hacks in 20-time.sh => I think the only way to improve UX in this case is #15168.

#22 Updated by intrigeri 10 months ago

  • Related to deleted (Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC)

#23 Updated by intrigeri 10 months ago

  • Blocked by Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC added

#24 Updated by intrigeri 10 months ago

  • Assignee deleted (goupille)
  • Target version deleted (Tails_3.11)
  • QA Check deleted (Info Needed)

#25 Updated by intrigeri 10 months ago

  • Type of work changed from Research to Code

#26 Updated by intrigeri 10 months ago

  • Related to Bug #15743: Too much log when using obfs4 in Tails added

#27 Updated by mercedes508 6 months ago

a89500911ff03e90098e12848a5d639e
f4df66ca-7cf4-2484-7ef9-69586df3d173 (message ID from non WB report)

#28 Updated by intrigeri about 1 month ago

#29 Updated by intrigeri about 1 month ago

  • Related to deleted (Bug #9268: obfs4 bridges often don't work (maybe MTU?))

#30 Updated by intrigeri about 1 month ago

  • Duplicated by Bug #9268: obfs4 bridges often don't work (maybe MTU?) added

#31 Updated by intrigeri about 1 month ago

  • Related to Bug #16972: tordate sometimes breaks obfs4 by messing with a correct clock added

#32 Updated by intrigeri about 1 month ago

intrigeri wrote:

I'm attaching these logs from a user experiencing this issue.

Could you reproduce it?

The timezone is UTC.

You mean the hardware clock?

Dear @mercedes508, AFAICT you did not answer these questions. I guess it's now too late to find out. Still, analyzing the logs lead me to discover a potential bug (#16972) in this area. I'll propose a branch that might help for such situations.

This being said, let's now keep this ticket focused on the "hardware clock too far away from UTC". If you folks get bug reports about obfs4 and a hardware clock that has the correct UTC time (better ask the user to check in the BIOS), please file new tickets :)

#33 Updated by intrigeri about 1 month ago

  • Related to Feature #15599: Improve known issues about clock going backwards added

#34 Updated by intrigeri about 1 month ago

  • % Done changed from 100 to 0

#35 Updated by goupille 8 days ago

  • Assignee set to goupille

Bug report: f24d7bd837478c87bd0afc8b2f8c8460

Also available in: Atom PDF