Project

General

Profile

Bug #15548

Tails can't establish a connection with obfs4 bridges and a hardware clock too far away from UTC

Added by goupille about 1 year ago. Updated about 2 months ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Tor configuration
Target version:
-
Start date:
05/09/2018
Due date:
% Done:

100%

QA Check:
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Several users reported having issues with obfs4 since a few weeks. I tried with the following bridges


obfs4 76.74.178.195:9443 406A8B5869B72221036291407EC3688C69995F80 cert=FY2R16JOoE2VNCU2gVLWBj6Gg+YBP7mTLU5zl12Fz9iC5TQG6SqE71CFhD3zIuJcEFrcMQ iat-mode=0
obfs4 91.89.79.175:43099 681DD069768F8DF5C94284FFE8CEA600C6EAFB86 cert=Dux9bBro6xfxb7rUr8wdp0TephA7wMV0MjPMAcNqkU/r1Q+56EpHypOFZHRZFDukZNCHYg iat-mode=0
obfs4 34.203.58.40:9443 A3C8B95009DB61BE34B2628C9B9B4E66FCA32FA3 cert=etaJXxltJNQxJlKxVgGXWakop242cHNtzxG/GKroXYPTlzvmsYh818WrcIJ5cZ4DbZREJA iat-mode=0

and ended up with Tor launcher stuck on "Loading Network Consensus" for at least 15 minutes. I saw this behaviour two times on three trials.

I attached a whisperback report and the content of /var/log/tor/log, they are showing a weird jump back in the time :

Apr 20 14:17:56.000 [info] connection_edge_process_inbuf(): data from edge while in 'waiting for connect response' state. Sending it anyway. package_partial=1, buflen=214
Apr 20 14:17:56.000 [info] handle_control_authenticate(): Authenticated control connection (23)
Apr 20 11:30:00.000 [notice] Interrupt: exiting cleanly.
Apr 20 11:30:00.000 [info] or_state_save(): Saved state to "/var/lib/tor/state" 

bug report - obfs4 (554 KB) goupille, 04/20/2018 02:24 PM

journal.txt View (401 KB) mercedes508, 06/15/2018 12:22 PM

torlog.txt View (303 KB) mercedes508, 06/15/2018 12:22 PM


Subtasks

Feature #15599: Improve known issues about clock going backwardsResolved


Related issues

Related to Tails - Bug #9268: obfs4 bridges often don't work (maybe MTU?) Confirmed 04/21/2015
Related to Tails - Bug #15743: Too much log when using obfs4 in Tails Resolved 07/19/2018
Blocked by Tails - Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC In Progress 01/15/2018

History

#1 Updated by intrigeri about 1 year ago

  • Subject changed from Tails can't establish a connection with obfs4 bridges to Tails can't establish a connection with obfs4 bridges and a hardware clock >> UTC
  • Category set to Tor configuration
  • Assignee changed from intrigeri to goupille
  • Target version set to Tails_3.7
  • QA Check set to Info Needed
Apr 20 14:17:55 amnesia time[16223]: Waiting for a Tor consensus file to contain a valid time interval
Apr 20 14:17:55 amnesia nm-dispatcher[7795]: /var/lib/tor/ CLOSE_WRITE,CLOSE cached-descriptors.new
Apr 20 14:17:56 amnesia nm-dispatcher[7795]: /var/lib/tor/ CLOSE_WRITE,CLOSE unverified-microdesc-consensus.tmp
Apr 20 14:17:56 amnesia time[16259]: A Tor consensus file now contains a valid time interval.
Apr 20 14:17:56 amnesia time[16262]: We do not have a Tor verified consensus, let's use the unverified one.
Apr 20 14:17:56 amnesia time[16263]: Waiting for the chosen Tor consensus file to contain a valid time interval...
Apr 20 14:17:56 amnesia time[16265]: The chosen Tor consensus now contains a valid time interval, let's use it.
Apr 20 14:17:56 amnesia time[16269]: Tor: valid-after=2018-04-20 10:00:00 | valid-until=2018-04-20 13:00:00
Apr 20 14:17:56 amnesia time[16272]: Current time is 2018-04-20 14:17:56
Apr 20 14:17:56 amnesia time[16277]: Current time is not in valid Tor range, setting to middle of this range: [2018-04-20 11:30:00]
Apr 20 11:30:00 amnesia systemd[1]: Stopping Anonymizing overlay network for TCP...
-- Subject: Unit tor@default.service has begun shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has begun shutting down.
Apr 20 11:30:00 amnesia systemd[1]: Stopped Anonymizing overlay network for TCP.
-- Subject: Unit tor@default.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has finished shutting down.
Apr 20 11:30:00 amnesia systemd[1]: Starting Anonymizing overlay network for TCP...
-- Subject: Unit tor@default.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit tor@default.service has begun starting up.

So there was a mismatch between the system clock (that we initialize to whatever the hardware clock says) and the Tor consensus we got.
To me this looks like "Problems when the system clock goes backwards" from https://tails.boum.org/support/known_issues/.

Could you please ensure the hardware clock is set to UTC (not local time) and retry?

#2 Updated by intrigeri about 1 year ago

  • Related to Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC added

#3 Updated by intrigeri about 1 year ago

Assuming goupille confirms this was caused by a RTC set to localtime >> UTC: if we get something done on #15168 soon, great, this will fix this problem; otherwise, I'm starting to think we should notice when we would like to set the clock backwards and point the user to our doc that explains how they should set their RTC to UTC.

#4 Updated by goupille about 1 year ago

  • Assignee changed from goupille to intrigeri

those logs are coming from a VM, I rebooted the VM three times and could reproduce the error the first and the third times. the lgos are coming from the last session. libvirt says that the VM clock is set to UTC.

jumping to 11:30:00 exactly seems really weird, iirc, gnome's time was set to paris time when the error occured, so UTC should have been 12:17:56, no ?

#5 Updated by intrigeri about 1 year ago

  • Assignee changed from intrigeri to goupille

goupille wrote:

jumping to 11:30:00 exactly seems really weird,

That's not random, it's our time sync system in play: Current time is not in valid Tor range, setting to middle of this range: [2018-04-20 11:30:00]. This happens because "Current time is not in valid Tor range", which should never happen if the RTC is in UTC and the host system's clock is correct.

iirc, gnome's time was set to paris time when the error occured

My Tails VMs (with RTC in UTC) display UTC time before the Tails time sync' is done, as expected. So if you're seeing localtime instead, then it strongly suggests that either your VM actually has a RTC set to Paris time (and not to UTC) — which could be a configuration error or a libvirt/QEMU bug — or the host system has a wrong clock or timezone settings. In any case, the result is the same: Tails is given Paris time and believes it's UTC, which results in "Current time is not in valid Tor range".

Please share:

  • the content of /etc/timezone on the host system
  • the output of date && sudo hwclock on the host system; and check that the time is correct using whatever online clock service you want
  • the full clock section of your test VM's definition (virsh dumpxml $VM)

#6 Updated by intrigeri about 1 year ago

Also, if you want to double check, start Tails, enable offline mode in the Greeter, and check the system time after login: it should be the correct UTC time; if it's not, then the underlying virtualization stack or host system is either buggy or badly configured.

#7 Updated by goupille about 1 year ago

intrigeri wrote:

My Tails VMs (with RTC in UTC) display UTC time before the Tails time sync' is done, as expected.

my mistake, my VMs are displaying UTC time even before the time sync...

  • the content of /etc/timezone on the host system
  • the output of date && sudo hwclock on the host system; and check that the time is correct using whatever online clock service you want
  • the full clock section of your test VM's definition (virsh dumpxml $VM)

I sent you an email with those informations (ticket number in the subject)

#8 Updated by intrigeri about 1 year ago

my mistake, my VMs are displaying UTC time even before the time sync...

OK, then please retry and upon failure, send me:

  • the content of the Journal
  • the Tor log
  • the accurate time (with timezone) from outside of Tails

Rationale: I want to check if the Tor consensus is OK of not.

#9 Updated by Mackpetrova about 1 year ago

Set network proxy to manual and use those settings

#10 Updated by Anonymous about 1 year ago

this has been a problem for soooo long.
time syncing has been failing for months if not years, whenever a bridge (obfs4) is selected right at the startup (non persistent mode).
would be great if this would fixed for good.

it is hilariously stupid to need to select "use a bridge" at startup, connected to a regular tor node until time syncing is done, disconnect again, open up the tor settings fast enough and THEN to insert the bridge info... (_)

#11 Updated by bertagaz about 1 year ago

  • Target version changed from Tails_3.7 to Tails_3.8

#12 Updated by mercedes508 11 months ago

intrigeri wrote:

my mistake, my VMs are displaying UTC time even before the time sync...

OK, then please retry and upon failure, send me:

  • the content of the Journal
  • the Tor log
  • the accurate time (with timezone) from outside of Tails

Rationale: I want to check if the Tor consensus is OK of not.

I'm attaching these logs from a user experiencing this issue.
The timezone is UTC.

#13 Updated by intrigeri 11 months ago

I'm attaching these logs from a user experiencing this issue.

Could you reproduce it?

The timezone is UTC.

You mean the hardware clock?

#14 Updated by intrigeri 11 months ago

  • Target version changed from Tails_3.8 to Tails_3.9

#15 Updated by u 9 months ago

Anonymous wrote:

it is hilariously stupid to need to select "use a bridge" at startup, connected to a regular tor node until time syncing is done, disconnect again, open up the tor settings fast enough and THEN to insert the bridge info... (_)

Please read our code of conduct: https://tails.boum.org/contribute/working_together/code_of_conduct/
We very much appreciate being excellent to each other, thank you.

#16 Updated by u 9 months ago

hi goupille, is this still a thing?

#17 Updated by u 9 months ago

  • Related to Bug #9268: obfs4 bridges often don't work (maybe MTU?) added

#18 Updated by intrigeri 9 months ago

  • Target version changed from Tails_3.9 to Tails_3.10.1

#19 Updated by intrigeri 7 months ago

  • Target version changed from Tails_3.10.1 to Tails_3.11

#20 Updated by intrigeri 6 months ago

Dear help desk, it would be nice to have conclusive, actionable input here: we've been trying to confirm a correlation between the hardware clock timezone and this issue (and #9268) for more than a year, they regularly make it into the help desk's hot topics, and it's still not clear to me what I should do about it.

Let's focus on this simple question: are there users for whom obfs4 never works, despite having a hardware clock set to the correct UTC time?

If the answer is "yes", then we need to investigate further here.

Else, if the answer is "no" or "we don't know" or calming silence, let's mark this ticket as blocked by #15168 (or even reject it) and prioritize that one higher: at least when we'll have fixed #15168 we'll see whether users still have issues with obfs4 :)

Thanks in advance!

#21 Updated by intrigeri 6 months ago

  • Subject changed from Tails can't establish a connection with obfs4 bridges and a hardware clock >> UTC to Tails can't establish a connection with obfs4 bridges and a hardware clock too far away from UTC

Let's focus on this simple question: are there users for whom obfs4 never works, despite having a hardware clock set to the correct UTC time?
If the answer is "yes", then we need to investigate further here.
Else, if the answer is "no" or "we don't know" or calming silence, let's mark this ticket as blocked by #15168 (or even reject it) and prioritize that one higher: at least when we'll have fixed #15168 we'll see whether users still have issues with obfs4 :)

Actually, let's do this instead: if the answer is "yes", file a new ticket. If the answer is "no", then I won't need more input and I'll keep working on it here and on #15168. Thanks!

While looking at #15743 I worked a bit on this today. I got a fresh set of obfs4 bridges that work fine with a RTC in UTC.

If I set the RTC to UTC+1 (<clock offset='variable' adjustment='3600' basis='utc'/> in libvirt), tor never manages to bootstrap. I see:

  • either connection_or_note_state_when_broken(): Connection died in state 'handshaking (proxy) with SSL state (No SSL object)'
  • or clock_skew_warning(): Received directory with skewed time (DIRSERV:148.163.90.126:443): It seems that our clock is ahead by 59 minutes, 58 seconds, or that theirs is behind, or they are sending us the wrong time. Tor requires an accurate clock to work: please check your time, timezone, and date settings

If I set the RTC to UTC-1, tor does bootstrap as I was lucky enough to still be within the consensus validity bounds, which is not guaranteed with a 1-hour incorrect system time, and I see: [info] clock_skew_warning(): Received NETINFO cell with skewed time (OR:148.163.90.126:443): It seems that our clock is behind by 1 hours, 0 minutes, or that theirs is ahead, or they are sending us the wrong time. Tor requires an accurate clock to work: please check your time, timezone, and date settings.

So at least this confirms that we have a problem for obfs4 if the RTC is incorrect or set to a timezone too far away from UTC. What's interesting is that in some cases, we don't even manage to establish a SSL connection to the bridge, so we don't get any time info from it at all, so that's not a failure mode we can recover from even with our dirty "tordate" hacks in 20-time.sh => I think the only way to improve UX in this case is #15168.

#22 Updated by intrigeri 6 months ago

  • Related to deleted (Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC)

#23 Updated by intrigeri 6 months ago

  • Blocked by Bug #15168: Improve UX when hardware clock is set to localtime in a timezone too far from UTC added

#24 Updated by intrigeri 6 months ago

  • Assignee deleted (goupille)
  • Target version deleted (Tails_3.11)
  • QA Check deleted (Info Needed)

#25 Updated by intrigeri 6 months ago

  • Type of work changed from Research to Code

#26 Updated by intrigeri 6 months ago

  • Related to Bug #15743: Too much log when using obfs4 in Tails added

#27 Updated by mercedes508 about 2 months ago

a89500911ff03e90098e12848a5d639e
f4df66ca-7cf4-2484-7ef9-69586df3d173 (message ID from non WB report)

Also available in: Atom PDF