Project

General

Profile

Bug #17110

WhisperBack cannot send reports since Sep 10: expired (pinned) certificate

Added by numbat 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Infrastructure
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
bugfix/17110-whisperback-drop-tls;whisperback:bugfix/17110-whisperback-drop-tls
Type of work:
Code
Blueprint:
Starter:
Affected tool:
WhisperBack

Description

Whisperback can't send bug reports. Users have reported this and we've noticed this first hand as well.

amnesia@amnesia:~$ whisperback
[WARNING] whisperback.py:150 __load_conf: Failed to load conf /home/amnesia/.whisperback/config.py
[WARNING] whisperback.py:150 __load_conf: Failed to load conf /home/amnesia/config.py
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/whisperBack/whisperback.py", line 245, in save_exception
    func(*args)
  File "/usr/lib/python3/dist-packages/whisperBack/mail.py", line 77, in send_message_tls
    (resp, reply) = smtp.starttls(context=ssl_context)
  File "/usr/lib/python3.5/smtplib.py", line 766, in starttls
    server_hostname=self._host)
  File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 760, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

Related issues

Duplicated by Tails - Bug #17120: No WhisperBack reports since September 10 Duplicate
Blocks Tails - Feature #13242: Core work: Sysadmin (Maintain our already existing services) Confirmed 06/29/2017

Associated revisions

Revision 2b3b51a9 (diff)
Added by segfault 2 months ago

Remove unused WhisperBack TLS certificate (refs: #17110)

We removed TLS support from WhisperBack.

Revision 502a5ad2 (diff)
Added by intrigeri 2 months ago

Enable the bugfix-17110-whisperback-drop-tls APT overlay (refs: #17110).

Revision 2bf52bc0
Added by intrigeri 2 months ago

Merge branch 'bugfix/17110-whisperback-drop-tls' into devel (Closes: #17110)

History

#1 Updated by u 2 months ago

  • Affected tool set to WhisperBack

#2 Updated by goupille 2 months ago

  • Assignee set to intrigeri

#3 Updated by intrigeri 2 months ago

  • Assignee changed from intrigeri to Sysadmins
  • Priority changed from Elevated to High
  • Target version set to Tails_4.0

#4 Updated by intrigeri 2 months ago

  • Duplicated by Bug #17120: No WhisperBack reports since September 10 added

#5 Updated by intrigeri 2 months ago

  • Blocks Feature #13242: Core work: Sysadmin (Maintain our already existing services) added

#6 Updated by intrigeri 2 months ago

sajolida says on #17120 that he received no WhisperBack report since September 10.

#7 Updated by groente 2 months ago

  • Assignee changed from Sysadmins to intrigeri
  • Type of work changed from Sysadmin to Code

the ssl error seems to be given on a connection to the local exim running on tails instances, that's a bit beyond the scope of sysadmin work...

#8 Updated by intrigeri 2 months ago

the ssl error seems to be given on a connection to the local exim running on tails instances, that's a bit beyond the scope of sysadmin work...

@groente: thanks. I understand this is not a sysadmin problem so I'll look into it.

FTR there's no local Exim running in Tails: WhisperBack delivers email directly over SMTP to a Onion service (postfix-hidden) hosted on whisperback.lizard.

#9 Updated by intrigeri 2 months ago

intrigeri wrote:

groente: thanks. I understand this is not a sysadmin problem so I'll look into it.

The bug comes from (an ancient part of our) infra, and leaks into Tails images in order to implement a very strict security setup:

  • On the sysadmin side, tails_secrets_whisperback.git:files/ssl/4mvq3pnvid3awjln.onion.pem, which is used by our WhisperBack SMTP relay (that runs on whisperback.lizard), indeed expired on September 10 ⇒ adding my fellow sysadmins to the loop.
  • A copy of that file lives in tails.git: source:config/chroot_local-includes/etc/whisperback/4mvq3pnvid3awjln.onion.pem, which WhisperBack pins when establishing TLS connections to the WhisperBack SMTP relay ⇒ oh shiiiiiit, there's no way we fix that without releasing a new Tails.

To fix the problem in time for 4.0~rc1, I see these options:

  • Update the certificate (or replace the key pair + self-sign again). Can we have a self-signed cert with no expiration date at all?
  • Drop TLS for WhisperBack: OpenPGP-encrypted email over a Tor onion service should provide enough security, no? Optionally, upgrade the onion service to HSv3 to increase our confidence in the authentication and in the 2nd layer of encryption it provides.

The first option is cheaper to implement right now but it has a higher maintenance cost and will require doing extra work to ensure this problem never happens again.
The second option requires a bit more work right now but then we're done.

Personally, I think TLS is quite overkill here, and I'm leaning towards the second option. groente, segfault, zen: any thoughts on this?

And then, depending on how we decide to fix the problem, we may or may not need to take extra action to ensure that the problem never comes back. For example, if we decide to keep TLS with an expiration date in that loop:

  • Apparently we have no check anywhere to alert us when this certificate will expire soon → we should add either a monitoring check for this on the infra side (do we already have checks for upcoming certs expiration that we could easily extend?), or add a test case to the Tails automated test suite (where we already check that various OpenPGP keys expire in more than 3 months).
  • Make the monitoring check for the WhisperBack SMTP relay, which still passes flawlessly (ahem), test something that better reflects the actual use case for this service. Granted, this wouldn't have alerted us in time to fix the problem in Tails 3.16, but still, I would have preferred having a full month to fix the problem, instead of 6 days.

Post-mortem: this piece of infra was set up in September 2009, a few weeks after the first public release of amnesia. For some reason we wanted TLS on top of encrypting email with OpenPGP + the end-to-end authentication + encryption provided by Tor onion services. So we created a TLS key pair. The self-signed certificate was set to expire 10 years later, that is, last month. The project was way too immature back then to think about what would happen 10 years later and set up a process to update this certificate. Oh well :)

#10 Updated by intrigeri 2 months ago

  • Subject changed from Whisperback - certificate verify failed to WhisperBack cannot send reports since Sep 10: expired (pinned) certificate
  • Status changed from Confirmed to In Progress

#11 Updated by intrigeri 2 months ago

#12 Updated by anonym 2 months ago

intrigeri wrote:

The second option requires a bit more work right now but then we're done.

Well, if we go with option 1 we will have a clear deadline for when we need to do option 2 (or update the cert yet again) so I think it is fine to do the cheaper option now and add clear plans for dealing with 2 later, but in good time before the next cert expiry.

Personally, I think TLS is quite overkill here, and I'm leaning towards the second option.

100% agreed. If there is anything for us to improve in this area's security story it is to update to v3 onions (v2 onions, among other things, depend on rsa1024 which is starting to get a bit too weak).

#13 Updated by segfault 2 months ago

  • Assignee changed from intrigeri to segfault

#14 Updated by segfault 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (segfault)
  • Feature Branch set to bugfix/17110-whisperback-drop-tls;whisperback:bugfix/17110-whisperback-drop-tls

#15 Updated by intrigeri 2 months ago

  • Assignee set to intrigeri

LGTM at first glance.

Next steps:

  1. drop TLS server-side
  2. do a more proper code review and test
  3. if time allows, switch to HSv3 (requires changes on both client and server side + in our infra monitoring); else, file a ticket to do that later

#16 Updated by segfault 2 months ago

Note that I tested this by copying the modified files to a running Tails and sending a test report. WhisperBack said that the report was sent successfully.

#17 Updated by intrigeri 2 months ago

  • Status changed from Needs Validation to In Progress

#18 Updated by intrigeri 2 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

#19 Updated by intrigeri 2 months ago

  • Status changed from Resolved to Needs Validation
  • Assignee changed from intrigeri to groente

Next steps:

  1. drop TLS server-side
  2. do a more proper code review and test
  3. if time allows, switch to HSv3 (requires changes on both client and server side + in our infra monitoring)

All done!

While we certainly could have postponed the migration to a HSv3, I figured that now is a good time to do it: the WhisperBack feedback loop is broken already, so we can introduce backwards-incompatible changes without having to deal with the complexity of a transition period, during which we would have to support both the old v2 Onion service and the new v3 one.

Dear groente, can you please review the Puppet code changes? They've all been done today and the most relevant ones are tagged with this ticket ID.

#20 Updated by intrigeri about 2 months ago

#21 Updated by intrigeri about 2 months ago

  • Target version changed from Tails_4.0 to Tails_4.1

#22 Updated by groente about 1 month ago

  • Status changed from Needs Validation to Resolved

looks good!

Also available in: Atom PDF