Project

General

Profile

Feature #8143

Use apt-transport-https to protect against security issues in APT?

Added by hans over 4 years ago. Updated over 2 years ago.

Status:
Rejected
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Start date:
10/16/2014
Due date:
% Done:

0%

QA Check:
Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

Update: this ticket has gone wild, and then has been focussed again. See comment 14 for a current summary, and next things to do.

Initial description follows.

For people who use apt-get in TAILS, like for adding packages to the persistent store, there can be a lot of metadata leaked to the internet when apt-get contacts Debian repositories using a default config. Mostly, that is because by default, the connects are unencrypted (http, ftp, rsync). Chance are that TAILS users using apt-get will each have a unique set of packages that they install, so that provides a unique fingerprint for that TAILS instance.

Luckily, there are some relatively easy steps that greatly reduce the amount of metadata that is leaked: using HTTPS connections to the mirrors, and setting apt-get to pipeline as many transactions into a given HTTPS session. The downside is that it is not common for Debian machines to connect to apt mirrors using HTTPS, so that potentially marks the install as a TAILS install. There are more and more HTTPS mirrors, and more interest in using them, so I think in time, that will only lessen as a concern.

On that note, here is the config that I have been using on a number of Debian-deriv machines, and it has been working well. It requires apt-transport-https.

$ cat /etc/apt/apt.conf.d/99force-tor
# force everything through privoxy HTTP proxy to tor
Acquire::ftp::Proxy "http://127.0.0.1:8118";
Acquire::http::Proxy "http://127.0.0.1:8118";
Acquire::https::Proxy "http://127.0.0.1:8118";

# re-use HTTPS connections to make profiling harder
Acquire::http::Pipeline-Depth "50";
Acquire::https::Pipeline-Depth "50";

Acquire::https::SslForceVersion "TLSv1";

I have found about 10 official Debian mirrors that have reliable HTTPS. Then I have a script that finds all of them (attached), but many have self-signed certs and other issues. A number of the HTTPS mirrors also mirror the "security" archive, but I recommend that the http URL to the official security.debian.org repo is still included to make sure that security updates are promptly available.

I also have a test security repo running that is only available via an .onion address. I hope to encourage people to run official mirrors on a Tor Hidden Service, then HTTPS is not needed. Note that apt-transport-tor is not required if a tor proxy is setup. To try mine, add it to your sources.list (and make sure apt-get is somehow using Tor). The order is important, that determines the priority of where apt-get will get the package from is all other variables are the same.

deb http://dju2peblv7upfz3q.onion/debian-security/ wheezy/updates main
deb http://security.debian.org/ wheezy/updates main

find-https-debian-archives.py View (3.48 KB) hans, 10/16/2014 08:58 AM


Related issues

Related to Tails - Feature #11556: Use Onion Services for APT Resolved 07/03/2016
Related to Tails - Feature #8192: Have HTTPS on all the subdomains of tails.boum.org Resolved 10/31/2014

History

#1 Updated by hans over 4 years ago

I should add, I've had good luck with these HTTPS Debian mirrors:
  • mirrors.ece.ubc.ca
  • mirror.cse.unsw.edu.au
  • mirrors.kernel.org
  • spout.ussg.indiana.edu
Also, there is more discussion on this topic here:

#2 Updated by sajolida over 4 years ago

  • QA Check set to Info Needed

Actually, APT is torified too. From what I read this was unclear to you.
Does you analysis still applies to torified APT traffic?

#3 Updated by hans over 4 years ago

Yeah, sorry, I knew that TAILS forces everything through Tor, I should have removed that part from this post (I'm boiling down discussions from elsewhere).

Given all of the proven fingerprinting approaches, like using the font list from the browser, I think its a safe assumption that the apt-get metadata will also provide similar fingerprinting opportunities. For basic TAILS use, this is all avoided since updates are done via ISO images. But once a user installs packages via apt-get, that changes.

So for example, TAILS does not use the default set of mirrors that a standard Debian install uses. It actually uses a large, customized set of apt sources, and it seems quite likely that the set of mirrors and the order in which they are run will provide a way to identify the system as TAILS.

As for identifying individual machines, apt-get sends a lot of metadata, like language that the system is using, which packages need updates, etc. On top of the set of mirrors used, there is potentially enough info there to fingerprint the individual machine.

So then the question is how the apt-get downloads map to different Tor circuits. If all of the traffic from a given apt-get session goes over a single Tor circuit, then the exit node, the mirror server, and any network observer that can see the traffic between those two can use that as the fingerprint.

#4 Updated by hans over 4 years ago

To expand on this, if TAILS fetched all of its apt sources (wheezy, backports, testing, etc) via HTTPS from the same mirror (e.g. mirrors.kernel.org), then the exit node and network observer could not really distinguish which distro was making the connection since it hosts many distro mirrors. I think this setup would also prevent the custom pattern of repos from being distinguished since it would just show as downloading some series of files, and those files could be packages, repo metadata, perl modules, source tarballs, etc.

#5 Updated by sajolida over 4 years ago

Regarding the set of custom APT sources, you said that this would allow
identifying the system as Tail.

First of all, we use circuit isolation by default for APT as explained
in our design document:

https://tails.boum.org/contribute/design/stream_isolation/

So all the connections to different APT sources will not be processed by
a single exit node.

Furthermore, it is already pretty hard to hide the fact that you are
using Tails in the first place. See
https://tails.boum.org/doc/about/fingerprint/ so this is not something
that we can guarantee. We would like to improve on this situation to
hide Tails usage on the local network which is much more problematic
than to the exit node. Note that your issue only addresses
fingerprinting by the exit node.

In a similar way, we do tons of networking activity through the Tor
network that can only be Tails networking activity (checking for
upgrade for example). So I don't really understand what kind of concrete
and new attacks, being able to spot some APT traffic as coming from
Tails would allow... Could you elaborate on this?

#6 Updated by hans over 4 years ago

I didn't realize that apt-get was setup to use separate Tor circuits for each host/port. That definitely helps quite a bit.

It looks to me that TAILS uses four hosts for apt sources:
  1. ftp.us.debian.org (wheezy, wheezy-backports, wheezy-proposed-updates, testing, unstable, experimental)
  2. security.debian.org (wheezy/updates)
  3. deb.torproject.org (wheezy, sid)
  4. deb.tails.boum.org (1.2)

So the Tor stream isolation would isolate the traffic to/from those hosts. That reduces the identification of TAILS, since Debian developers and pro users often include multiple Debian releases on a machine, like TAILS does. I think there will still be cases where there could be a recognizable, individual fingerprint of packages from looking at the traffic only on ftp.us.debian.org or security.debian.org. That fingerprint would be unique to the given instances of TAILS, based on extra packages that users have installed. It is not uncommon to receive lots of updates from each stable point release (7.1, 7.2, etc), that would probably be the area of highest risk of fingerprinting. Using an HTTPS connection to the mirror would reduce that risk.

It is also not uncommon to get regular updates of lots of packages from security.debian.org. security.debian.org is only available via HTTP, but some of the HTTPS mirrors also include wheezy/updates, so that could be put as a higher priority apt source. That makes apt-get get as many security updates via the HTTPS mirror, while only getting the newest ones from http://security.debian.org, i.e. the ones that have not synced to the mirrors yet.

#7 Updated by hans over 4 years ago

Another possibility is to use a specific mirror for each Debian section, i.e. something like:

deb https://mirrors.kernel.org/debian wheezy main
deb https://mirror.cse.unsw.edu.au/debian wheezy-backports main
deb https://mirrors.ece.ubc.ca/debian wheezy-proposed-updates main
deb https://spout.ussg.indiana.edu/debian testing main
deb https://mirrors.kernel.org/debian unstable main
deb http://ftp.us.debian.org/debian experimental main
deb http://security.debian.org/ wheezy/updates main

This reminds me, since the user can possibly install different versions of a package from different sections (wheezy, backports, experimental, etc.), that will generate more unique fingerprint information.

#8 Updated by sajolida over 4 years ago

  • Type of work changed from Debian to Discuss

You're not answering my question. I still don't understand what kind of concrete and new attacks, being able to spot some APT traffic as coming from Tails on the exit node would allow... Could you elaborate on this?

#9 Updated by hans over 4 years ago

I've always been a defensive thinker, so I'm particularly good at devising concrete attacks. In this case, I can see metadata that can provide strong fingerprinting opportunities. Plus, it turns out that it is straightforward and relatively low risk to fix this issue. I don't think its high priority, but I also don't think it should be ignored.

#10 Updated by sajolida over 4 years ago

To be more explicit, I'm not sure that being able to fingerprint APT
traffic coming from Tails on the exit node is a real threat.

First of all, a vast majority of Tails users won't ever run apt-get.

Second, even if an exit node is able to identify apt traffic as coming
from Tails, the OpenPGP mechanisms of APT would prevent it from
providing rogue packages or malware. So remote exploit is not a risk in
that case I think.

In case a Tails user installs custom software, at most, the exit node
could tell which software packages are installed by a same Tails user.
But without knowing where this user since it comes through Tor and who
she is. Also note that if this user is installing those packages on a
regular basis she might have them stored in the persistent volume and
only download them if a new version appears. That limits the
possibilities of time correlation on those download.

Honestly, I think that all of that is no big deal.

#11 Updated by hans over 4 years ago

I agree that this only applies to the minority of people who are installing extra packages. But if that is a fully supported feature, it should be vetted as much as the rest of TAILS.

Here's one very specific threat that only involves apt: using the fingerprint provided to the mirror to identify a specific computer, a QUANTUMINSERT-style attack could then be used to redirect that computer's traffic intended for ftp.us.debian.org. Since TAILS uses HTTP and a static host as the main apt mirror, it makes it easy to track and run this kind of attack on. The software inserted could then check for identifying information on the local computer, then uninstall if it was the wrong target.

If there was a published update for apt, then this would make this attack much easier. There would be an delay from when the apt update is published to ftp.us.debian.org and the time that TAILS received the update. So when apt connects and asks for the update, the attacker knows that the machine is vulnerable.

This kind of attack could also be applied by someone who gains access to ftp.us.debian.org.

#12 Updated by sajolida over 4 years ago

APT packages are cryptographically signed by their developers using
OpenPGP. They are verified by the client before installation. So
delivering malware through rogue packages is not really an issue that
depends on whether HTTP or HTTPS in use to download them.

#13 Updated by hans over 4 years ago

I didn't mean delivering a bad package, I meant exploiting a vulnerability in apt as it connects via HTTP.

#14 Updated by intrigeri over 4 years ago

Wow, what a discussion. It seems to have drifted from "metadata leakage" to "exploiting remote security issues in APT", I see people not understanding each other, and I see too little threat modeling for my taste. I think we really need a summary. I'll try to provide a not-too-biased one.

First of all, it seems that I should debunk some myths:

  • "APT packages are cryptographically signed by their developers using OpenPGP. They are verified by the client before installation." => yes and no. The OpenPGP signature created by whoever uploads a package to Debian is not visible by the end-user. What the end-user's APT sees is cryptographic signatures of the index files, created by the Debian infrastructure.
  • APT's cryptographic code is not perfect. Very recently (read: in the last few weeks), quite a few issues of the "doesn't properly verify signatures" were discovered. It wasn't the first time, and I wouldn't be surprised if there were more to come. So, some defense in depth can definitely make sense IMO.

Below, I'll be assuming an attacker who:

  • can't forge arbitrary X.509 certificates
  • may run Tor exit nodes
  • may run Debian mirrors
  • may sit in between Tor exit nodes and Debian mirrors

I'll also assume that apt-transport-https correctly verifies X.509 certificate chains.

fingerprinting Tails users

apt-get update

  • currently: probably leaks to the attacker that a Tails user is doing apt-get update using that exit node
  • over HTTPS, minimal number of mirrors: basically the same, since the traffic patterns (size of the downloaded files, etc.) are easily fingerprintable, I guess
  • over HTTPS, spread over a maximal number of mirrors: a bit more complicated, since circuit isolation makes it so we leak less metadata to attackers; but still, that's someone using APT to talk to HTTPS Debian mirrors over Tor... that is, most likely a Tails user

=> not worth bothering; we're not trying to protect against this attack anyway.

apt-get install pkg1 pkg2 pkg3 ...

Here, the attacker wants to fingerprint a given Tails user who has configured "additional software packages".

Below, I'll assume that most packages come from the primary Debian archive (currently: ftp.us.d.o, Wheezy, main+contrib+non-free), and that the list of installed packages coming from this single archive is enough to identify a given Tails user's list of "additional software packages". It's the worst case.

  • currently: an attacker sitting somewhere in between ftp.us.d.o and the exit node used to reach it (inclusive) can probably notice whenever this user boots Tails; this can perhaps be used for confirmation attacks; however, this is somewhat mitigated by circuit isolation (if what the attacker wants to confirm is not publicly visible, then they also have to monitor the traffic they want to confirm on another side of the Internet)
  • HTTPS, regardless of the number of mirrors: basically the same, since the traffic patterns (size of the downloaded files, etc.) are easily fingerprintable, I guess

=> not worth bothering

exploiting remote exploits in APT

  • currently: the attacker can exploit the default HTTP transport, and APT code that handles the downloaded data
  • over HTTPS, regardless of the number of mirrors: the attacker can't exploit the default HTTP transport anymore; instead, they can now exploit apt-transport-https; and anyway, if they control one of the configured APT repositories, they can still exploit the APT code that handles the downloaded data

=> it could be worth using HTTPS

exploiting signature validation issues in APT

This applies to both apt-get update and apt-get install.

  • currently: easy
  • over HTTPS, regardless of the number of mirrors: the attacker needs to control at least mirror configured to be used in Tails

=> it could be worth using HTTPS

(Temporary) conclusion

  • I see no serious fingerprinting attack that's part of our threat model.
  • If apt-transport-https' code is at least as solid as APT's default HTTP transport, then it could be worth using HTTPS to protect against security issues in APT.
  • It doesn't seem useful to spread the load over several mirrors (and taking the risk of desynchronization, and more maintenance work).

So, let's say we do want to use HTTPS to connect to as many APT repositories as possible:

  • Who verifies that the apt-transport-https code is solid enough?
  • Is it compatible with using apt-cacher-ng on build boxes? Apparently apt-cacher-ng doesn't support proxying HTTPS, so we probably have to use a different APT sources configuration at build time from the one configured at runtime. Doable, but needs more patches.
  • Who writes the patches?

#15 Updated by aguelier@ruggedinbox.com over 4 years ago

These are general vulnerabilities that affect the entire clearnet. Facebook has rightly ascertained that the best way to fix it is by running a Tor hidden service and any other site that doesn't want to be vulnerable in this way should really be doing the same (including Debian, the Tor project, Tails, etc).

#16 Updated by hans over 4 years ago

@intrigeri that was a good breakdown of the issues, I agree with your conclusions.

It would be good to be able to access a Debian mirror via a Tor Hidden Service, but there are no stable mirrors with .onion addresses that I could find. I run a security-only mirror on a hidden service, but that's not something for TAILS to rely on, its meant only as a test to encourage mirror operators to set up .onion addresses on their mirrors.

So further discussion about reducing fingerprinting of apt traffic and Tor Hidden Services for Debian mirrors should happen elsewhere, and this thread should be focused on the points that intregeri outlined. I hope to have some time to look into apt-transport-https more in the not too distant future, but don't wait on me.

#17 Updated by intrigeri over 4 years ago

  • Subject changed from reducing metadata leakage from apt-get to Use apt-transport-https to protect against security issues in APT?
  • Category deleted (Persistence)
  • Status changed from New to Confirmed
  • Priority changed from Normal to Low
  • Type of work changed from Discuss to Research

OK, refocussing this ticket on the actual problems that might be mitigated by using apt-transport-https. Next steps are listed on comment 14.

#18 Updated by intrigeri over 4 years ago

  • Description updated (diff)

#19 Updated by hans over 3 years ago

So weasel added a Tor Hidden Service as an official Debian mirror:
http://richardhartmann.de/blog/posts/2015/08/25-Tor-enabled_Debian_mirror_part_2/

So this could greatly improve the issues discussed here by using the onion address to provide end-to-end encryption. I think TAILS is already using Tor TransProxy, so it should just be a matter of making sure that apt is also forced over that, then no other software should be needed (e.g. apt-transport-tor or apt-transport-https)

#20 Updated by intrigeri over 3 years ago

I think TAILS is already using Tor TransProxy,

FYI Tails uses apt-transport-tor.

#21 Updated by intrigeri almost 3 years ago

#22 Updated by hans almost 3 years ago

FYI, weasel just added an official Tor Onion Service mirror of the security archive too. If that is listed as an apt source before security.debian.org, then most security updates will come via the Tor Onion Service, with the classic HTTP mirror as a failsafe backup.

https://onion.debian.org/

#23 Updated by intrigeri over 2 years ago

Is it compatible with using apt-cacher-ng on build boxes? Apparently apt-cacher-ng doesn't support proxying HTTPS, so we probably have to use a different APT sources configuration at build time from the one configured at runtime. Doable, but needs more patches.

I've looked into it. apt-cacher-ng now supports passing HTTPS connections through, but of course it can't cache the corresponding data, so indeed we cannot afford using apt-transport-https at build time. Now, we already reconfigure APT source at boot time (source:config/chroot_local-includes/lib/live/config/1500-reconfigure-APT) so we could as well enable HTTPS there.

#24 Updated by intrigeri over 2 years ago

  • Related to Feature #8192: Have HTTPS on all the subdomains of tails.boum.org added

#25 Updated by hans over 2 years ago

For the record, here is a real world exploit that using Tor and/or HTTPS for all apt sources would protect against:
https://www.debian.org/security/2016/dsa-3733

#26 Updated by intrigeri over 2 years ago

  • Assignee set to intrigeri
  • Target version set to Tails 2.10

I'll give it a try for one of the next 2 major releases.

#27 Updated by intrigeri over 2 years ago

  • QA Check deleted (Info Needed)

#28 Updated by intrigeri over 2 years ago

  • Status changed from Confirmed to Rejected
  • Assignee deleted (intrigeri)
  • Target version deleted (Tails 2.10)

As explained on #11556#note-6, #11556 is cheaper to implement. It also brings essentially the same advantages. HTTPS can come later, on top of the Onion services, if/when they provide such a thing some day.

Also available in: Atom PDF