Project

General

Profile

Bug #7161

Support more than 24 HTTP mirrors

Added by intrigeri over 4 years ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Infrastructure
Target version:
Start date:
01/09/2015
Due date:
04/15/2016
% Done:

100%

QA Check:
Feature Branch:
Type of work:
Code
Starter:
No
Affected tool:

Description

We're currently blocked from adding more that 29 mirrors in our dl.amnesia.boum.org round-robin DNS pool.

Either https://trac.torproject.org/projects/tor/ticket/11741 is resolved quickly, or we'll have to find another way to point to, or redirect to, mirrors.


Subtasks

Feature #8635: Make each mirror provide a unique virtualhost nameResolved

Feature #11054: Update contribute/how/mirror documentation with examples for the web servers we supportResolved

Bug #11055: Ask all mirror operators to implement new virtual hostnameResolved

Bug #11056: Wait for most mirrors to implement new virtual hostnameResolved

Feature #11334: Document how HTTP mirrors get a unique virtualhost nameResolved

Feature #11335: Set up the process and tools to allocate unique *.dl.amnesia.boum.org hostnamesResolved

Feature #8637: Design how to convey the mirror pool's configuration to the dispatcher scriptResolved

Feature #8638: Implement how to convey the mirror pool's configuration to the dispatcher scriptResolved

Feature #8639: Write a mirror pool dispatcher scriptResolved

Feature #8640: Have the mirror pool dispatcher library auditedResolved

Bug #11405: Test mirror pool dispatcher script in different browsersResolved

Feature #8642: Enable the mirror pool dispatcher on all website pages that need itResolved

Feature #8643: Clean up the remainers of the old mirror pool setupResolved

Feature #10284: Design how the IA+DAVE will support the new mirror pool systemResolved

Feature #10294: Define format for per-mirror hostnameResolved

Feature #10295: Build a list of fast and reliable HTTP mirrorsResolved

Feature #11079: Publish a call for more HTTP mirrorsResolved

Feature #11109: Have DAVE build the ISO URL using our mirrors pool configurationResolved

Feature #8641: Have the mirror pool dispatcher script deployed on tails.b.oResolved

Feature #11122: Update check-mirrors to support the new mirror pool designResolved

Feature #11123: Have the Upgrader use the new mirror pool designResolved

Feature #11283: Find which of our mirrors should stay in the fallback DNS round-robin poolDuplicate

Feature #11284: Update the DNS round-robin mirror pool to only include a few fast and reliable HTTP mirrorsResolved

Bug #11329: Mark all links on the website which should use the mirror poolResolved

Feature #11384: Record the current state of our mirror pool in JSONResolved

Bug #11482: Add a random sleep time to the rsync cronjob we suggest mirror operators useResolved

Feature #11902: Move the new mirror pool design doc to contribute/designResolved

History

#1 Updated by intrigeri over 4 years ago

The Tor ticket I've created is apparently a duplicate of https://trac.torproject.org/projects/tor/ticket/4734, which clearly won't be resolved quickly. So, we really have to move another way to point to, or redirect to, mirrors.

#2 Updated by ToBeFree over 4 years ago

Here are some alternatives I could think of. Some of them might not really be feasible, though:

  • Modify the DNS server, so that it only serves 29 randomly selected A records from a possibly much larger pool
  • Create an interstitial website which lists all available mirrors, or which automatically selects one of them for an automatic download (SourceForge does that, for example)
  • Create A records for a.dl.amnesia.boum.org, b.dl.amnesia.boum.org, ... - each of them should have at most 29 A records. The DNS server for dl.amnesia.boum.org (which probably needs to be modified for that) now only needs to serve a random CNAME record to one of these subdomains. dl.amnesia.boum.org might point to a.dl.amnesia.boum.org, and that could point to 29 IP addresses.
  • Instead of creating an interstitial website, replace all occurences of links to dl.amnesia.boum.org with [randomcharacter].dl.amnesia.boum.org; imho best implemented using JavaScript and a fallback to dl.amnesia.boum.org
  • Create a script which automatically replaces the current A records with 29 new, random ones from a possibly much larger pool, every x minutes/hours/days, using a simple crontab entry. This might be the easiest solution, and I guess that it would actually be quite powerful. Especially for large .iso images, changing the A records every minute will probably actually have a load distributing effect similar to a huge number of A records.

#3 Updated by Sagi over 4 years ago

Just my 2 cents, some questions:
To keep up with increasing demand, does tails intend to:
  1. spread its download infrastructure as widely as possible (say, bazillion mirrors)
    or
  2. up the requirements for the limited set of mirrors
Some considerations:
  • The disadvantage of uniformly selecting mirrors is that capacity constraints are not taken into account, which becomes a problem once demand is greater than the smallest of mirrors.
  • The disadvantagea of a (large) mirror pool without some form of monitoring is the increasing chance of unmaintained/intermittently failing mirrors getting traffic.

Have you guys taken a look at something like mirrorbrain [1]? I suppose the geo-location functionality is less noticable over Tor, but the ability to assign weights to mirrors and/or specific locations to files might prove useful.

[1] http://mirrorbrain.org/

#4 Updated by sajolida over 4 years ago

Have you guys taken a look at something like mirrorbrain [1]? I suppose the geo-location functionality is less noticable over Tor, but the ability to assign weights to mirrors and/or specific locations to files might prove useful.

[1] http://mirrorbrain.org/

That looks cool.

http://mirrorbrain.org/features/

Unfortunately, it is not available in Debian. But there are unofficially
Debian packages:

http://download.opensuse.org/repositories/Apache:/MirrorBrain/Debian_7.0/

#5 Updated by Sagi over 4 years ago

I guess the disadvantage of such a centralized system is its resilience to interference relative to e.g. DNS-records with high TTL's. But I guess that's part of the problem space here. I'm not sure what the relevant trade-offs are for Tails.

#6 Updated by foob over 4 years ago

I opened a thread on tails-dev@ with some additional informations to this ticket: https://mailman.boum.org/pipermail/tails-dev/2014-May/005733.html . If you would like to participate ... :)

#7 Updated by intrigeri over 4 years ago

Thanks, foob!

For the record, I believe tails-dev@ will be a way better place to discuss this than a ticket: I expect the discussion to need a few dozen emails, and we'll definitely want sub-threads, that are not supported in Redmine.

#8 Updated by intrigeri over 4 years ago

  • Blueprint set to https://tails.boum.org/blueprint/HTTP_mirror_pool/

#9 Updated by intrigeri over 4 years ago

  • % Done changed from 0 to 10

Anyone interested, please add your information (ideally, in an organized way) on the blueprint (either via the web interface, if it works, or as pull requests sent to ), and participate in the discussion on the tails-dev mailing list. Thanks in advance!

#10 Updated by sajolida over 4 years ago

  • Target version set to Sustainability_M1

Tails 2.0 if mission critical, otherwise consider it as non-blocker

#11 Updated by ToBeFree over 4 years ago

(% Done can be at least 50% now imho)

I created the needed JavaScript code now; all that would be left to be done is telling the mirror owners to do a small configuration change, depending on what solution you prefer. :)

There is also an example website for both solutions linked on the blueprint page.

https://tails.boum.org/blueprint/HTTP_mirror_pool/#index3h1

Best regards,
Tobias Frei

#12 Updated by intrigeri over 4 years ago

Replied on tails-dev@. Let's keep the discussion there for now :)

#13 Updated by sajolida about 4 years ago

  • Category changed from Infrastructure to 214

#14 Updated by sajolida almost 4 years ago

  • Subject changed from Support 30+ HTTP mirrors to Support more than 24 HTTP mirrors

#15 Updated by BitingBird almost 4 years ago

  • Category changed from 214 to Infrastructure

#18 Updated by intrigeri over 3 years ago

  • Assignee set to u
  • Target version changed from Sustainability_M1 to Tails_2.3
  • Type of work changed from Research to Code

#20 Updated by u about 3 years ago

We decided to implement a two-way strategy for this feature:

  • keep DNS as is, serving 3 highly reliable mirrors only. This will be the fallback for people who do not use JS.
  • use Vanilla JS (no frameworks) to modify the link on the download page. This JS will be hosted in a git overlay, so not every contributor can interfere with it. It'll load a JSON file, probably hosted on another overlay, with a list of mirrors and an assigned weight. weight 0 would mean that the mirror is currently not active. We'll use JSON and not YAML to avoid 3rd party parsers.

I've updated the blueprint with my working prototype.

#21 Updated by anonym over 2 years ago

  • Target version changed from Tails_2.3 to Tails_2.4

#23 Updated by anonym over 2 years ago

  • Target version changed from Tails_2.4 to Tails_2.5

#24 Updated by BitingBird over 2 years ago

  • Status changed from Confirmed to In Progress

#25 Updated by intrigeri over 2 years ago

  • Target version changed from Tails_2.5 to Tails_2.7

#26 Updated by intrigeri about 2 years ago

  • QA Check set to Ready for QA

(The only two remaining subtasks are Ready for QA, so once they pass their review we can close this ticket as resolved! :)

#27 Updated by u about 2 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (u)

woohoo!

#28 Updated by intrigeri 7 months ago

  • QA Check deleted (Ready for QA)

Also available in: Atom PDF