Project

General

Profile

Feature #15875

Make it possible to resume an IUK download from within Tails

Added by Anonymous over 1 year ago. Updated 19 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
08/30/2018
Due date:
% Done:

0%

Feature Branch:
feature/15875-upgrader-resume-iuk-download
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Upgrader

Description

reported by users, but we do it only if it's cheap to implement because it might be superseded if we look into replacing our upgrader.

Downloading target files should be more robust on flaky Internet connections.

E.g. when a target file download times out, it should be retried a few times. Ideally, the download should be resumed using the already downloaded partial content.

Tails::IUK::TargetFile::Download currently uses Tails::IUK::LWP::UserAgent::WithProgress, which is a child class of LWP::UserAgent.

Possible implentation ideas:

If we instead used Curl:

Alternatively, there could be a button for manually retrying or resuming an interrupted download.


Related issues

Related to Tails - Feature #14544: Spend software developer time on smallish UX improvements In Progress 08/31/2018
Related to Tails - Feature #15277: Update our survey of non-NIH system upgrade solutions Confirmed 02/03/2018
Related to Tails - Bug #14754: Partially applied incremental upgrades cause all kinds of trouble Confirmed 03/21/2019
Duplicated by Tails - Feature #5825: Retry incremental upgrade downloads Duplicate

Associated revisions

Revision 771f8ab9 (diff)
Added by intrigeri 3 months ago

WIP: Make it possible to resume an IUK download from within Tails (refs: #15875)

Revision 9ca8b5ce (diff)
Added by intrigeri 3 months ago

WIP: Make it possible to resume an IUK download from within Tails (refs: #15875)

Revision 83cfbe15 (diff)
Added by intrigeri 29 days ago

WIP: Make it possible to resume an IUK download from within Tails (refs: #15875)

History

#1 Updated by sajolida over 1 year ago

  • Related to Feature #14544: Spend software developer time on smallish UX improvements added

#2 Updated by intrigeri over 1 year ago

  • Related to Feature #15277: Update our survey of non-NIH system upgrade solutions added

#3 Updated by intrigeri over 1 year ago

  • Duplicated by Feature #5825: Retry incremental upgrade downloads added

#4 Updated by intrigeri over 1 year ago

  • Description updated (diff)

#5 Updated by intrigeri over 1 year ago

  • Affected tool set to Upgrader

#6 Updated by intrigeri over 1 year ago

  • Description updated (diff)

#7 Updated by sampalmer over 1 year ago

...
Alternatively, there could be a button for manually retrying or resuming an interrupted download.

This is a great initiative. The buttons for the user to click would be most simple to implement.

If a download stalls, the user is told. The user may click "Resume". It might fail, but the user might know that's because their internet is down, so the user waits longer before resuming again. Then it fails again, despite other internet systems working. The user then decides to restart the download: "Restart".

As you can see here, putting the user in charge means we don't need to anticipate design for many scenarios. Put the user in the loop, given them recommendation of the action they should take, and then let them have control.

This should result in less "errors" in this feature.

In the future, we might revisit this and make it "better". But I suspect that having the buttons for the user to control will be enough and also the best final solution.

Agreed?

#9 Updated by intrigeri 4 months ago

  • Target version changed from 2019 to Tails_4.2

The plan is to try to do this if it fits in the Upgrader work for #15281. Let's see how it goes in December.

#10 Updated by intrigeri 3 months ago

  • Description updated (diff)

#11 Updated by intrigeri 3 months ago

  • Description updated (diff)

#12 Updated by intrigeri 3 months ago

  • Description updated (diff)

#13 Updated by intrigeri 3 months ago

tl;dr, primarily for @sajolida: if you don't want to read my full reasoning, you can check what I'm proposing and tell me if that seems OK → look for "I propose" below.

The baseline is that when the download fails, we tell this to the user:

The upgrade could not be downloaded.

Check your network connection, and restart Tails to try upgrading again.

If the problem persists, go to file:///usr/share/doc/tails/website/doc/upgrade/error/download.en.html

That is, we already offer a (painful) retry-from-scratch mechanism. Moreover, the reason why we put this ticket near the top of my plate recently was #15281: "the upgrades will be bigger, so both the risk and impact of an interrupted download are higher".

So, while it seems rather cheap to offer retrying the download from scratch when it fails, I'm not sure if it's worth the effort in itself, and I don't think that's the right place where we should focus. I believe the biggest UX win here would come from resuming the download, keeping the data that was already downloaded.

@sajolida, are we on the same page so far?

In terms of implementation cost, from lower to higher:

  • (A) If the download fails, the user is offered to retry it from scratch.
  • (A + new circuit) Same as A, but a different Tor circuit is used for every retry.
  • (A + new mirror) Same as A, but a different mirror is used for every retry.
  • (B) If the download fails, it is silently retried N times from the beginning of the file and from the same mirror. The user is not told anything explicit about what's going on, apart of the fact the progress bar restarts from 0.
  • (B + new circuit) Same as B, but a new Tor circuit is used for every retry.
  • (C) If the download fails, it is silently resumed, reusing the previously downloaded data, from the same mirror. The user is not told anything explicit about what's going on. The progress bar keeps moving towards 100% monotonically, likely with pauses between download attempts.
  • (C + new circuit) Same as C, but a new Tor circuit is used for every retry.
  • (C + fallback DNS mirrors pool) Same as C, but the 2nd attempt uses the fallback DNS pool of mirrors, which are fast and reliable.
  • (C + new circuit + fallback DNS mirrors pool)
  • (B + new mirror) Same as B, but a different mirror is used for every retry.
  • (C + new mirror) Same as C, but a different mirror is used for every retry.
  • Other ideas?

IMO:

  • As I've argued above, I doubt (A) and its variants are worth the effort. I'd rather spend my time on something that makes a bigger difference for our users.
  • The UX of (B) and its variants is super messy and confusing.
  • The sweet spots in terms of implementation cost / UX benefit are (C + new circuit) and (C + new circuit + fallback DNS pool). But doing only (C) would already be a pretty good start.

⇒ I propose to implement (C) and then, if time allows, extend it with the "+ new circuit" and/or "+ fallback DNS mirrors pool" bonuses. What do you think?

Given that's consistent with what the description of this ticket says since 14 months, and in the meantime we added this ticket as-is to our S02 plans, I'll optimistically assume this is consistent with what we already agreed upon. So if time allows, I'll start work in this direction this week, without waiting for an answer, in the hope I can get something ready in time for 4.2, which would improve UX when users upgrade to 4.3.

Note that given how our current Upgrader is implemented, there's a delay between the time when we hide a dialog and when we display the next one. This delay is generally pretty short but on slower computer it can last a few seconds. We know that when this happens, some users sometimes (mis)understand that the upgrade was applied, and then restart their computer. Adding more interaction with the user implies more dialogs popping up and disappearing, and thus more times when the screen is empty and it's unclear what's going on. So I have slight doubts about whether the options that introduce more interaction would improve UX more than they would harm it; it depends on the user though.

Frankly, at this point I'm starting to be really annoyed by Upgrader's half-assed GUI based on zenity dialogs. It really restricts a lot what we can do; it's the cause for the delays I've talked about above, and for some other UX bugs in the upgrade process. I would love to replace this with a proper GUI that continuously keeps the user aware of what's going on, but given our plans/hopes to ditch this Upgrader as soon as we can (with the added benefit that more people would feel comfortable working on it), I don't think it would be sensible to invest lots of time in designing and implementing this now. Too bad, we should perhaps have done this a few years back, but oh well, we had other priorities.

#14 Updated by intrigeri 3 months ago

I've updated our check-mirrors.rb script so it verifies that all mirrors support Range requests: we don't want resuming to randomly fail depending on which mirror was picked.

If we end up implementing the "fallback DNS mirrors pool" feature mentioned in my previous comment, then technically this check will be stricter than needed: we would only need mirrors that are in the fallback DNS mirrors pool to accept Range requests. But in theory our mirror pool operators could add any mirror to that DNS pool at any time, so if we dropped this check, they would have to think about checking this before adding a mirror to the DNS pool, which feels tedious and error-prone.

FTR, at the moment all our active mirrors support Range requests. All commonly used webservers support it by default and apparently it's rarely disabled, so I won't bother documenting this in contribute/how/mirror: if the operator of a new candidate mirror has disabled Range requests, we'll notice when we run check-mirrors.rb on it and it'll be a good time to explain them, without having to make the doc more complex for everybody else.

#15 Updated by sajolida 3 months ago

It seems like your check-mirror.git:5d78d66 broken the cron job. See <>.

#16 Updated by intrigeri 3 months ago

It seems like your check-mirror.git:5d78d66 broken the cron job. See <>.

Gah, Ruby 2.3's Regexp class has no match? method :/ Should be fixed in Git, let's see what the next cronjob run has to say.

#17 Updated by intrigeri 3 months ago

  • Status changed from Confirmed to In Progress
  • Target version changed from Tails_4.2 to Tails_4.3
  • Feature Branch set to feature/15875-upgrader-resume-iuk-download

Meanwhile, I've implemented (C). It passes the corresponding test suites. I'm not 100% happy with the implementation and interestingly, there's a chance that (C + new circuit) and/or (C + fallback DNS pool) allow me to improve on this.

This is not going to make it into 4.2: until yesterday I had no idea if #15281 would make it into 4.2, and I'd like sajolida's input before I work on this again (best case: around Jan 25-28) ⇒ postponing.

#18 Updated by intrigeri 3 months ago

  • Description updated (diff)

#19 Updated by intrigeri 3 months ago

  • Assignee changed from intrigeri to sajolida
  • Type of work changed from Code to User interface design

intrigeri wrote:

I'd like sajolida's input before I work on this again (best case: around Jan 25-28) ⇒ postponing.

Please reassign to me once you've provided this input :)

#20 Updated by sajolida about 2 months ago

Rolling back a bit, I couldn't find notes on what user scenarios we wanted to handle with this resume.

You're been working on:

  • (S1) The network connection or the mirror fails, cf. the "flaky" in the description of this ticket.

For this, indeed (C) and its variants are the sweet spot and this will already be a great improvement.

I've also had in mind other user stories when people try to upgrade on a super slow connection and downloading takes several hours:

  • (S2) I don't have the time to download everything in 1 session. Maybe I'm traveling (Cris) or maybe I have limited time on Tails (Kim). In this case, I would like to resume my download the next time I start Tails.
  • (S3) The download is eating all my bandwidth and I need to pause it momentarily to do something else. In this case, I would like to be able to pause the download myself and resume it when convenient.

We discussed both user stories with intrigeri and they seemed doable when the user has a Persistent Storage unlocked. Instead of downloading to /tmp, Tails Upgrader could download to the Persistent Storage.

To solve (S2):

  • Tails Upgrader would offer to do an upgrade again next time, maybe with a slightly adjusted prompt, and then resume the download and the progress bar where it stopped.

To solve (S3):

  • If the user tries to close Tails Upgrader, another dialog could appear and offer to resume the download and the upgrade.

Solving (S2) and (S3) without a Persistent Storage would be quite harder.

#21 Updated by anonym about 2 months ago

  • Target version changed from Tails_4.3 to Tails_4.4

#22 Updated by intrigeri 29 days ago

  • Assignee changed from sajolida to intrigeri

Oops, we did not reassign this to me so it got kinda off my radar.

#23 Updated by intrigeri 29 days ago

  • Target version changed from Tails_4.4 to Tails_4.5

No way this gets into 4.4, but 4.5 might be feasible at least for a first iteration.

#24 Updated by intrigeri 28 days ago

  • Related to Bug #14754: Partially applied incremental upgrades cause all kinds of trouble added

#25 Updated by intrigeri 22 days ago

  • Type of work changed from User interface design to Code

#26 Updated by intrigeri 19 days ago

  • Target version changed from Tails_4.5 to Tails_4.6

Also available in: Atom PDF