Project

General

Profile

Bug #14754

Partially applied incremental upgrades cause all kinds of trouble

Added by goupille about 1 year ago. Updated 5 days ago.

Status:
In Progress
Priority:
Elevated
Assignee:
Category:
Hardware support
Target version:
Start date:
10/01/2017
Due date:
% Done:

30%

QA Check:
Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:
Upgrader

Description

multiple users reported that after an upgrade to 3.2, the mouse and keyboard are not working anymore in the greeter

we do not have a lot of data, but, even if it looks like #13576, other hardware are affected (Acer Aspire 1410, Asus Eee PC...)

To help debugging this, see #14754#note-2.

To apply the documented workaround, follow https://tails.boum.org/support/known_issues/#partial-upgrade


Related issues

Related to Tails - Bug #11839: After automatic upgrade from Tails 2.5 to 2.6, keyboard and mouse stop working Rejected 09/24/2016
Related to Tails - Bug #15557: Document workaround for partially applied automatic upgrade Resolved 04/30/2018
Related to Tails - Bug #8449: Tails Upgrader fails to install some IUKs Resolved 12/16/2014
Related to Tails - Feature #15282: Add a prompt right before applying an IUK Fix committed 02/05/2018

Associated revisions

Revision fa1dd840 (diff)
Added by intrigeri 24 days ago

Import tails-iuk's bugfix/14574-partially-applied-upgrades branch (refs: #14754, #15282).

… at commit a8b4f2e6d0c979ca13785a0132aae2165709777e.

Revision 675a8ff8 (diff)
Added by intrigeri 5 days ago

Import tails-iuk's bugfix/14754-partially-applied-upgrades branch (refs: #14754, #15282).

… at commit 267db7d253830625a4f3cf8562640cf53dc935fc.

Revision b7ea41c2 (diff)
Added by intrigeri 5 days ago

Test suite: update picture (refs: #14754)

This text is not bold anymore since
iuk.git:267db7d253830625a4f3cf8562640cf53dc935fc.

Revision 2990428a (diff)
Added by intrigeri 5 days ago

Replace imported tails-iuk patch with the bugfix-14754-partially-applied-upgrades APT overlay (refs: #14754).

The changes that were in this patch have been released in tails-iuk 3.4-1.

Revision 1a742b7a
Added by intrigeri 5 days ago

Merge branch 'bugfix/14754-partially-applied-upgrades' into stable

Fix-committed: #15282
Refs: #14754

History

#1 Updated by goupille about 1 year ago

on the same computers, a clean install of tails 3.1 then an upgrade to 3.2 solved the issue.

#2 Updated by intrigeri about 1 year ago

  • Assignee changed from intrigeri to goupille
  • QA Check set to Info Needed

So I read:

multiple users reported that after an upgrade to 3.2, the mouse and keyboard are not working anymore in the greeter

and later:

on the same computers, a clean install of tails 3.1 then an upgrade to 3.2 solved the issue.

These two situations look very different from where I stand.

How exactly do they differ? In other words, how exactly was the upgrade applied in each case?

I suspect this is related to #11839 so please gather the info I had requested on #11839#note-11 (these commands should be run from a working system, with the faulty Tails device mounted on /media/amnesia/Tails, otherwise they should be adapted).

#3 Updated by intrigeri about 1 year ago

  • Related to Bug #11839: After automatic upgrade from Tails 2.5 to 2.6, keyboard and mouse stop working added

#4 Updated by intrigeri about 1 year ago

  • Priority changed from Normal to Elevated

This was 3rd on the list of hot topics for help desk in September => bumping priority.

#5 Updated by songoku about 1 year ago

I'm using tails in VirtualBox (I know - not recommended but anyway...) and having that exact same issue. So maybe I can provide some information for fixing this quite annyoing bug.

#6 Updated by mercedes508 about 1 year ago

I just received a similar report for upgrade from 3.2 to 3.3, and asked the info from #11839#note-11

#7 Updated by mercedes508 about 1 year ago

  • Status changed from New to Confirmed

#8 Updated by jgallias about 1 year ago

I seem to have something very similar for Tails 3.3 on Lenovo 11e Yoga (Intel)

The behavior first appeared doing an in-place upgrade over Tor to 3.3.
Thinking that maybe something went wrong with the auto-upgrade, I used Bittorrent to download the ISO, PGP verify, then latest Universal-USB-Installer.exe on Windows.
I pulled a new USB Stick out of the package to make sure it wasn't a failing drive.
The resulting Intermediate Tails has no touchpad, no matter how many attempts I make across both drives.

Greeter and GNOME will only accept keyboard/touchscreen input, no touchpad.

I've submitted it through WhisperBack, but this was the closest existing bug I could find.

#9 Updated by u 11 months ago

@helpdesk:

  • Does this only happen for upgrades?
  • Is this still an issue for many users?

#10 Updated by goupille 11 months ago

  • Does this only happen for upgrades?

as far as I remember, it was only reported happening after upgrades

  • Is this still an issue for many users?

I don't think so, there have been some reports these past months but not as much as when I opened the ticket. several users were asked for the debug infos, but most of them fixed their issue before by installing Tails from scratch.

#11 Updated by songoku 11 months ago

I still have it - how can I help?

#12 Updated by goupille 11 months ago

songoku wrote:

I still have it - how can I help?

we still need the information asked by intrigeri in comment #2, could you start your computer with another tails device, plug in the problematic one, open a terminal and gather the output of the commands listed in #11839#note-11 ?

#13 Updated by songoku 11 months ago

Those commands won't work, because there's no Tails directory there, just TailsData - If I use this directory, there's no live directory, hence no Tails.module file

------------------------------------------
Here the commands for reference:
cat /media/amnesia/Tails/live/Tails.module
sha256sum /media/amnesia/Tails/live/*

But also:

ls -l /media/amnesia/Tails/live/
df /media/amnesia/Tails/live/
the date/time when the upgrade was applied

#14 Updated by goupille 8 months ago

there were several users reporting this issue after an upgrade from 3.6.1 to 3.6.2. I asked them the output of those commands, hopefully I'll have something to forward to the FT.

#15 Updated by goupille 8 months ago

  • Assignee changed from goupille to intrigeri
  • QA Check changed from Info Needed to Dev Needed

I forwarded you the info you asked for, the ticket number is in the subject

#16 Updated by intrigeri 8 months ago

So I got two reports with identical symptoms:

  • automatic upgrade from 3.6.1 to 3.6.2
  • 3.6.2.squashfs, initrd.img and vmlinuz have the right size and hash
  • Tails.module was not updated and still lists only filesystem.squashfs

=> the fix I was refering to in #11839#note-12 is not sufficient :/

Dear help desk, please ask the affected users to try this workaround: when upgrading to 3.6.2, add a line that contains "3.6.2.squashfs" (without the quotes) at the end of live/Tails.module. Adjust the version number depending on what version one is upgrading to, e.g. if users have the same problem after upgrading to 3.7, they should add a line that contains "3.7.squashfs" instead.

#17 Updated by intrigeri 8 months ago

  • Target version set to Tails_3.11

Also note that in these two reports, the mtime of Tails.module indicates it has not been modified during the upgrade.

https://git-tails.immerda.ch/iuk/commit/?h=feature/14594-asp-gui&id=968cbd969c3b77bd48865e2ebd3785ca64fb5ab4 might fix this problem. I applied this change in an unrelated branch targetted at our next major release, so we'll only know after this makes it into Tails 3.9, once users upgrade to Tails 3.10.

I've read the corresponding code again and still can't find out why it would sometimes fail (assuming the upgrade did complete successfully, even if one shuts down Tails in an unclean way). So I'll create a ticket to document the workaround and will wait for reports about the 3.9→3.10 upgrade.

#18 Updated by sajolida 7 months ago

intrigeri: Do you think that we should do something specific about this issue on top of #15557?

#19 Updated by intrigeri 7 months ago

sajolida wrote:

intrigeri: Do you think that we should do something specific about this issue on top of #15557?

I don't see what else we can do than wait and see if my fix works (and in the meantime, document the workaround) :/

#20 Updated by intrigeri 6 months ago

  • Related to Bug #15557: Document workaround for partially applied automatic upgrade added

#21 Updated by intrigeri about 2 months ago

I'm told a user reported this problem after upgrading to 3.10.1 :/

#22 Updated by goupille about 2 months ago

intrigeri wrote:

I'm told a user reported this problem after upgrading to 3.10.1 :/

indeed, we got a lot of users reporting that...

I'm wondering if it could just be that they rebooted before the end of the upgrade, tho...

#23 Updated by intrigeri 28 days ago

goupille wrote:

I'm wondering if it could just be that they rebooted before the end of the upgrade, tho...

A friend of mine reported to me "I've upgraded my Tails but it still tells me it needs to be upgraded" so we tried to reproduce the problem together. What happens is that our Upgrader displays a progress bar while downloading, then turns off the network and displays nothing whatsover while it's applying the upgrade, which can take several minutes. It was super unclear to this user that they were supposed to wait before rebooting. I suspect this UX problem is exactly what leads to all these partially applied upgrades.

#24 Updated by intrigeri 25 days ago

  • Related to Bug #8449: Tails Upgrader fails to install some IUKs added

#25 Updated by intrigeri 25 days ago

goupille wrote:

I'm wondering if it could just be that they rebooted before the end of the upgrade, tho...

I'm starting to wonder if we have only one problem left here ("reboot before the end of the upgrade", which is definitely a problem in the real world as shown by my last comment) or if there's still a bug in the way our Upgrader updates Tails.module. I now suspect these bugs were fixed already but have been red herrings so far, that prevented us from zooming out at the overall UX of the upgrade process enough to identify the lack of feedback as another possible cause of trouble.

So in doubt, I think it makes more sense to first focus on fixing the first, obvious problem that we know does exist and can cause the reported behaviour: no feedback while applying the downloaded upgrade leads users to restart the system before the upgrade is fully applied. I'll look at the implementation, will propose something and will ask sajolida to review the UX of my proposal.

And if that's not enough to avoid partially applied upgrades: I took yet another look at the upgrade_modules_file method and have a couple ideas that could possibly fix it and would be worth trying (note to myself: create the temporary modules file in $self->tempdir i.e. on the system partition and then move it atomically; or replace all the code in that method with one single call to run_as_root(nocache + some command that appends the line to the modules file without bothering about replacing it atomically). These should be cheap so I might find time to sneak one of them in the same branch as the obvious UX fix.

#26 Updated by intrigeri 25 days ago

intrigeri wrote:

So in doubt, I think it makes more sense to first focus on fixing the first, obvious problem that we know does exist and can cause the reported behaviour: no feedback while applying the downloaded upgrade leads users to restart the system before the upgrade is fully applied. I'll look at the implementation, will propose something and will ask sajolida to review the UX of my proposal.

Meh. The code to give such feedback is there. So I guess I "just" need to figure out why it doesn't (always?) work and fix it.

#27 Updated by intrigeri 24 days ago

  • Related to Feature #15282: Add a prompt right before applying an IUK added

#28 Updated by intrigeri 24 days ago

  • Feature Branch set to bugfix/14574-partially-applied-upgrades, iuk:bugfix/14574-partially-applied-upgrades

#29 Updated by intrigeri 24 days ago

  • Status changed from Confirmed to In Progress

#30 Updated by intrigeri 24 days ago

  • % Done changed from 0 to 10
  • QA Check deleted (Dev Needed)

#31 Updated by intrigeri 24 days ago

  • Subject changed from Keyboard and mouse do not work after upgrading to tails 3.2 to Partially applied incremental upgrades cause all kinds of trouble

#32 Updated by emmapeel 24 days ago

I am not very sure... Well, maybe this upgrade was longer to apply than usual, but I think there are more reports of this ticket during this upgrade, and we had a lot of reports back in 3.1

(I am not so sure we get the same amount of reports in every update)

#33 Updated by intrigeri 24 days ago

I am not very sure... Well, maybe this upgrade was longer to apply than usual, but I think there are more reports of this ticket during this upgrade, and we had a lot of reports back in 3.1

Interesting! I suspect the regression (lack of feedback while applying the upgrade) was introduced in 3.0, which might explain a sudden surge of such reports when we released 3.1. OTOH we did release 3.0.1 so if my hunch was really correct, the problem should have shown up at that time. So I'm not sure either. At this point, without any clue wrt. how to reproduce the possibly remaining bug, the best I can do is to fix issues I understand first, see if it helps, then try random things that might fix issues I can't reproduce.

#34 Updated by intrigeri 24 days ago

Note to myself: once I have something that fixes the UI + adds #15282, document for sajolida how he can test it for real and ask him to.

#35 Updated by intrigeri 24 days ago

  • Assignee changed from intrigeri to sajolida
  • % Done changed from 10 to 20
  • QA Check set to Ready for QA

As discussed on XMPP, please test this + #15282. Here's how. All commands must be run as the amnesia user.

  1. get an ISO from https://nightly.tails.boum.org/build_Tails_ISO_bugfix-14754-partially-applied-upgrades/lastSuccessful/archive/build-artifacts/
  2. install that ISO to a USB stick with Tails Installer and boot it
  3. set an administration password
  4. run sudo sed -i 's/^TAILS_VERSION_ID=.*$/TAILS_VERSION_ID="1.0~test"/' /etc/os-release
  5. wait for "Tor is ready" then wait 2 or 3 more minutes
  6. run systemctl --user restart tails-upgrade-frontend.service
  7. you should be asked to upgrade to 1.1~test, click through the dialogs and profit
  8. tell me how to make it better and I'll see what I can do & when :)

Note that the "fake" IUK you'll be using is very small so the dialog I got back on this branch appears only for a short time. You might want to use a screen recorder. Or, if you want to test this in a more real world style, instead boot Tails 3.9.1, replace /usr/share/perl5/Tails/IUK/Frontend.pm with https://git.tails.boum.org/iuk/plain/lib/Tails/IUK/Frontend.pm?h=bugfix/14574-partially-applied-upgrades, and then follow the instructions from step 5.

#36 Updated by intrigeri 22 days ago

  • Description updated (diff)

#37 Updated by intrigeri 7 days ago

Scheduling note: this will also need to get a code review once you've done the UX part, and the deadline for merging this in time for 3.11 is at the end of the week.

#38 Updated by sajolida 6 days ago

  • Assignee changed from sajolida to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

It looks great. A few nitpicking comments, while we're modifying strings:

  • "It is recommended to close all the open applications during the upgrade.\n" could become "We recommend you close all other applications during the upgrade.\n which is shorter, more personal, and more consistent with the language in the new dialog. I also wondered if we could remove it. Is closing all other application relevant as well while downloading?
  • It seems like we're using everywhere sentence-style capitalization in button lables instead of header capitalization as per the GNOME guidelines: https://developer.gnome.org/hig/stable/buttons.html.en. This is serious nitpicking, so if changing everything breaks the test suite or translations we should stick to use sentence-style capitalization consistently within this app. Sorry I didn't spot this ealier.
  • Please remove the bold on "The upgrade was successfully downloaded.": this information is already in the title and not the most important thing for the user to learn here.

#39 Updated by intrigeri 6 days ago

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Dev Needed to Ready for QA

Thanks sajolida!

anonym, can you please do a code review? I'll implement the strings change sajolida requested tomorrow, which should not invalidate your review. I'm concerned that if I don't send this for code review now, this certainly won't make it into 3.11. No pressure, if you can't make it, no big deal, it'll wait for 3.12. I should have asked a code review at the same time as the UI one.

#40 Updated by intrigeri 6 days ago

  • Feature Branch changed from bugfix/14574-partially-applied-upgrades, iuk:bugfix/14574-partially-applied-upgrades to bugfix/14754-partially-applied-upgrades, iuk:bugfix/14754-partially-applied-upgrades

Oops, sorry about the wrong branch name.

#41 Updated by intrigeri 5 days ago

  • "It is recommended to close all the open applications during the upgrade.\n" could become "We recommend you close all other applications during the upgrade.\n which is shorter, more personal, and more consistent with the language in the new dialog.

Done. That's not a new string but it was cheap to change. We have the exact same phrasing in the doc (wiki/src/doc/first_steps/upgrade.mdwn) so I've updated it as well: even though the 2 other bullet points in the same bullet list are in the passive voice, I figured that consistency between the doc and the app is more important than intra-list consistency. Revert at any time if you disagree :)

Both changes will break translations but that's life.

I also wondered if we could remove it. Is closing all other application relevant as well while downloading?

It is: an already open application can start eating much more RAM (say, Tor Browser if you open a bunch of tabs) which can make the download fail.

  • It seems like we're using everywhere sentence-style capitalization in button lables instead of header capitalization as per the GNOME guidelines: https://developer.gnome.org/hig/stable/buttons.html.en. This is serious nitpicking, so if changing everything breaks the test suite or translations we should stick to use sentence-style capitalization consistently within this app.

That will break translations and require ~1h of work to update the test suite go through QA.

Sorry I didn't spot this ealier.

Sorry Tails Upgrader is not good enough for you to use it more often :/

  • Please remove the bold on "The upgrade was successfully downloaded.": this information is already in the title and not the most important thing for the user to learn here.

I'm on it (anonym: I'll have to update a test suite image for this so it's expected that what I'm pushing now breaks the test suite).

#42 Updated by intrigeri 5 days ago

intrigeri wrote:

  • Please remove the bold on "The upgrade was successfully downloaded.": this information is already in the title and not the most important thing for the user to learn here.

I'm on it (anonym: I'll have to update a test suite image for this so it's expected that what I'm pushing now breaks the test suite).

Adjusted the test suite, so all done here from my PoV as far as this first attempt for 3.11 is concerned.

#43 Updated by anonym 5 days ago

  • % Done changed from 20 to 30

intrigeri wrote:

anonym, can you please do a code review?

Code looks good!

I am under the impression this is ready to be merge now too, right? I'll optimistically work with that assumption, so I guess you have at least an hour (so I can build and test) to tell me not to in case there's more.

#44 Updated by anonym 5 days ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

anonym wrote:

I am under the impression this is ready to be merge now too, right? I'll optimistically work with that assumption, so I guess you have at least an hour (so I can build and test) to tell me not to in case there's more.

Clarified over XMPP. Back to intri's plate!

#45 Updated by intrigeri 5 days ago

  • Target version changed from Tails_3.11 to Tails_3.13
  • QA Check deleted (Dev Needed)
  • Feature Branch deleted (bugfix/14754-partially-applied-upgrades, iuk:bugfix/14754-partially-applied-upgrades)
  • Affected tool set to Upgrader

OK, merged, let's see how it goes for 3.11→3.12 automatic upgrades.

#46 Updated by intrigeri 5 days ago

(I'll reassign to help desk who can gather the info I need, once I know who'll be on duty the 2 weeks that follow the 3.12 release.)

Also available in: Atom PDF