Project

General

Profile

Feature #14570

Feature #14568: Additional Software Packages

Implement Offline Mode for Additional Software Packages

Added by u over 1 year ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
01/17/2016
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
bugfix/14570-asp-offline-vs-incomplete-previous-upgrade
Type of work:
Code
Starter:
Affected tool:
Additional Software Packages

Description

(B5)

The additional software feature currently only works reliably when connected to the Internet and we want to remove this limitation.


Subtasks

Bug #10958: Disable automatic removal of debs after installResolved


Related issues

Related to Tails - Feature #9819: Check for updates of "additional software" only once Resolved 07/29/2015
Related to Tails - Feature #14571: Code review for Implement Offline Mode Resolved 08/30/2017 01/15/2018
Duplicated by Tails - Feature #6260: Offline additional software Duplicate 09/07/2013
Blocks Tails - Feature #14581: Install caff (signing-party) by default In Progress 09/01/2017
Blocks Tails - Bug #15198: Convert ASP to Python3 and follow PEP-8 Resolved 01/19/2018
Blocked by Tails - Feature #14572: Implement automated tests for Additional Software/Offline Mode Resolved 08/30/2017 01/15/2018

Associated revisions

Revision 58e5e0bc (diff)
Added by intrigeri 12 months ago

ASP: start dumping our analysis (refs: #14570)

Revision a838a5eb (diff)
Added by intrigeri 12 months ago

ASP: drop obsolete possible solutions and merge sections (refs: #14570)

Revision fac21d9d (diff)
Added by intrigeri 12 months ago

ASP: describe better the 2 "new" problems (refs: #14570)

Revision 6fe29852 (diff)
Added by intrigeri 12 months ago

ASP: fix & improve testing procedure (refs: #14570).

Revision c391d839 (diff)
Added by intrigeri 12 months ago

ASP: stuff out the testing procedure (refs: #14570)

Revision 7a0f8800 (diff)
Added by intrigeri 12 months ago

Additional software packages: fix the "incomplete online upgrade process" cause of breakage in offline mode (refs: #14570)

Problem: assume that during an online Tails session, the APT indices are
successfully updated, but then Tails is shut down before the Debian packages
that need upgrading have been downloaded. Then, if Tails is started offline the
next time, the packages that needed to be upgraded could not be installed.

Solution: use updated APT lists when installing packages (before running
`apt-get update`, and possibly offline) only after we know our package cache has
been updated too. In more details:

- Upgrade operation (run once Tor has bootstrapped): save a copy of
`/var/lib/apt/lists/` before running the `apt-get update`;
once `apt-get upgrade` has succeeded, remove this copy of the old
APT lists.
- Installation operation: if there is a copy of the old APT lists, restore it
before running `apt-get install`.

Revision d15ad234 (diff)
Added by alant 11 months ago

ASP: remove now unused import glob (refs: #14570)

It's not used since 05aca68.

Revision 53bdfcdb (diff)
Added by bertagaz 11 months ago

Test suite: add additional software packages feature.

This first iteration implements test of the automatic installation of
additional software packages when Tails is booted in offline mode.

Refs: #14572, #14570, #9059

Revision 7dfa9761
Added by bertagaz 10 months ago

Merge remote-tracking branch 'origin/feature/14572-automated-tests-for-ASP-offline-mode' into devel

Fix-committed: #14570, #14572

History

#1 Updated by u over 1 year ago

#2 Updated by u over 1 year ago

  • Assignee changed from intrigeri to alant

#4 Updated by u over 1 year ago

  • Description updated (diff)

#5 Updated by u over 1 year ago

  • Blocked by Feature #14571: Code review for Implement Offline Mode added

#6 Updated by u over 1 year ago

  • Blocked by deleted (Feature #14571: Code review for Implement Offline Mode)

#7 Updated by u over 1 year ago

#8 Updated by BitingBird over 1 year ago

  • Target version set to 2018

#9 Updated by u over 1 year ago

  • Due date set to 01/15/2018

#10 Updated by u over 1 year ago

  • Related to Feature #14572: Implement automated tests for Additional Software/Offline Mode added

#11 Updated by intrigeri over 1 year ago

#13 Updated by intrigeri over 1 year ago

  • Related to Feature #9819: Check for updates of "additional software" only once added

#14 Updated by intrigeri over 1 year ago

  • Subject changed from Implement Offline Mode to Implement Offline Mode for Additional Software Packages

#17 Updated by u over 1 year ago

  • Affected tool set to Additional Software Packages

#18 Updated by u over 1 year ago

  • Target version changed from 2018 to Tails_3.5

#19 Updated by u about 1 year ago

  • Description updated (diff)

#20 Updated by alant about 1 year ago

  • Blueprint set to https://tails.boum.org/blueprint/additional_software_packages_offline_mode/

#21 Updated by alant about 1 year ago

#22 Updated by intrigeri 12 months ago

  • Related to deleted (Feature #6260: Offline additional software)

#23 Updated by intrigeri 12 months ago

#24 Updated by intrigeri 12 months ago

  • Description updated (diff)

#25 Updated by alant 12 months ago

I tried installing packages offline in Tails 3.3 using the following procedure :

  • start Tails online with persistence of apt packages and apt lists
  • install optipng (currently pulled from stretch/updates, which expires on 22 Dec 2017) and wdiff (from stretch, which doesn't expire) and add them to additional software list
  • reboot offline
  • the install works and optipng version is the one from stretch/updates
  • set the date 1 year in the future in the BIOS
  • reboot offline
  • the install works and optipng version is the one from stretch/updates

I went through the entire procedure 3 times and got the same results. Basic offline operation is thus already working, and #6260 seems to be already resolved : recent APT doesn't check Valid-Until on package installation.

#26 Updated by intrigeri 12 months ago

We've identified 3 causes of issues with offline usage of Additional Software Packages, and described them on the blueprint. Two of these problems have been fixed and Alan will now look closer at the last one.

#27 Updated by intrigeri 12 months ago

  • Status changed from Confirmed to In Progress

#28 Updated by alant 12 months ago

  • Blueprint changed from https://tails.boum.org/blueprint/additional_software_packages_offline_mode/ to https://tails.boum.org/blueprint/additional_software_packages/offline_mode

#29 Updated by intrigeri 12 months ago

  • Description updated (diff)

#30 Updated by intrigeri 12 months ago

  • QA Check set to Ready for QA
  • Feature Branch set to bugfix/14570-asp-offline-vs-incomplete-previous-upgrade

Please:

  • make sure this implements the design you came up with correctly
  • make sure I got right the bits that your design did not specify (e.g. failure modes, when to delete the copy of the old APT lists, etc.)
  • do a first code review so I learn some more Python :)

Then once happy, we'll reassign to segfault or u, who are supposed to review our code before sending to a Foundations Team member's plate for the last (presumably very quick) review and merge.

#31 Updated by intrigeri 12 months ago

intrigeri wrote:

once happy, we'll reassign to segfault or u, who are supposed to review our code

This is supposed to happen on #14571. I'm not sure I understand the intended Redmine workflow/semantics which are different from what we would do (Ready for QA, reassign to the reviewer), but let's try :)

before sending to a Foundations Team member's plate for the last (presumably very quick) review and merge.

Actually, I would like automated tests to be ready before we merge this branch.

#32 Updated by alant 12 months ago

  • Assignee changed from alant to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

I reviewed the code. I fixed 2 whitespace errors and removed the f_ prefix in restore_old_apt_lists, which makes the code less readable to me.

I also think it would be more clean to move content of srcdir to dstdir instead of copying it in restore_old_apt_lists, then call delete_old_apt_lists. If you agree, please do it.

Else, it looks fine. I didn't try the code inside an ISO.

#33 Updated by intrigeri 12 months ago

  • Assignee changed from intrigeri to alant

alant wrote:

I reviewed the code. I fixed 2 whitespace errors and removed the f_ prefix in restore_old_apt_lists, which makes the code less readable to me.

Thanks!

I also think it would be more clean to move content of srcdir to dstdir instead of copying it in restore_old_apt_lists, then call delete_old_apt_lists. If you agree, please do it.

I agree, please do it (you have vastly more time allocated on this ticket than me).

#34 Updated by intrigeri 11 months ago

  • Blocks Feature #14581: Install caff (signing-party) by default added

#35 Updated by u 11 months ago

Hi Alan, when done, please reassign to segfault and point him to #14571. Thanks!

#36 Updated by alant 11 months ago

  • Assignee changed from alant to segfault
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:

alant wrote:

I also think it would be more clean to move content of srcdir to dstdir instead of copying it in restore_old_apt_lists, then call delete_old_apt_lists. If you agree, please do it.

I agree, please do it (you have vastly more time allocated on this ticket than me).

Done and tested inside a running Tails. Please review and merge.

#37 Updated by intrigeri 11 months ago

Please review and merge.

s/and merge/and reassign to anonym for merging/

(and I expect anonym to have a quick look at the diff too as the RM)

#38 Updated by segfault 11 months ago

The code looks good to me, except for some nitpicking:

1. Commit 7a0f8800 introduces a broad except: clause which doesn't log the exception message. I would fix this with the attached patch.

2. According to PEP 8, one-line docstrings should have the closing """ on the same line.

Also, there is some bad code (an unclosed file, statements without effect, a potential reference before assignment) in tails-additional-software, which was not added in this branch, and which gets mostly deleted by bugfix/9059-additional-software-after-session. I would like to take another quick look at it after the two branches are merged.

I'm currently building the ISO to test it. Will report back later.

#39 Updated by u 11 months ago

@segfault: thanks! Once done, pleasr reassign to alant. I'm also adding @anonym as a watcher, as he should at least have a quick look too.

#40 Updated by alant 11 months ago

1. Commit 7a0f8800 introduces a broad except: clause which doesn't log the exception message. I would fix this with the attached patch.

Applied, thanks

2. According to PEP 8, one-line docstrings should have the closing """ on the same line.

I agree, but fixing it here without doing it in all the file looks wierd, so I propose to forget about it for now, and do it later on when converting this file to python3. I plan to do that after all branches are merged and tests are written.

#41 Updated by segfault 11 months ago

  • Assignee changed from segfault to anonym

I'm currently building the ISO to test it. Will report back later.

Seems to work fine.

#42 Updated by u 11 months ago

2. According to PEP 8, one-line docstrings should have the closing """ on the same line.

I agree, but fixing it here without doing it in all the file looks wierd, so I propose to forget about it for now, and do it later on when converting this file to python3. I plan to do that after all branches are merged and tests are written.

I'll create a ticket for that so it's not forgotten.

#43 Updated by u 11 months ago

  • Related to Bug #15198: Convert ASP to Python3 and follow PEP-8 added

#44 Updated by anonym 11 months ago

  • Target version changed from Tails_3.5 to Tails_3.6

#45 Updated by anonym 11 months ago

  • Assignee changed from anonym to bertagaz
  • QA Check changed from Ready for QA to Pass

Looks great! I pushed some minor fixes to whitespaces and comments only. I'd merge it, if I hadn't pushed quite a bit of stuff for #14572, so I'm handing over the merge to the RM.

#46 Updated by u 10 months ago

  • Blocks deleted (Feature #14571: Code review for Implement Offline Mode)

#47 Updated by u 10 months ago

  • Related to Feature #14571: Code review for Implement Offline Mode added

#48 Updated by u 10 months ago

  • Related to deleted (Bug #15198: Convert ASP to Python3 and follow PEP-8)

#49 Updated by u 10 months ago

  • Blocks Bug #15198: Convert ASP to Python3 and follow PEP-8 added

#50 Updated by u 10 months ago

  • Blocks deleted (Bug #15198: Convert ASP to Python3 and follow PEP-8)

#51 Updated by u 10 months ago

  • Blocks Bug #15198: Convert ASP to Python3 and follow PEP-8 added

#52 Updated by intrigeri 10 months ago

  • Related to deleted (Feature #14572: Implement automated tests for Additional Software/Offline Mode)

#53 Updated by intrigeri 10 months ago

  • Blocked by Feature #14572: Implement automated tests for Additional Software/Offline Mode added

#54 Updated by bertagaz 10 months ago

  • Status changed from In Progress to Fix committed

#55 Updated by bertagaz 10 months ago

  • Assignee deleted (bertagaz)

That's now merged into devel and will go into 3.6, woohoo!

#56 Updated by bertagaz 9 months ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF