Project

General

Profile

Feature #15091

Upgrade to Thunderbird 60

Added by intrigeri over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
05/09/2018
Due date:
% Done:

100%

Estimated time:
10.00 h
Feature Branch:
kibi:feature/15091-thunderbird-60, icedove:feature/15091-thunderbird-60
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Email Client

Description

See https://wiki.mozilla.org/Thunderbird/Add-ons_Guide_57: legacy add-ons are still supported but may require adjustments as many interfaces are being removed in Thunderbird 57, 58 and 59.

The next ESR will be Thunderbird 60, which we'll probably have to upgrade to in Tails 3.9 because it'll be the only way for us to get Thunderbird security updates from Debian.

FWIW:

Rough upstream timeline:

  • done: release 60.0b9 to fix https://bugzilla.mozilla.org/show_bug.cgi?id=394322
  • done (July 10): release 52.9.1 for security updates that were not ready for 52.8.0 (52.9 was messed up and won't be released)
  • done (July 10): release 60.0b10
  • release 60.0 for manual updates only (probably in July)
  • release 60.1.0 and 52.9.0 (July)
  • release 60.2.0 and end of life for 52 (September)

Subtasks

Bug #15600: Upgrade to Torbirdy 0.2.5Resolved


Related issues

Related to Tails - Bug #15791: Update test suite for Thunderbird 60 Resolved 08/15/2018
Related to Tails - Feature #15792: Upgrade to Thunderbird 60 final Resolved 08/15/2018
Blocks Tails - Bug #11082: Replace Liferea Resolved 05/05/2016
Blocks Tails - Feature #15334: Core work 2018Q3: Foundations Team Resolved 02/20/2018
Blocks Tails - Feature #15745: Update doc wrt. Liferea → Thunderbird as the default feed reader Resolved 07/22/2018
Blocks Tails - Bug #15602: Fix EFAIL Resolved 05/14/2018
Blocks Tails - Bug #15551: Generating a revocation certificate with Enigmail fails Resolved 04/22/2018
Blocks Tails - Bug #15775: Adjust persistence setup to Thunderbird being recommended for feeds Resolved 08/08/2018

Associated revisions

Revision 8921d19b (diff)
Added by intrigeri about 1 year ago

Enable the feature-15091-thunderbird-60 APT overlay (refs: #15091).

Revision 25faf4ed (diff)
Added by Cyril Brulebois about 1 year ago

Install Torbirdy from stretch-backports (refs: #15091).

Torbirdy 0.2.5, compatible with TB60 (see #15600), is available from
stretch-backports. Let's adjust APT pinning to use this version.

Revision 3c84e995 (diff)
Added by Cyril Brulebois about 1 year ago

Torbirdy: drop local patch following the bump to 0.2.5 (refs: #15091, #15600).

The chrome/content/preferences.js part is spotted as being already
applied, while the components/torbirdy.js part cannot be applied.
Upon inspection, the diff was applied upstream, the failure to apply
was triggered by a context change.

Revision c4c8d79b (diff)
Added by Cyril Brulebois about 1 year ago

Torbirdy: update another local patch following the bump to 0.2.5 (refs: #15091, #15600).

Most of these changes were merged upstream, except the specific proxy
setting: default to socks5h://127.0.0.1:9050 proxy unless anonService is
set to “jondo”.

Revision 05e62eed (diff)
Added by Cyril Brulebois about 1 year ago

Thunderbird: Refresh apparmor profile for TB60 (refs: #15091).

Revision 549301c7 (diff)
Added by Cyril Brulebois about 1 year ago

Torbirdy: Enable the email wizard in the system configuration (refs: #15091, #15600).

There's:
/usr/share/xul-ext/torbirdy/defaults/preferences/000system.js → /etc/xul-ext/torbirdy.js

which is read after:
/usr/share/xul-ext/torbirdy/defaults/preferences/prefs.js

but setting extensions.torbirdy.emailwizard to true in the former
doesn't seem sufficient, even though strace confirms it's being read.

Revision 01df2729 (diff)
Added by Cyril Brulebois about 1 year ago

Thunderbird: Import locale selection handling from Tor Browser (refs: #15091).

Since TB60, intl.locale.matchOs is no longer supported (as documented in
its README.Debian), so let's adapt what's being used in Tor Browser.

Revision b6e401f9 (diff)
Added by Cyril Brulebois about 1 year ago

Thunderbird: Fix regression in defaultprotocol handling (refs: #15091).

Preferences handling seems to have changed heavily in TB60, so switch to a
user_pref in prefs.js instead of a regular pref in preferences/0000tails.js

Another possibility would be to use /etc/thunderbird/pref/thunderbird.js but
that would need to happen in gdm's PostLogin (root privileges would be
required).

Revision fdcfa34e (diff)
Added by Cyril Brulebois about 1 year ago

Torbirdy: Enable the gpg_already_torified setting in the system configuration (refs: #15091, #15600).

There's:
/usr/share/xul-ext/torbirdy/defaults/preferences/000system.js → /etc/xul-ext/torbirdy.js

which is read after:
/usr/share/xul-ext/torbirdy/defaults/preferences/prefs.js

but setting extensions.torbirdy.gpg_already_torified to true in the former
doesn't seem sufficient, even though strace confirms it's being read.

Without this setting, fetching key from a keyserver doesn't work out of the
box.

Revision d6e440b0 (diff)
Added by Cyril Brulebois about 1 year ago

Torbirdy: Drop obsolete system configuration file (refs: #15091, #15600).

After the configuration handling changes implemented in TB 60, it seems
that this file is read but its contents seem ignored, so remove it
entirely and rely on manual patches (see previous commits) to set these:
- extensions.torbirdy.emailwizard: true
- extensions.torbirdy.gpg_already_torified: true

Revision 5f202052 (diff)
Added by intrigeri about 1 year ago

Thunderbird: update auto config prefs to match updated patch set (refs: #15091).

The new patch set introduces, or modifies the behaviour of, 4 prefs:

  • mailnews.auto_config.account_constraints.allow_oauth2 and
    mailnews.auto_config.account_constraints.ssl_only have been renamed; this
    commit updates our prefs accordingly
  • mailnews.auto_config.fetchFromISP.ssl_only → brand new, false by default;
    let's enable it to get the main benefit from our Thunderbird patches
  • mailnews.auto_config.guess.timeout → brand new, introduced precisely so that
    we can use a higher value, which I'm doing here.

Revision ad91a77e (diff)
Added by Cyril Brulebois about 1 year ago

Thunderbird: check the lang packs are suffixed @firefox.mozilla.org (refs: #15091).

It was slightly surprising to spot @firefox.mozilla.org as a suffix
instead of the traditional @thunderbird.mozilla.org one. This was
reported to Debian#906176 (https://bugs.debian.org/906176).

Let's check how many language packages are found with the suffix
used in tails-shell-library/thunderbird.sh, to make sure l10n has
a chance to work.

Revision 182dc799
Added by intrigeri about 1 year ago

Merge remote-tracking branch 'origin/feature/15091-thunderbird-60' into devel (refs: #15091, #15600)

Revision 0c2bfb6e (diff)
Added by Cyril Brulebois 12 months ago

Revert "Thunderbird: Import locale selection handling from Tor Browser (refs: #15091)." (refs: #15959).

This reverts commit 01df272943269352358aa1974cb6d90ddb36e926.

By merging thunderbird 1:60.0-3 from Debian, we should get a proper
setting for intl.locale.requested, which is an empty string. This
replaces the old “intl.locale.matchOs = true” behaviour.

This also makes our efforts to set this parameter based on the locale
settings moot, so let's revert it all and let the built-in code do its
job instead.

History

#1 Updated by intrigeri over 1 year ago

#2 Updated by intrigeri over 1 year ago

  • Assignee set to anonym
  • Target version set to Tails_3.6

anonym, do you think you can handle the initial assessment (what needs to be done and when, which probably requires testing the add-ons we ship on the latest Thunderbird non-ESR/beta releases) in 2018Q1?

#3 Updated by anonym over 1 year ago

intrigeri wrote:

anonym, do you think you can handle the initial assessment (what needs to be done and when, which probably requires testing the add-ons we ship on the latest Thunderbird non-ESR/beta releases) in 2018Q1?

Certainly!

#4 Updated by anonym over 1 year ago

  • Assignee changed from anonym to intrigeri
  • QA Check set to Info Needed

intrigeri wrote:

I don't know what the timeline for this is. Likely we need to be ready for the next ESR. Assuming it's Thunderbird 60, it depends how much overlap there is (if any) in support for the old (52) and new (60) ESR branches.

Since the stable Thunderbird releases all are ESR, we (and Debian) won't get any of versions 57-59, but only 60. And when that one is released and imported into Debian I suppose the Debian package maintainer for enigmail would be forced to import a compatible version (i.e. a version after the conversion to an embedded WebExtension). What can we expect on the Debian side in this situation?

#5 Updated by intrigeri over 1 year ago

  • Assignee changed from intrigeri to u

Since the stable Thunderbird releases all are ESR, we (and Debian) won't get any of versions 57-59, but only 60. And when that one is released and imported into Debian I suppose the Debian package maintainer for enigmail would be forced to import a compatible version (i.e. a version after the conversion to an embedded WebExtension). What can we expect on the Debian side in this situation?

Good question! Better ask it to the maintainer of torbirdy in Debian ⇒ reassigning.

u, if you need help (or a rubber-duck) for this, I'm here :)

#6 Updated by intrigeri over 1 year ago

  • Assignee changed from u to anonym
  • QA Check deleted (Info Needed)

anonym: looks like I got confused and misread Torbirdy when you wrote Enigmail.

So yeah, on the Enigmail side I'm not very concerned because it's actively developped and I'm confident dkg will import whatever upstream release is needed to keep it working. There's already been an update of Enigmail in Stretch (or was it Jessie?) to keep it working with the Thunderbird security update.

I'm much more concerned about Torbirdy upstream. In Debian we can't do much better than importing upstream stuff. And same as Enigmail, if it's broken in stable due to a Thunderbird security update, we will need to either update Torbirdy there too, or drop it from stable.

#8 Updated by anonym over 1 year ago

Sukhbir intends to maintain TorBirdy, and is just waiting for the situation to become clearer for him.

#9 Updated by intrigeri over 1 year ago

  • Target version changed from Tails_3.6 to Tails_3.7
  • Type of work changed from Research to Communicate

#10 Updated by anonym over 1 year ago

  • Target version changed from Tails_3.7 to Tails_3.6
  • Type of work changed from Communicate to Research

Now that Thunderbird 57 is out, I've asked Sukhbir for a more concrete plan/schedule.

#11 Updated by anonym over 1 year ago

  • Target version changed from Tails_3.6 to Tails_3.7
  • Type of work changed from Research to Communicate

#12 Updated by intrigeri over 1 year ago

Thunderbird 58 (beta3) is now in Debian experimental so one possible next step would be to test the add-ons we use and see how easily our patches can be rebased. This won't tell us everything about what Thunderbird 60 will break for us but it might already highlight a few issues and allow us to work on them incrementally, which could be an efficient way to avoid having to enter sacrifice mode in July/August even more than planned :)

#13 Updated by intrigeri over 1 year ago

#14 Updated by intrigeri over 1 year ago

#15 Updated by intrigeri over 1 year ago

#16 Updated by intrigeri over 1 year ago

  • Subject changed from Get ready for Thunderbird 57+ to Get ready for Thunderbird 60+

#17 Updated by intrigeri over 1 year ago

Thunderbird 60 beta1 was released. I guess it should land in Debian experimental soonish.

#18 Updated by intrigeri over 1 year ago

Regarding the timeline, my understanding is that Stretch will get Thunderbird 60 as soon as it's out. Chances are that it'll be uploaded to Debian stable-security too late for Tails 3.7 but for 3.8 the only way we'll have to upgrade Thunderbird and get security fixes will be to upgrade to ESR 60.

#19 Updated by intrigeri over 1 year ago

  • Description updated (diff)

#20 Updated by intrigeri over 1 year ago

  • Type of work changed from Communicate to Code

#22 Updated by intrigeri over 1 year ago

  • Description updated (diff)

#23 Updated by intrigeri over 1 year ago

  • Description updated (diff)

#24 Updated by intrigeri over 1 year ago

Next step is to assess what work needs to be done by testing Thunderbird 60~betaN in the context of Tails i.e.:

  1. try to build a package for Stretch with our patchset: generate them with git format-patch upstream/60.0_b2..${TAILS_REMOTE}/secure_account_creation-60.0_b2 and refresh them against the latest version of Thunderbird 60 that's available in Debian (keeping the backport-to-Stretch bits if there are some)
  2. install this new package in a running Tails
  3. upgrade add-ons as needed (see ticket description)
  4. if needed, disable strict add-ons compatibility mode
  5. test Thunderbird manually; in particular, pay attention to anything Torbirdy-related
  6. prepare a branch against tails.git with whatever changes that were needed
  7. ask intrigeri to upload your package to our custom APT repo
  8. build an ISO from your branch
  9. run our source:features/thunderbird.feature automated test suite and list what's broken

#25 Updated by intrigeri over 1 year ago

  • Assignee changed from anonym to CyrilBrulebois
  • Estimated time set to 6.00 h

#26 Updated by intrigeri over 1 year ago

  • Due date set to 05/15/2018

#27 Updated by bertagaz over 1 year ago

  • Target version changed from Tails_3.7 to Tails_3.8

#28 Updated by intrigeri over 1 year ago

  • Description updated (diff)

(added upstream timeline for 32.x and 60.x releases)

#29 Updated by intrigeri about 1 year ago

  • Description updated (diff)

#30 Updated by intrigeri about 1 year ago

  • Target version changed from Tails_3.8 to Tails_3.9

#32 Updated by intrigeri about 1 year ago

kibi will make progress this week and update us by next Tuesday.

#33 Updated by intrigeri about 1 year ago

  • Due date changed from 05/15/2018 to 07/03/2018

#34 Updated by intrigeri about 1 year ago

  • Description updated (diff)

#35 Updated by intrigeri about 1 year ago

  • Description updated (diff)

#36 Updated by intrigeri about 1 year ago

  • Description updated (diff)

#37 Updated by intrigeri about 1 year ago

  • Description updated (diff)

#38 Updated by intrigeri about 1 year ago

intrigeri wrote:

kibi will make progress this week and update us by next Tuesday.

This did not happen and I'm leaving AFK July 13-21. The next synchronization point (when we can decide who does the next steps and when) will be our next team meeting, on July 27. It would be great if #15091#note-24 was done before that meeting so I have time to take a look and ask any clarification I might need to help us organize the next steps.

The end goal being to have a branch that migrates us to ESR60 merged into devel by August 15: I don't think there'll be another ESR52 release. I'll be super busy between the next FT meeting and August 15 so if you want to schedule a few days to handle these next steps yourself, it would be awesome. If you can't, we'll find another solution :)

#39 Updated by intrigeri about 1 year ago

  • Due date changed from 07/03/2018 to 07/21/2018

#40 Updated by intrigeri about 1 year ago

Asked Carsten about his tentative timeline / guts feelings wrt. ESR60 in Stretch (Cc'd you).

#41 Updated by intrigeri about 1 year ago

#42 Updated by intrigeri about 1 year ago

  • Estimated time changed from 6.00 h to 10.00 h

I suspect 6 hours of focussed work (that excludes ISO build & automatic test time) might not be enough to do #15091#note-24. So bumping that number in case if you need more and I'm not around to validate it.

#43 Updated by intrigeri about 1 year ago

#44 Updated by intrigeri about 1 year ago

#45 Updated by intrigeri about 1 year ago

#46 Updated by intrigeri about 1 year ago

  • Blocks Feature #15745: Update doc wrt. Liferea → Thunderbird as the default feed reader added

#47 Updated by intrigeri about 1 year ago

Hi kibi!

intrigeri wrote:

intrigeri wrote:

kibi will make progress this week and update us by next Tuesday.

This did not happen and I'm leaving AFK July 13-21. The next synchronization point (when we can decide who does the next steps and when) will be our next team meeting, on July 27. It would be great if #15091#note-24 was done before that meeting so I have time to take a look and ask any clarification I might need to help us organize the next steps.

Any chance #15091#note-24 is done before July 27?

#48 Updated by CyrilBrulebois about 1 year ago

Hi intrigeri,

Sorry, I haven't been able to make progress on this issue yet (see out-of-band message for details).

On a related note, it seems stretch might get new rustc/cargo packages soon-ish, which might help us with Thunderbird.

Hoping to get back to Tails/Thunderbird work this very week-end and following week (finally free from in-person meeting/teaching for a while).

Sorry for the lag.

#49 Updated by intrigeri about 1 year ago

Depending on kibi's progress by Aug 7, I might chime in and help on Aug 8-9.

#50 Updated by CyrilBrulebois about 1 year ago

To keep you posted: I've had to refresh almost all patches because files were moved around in a later beta, and a few patches also needed context updates. I've also had to deal with a few versioned build dependencies (mostly by trivially-backporting bits and pieces from testing/unstable), and I managed to have a full thunderbird build in a stretch-based chroot. I'll perform some clean-up on the patch series, but also move on to installing the resulting binary inside Tails and switching to run-time testing.

While I'm at it: I think this is the kind of packages that some trusted Tails developer might be building locally and uploading to the custom APT repository, right?

FWIW, I know renaming things is always a pain, but I was a bit surprised not to find the git repository by searching for “bird”. Turned out the repository is still called icedove at the moment → https://git.tails.boum.org/icedove/ ;)

#51 Updated by intrigeri about 1 year ago

While I'm at it: I think this is the kind of packages that some trusted Tails developer might be building locally and uploading to the custom APT repository, right?

Right.

#52 Updated by CyrilBrulebois about 1 year ago

  • Feature Branch set to https://salsa.debian.org/kibi/thunderbird/tree/feature-15091-get-ready-for-thunderbird-60-plus

(Sorry, I should have read the whole ticket before asking…)

Here's a report after roughly 6 hours spent on TB.

Building

I've opened a bug report against the experimental version of src:thunderbirdhttps://bugs.debian.org/905465 with my findings regarding build dependencies and how to backport thunderbird to stretch.

I've also rebased the 18 Tails-specific patches from b2 on top of b10. Note: files were moved around between two beta releases, so it's really important to use git rebase with appropriate parameters instead of exporting the patches and applying them (filenames would need extensive tweaking).

Only two patches needed manual updates, because the context changed a bit:
  • 0005 = Document that mailnews.{auto_config,mx_service}_url can be disabled.
  • 0013 = Add pref to force SSL for ISP fetch during autoconfiguration.

As documented in the Debian bug report, I've built the packages in a stretch chroot, with stretch-proposed-updates enabled for some build dependencies (cargo etc.).

Since the patches touch upstream files directly, dpkg-source refuses to create a source package, so I've built the binaries by running dpkg-buildpackage -b manually, in an sbuild-createchroot'd chroot, with build dependencies installed manually; note: /dev/shm is required (to avoid EPERM in the Python multiprocessing module), so I went for a bind-mount.

Binaries and git branch

The resulting binaries are available in the following repository, signed with my “corporate” key:

The git branch is available on salsa (thunderbird.git is a bit heavy so I didn't publish that one on my own server) → https://salsa.debian.org/kibi/thunderbird/tree/feature-15091-get-ready-for-thunderbird-60-plus

Running

Installing these binaries on Tails 3.8, three l10n packages go away (-bn-bd, -pa-in, -ta-lk), while two get installed (-kk, -ms), along with a new dependency: libjsoncpp1.

Upon start-up, the account creation screen shows up, and an attempt is made to derive the right settings from my mail address.

On the addons side, 2 are installed:

  • enigmail: 2.0.7, last updated 2018-06-14 → no worries
  • torbirdy: 0.2.1, last updated 2016-11-24; disabled, incompatible with TB 60.0

The addons screen doesn't make it possible to perform an online search, so I had to download the .xpi manually to load it from a local file; after a restart, torbirdy is successfully enabled (as reported with green indication at the bottom: “TorBirdy Enabled: tor”).

Trying to create an account results in a different screen: no autodetection is performed this time, and I have to pick between IMAP and POP protocols. After setting it up, I've been able to download mails from my server using IMAP. I've also managed to configure an SMTP server and to send mail from there.

Message seen in the terminal thunderbird was started from: it tries to read from /etc/dconf/db/local and this isn't possible; a warning regarding performance issues is printed accordingly; in dmesg, the same issue can be seen, through DENIED log lines.

Note: I cannot seem to be able to easily extract the apt log, but I chose to install the new profile (/etc/apparmor.d/usr.bin/thunderbird) instead of keeping the old one. I suppose an update is in order?

#53 Updated by CyrilBrulebois about 1 year ago

  • Assignee deleted (CyrilBrulebois)
  • QA Check set to Info Needed

I'd be happy get some feedback on my findings. :)

FWIW, TB60 (final) was just released → https://www.thunderbird.net/en-US/thunderbird/60.0/releasenotes/

#54 Updated by intrigeri about 1 year ago

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri

(Better assign to the person you're requesting info from :)

#55 Updated by u about 1 year ago

  • Related to Bug #15600: Upgrade to Torbirdy 0.2.5 added

#56 Updated by intrigeri about 1 year ago

  • Assignee changed from intrigeri to CyrilBrulebois
  • QA Check changed from Info Needed to Dev Needed

Woohoo, awesome progress! \o/

Here's a report after roughly 6 hours spent on TB.

Glad there's some time left. We'll update the budget dynamically in the next days, keep talking to me along the way :)

I've also rebased the 18 Tails-specific patches from b2 on top of b10. Note: files were moved around between two beta releases, so it's really important to use git rebase with appropriate parameters instead of exporting the patches and applying them (filenames would need extensive tweaking).

Congrats! I wonder if it's worth also preparing a secure_account_creation-60.0_b10 branch forked off the upstream/60.0_b10 tag with these patches. Probably not very useful since the patch series is tracked via quilt in the branch you've prepared already + I doubt anonym will work on upstreaming it any time soon. Now, if you think it's a matter of 15 min of work, please do it.

I'll take a look at the updated patch series later this week.

Since the patches touch upstream files directly, dpkg-source refuses to create a source package, so I've built the binaries by running dpkg-buildpackage -b manually, in an sbuild-createchroot'd chroot, with build dependencies installed manually; note: /dev/shm is required (to avoid EPERM in the Python multiprocessing module), so I went for a bind-mount.

I'm not sure I understand. I'll try building in the way we document and will see.

Upon start-up, the account creation screen shows up, and an attempt is made to derive the right settings from my mail address.

Sounds good! Does it show a "only use secure protocols" (or similar) checkbox? (i.e. are our patches still effective?)

On the addons side, 2 are installed:

  • enigmail: 2.0.7, last updated 2018-06-14 → no worries
  • torbirdy: 0.2.1, last updated 2016-11-24; disabled, incompatible with TB 60.0

The addons screen doesn't make it possible to perform an online search, so I had to download the .xpi manually to load it from a local file; after a restart, torbirdy is successfully enabled (as reported with green indication at the bottom: “TorBirdy Enabled: tor”).

Good! Next step: install the version from Debian. To do so, please create a feature/15091-thunderbird-60 branch (forked off tails.git's devel branch) that installs torbirdy 0.2.5. It's been in testing/sid for some time and Ulrike uploaded it to stretch-backports today.

Message seen in the terminal thunderbird was started from: it tries to read from /etc/dconf/db/local and this isn't possible; a warning regarding performance issues is printed accordingly; in dmesg, the same issue can be seen, through DENIED log lines.

Looks like an AppArmor problem. Let's take a closer look once the profile is updated i.e.:

Note: I cannot seem to be able to easily extract the apt log, but I chose to install the new profile (/etc/apparmor.d/usr.bin/thunderbird) instead of keeping the old one. I suppose an update is in order?

So far we've adjusted the AppArmor profile for our needs via config/chroot_local-patches/apparmor-adjust-thunderbird-profile.diff (tails.git). This avoids having to build & upload src:thunderbird every time we want to patch it. So we need to update that patch on the aforementioned feature/15091-thunderbird-60 branch. Then that branch will FTBFS (the updated patch won't apply) until it installs Thunderbird 60, so I'll try to build & upload a package to an overlay APT suite ASAP, which should unblock your next steps :)

Meta: I think we'll need to start filing subtasks soonish, before this ticket becomes too complex and makes it impossible to track what's done vs. TBD.

#57 Updated by intrigeri about 1 year ago

Since the patches touch upstream files directly, […]

I'm not sure I understand. I'll try building in the way we document and will see.

Now I understood: you've imported the patch series directly on your Debian packaging branch instead of doing so as quilt patches (which we've done so far, using the secure-account-creation "topic"). I'll fix that so I can build the package without doing crazy stuff.

#58 Updated by intrigeri about 1 year ago

  • Feature Branch changed from https://salsa.debian.org/kibi/thunderbird/tree/feature-15091-get-ready-for-thunderbird-60-plus to feature/15091-thunderbird-60, icedove:feature/15091-thunderbird-60

Now I understood: you've imported the patch series directly on your Debian packaging branch instead of doing so as quilt patches (which we've done so far, using the secure-account-creation "topic"). I'll fix that so I can build the package without doing crazy stuff.

I've:

  1. enabled the feature-15091-thunderbird-60 APT overlay suite on a new feature/15091-thunderbird-60 branch forked off devel (tails.git)
  2. imported the patch series into debian/patches/secure-account-creation (gbp pq FTW)
  3. rewrote Git history to avoid reverting 18 commits and the confusion it would cause
  4. created a fresh feature/15091-thunderbird-60 branch in icedove.git
  5. merged my rewritten feature-15091-get-ready-for-thunderbird-60-plus into feature/15091-thunderbird-60; while resolving conflicts I've dropped most of the previous tails/stretch delta (that comes from Debian's debian/stretch branch) to remain as close as possible to the branch you've tested; that delta wll come back once Thunderbird 60 is in Stretch and we resume merging from Debian's debian/stretch branch, though.
  6. created a sbuild chroot with pu enabled (sudo sbuild-createchroot --arch=amd64 --include=debhelper,eatmydata --components=main,contrib,non-free --extra-repository='deb http://deb.debian.org/debian stretch-proposed-updates main' --chroot-prefix=stretch-pu stretch /var/cache/sbuild/stretch-pu-amd64-sbuild http://localhost:3142/debian and then tweaked the new config file in /etc/schroot/chroot.d/ to suit my taste)

Next steps on my side:

  1. build a package with gbp buildpackage --git-debian-branch=feature/15091-thunderbird-60 --git-export=WC --git-builder='sbuild --chroot=stretch-pu-amd64-sbuild' or similar
  2. upload to the package to the feature-15091-thunderbird-60 APT overlay suite

I'll keep you updated.

And then Cyril can (some of that stuff can be done already though):

  1. have the branch install a compatible Torbirdy (now tracked on #15600)
  2. update the AppArmor patch (shout if you need help understanding/rebasing our diff)
  3. do the remaining parts of #15091#note-24
  4. check if we're using the updated patch series correctly (IIRC I've noticed a bunch of prefs and I think some of them are new, possibly off by default, and meant to be enabled or tweaked in Tails)

#59 Updated by intrigeri about 1 year ago

  • Subject changed from Get ready for Thunderbird 60+ to Upgrade to Thunderbird 60

("Get ready" made sense back in May but we're not there anymore.)

#60 Updated by intrigeri about 1 year ago

#61 Updated by intrigeri about 1 year ago

#62 Updated by intrigeri about 1 year ago

Next steps on my side:

  1. build a package with gbp buildpackage --git-debian-branch=feature/15091-thunderbird-60 --git-export=WC --git-builder='sbuild --chroot=stretch-pu-amd64-sbuild' or similar
  2. upload to the package to the feature-15091-thunderbird-60 APT overlay suite

I'll keep you updated.

Done!

#63 Updated by intrigeri about 1 year ago

  • Blocks Bug #15551: Generating a revocation certificate with Enigmail fails added

#64 Updated by intrigeri about 1 year ago

#65 Updated by intrigeri about 1 year ago

  • Blocks Bug #15775: Adjust persistence setup to Thunderbird being recommended for feeds added

#66 Updated by intrigeri about 1 year ago

Quickly tested at commit:4ee4dc5713801fcc629b90e36981d1b821b871e8:

  • There's no "use only secure protocols" checkbox; I dunno if the latest patchset has removed it or if we need to enable a pref to get it back. But perhaps it's because of:
  • I'm told "TorBirdy has disabled Thunderbird's auto-configuration wizard to protect your anonymity" while we're supposed to have a pref (extensions.torbirdy.emailwizard I think) set to enable it in Tails. Did the pref get removed/renamed/broken in upstream Torbirdy? Or is /etc/xul-ext/torbirdy.js ignored? (For Firefox 60, IIRC we had to move prefs somewhere else, see the branch for #15023. Perhaps config/chroot_local-includes/etc/thunderbird/pref/thunderbird.js would work better.) I see that extensions.torbirdy.emailwizard is set to false in about:config, so probably our pref is not taken into account anymore.
  • I can read email over IMAP :)
  • I did not test Enigmail functionality further than going through its config wizard (which worked fine).

#67 Updated by CyrilBrulebois about 1 year ago

My first attempt at figuring out whether preferences were taken into account was strace-ing Thunderbird, but that's not allowed within Tails. I assumed it was due to AppArmor, so I've tried disabling it entirely with apparmor=0 on the kernel command line, but that didn't allow me to strace it anyway.

So I moved to debugging Thunderbird & TorBirdy in a stretch chroot, and the resulting strace was rather surprising: no configuration files read from /etc

How Thunderbird and TorBirdy ship/read their configuration

It's still possible to override settings in Thunderbird and TorBirdy though, since both have symlinks to places under /etc:

Thunderbird ships preferences this way:
  • /etc/thunderbird/pref/thunderbird.js
  • /usr/share/thunderbird/defaults/pref/channel-prefs.js
  • /usr/share/thunderbird/defaults/pref/vendor.js
The first one under /etc is taken into account thanks to this symlink:
  • /usr/share/thunderbird/defaults/syspref/etc/thunderbird/pref (directory)
TorBirdy ships preferences this way:
  • /etc/xul-ext/torbirdy.js
  • /usr/share/xul-ext/torbirdy/defaults/preferences/prefs.js
The first one under /etc is taken into account thanks to this symlink:
  • /usr/share/xul-ext/torbirdy/defaults/preferences/000system.js/etc/xul-ext/torbirdy.js (file only this time)

What's in Tails

In our Tails build, we have a custom:
  • /etc/xul-ext/torbirdy.js

as expected, which enables the email wizard and announces that gpg is torified already.

In our Tails build still, we have a custom
  • /etc/thunderbird/pref/thunderbird.js

which contains settings by intrigeri regarding EFAIL, and doesn't disable the email wizard.

I'll double check these findings in a clean environment, but all in all I think the way we ship settings is fine or at least supposed to work. It might rather be some issues/changes on the code side instead. This is what I'll be working on now, along with checking what happens if all settings are merged into the main Thunderbird configuration file.

Extra stuff

Thunderbird's README.Debian mentions possible issues regarding l10n packs. I cannot really dig into that right now, but it might make sense to open an issue to make sure we don't forget to check what's happening there.

#68 Updated by intrigeri about 1 year ago

My first attempt at figuring out whether preferences were taken into account was strace-ing Thunderbird, but that's not allowed within Tails.

I think that's due to /etc/sysctl.d/ptrace_scope.conf:kernel.yama.ptrace_scope=2.

Thunderbird's README.Debian mentions possible issues regarding l10n packs. I cannot really dig into that right now, but it might make sense to open an issue to make sure we don't forget to check what's happening there.

Yes, a subtask to test Thunderbird in various languages would be useful :)

#69 Updated by CyrilBrulebois about 1 year ago

  • Feature Branch changed from feature/15091-thunderbird-60, icedove:feature/15091-thunderbird-60 to kibi/feature/15091-thunderbird-60, icedove:feature/15091-thunderbird-60

Thanks for the sysctl setting, using 0 instead of 2 indeed lets me check TB's behaviour. :)

So: even if both configuration files are indeed read successfully, only the setting in /usr/share/xul-ext/torbirdy/defaults/preferences/prefs.js seems to matter. So I've added a local patch as discussed.

Confirmed to work fine with this commit in my feature/15091-thunderbird-60 branch (https://mraw.org/git/?p=tails.git;a=shortlog;h=refs/heads/feature/15091-thunderbird-60):

commit 7f1fb753251b352a7af0eae52bc865c899846632
Author: Cyril Brulebois <cyril@debamax.com>
Date:   Tue Aug 14 12:57:10 2018 +0200

    Torbirdy: Enable the email wizard in the system configuration.

    There's:
      /usr/share/xul-ext/torbirdy/defaults/preferences/000system.js → /etc/xul-ext/torbirdy.js

    which is read after:
      /usr/share/xul-ext/torbirdy/defaults/preferences/prefs.js

    but setting extensions.torbirdy.emailwizard to true in the former
    doesn't seem sufficient, even though strace confirms it's being read.

#70 Updated by CyrilBrulebois about 1 year ago

  • Feature Branch changed from kibi/feature/15091-thunderbird-60, icedove:feature/15091-thunderbird-60 to kibi:feature/15091-thunderbird-60, icedove:feature/15091-thunderbird-60

Just pushed a refresh branch which includes borrowing code from Tor Browser to set the locale according to $LANG. That seems to work fine for a few languages, like French, German, Chinese, and Japanese.

Latest commit:

commit b6e401f9b5a843410500fff07330abe8fa583f24
Author: Cyril Brulebois <cyril@debamax.com>
Date:   Tue Aug 14 17:17:40 2018 +0200

    Thunderbird: Fix regression in defaultprotocol handling (refs: #15091).

    Preferences handling seems to have changed heavily in TB60, so switch to a
    user_pref in prefs.js instead of a regular pref in preferences/0000tails.js

    Another possibility would be to use /etc/thunderbird/pref/thunderbird.js but
    that would need to happen in gdm's PostLogin (root privileges would be
    required).

It would be nice to test what happens when persistency is enabled, as I haven't exactly reproduced the issue as it is. I've just inferred it from what was happening with language settings.

For reference, mailnews.auto_config.fetchFromISP.ssl_only is set to false, and I'm not sure I've successfully tested what's happening there. I'll follow up later on that.

#71 Updated by CyrilBrulebois about 1 year ago

Just opened https://labs.riseup.net/code/issues/15788 related to insecure protocols.

#72 Updated by CyrilBrulebois about 1 year ago

On the Enigmail front:
  • creation looks good (default/normal mode)
  • signing looks good
  • exporting public key and reimporting elsewhere to verify the signed mail looks good
  • searching keys on keyserver looks BROKEN: instantaneous answer, with no key found. Not an apparmor issues.

→ I will need to revisit Thunderbird & TorBirdy patches.

The assistant looks a bit buggy, but that's probably not a new thing:
  • window size on 800x600 desktop is too small, with huge margins → scrollbars appear
  • selecting “cancel” when creating the revocation certificate leads to a pop-up, then a file dialog, to save the revocation certificate that wasn't created.

Both of those issues should probably be checked against non-Tails Enigmail, and possibly reported upstream.

#73 Updated by CyrilBrulebois about 1 year ago

OK, searching/fetching public keys with Enigmail fixed: same story for the gpg_already_torified as we had with emailwizard (both being under the extensions.torbird. namespace); another commit deals with gpg_already_torified through a local patch against the installed prefs.js, and another commit removes the local include of torbirdy.js (which seems read but ignored, for both settings).

I've also cherry-picked the apparmor patch allowing /etc/ld.so.conf* on top of the branch. Last commit:

commit 66fcc833bf997cbd43facc062b1ad7a3979b0e9e
Author: intrigeri <intrigeri@boum.org>
Date:   Tue Aug 14 10:34:22 2018 +0000

    AppArmor: allow /etc/ld.so.conf and friends in the "base" abstraction.

    Thunderbird 60 needs access there and who knows what might break
    in surprising, hard to debug ways, if we block it.

    Backport from upstream commit 6d22c871bfb17da7620efab4ad5c41960d1efd41,
    first released in Debian via apparmor 2.13-1.

#74 Updated by intrigeri about 1 year ago

Code review looks good up to, and including, 66fcc833bf997cbd43facc062b1ad7a3979b0e9e. Woohoo! \o/

I was surprised to see that the langpacks are called *firefox.mozilla.org.xpi so I've double-checked and indeed, it's the case. Let's hope it's not a packaging bug whose future fix would break our stuff (maybe it's worth adding a build-time check that aborts the build if there's no file matching this pattern in $TB_EXT? I'm unsure.).

#75 Updated by intrigeri about 1 year ago

Also, please see 98954ee6e47da3d400db098021ecebe24c1a9aff that re-introduces some work we had on feature/15091-thunderbird-60 in the official repo, which got lost while rebasing your own branch on devel.

#76 Updated by intrigeri about 1 year ago

The branch introduces, or modifies the behaviour of, 4 prefs:

  • mailnews.auto_config.account_constraints.allow_oauth2 → it's been renamed so I think /etc/thunderbird/pref/thunderbird.js needs to be updated
  • mailnews.auto_config.fetchFromISP.ssl_only → brand new, false by default; we need to enable it
  • mailnews.auto_config.account_constraints.ssl_only → it's been renamed so I think /etc/thunderbird/pref/thunderbird.js needs to be updated
  • mailnews.auto_config.guess.timeout → brand new, I think the default timeout (10) is too low when using Tor, we should bump it in our prefs

#77 Updated by CyrilBrulebois about 1 year ago

intrigeri wrote:

Code review looks good up to, and including, 66fcc833bf997cbd43facc062b1ad7a3979b0e9e. Woohoo! \o/

I was surprised to see that the langpacks are called *firefox.mozilla.org.xpi so I've double-checked and indeed, it's the case. Let's hope it's not a packaging bug whose future fix would break our stuff (maybe it's worth adding a build-time check that aborts the build if there's no file matching this pattern in $TB_EXT? I'm unsure.).

I've pushed an extra commit to my branch that does a check on the number of lang packs found, and yells+aborts when there are none. Filed a bug report on the Debian side too to get some feedback from the maintainer. → https://bugs.debian.org/906176

From his first reply: getting l10n to work at all was quite hard in the first place; also: we might get a better support in 60.1. We'll see!

#78 Updated by u about 1 year ago

Are you interested in the latest set of patches, that already takes SSL into account? Will send it to you both by email.

#79 Updated by intrigeri about 1 year ago

CyrilBrulebois wrote:

I've pushed an extra commit to my branch that does a check on the number of lang packs found

LGTM. Added another commit on top.

#80 Updated by intrigeri about 1 year ago

I've filed subtasks for the remaining things to do, that I won't block on for merging into devel for 3.9~rc1 (just like we'll merge the VeraCrypt and ASP branch without automated tests yet). I'll do the automated tests (and relevant parts of the manual tests) by hand and will then merge if happy!

#81 Updated by intrigeri about 1 year ago

  • Related to Bug #15791: Update test suite for Thunderbird 60 added

#82 Updated by intrigeri about 1 year ago

#83 Updated by intrigeri about 1 year ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (CyrilBrulebois)
  • QA Check changed from Dev Needed to Pass

intrigeri wrote:

I'll do the automated tests (and relevant parts of the manual tests) by hand and will then merge if happy!

Merged! See you on #15791 and #15792 (that I've unparented to be able to mark this one as fix committed).

#84 Updated by intrigeri about 1 year ago

  • Status changed from Fix committed to In Progress

#85 Updated by intrigeri about 1 year ago

  • Status changed from In Progress to Fix committed

#86 Updated by intrigeri about 1 year ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF