Project

General

Profile

Feature #16356

Upgrade to Tor Browser 9.0 (based on Firefox 68)

Added by intrigeri 11 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
Start date:
01/14/2019
Due date:
% Done:

100%

Feature Branch:
feature/16356-tor-browser-9.0+force-all-tests, torbrowser-launcher:feature/16356-tor-browser-9.0+force-all-tests
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Browser


Subtasks

Feature #17035: Upstream Tor Browser 9.0 fixes to the torbrowser-launcher AppArmor profilesResolved

Bug #17036: uBlock is not enabled in Tor Browser 9Resolved

Bug #17038: Update test suite for Tor Browser 9's Tor LauncherResolved

Bug #17040: "Persistent browser bookmarks" test fails with Tor Browser 9Resolved

Feature #17044: Update documentation for Tor Browser 9.0Resolved

Feature #17055: Update Unsafe Browser tweaks for Tor Browser 9Resolved

Feature #16357: Deal with Torbutton being integrated into Tor BrowserResolved

Bug #17039: Unsafe Browser does not start (segfault) with Tor Browser 9Resolved

Bug #17056: Make test suite robust with Tor Browser 9.0Resolved

Bug #17114: Adjust Tor Browser updates settings for 9.0Resolved

Bug #17121: Tor Browser 9 sometimes won't load new URLsResolved

Feature #17130: Unsafe Browser based on Tor Browser 9.0a7 makes connections to the Internet which are not user initiatedResolved

Bug #17140: Tor Browser 9.0a7 fails to start in ko_KR.utf8 localeResolved

Bug #17142: New Unsafe Browser tabs have the "Private Browsing" name and the Tor Browser iconResolved

Bug #17150: Spellchecking only available for English in Tor Browser 9Resolved

Feature #17157: Upgrade to Tor Browser 9.0a8Resolved


Related issues

Related to Tails - Feature #10422: Grant Tor Browser access to files as designated by the user Confirmed 08/30/2018
Related to Tails - Bug #17042: uBlock buttons disapear when Security level is safest Resolved
Related to Tails - Bug #17159: Tor Browser displays an "Update Failed" pop-up Confirmed
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed
Blocks Tails - Bug #16693: Upgrade to Tor Browser based on Firefox 68.2 Resolved

Associated revisions

Revision b88bb369 (diff)
Added by anonym 7 months ago

Upgrade Tor Browser to 9.0a1.

Refs: #16356

Revision af719e9d (diff)
Added by anonym 7 months ago

Adapt Tor Launcher installation vs Tor Browser 9.0a1.

As of Tor Browser 9.0a1 the Tor Launcher extension is a "system"
add-on (like pdfjs) but we can still convert it to something that can
be run as a XUL standalone application by just moving things around a
bit.

Refs: #16356, #15709

Revision 7c96cb58 (diff)
Added by anonym 7 months ago

Tor Browser: refresh patch for apply_extension_code_signing_hacks().

Refs: #16356

Revision 1744b37b (diff)
Added by anonym 7 months ago

Unsafe/Tor Browser: disable the integrated Tor Launcher.

Refs: #16356, #15709

Revision f74cec4d (diff)
Added by anonym 7 months ago

onion-grater: retry connecting to the real control port.

This should not be necessary and is just placed here as a workaround
until I can figure out the real issue (bug in onion-grater? bug in
stem?).

When working on the Tor Browser 9.0 migration there were issues with
Tor Launcher. It starts and does its initial connection to the control
port where it successfully fetches CONFs etc. When clicking "Connect"
it successfully sends a few more things and disconnects, only to
reconnect immediately later (due to some implementation detail in Tor
Launcher). This reconnection fails:

[...]
/usr/local/lib/tor-browser/firefox-unconfined (pid: 8865, user: tor-launcher, port: 57472, filter: tor-launcher): > SAVECONF
/usr/local/lib/tor-browser/firefox-unconfined (pid: 8865, user: tor-launcher, port: 57472, filter: tor-launcher) disconnected: Client closed its socket
[ Here comes the reconnect: ]
/usr/local/lib/tor-browser/firefox-unconfined (pid: 8865, user: tor-launcher, port: 57476, filter: tor-launcher) connected: loaded filter: tor-launcher
Final rules: [...]
Unable to connect to tor. Maybe it's running without a ControlPort?
/usr/local/lib/tor-browser/firefox-unconfined (pid: 8865, user: tor-launcher, port: 57476, filter: tor-launcher) disconnected: client quit
---------------------------------------

Exception happened during processing of request from ('127.0.0.1', 57476)
Traceback (most recent call last):
File "/usr/lib/python3.5/socketserver.py", line 625, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.5/socketserver.py", line 354, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.5/socketserver.py", line 681, in init
self.handle()
File "/usr/local/lib/onion-grater", line 629, in handle
self.controller = self.connect_to_real_control_port()
File "/usr/local/lib/onion-grater", line 570, in connect_to_real_control_port
stem.connection.authenticate_cookie(controller, cookie_path=global_args.control_cookie_path)
File "/usr/lib/python3/dist-packages/stem/connection.py", line 803, in authenticate_cookie
auth_response = _msg(controller, msg)
File "/usr/lib/python3/dist-packages/stem/connection.py", line 1055, in _msg
return controller.msg(message)
AttributeError: 'NoneType' object has no attribute 'msg'
----------------------------------------

But if we just connect yet again, it works, hence the workaround.

Refs: #16356, #15709

Revision 3b6c5dbc (diff)
Added by anonym 6 months ago

Upgrade Tor Browser to 9.0a2 (refs: #16356).

Revision c1ce06c0 (diff)
Added by intrigeri 3 months ago

Update patches for the Tor Browser add-ons signing hack (refs: #16356).

Revision ad2c198a (diff)
Added by intrigeri 3 months ago

Update patches for the Tor Browser add-ons signing hack (refs: #16356).

Revision eaa4e140 (diff)
Added by intrigeri 3 months ago

Enable localization for new locales introduced in Tor Browser 9.0 (refs: #16356)

Revision d74f4a63 (diff)
Added by intrigeri 3 months ago

Tor Browser: use the bundled libstdc++.so.6 (refs: #16356)

Stretch lacks a recent enough version for Tor Browser 9.0a6:

XPCOMGlueLoad error for file /usr/local/lib/tor-browser/libxul.so:
/usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.23' not found (required by /usr/local/lib/tor-browser/libxul.so)
Couldn't load XPCOM.

I missed this when importing 9.0a6 because abicheck.cc has not been updated yet:
https://trac.torproject.org/projects/tor/ticket/31646

Revision f86f3bdf (diff)
Added by intrigeri 3 months ago

Tor Browser: update path of the bundled libstdc++.so.6 (refs: #16356)

We can copy it to the same place as before and Tor Browser will find it, thanks
to our custom $LD_LIBRARY_PATH. We were already installing it to a different
location than where it lives in the Tor Browser tarball anyway, last time
we needed to use that bundled copy.

Revision 5066426d (diff)
Added by intrigeri 3 months ago

Make newly open tabs blank (refs: #16356)

The Tor Browser is apparently going to do the same. But let's not wait for their
next alpha, as I suspect the about:newtab we currently get is what makes our
test suite brittle in new tabs (pressing Enter after pasting a URL in the
address bar, in a newly opened tab, often has no visible effect).

Revision 06625a1e (diff)
Added by segfault 3 months ago

Refresh patch to apply on top of webext-ublock-origin 1.22.2+dfsg-1 (refs: #16356)

Revision f98bbcea (diff)
Added by intrigeri 2 months ago

Revert "Make newly open tabs blank (refs: #16356)"

This reverts commit 5066426db1f6c30bffc5555f60ea25d013023814.

Tor Browser 9.0a7 now does what we want by default:

- https://trac.torproject.org/projects/tor/ticket/31575#comment:12
- https://trac.torproject.org/projects/tor/ticket/30662

Revision 3758e881 (diff)
Added by intrigeri 2 months ago

Update comment (refs: #16356)

Revision b28259f1 (diff)
Added by intrigeri 2 months ago

Test suite: update reference image for Tor Browser 9 (refs: #16356)

Revision d5fc4533 (diff)
Added by segfault 2 months ago

Increase memory reserved for root processes and PF_MEMALLOC allocations (refs: #16356)

Revision aa90a19a
Added by intrigeri 2 months ago

Merge branch 'feature/16356-tor-browser-9.0+force-all-tests' into devel (refs: #16356)

History

#1 Updated by intrigeri 11 months ago

#2 Updated by intrigeri 9 months ago

#3 Updated by intrigeri 9 months ago

#4 Updated by intrigeri 8 months ago

  • Blocks Bug #16693: Upgrade to Tor Browser based on Firefox 68.2 added

#5 Updated by anonym 7 months ago

  • Status changed from Confirmed to In Progress
  • Assignee set to anonym

Gonna import 9.0a1 as a first step.

#6 Updated by anonym 7 months ago

  • Feature Branch set to feature/16356-tor-browser-9.0

Unless I messed up my rebase, the feature branch should build and result in a working Tor Browser and Unsafe Browser, with Tor Launcher disabled. I've tested them minimally without issue, so that's progress!

Unfortunately Tor Launcher doesn't start properly -- its window starts as just a few pixels but can at least be resized, but then you see it's just all black and uninteractable. I think my last commit will fix this. If not, the script below (from my experimenting on #15709) succeeds at what 10-tbb.sh apparently fails at and should be useful to debug this (e.g. compare the resulting dir with what 10-tbb.sh puts into the builds):

rm -rf /usr/local/lib/tor-launcher-standalone /tmp/browser-omni /home/tor-launcher/.tor-launcher/
7z x -o/tmp/browser-omni /usr/local/lib/tor-browser/browser/omni.ja
cp -a /tmp/browser-omni/chrome/torlauncher/ /usr/local/lib/tor-launcher-standalone
cp /lib/live/mount/rootfs/filesystem.squashfs/usr/local/lib/tor-launcher-standalone/application.ini /usr/local/lib/tor-launcher-standalone/application.ini
mkdir /usr/local/lib/tor-launcher-standalone/chrome
mv /usr/local/lib/tor-launcher-standalone/{content,locale,skin} /usr/local/lib/tor-launcher-standalone/chrome
mkdir -p /usr/local/lib/tor-launcher-standalone/defaults/preferences
cp /tmp/browser-omni/defaults/preferences/torlauncher-prefs.js /usr/local/lib/tor-launcher-standalone/defaults/preferences/prefs.js
grep torlauncher /tmp/browser-omni/chrome/chrome.manifest \
  | sed --regexp-extended \
    -e 's@^(content|locale|skin) (torlauncher.*) torlauncher/(.*)$@\1 \2 chrome/\3@' \
    -e 's@^(component) (\S+) torlauncher/(.+)$@\1 \2 \3@' \
    -e 's@^(resource torlauncher) .*$@\1 ./@' \
  > /usr/local/lib/tor-launcher-standalone/chrome.manifest
chmod -R a+rX /usr/local/lib/tor-launcher-standalone
tails-tor-launcher

#7 Updated by anonym 7 months ago

Note to self: I never actually successfully configured anything with Tor Launcher, I've just seen it started. Trying to configure anything actually just says there are problems with the control port. In the journal I see:

stem.ProtocolError: GETINFO response didn't have an OK status:
Command filtered

so I probably just have to fix something in the onion-grater filter.

#8 Updated by anonym 7 months ago

NOTE TO SELF: These results were from my test setup, not Tails, so this problem has never occurred in Tails.

Eh, from what I can tell, when I press "Connect" in Tor Launcher, onion-grater tries to connect to itself and (naturally) fails:

pgrep onion-grater
23458

[... normal stuff, where the tor-launcher filter is loaded and there's some back-and-forth ...]
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None) connected: loaded filter: None
Final rules:
commands: {}
events: {}
restrict-stream-events: false

/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> PROTOCOLINFO 1
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250-PROTOCOLINFO 1
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250-AUTH METHODS=NULL
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250-VERSION Tor="0.3.5.8" 
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250 OK
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> AUTHENTICATE
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250 OK
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> SETEVENTS
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 250 OK
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> GETCONF __owningcontrollerprocess
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): command filtered: GETCONF __owningcontrollerprocess
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 510 Command filtered
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): -> GETINFO version
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): command filtered: GETINFO version
/usr/bin/python3.5 (pid: 23458, user: root, port: 59818, filter: None): <- 510 Command filtered
/usr/bin/python3.5 (pid: 23458, user: root, port: 59812, filter: None) disconnected: client quit
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 59812)
Traceback (most recent call last):
  File "/usr/lib/python3.5/socketserver.py", line 625, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.5/socketserver.py", line 354, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.5/socketserver.py", line 681, in __init__
    self.handle()
  File "./onion-grater", line 629, in handle
    self.controller = self.connect_to_real_control_port()
  File "./onion-grater", line 569, in connect_to_real_control_port
    controller = stem.connection.connect(control_socket=global_args.control_socket_path)
  File "/usr/lib/python3/dist-packages/stem/connection.py", line 287, in connect
    return _connect_auth(control_connection, password, password_prompt, chroot_path, controller)
  File "/usr/lib/python3/dist-packages/stem/connection.py", line 371, in _connect_auth
    return controller(control_socket, is_authenticated = True)
  File "/usr/lib/python3/dist-packages/stem/control.py", line 1041, in __init__
    self.add_event_listener(_sighup_listener, EventType.SIGNAL)
  File "/usr/lib/python3/dist-packages/stem/control.py", line 2997, in add_event_listener
    if event_type and (self.get_version() < event_type._VERSION_ADDED):
  File "/usr/lib/python3/dist-packages/stem/control.py", line 454, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/stem/control.py", line 1235, in get_version
    version = stem.version.Version(self.get_info('version'))
  File "/usr/lib/python3/dist-packages/stem/control.py", line 454, in wrapped
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/stem/control.py", line 1163, in get_info
    stem.response.convert('GETINFO', response)
  File "/usr/lib/python3/dist-packages/stem/response/__init__.py", line 132, in convert
    message._parse_message(**kwargs)
  File "/usr/lib/python3/dist-packages/stem/response/getinfo.py", line 40, in _parse_message
    raise stem.ProtocolError("GETINFO response didn't have an OK status:\n%s" % self)
stem.ProtocolError: GETINFO response didn't have an OK status:
Command filtered

How/why does it connect to itself????

#9 Updated by anonym 7 months ago

I think I have stumbled upon a bug in stem (or onion-grater?) that I had to workaround with f74cec4d7fc44d9b9eec37090990e1868ba592db so we can get some test results. That commit should be reverted, or the code it adds should be made saner (e.g. don't retry indefinitely) if we really need something like that.

Any way, as of now, everything seems to work on the surface level! \o/ Let's see what Jenkins thinks of the non-fragile tests -- later I should repush with +force-all-tests@ and compare those tests too.

#10 Updated by anonym 7 months ago

  • Feature Branch changed from feature/16356-tor-browser-9.0 to feature/16356-tor-browser-9.0+force-all-tests

anonym wrote:

Let's see what Jenkins thinks of the non-@fragile tests

Ah, it turns out those already passes: https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-16356-tor-browser-9.0/1/

later I should repush with +force-all-tests and compare those tests too.

So I'll repush now.

#11 Updated by anonym 7 months ago

The first full test suite run went pretty well:

Failing Scenarios:
cucumber features/additional_software_packages.feature:69 # Scenario: Recovering in offline mode after Additional Software previously failed to upgrade and then succeed to upgrade when online
cucumber features/electrum.feature:15 # Scenario: Using a persistent Electrum configuration
cucumber features/tor_stream_isolation.feature:9 # Scenario: tails-security-check is using the Tails-specific SocksPort
cucumber features/totem.feature:50 # Scenario: Watching a WebM video over HTTPS

219 scenarios (4 failed, 215 passed)
1658 steps (3 failed, 1655 passed)
346m16.087s

No failure is related to the browser, so we're pretty in great shape!

#12 Updated by anonym 6 months ago

Updated to Tor Browser 9.0a2. What do you think, Jenkins?

#13 Updated by intrigeri 5 months ago

  • Description updated (diff)

#14 Updated by intrigeri 4 months ago

#15 Updated by segfault 4 months ago

  • Description updated (diff)

#16 Updated by intrigeri 4 months ago

  • Priority changed from Normal to Elevated

Nightly builds of Tor Browser 9 based on esr68 are now available: http://f4amtbsowhix7rrf.onion/tor-browser-builds/.

Bumping priority as this is our (FT) next big thing for Sept-Oct. Ideally, we would be able to include this in 4.0~rc1, that should be built around Oct 10.

#17 Updated by intrigeri 3 months ago

  • Description updated (diff)
  • Assignee deleted (anonym)

(anonym made it clear we should not count on him now.)

#18 Updated by intrigeri 3 months ago

  • Related to Feature #10422: Grant Tor Browser access to files as designated by the user added

#19 Updated by intrigeri 3 months ago

9.0a6, based on 68.1.0esr, is now available: https://blog.torproject.org/new-release-tor-browser-90a6

#20 Updated by intrigeri 3 months ago

  • Assignee set to intrigeri

I'll at least update the branch so it uses 9.0a6, based on 68.1.0esr, and then we'll see.

#21 Updated by intrigeri 3 months ago

  • Feature Branch changed from feature/16356-tor-browser-9.0+force-all-tests to feature/16356-tor-browser-9.0+force-all-tests, torbrowser-launcher:feature/16356-tor-browser-9.0+force-all-tests

#22 Updated by intrigeri 3 months ago

  • Blocks Bug #17018: Tor Browser cannot load libmozsandbox.so added

#23 Updated by intrigeri 3 months ago

OK, the branch now builds & Tor Browser starts. There are a number of serious issues that are tracked in subtasks. I'll take a look at the test suite results, will file more subtasks if needed, and then will give this ticket back to the collective pool, hoping that this clarification of what needs to be done will help us organize this work :)

#24 Updated by intrigeri 3 months ago

  • Assignee deleted (intrigeri)

intrigeri wrote:

I'll take a look at the test suite results, will file more subtasks if needed, and then will give this ticket back to the collective pool, hoping that this clarification of what needs to be done will help us organize this work :)

Done.

#25 Updated by intrigeri 3 months ago

  • Target version changed from Tails_3.17 to Tails_4.0

#26 Updated by intrigeri 3 months ago

  • Related to Bug #17042: uBlock buttons disapear when Security level is safest added

#27 Updated by segfault 3 months ago

Pushed a commit which should fix the branch to FTBFS since the uBlock 1.22.2+dfsg-1 release.

#28 Updated by intrigeri 2 months ago

I see that 9.0a7 tarballs are now available and will upgrade the branch to this new alpha right away.

#29 Updated by intrigeri 2 months ago

intrigeri wrote:

I see that 9.0a7 tarballs are now available and will upgrade the branch to this new alpha right away.

Done, let's see how the test suite feels about it :)

#30 Updated by sajolida 2 months ago

This one is really nasty!

To open an URL, I typed it in LibreOffice Writer and right-clicked on it to open the link :)

#31 Updated by intrigeri 2 months ago

Hi @sajolida,

This one is really nasty!
To open an URL, I typed it in LibreOffice Writer and right-clicked on it to open the link :)

Did you mean to comment on #17121?

#32 Updated by intrigeri 2 months ago

  • Priority changed from Elevated to High

#33 Updated by sajolida 2 months ago

Indeed!

#34 Updated by anonym 2 months ago

Note that we still have f74cec4d7fc44d9b9eec37090990e1868ba592db (see #16356#note-9 and in particular the commit message for details) which was meant as a temporary workaround. Not understanding what's going on here is possibly bad enough to warrant an investigation, but there is also the potential for very real issues if tor dies, since onion-grater will endlessly try to reconnect as fast as possible which could lead to performance/battery drain, and (I think?) even a "memory leak" by filling the journal.

At the very least I think we should rate limit the retries after a few tries (to not needlessly delay the common case, where it works on the first retry), e.g.:

--- a/config/chroot_local-includes/usr/local/lib/onion-grater
+++ b/config/chroot_local-includes/usr/local/lib/onion-grater
@@ -567,8 +567,12 @@ class FilteredControlPortProxyHandler(socketserver.StreamRequestHandler):

     def connect_to_real_control_port(self):
         controller = None
+        tries = 0
         while not controller:
+            if tries >= 3:
+                time.sleep(1)
             controller = stem.connection.connect(control_socket=global_args.control_socket_path)
+            tries += 1
         stem.connection.authenticate_cookie(controller, cookie_path=global_args.control_cookie_path)
         return controller

#35 Updated by intrigeri 2 months ago

Hi @anonym,

Note that we still have f74cec4d7fc44d9b9eec37090990e1868ba592db (see #16356#note-9 and in particular the commit message for details) which was meant as a temporary workaround. Not understanding what's going on here is possibly bad enough to warrant an investigation,

Yes, quite possibly. I don't know if we'll get to the end of it by the time we want to close this very ticket, so I'd rather see this tracked in a dedicated ticket.

(I think?) even a "memory leak" by filling the journal.

journald has an upper limit for how much memory it'll use, but yes.

At the very least I think we should rate limit the retries after a few tries (to not needlessly delay the common case, where it works on the first retry), e.g.:

Sounds good, please go ahead!

#36 Updated by intrigeri 2 months ago

As discussed on XMPP with anonym and segfault today, I've merged this branch into devel so it's part of 4.0~rc1.

#37 Updated by intrigeri 2 months ago

  • Blocks deleted (Bug #17018: Tor Browser cannot load libmozsandbox.so)

#38 Updated by intrigeri about 2 months ago

  • Related to Bug #17159: Tor Browser displays an "Update Failed" pop-up added

#39 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Resolved

Ooh yeah.

Also available in: Atom PDF