Project

General

Profile

Feature #9833

Replace AdBlock Plus with uBlock Origin

Added by drizzt over 4 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Low
Assignee:
-
Category:
-
Target version:
Start date:
08/01/2015
Due date:
% Done:

100%

Feature Branch:
spriver:feature/9833-replace-adblock-with-ublock
Type of work:
Code
Blueprint:
Starter:
Affected tool:
Browser

Description

The discussion at https://mailman.boum.org/pipermail/tails-dev/2016-January/010146.html and https://mailman.boum.org/pipermail/tails-dev/2016-February/010163.html sums up a few good reasons to do the switch.

To start with, we should not use uBlock Origin's features (e.g. tracker blocking) others than ad blocking. One can configure it this way apparently: https://mailman.boum.org/pipermail/tails-dev/2016-February/010191.html and https://mailman.boum.org/pipermail/tails-dev/2016-February/010201.html


Related issues

Related to Tails - Feature #12153: Generate ublock.sqlite patterns file from scratch during build time Confirmed 01/18/2017
Related to Tails - Bug #13446: uBlock Origin connects to its github Rejected 07/08/2017

Associated revisions

Revision 5d7b901d
Added by anonym almost 3 years ago

Merge remote-tracking branch 'origin/feature/9833-replace-adblock-with-ublock' into devel

Fix-committed: #9833

History

#1 Updated by intrigeri over 4 years ago

  • Assignee set to goupille

I expect frontdesk to:

  1. check if uBlock is in Debian
  2. check what's the AdBlock Plus pref to disable "acceptable ads", and what is its value in current Tails

#2 Updated by BitingBird over 4 years ago

  • Affected tool set to Browser

#3 Updated by goupille over 4 years ago

Hi !

uBlock is not yet in Debian :
https://tracker.debian.org/search?package_name=ublock

but the process to include it is started :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=782074

as for the "acceptable ads" option, it is in add-ons > AdBlock > Preferences > filter preferences > Allow some non-intrusive advertising and it is set to 'No' (unchecked box) in Tails

#4 Updated by intrigeri over 4 years ago

  • Status changed from New to Rejected

as for the "acceptable ads" option, it is in add-ons > AdBlock > Preferences > filter preferences > Allow some non-intrusive advertising and it is set to 'No' (unchecked box) in Tails

Thanks, thus closing as this was the main argument in favour of replacing it with uBlock. Anyone, feel free to reopen once uBlock is in Debian and there are compelling arguments to switch.

#5 Updated by intrigeri almost 4 years ago

  • Status changed from Rejected to In Progress
  • Assignee deleted (goupille)

"[Tails-dev] Migrate Tail's browser AdBlock Plus to uBlock Origin"

#7 Updated by intrigeri almost 4 years ago

  • Subject changed from Remove AdBlock Plus and use uBlock to Replace AdBlock Plus with uBlock Origin
  • Description updated (diff)
  • Type of work changed from Discuss to Code

#8 Updated by BitingBird over 3 years ago

It's now in Debian testing. It should be backported to Jessie, or the work should be done in the Tails-Stretch branch.

#9 Updated by BitingBird over 3 years ago

  • Status changed from In Progress to Confirmed

#10 Updated by anonym over 3 years ago

  • Assignee set to spriver

spriver volunteered to give this a try. \o/

#11 Updated by anonym about 3 years ago

  • Blocks Bug #11419: Deal with mandatory extension signing post FF45esr added

#12 Updated by anonym about 3 years ago

  • Status changed from Confirmed to In Progress
  • Priority changed from Low to Normal
  • Target version set to Tails 2.10

It's be great if this happened in Tails 2.10, the last major release before we get mandatory extension signing (#11419) since this is a new feature, not bugfix. Still, if it cannot happen until 2.11 that's ok -- in that point release we'll introduce the new ESR series, which is way crazier for a point release.

Also, because of this, this ticket is not low prio any more.

Also, I know spriver has made progress on this privately! :)

#13 Updated by anonym about 3 years ago

  • Priority changed from Normal to Low
  • Target version deleted (Tails 2.10)

Apparently I was wrong: installing uBlock Origins via the .deb still results in an unsigned add-on.

#14 Updated by anonym about 3 years ago

  • Blocks deleted (Bug #11419: Deal with mandatory extension signing post FF45esr)

#15 Updated by spriver almost 3 years ago

  • Assignee changed from spriver to anonym
  • Target version set to Tails 2.10
  • QA Check set to Info Needed
  • Feature Branch set to spriver:feature/9833-replace-adblock-with-ublock

My findings so far with uBlock:
Installation after adding a pinning for xul-ext-ublock-origin is going fine. I also modified the script installing the Tor Browser. It's the same procedure as with adblock-plus, except that the symlink in ~/.tor-browser/profile.default/extensions/ wants to be called "". The addon is unsigned, though (see #11419). So far, so good.

But there's a thing that I'm having problems with solving, it's the patterns and settings file of uBlock (~/.tor-browser/profile.default/extension-data/ublock0.sqlite). This file contains the ad blocking patterns and settings the addon is using and is a sqlite database blob. The patterns can be updated in a similar way as it's being done while the release process of a Tails version is done: start Tails and TBB in it, update the file via the plugin, copy it into the git repo and commit it. However, we will pollute the git repo with a ~9MByte huge binary blob (containing EasyList, EasyList DE and EasyList FR).
There could be several approaches for solving the issue (apart from having the binary blob in our git, which I'd like to avoid):

  • The blob's created while building Tails. For doing this we need to include the EasyList text files and then create the database out ouf them, including the needed settings for the plugin (However, uBlock's using a feature creating pre-compiled filter lists for faster startup and filtering: https://github.com/gorhill/uBlock/wiki/Launch-and-filter-lists-load-performance. But ublock.sqlite contains both non- and precompiled patterns, so I think that having them not pre-compiled will "only" result in a bit slower speed. I need to investigate that further.) I'm certainly not a database pro, so I likely would need some help here for building the whole blob file.
  • While build time the blob will be created out of a database dump file which is stored as plain text. The dump can be saved in the repo without polluting it. This would require the RM to get the original sqlite file from inside a running Tails, then convert it into a text dump (which is a bash one-liner) and while build time it gets converted into the blob and included into the image being built. The git repo will then only include the text dump.
  • Maybe there's also the possibility that uBlock will fetch the filter lists after Tor Browser is being launched after each boot of Tails. uBlock has a feature for doing that, but my knowledge so far is that the update will only be applied every four days. The validity of EasyLists is four days, too, so maybe the update decision is being made upon the validity time of EasyLists (another thing I need to investigate). Maybe we can include a minimal sized blob file with outdated EasyLists that are getting updated by the addon itself after Tails is being booted and the browser was started.

I pushed now a first branch with a pre-installed uBlock (and having AdBlockPlus removed), which contains a sqlite database blob. This is only a temporary solution, as I'd like to avoid this.

#16 Updated by spriver almost 3 years ago

  • Maybe there's also the possibility that uBlock will fetch the filter lists after Tor Browser is being launched after each boot of Tails. uBlock has a feature for doing that, but my knowledge so far is that the update will only be applied every four days. The validity of EasyLists is four days, too, so maybe the update decision is being made upon the validity time of EasyLists (another thing I need to investigate). Maybe we can include a minimal sized blob file with outdated EasyLists that are getting updated by the addon itself after Tails is being booted and the browser was started.

This does not seem to be an option, as filter lists are updated ~4-5 days accordingly to https://github.com/gorhill/uBlock/wiki/Dashboard:-3rd-party-filters.

#17 Updated by spriver almost 3 years ago

The pre-compiled filter lists can be removed easily and with no problems for uBlock itself. uBlock will generate the pre-compiled filter lists after TBB will be started. So we could have a process like this:

  • Refresh the filter lists from a running Tails and copy the ublock0.sqlite to a working directory
  • Run a command e.g. echo '.dump' | sqlite3 ublock0.sqlite | grep -v "cached_asset_content://cache://compiled-" >> ublockdatabase.dump to convert the sqlite blob into a text file and remove all compiled filter lists after doing that
  • Recompile the textdump into a sqlite database while build time with something similar to cat ublockdatabase.dump | sqlite3 ublock0.sqlite and move it into config/chroot_local-includes/etc/tor-browser/profile/extension-data/ublock0.sqlite

I don't know if this actually works and I'm aware that this is a sort of hack-ish workaround.

Generating the sqlite database itself from scratch while boot time seems to be a bit complicated as the there'll be a checksum generated for each filter list. It'd also require some more of sqlite-voodoo I'm not capable of.

#18 Updated by anonym almost 3 years ago

spriver wrote:

The pre-compiled filter lists can be removed easily and with no problems for uBlock itself. uBlock will generate the pre-compiled filter lists after TBB will be started. So we could have a process like this:

  • Refresh the filter lists from a running Tails and copy the ublock0.sqlite to a working directory
  • Run a command e.g. echo '.dump' | sqlite3 ublock0.sqlite | grep -v "cached_asset_content://cache://compiled-" >> ublockdatabase.dump to convert the sqlite blob into a text file and remove all compiled filter lists after doing that

It might be worth throwing in | sort as well since that will reduce the diff further. (This could break the checksum mentioned below, however, and in that case we don't care about the less efficient diff, I guess.)

  • Recompile the textdump into a sqlite database while build time with something similar to cat ublockdatabase.dump | sqlite3 ublock0.sqlite and move it into config/chroot_local-includes/etc/tor-browser/profile/extension-data/ublock0.sqlite

I don't know if this actually works and I'm aware that this is a sort of hack-ish workaround.

It sounds both doable and like a good approach to me, not at all like a hack! :)

Generating the sqlite database itself from scratch while boot time seems to be a bit complicated as the there'll be a checksum generated for each filter list.

Let's just be really careful when we extract the filters, and also extract the checksum -- then when we restore the database the checksums should be ok. Right?

It'd also require some more of sqlite-voodoo I'm not capable of.

If you need help, you need to be more specific! :)

#19 Updated by spriver almost 3 years ago

anonym wrote:

spriver wrote:

The pre-compiled filter lists can be removed easily and with no problems for uBlock itself. uBlock will generate the pre-compiled filter lists after TBB will be started. So we could have a process like this:

  • Refresh the filter lists from a running Tails and copy the ublock0.sqlite to a working directory
  • Run a command e.g. echo '.dump' | sqlite3 ublock0.sqlite | grep -v "cached_asset_content://cache://compiled-" >> ublockdatabase.dump to convert the sqlite blob into a text file and remove all compiled filter lists after doing that

It might be worth throwing in | sort as well since that will reduce the diff further. (This could break the checksum mentioned below, however, and in that case we don't care about the less efficient diff, I guess.)

The filter lists are stored in one list per line (the patterns' values are being seperated by \n), so sort won't have any effect there.

  • Recompile the textdump into a sqlite database while build time with something similar to cat ublockdatabase.dump | sqlite3 ublock0.sqlite and move it into config/chroot_local-includes/etc/tor-browser/profile/extension-data/ublock0.sqlite

I don't know if this actually works and I'm aware that this is a sort of hack-ish workaround.

It sounds both doable and like a good approach to me, not at all like a hack! :)

How can this approach be done while building Tails? Where can I store the sqlite plaintext dump in the git repo? How do I ensure that sqlite3 is installed on the building system (or is sqlite3 already used in the building process?). And how can do the actual conversion from plaintext dump while building and include the resulting blob into the folder .tor-browser/profile.default/extension-data/?

Generating the sqlite database itself from scratch while boot time seems to be a bit complicated as the there'll be a checksum generated for each filter list.

Let's just be really careful when we extract the filters, and also extract the checksum -- then when we restore the database the checksums should be ok. Right?

Just creating the dump and removing lines, e.g. the pre-compiled filter patterns, is not a problem. I'll research later today if the checksums are needed at all (in my first tests it seems like that, but maybe this was also because of something different change I made) and if the lines containing the patterns can be modified without breaking uBlock.

It'd also require some more of sqlite-voodoo I'm not capable of.

If you need help, you need to be more specific! :)

I'd for now try to implement uBlock with the sqlite dump variant with generating the blob from the sqlite dump while build time. I'll frame the requirements for building the DB blob from scratch while building Tails during the next days. Maybe we should also contact the developer of uBlock Origin to find out, what is needed (e.g. the ublock0.sqlite contains fields that are unclear for me right now. For instance compiledMagic and selfieMagic which seem to be something like checksums).

#20 Updated by anonym almost 3 years ago

  • Assignee changed from anonym to spriver
  • QA Check deleted (Info Needed)

spriver wrote:

anonym wrote:

spriver wrote:

The pre-compiled filter lists can be removed easily and with no problems for uBlock itself. uBlock will generate the pre-compiled filter lists after TBB will be started. So we could have a process like this:

  • Refresh the filter lists from a running Tails and copy the ublock0.sqlite to a working directory
  • Run a command e.g. echo '.dump' | sqlite3 ublock0.sqlite | grep -v "cached_asset_content://cache://compiled-" >> ublockdatabase.dump to convert the sqlite blob into a text file and remove all compiled filter lists after doing that

It might be worth throwing in | sort as well since that will reduce the diff further. (This could break the checksum mentioned below, however, and in that case we don't care about the less efficient diff, I guess.)

The filter lists are stored in one list per line (the patterns' values are being seperated by \n), so sort won't have any effect there.

Ok, so then we need to insert newlines when extracting (so use sed and replace each filter separator with \n), and remove them when restoring (use sed to replace the newlines with the separator) -- otherwise it won't be much more line-diff friendly. Any way, the sorting will definitely break the checksum, so ignore that part.

  • Recompile the textdump into a sqlite database while build time with something similar to cat ublockdatabase.dump | sqlite3 ublock0.sqlite and move it into config/chroot_local-includes/etc/tor-browser/profile/extension-data/ublock0.sqlite

I don't know if this actually works and I'm aware that this is a sort of hack-ish workaround.

It sounds both doable and like a good approach to me, not at all like a hack! :)

How can this approach be done while building Tails? Where can I store the sqlite plaintext dump in the git repo?

We already store things like these in config/chroot_local-includes/usr/share/tails/. Just create a ublock-origin folder there. Remove that folder in config/chroot_local-hooks/98-remove_unwanted_files since it won't be needed post-build.

How do I ensure that sqlite3 is installed on the building system

Just apt install it in the hook (config/chroot_local-hooks) you'll create for the filter list restoration. And apt purge it when done.

(or is sqlite3 already used in the building process?).

At least in my last build sqlite3 was not installed (but libsqlite3-0 was).

And how can do the actual conversion from plaintext dump while building

Like I said above, just do it in a new build hook (config/chroot_local-hooks) e.g. generate-ublock-origin-filter.

and include the resulting blob into the folder .tor-browser/profile.default/extension-data/?

Just put the files in /etc/tor-browser/profile/extension-data/ (have a look at config/chroot_local-includes/usr/local/bin/tor-browser and config/chroot_local-includes/usr/local/lib/generate-tor-browser-profile).

Generating the sqlite database itself from scratch while boot time seems to be a bit complicated as the there'll be a checksum generated for each filter list.

Let's just be really careful when we extract the filters, and also extract the checksum -- then when we restore the database the checksums should be ok. Right?

Just creating the dump and removing lines, e.g. the pre-compiled filter patterns, is not a problem. I'll research later today if the checksums are needed at all (in my first tests it seems like that, but maybe this was also because of something different change I made) and if the lines containing the patterns can be modified without breaking uBlock.

Cool!

It'd also require some more of sqlite-voodoo I'm not capable of.

If you need help, you need to be more specific! :)

I'd for now try to implement uBlock with the sqlite dump variant with generating the blob from the sqlite dump while build time. I'll frame the requirements for building the DB blob from scratch

Do whatever is simplest. I suspect that is to dump the whole database to plaintext and not keep a partial binary blob with only the settings. I suspect the only tricky thing then is to know when to replace a newline with the "filter separator" (for the splitted filter lines) and when to not (for everything else).

while building Tails during the next days.

Note that the feature freeze for Tails 2.10 is on 2017-01-12 (noon CET) so if you're still aiming at that release, time is short! I'm available to accelerate things if you think it's possible.

Maybe we should also contact the developer of uBlock Origin to find out, what is needed (e.g. the ublock0.sqlite contains fields that are unclear for me right now. For instance compiledMagic and selfieMagic which seem to be something like checksums).

Sounds like a good idea!

#21 Updated by spriver almost 3 years ago

  • Assignee changed from spriver to anonym
  • QA Check set to Ready for QA

I pushed a branch implementing the steps from https://labs.riseup.net/code/issues/9833#note-20.
I needed to copy the dump file from /usr/share/tails/ublock-origin/ublock0.dump to /etc/tor-browser/profile/extension-data/ublock0.dump" for being able to create the sqlite blob (otherwise I got the error Error: unable to open database "/etc/tor-browser/profile/extension-data/": unable to open database file). The file will be deleted afterwards, but maybe there's a bit "cleaner" solution for that.

Updating of contribute/release_process.mdwn is required accordingly to the creation of the dump file once the branch will be reviewed/merged.

#22 Updated by anonym almost 3 years ago

From IRC:

11:55:10 PM - spriver: anonym: your commits looks good to me, uBlock's also
fine with the re-converted database. but it seems that uBlock is not keeping
the order of the lists consistently between updates which is going to create a
huge diff…

Ouch... can you email me the other database you tried with?

#23 Updated by anonym almost 3 years ago

I'm looking at:

git log --stat --follow config/chroot_local-includes/etc/tor-browser/profile/adblockplus/patterns.ini

(from any other branch than feature/9833-replace-adblock-with-ublock :)) and I see that ~10000 lines changed every month. Is it worse than that?

It actually looks to me like what we have in the dump are the raw eastlists, and they shouldn't use any different sorting. So, if your other uBlock Origin database was from October, i.e. three months ago, 30K changed lines would be identical. I actually expect that there is no problem here! :)

#24 Updated by anonym almost 3 years ago

  • Status changed from In Progress to 11
  • Assignee deleted (anonym)
  • % Done changed from 0 to 100
  • QA Check changed from Ready for QA to Pass

Phew! We just barely made it for Tails 2.10. Awesome! :)

#25 Updated by gorhill almost 3 years ago

spriver wrote:

Maybe we should also contact the developer of uBlock Origin to find out, what is needed (e.g. the ublock0.sqlite contains fields that are unclear for me right now. For instance compiledMagic and selfieMagic which seem to be something like checksums).

Hello, I am the developer of uBlock Origin ("uBO"), I just became aware that uBO is shipping in the latest RC version of Tails, I wish I could have been of help earlier.

Regarding what uBO needs in the SQL database depends on what are the specific settings you wish to be different than the ones set by default. Knowing this I will be able to provide some information.

Normally it should not be necessary to include the content of filter lists in the SQL file, as uBO will fetch the content of the enabled filter list(s) from their remote location(s) or the local installation (assets/thirdparties/). Some minutes after uBO is launched, it will update the enabled filter lists if needed, and their content will then be saved in the SQL database, along with a compiled version to speed up setup time next time uBO is launched.

#26 Updated by spriver almost 3 years ago

  • Related to Feature #12153: Generate ublock.sqlite patterns file from scratch during build time added

#27 Updated by anonym almost 3 years ago

  • Status changed from 11 to Resolved

#29 Updated by u over 2 years ago

  • Related to Bug #13446: uBlock Origin connects to its github added

Also available in: Atom PDF