Project

General

Profile

Bug #9412

Add SOCKS support to WhisperBack

Added by intrigeri over 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Elevated
Assignee:
Category:
-
Target version:
Start date:
05/16/2015
Due date:
07/15/2015
% Done:

100%

Feature Branch:
whisperback:feature/9412-add-socks-support-to-whisperback
Type of work:
Code
Blueprint:
Starter:
Affected tool:
WhisperBack

Description

We patch WhisperBack's .desktop file to start it with torsocks, and all I see is a long stream of:

WARNING torsocks[3823]: [syscall] Unsupported syscall number 240. Denying the call (in tsocks_syscall() at syscall.c:165)

... on stderr, and the WhisperBack GUI never appears. WhisperBack starts fine without torsocks, though.

monkeypatching-the-entire-connection-through-the-socks-proxy.patch View (2.02 KB) romeopapa, 08/10/2015 05:23 AM

0001-Monkeypatching-the-entire-connection-through-the-SOC.patch View (1.76 KB) romeopapa, 08/11/2015 08:07 AM

0002-The-SOCKS-host-and-port-can-be-configured-in-whisper.patch View (6.81 KB) romeopapa, 08/11/2015 09:15 PM

0003-mail.py-does-not-need-to-import-os-did-not-use-envir.patch View (1011 Bytes) romeopapa, 08/11/2015 09:36 PM

0004-Validates-SMTP-host-and-port-as-well-as-SOCKS-host-a.patch View (5.68 KB) romeopapa, 08/12/2015 03:29 PM

cleaned-0001-Monkeypatching-the-entire-connection-through-the-SOC.patch View - 229e894f (1.76 KB) romeopapa, 08/18/2015 05:32 AM

cleaned-0002-The-SOCKS-host-and-port-can-be-configured-in-whisper.patch View - 0b4d804f (4.72 KB) romeopapa, 08/18/2015 05:32 AM

cleaned-0003-Validates-hosts-and-ports-for-SOCKS-and-SMTP.patch View - a03ce284 (3.4 KB) romeopapa, 08/18/2015 05:32 AM


Related issues

Related to Tails - Bug #7755: Migrate WhisperBack to Python 3 Resolved 09/13/2014

Associated revisions

Revision c85411bb (diff)
Added by alant almost 4 years ago

Implement Tor stream isolation for WhisperBack

Will-fix: #9412

Revision a25da197 (diff)
Added by alant almost 4 years ago

Don't wrap WhisperBack with torsocks

It never worked under jessie and is not necessary now that WhisperBack supports
SOCKS proxies.

Will-fix: #9412

History

#1 Updated by intrigeri over 4 years ago

  • Related to Bug #7755: Migrate WhisperBack to Python 3 added

#2 Updated by intrigeri over 4 years ago

Perhaps the right way to fix that would be to give WhisperBack native SOCKS5 support?

#3 Updated by alant about 4 years ago

  • Subject changed from WhisperBack doesn't start under torsocks on Jessie to Add socks support to WhisperBack

intrigeri wrote:

Perhaps the right way to fix that would be to give WhisperBack native SOCKS5 support?

That looks quite easy to do, so let's do this way!

#4 Updated by intrigeri about 4 years ago

That looks quite easy to do, so let's do this way!

Woohoo :)

#5 Updated by intrigeri about 4 years ago

  • Subject changed from Add socks support to WhisperBack to Add SOCKS support to WhisperBack

#6 Updated by intrigeri about 4 years ago

  • Due date set to 07/15/2015

#8 Updated by alant about 4 years ago

What's your ETA on this one?

I wasn't aware of a deadline but I can very well do that before october 15th. Can you take care of the packaging?

#9 Updated by intrigeri about 4 years ago

I can very well do that before october 15th.

Excellent. Even better if it's done earlier, e.g. by the end of August or the end of September.

Can you take care of the packaging?

Sure!

#10 Updated by romeopapa about 4 years ago

Hi alant,

I was looking for simple things to do on python with jessie and intregeri said I might give you a hand on this high priority task.

I've based my work on the feature/python3 branch.

I've attached a very basic patch that should implement native SOCKS support, but I'm unable to test it as apparently it seems to fail upstream (see https://labs.riseup.net/code/issues/7755#note-38 )

Take care

#11 Updated by romeopapa about 4 years ago

Here's the updated patch, with which I've been able to successfully send a message earlier today.

I've also published the branch on gitlab, on a clone of whisperback's repo. Maybe it's more convenient for you: https://gitlab.com/romeopapa/whisperback/commits/feature/python3-9412-add-socks-support-to-whisperback

Take care,

#12 Updated by romeopapa about 4 years ago

To note, i was also going to supply a patch for tails/feature/jessie but intrigeri's commit 72647c7b2c674c7488a32d8942562d1d692b9d94 (Replace patches that wrapped apps with torsocks with dynamic patching with a hook.) has made that unnecessary.

#13 Updated by intrigeri about 4 years ago

  • QA Check set to Dev Needed

I've also published the branch on gitlab, on a clone of whisperback's repo.
Maybe it's more convenient for you:

Sure it is!

Before we review it in depth, may you please make the SOCKS port configurable? E.g. in Tails we're going to use something else than 9050. If you're not interested in that task, just say so :)

#14 Updated by romeopapa about 4 years ago

Sure, I can make the SOCKS posrt configurable.

E.g. in Tails we're going to use something else than 9050

  • Should it be a constant that could be overloaded (in order of priority) by: a command line argument and an environmnent variable and a config file somewhere?
  • or were you thinking about a GUI element?

#15 Updated by romeopapa about 4 years ago

<anonym>: read from the ENV something like WHISPERBACK_SOCKS_HOST and WHISPERBACK_SOCKS_PORT, but default to 127.0.0.1 and 9050 if not present

OK, going for environment variables WHISPERBACK_SOCKS_HOST and WHISPERBACK_SOCKS_PORT.

#16 Updated by romeopapa about 4 years ago

romeopapa wrote:

<anonym>: read from the ENV something like WHISPERBACK_SOCKS_HOST and WHISPERBACK_SOCKS_PORT, but default to 127.0.0.1 and 9050 if not present

Actually I've seen the way the config.py file works so I've also submitted a patch to tails (feature/jessie) so that the default values for SOCKS will be in there. Here's the commit on my gitlab fork:
https://gitlab.com/romeopapa/tails/commits/feature/jessie-9412-add-socks-support-to-whisperback

You will find attached 0002-The-SOCKS-host-and-port-can-be-configured-in-whisper.patch, or the whole series of commits to whisperback at my gitlab fork (ready to be merged into feature/python3)
https://gitlab.com/romeopapa/whisperback/commits/feature/python3-9412-add-socks-support-to-whisperback

Some notes:
  • I started with the environment variables and a system that would warn you in comprehensive terms if you weren't using the tail's hardcoded defaults (because I saw a wrongly configured socks proxy as a potential anonymity leak - but then I realised that so would be a misconfigured smtp server, etc..).
  • I've added some data validation. It tries to verify that it is a correct hostname (follows RFC 1123) or a correct IPv4 address (pysocks is not compatible with ipv6). I could finish and 'validate' the rest of the parameters loaded by config.py

#17 Updated by romeopapa about 4 years ago

Sorry it's probably not clear from my previous post, but I did not use any environment variables in the end.

Attached, one last patch (pushed on gitlab) that removes an unneeded import to 'os' I had just added.

#18 Updated by intrigeri about 4 years ago

Sure, I can make the SOCKS posrt configurable.

:)

E.g. in Tails we're going to use something else than 9050

  • Should it be a constant that could be overloaded (in order of priority) by: a command line argument and an environmnent variable and a config file somewhere?

See the "CONFIGURATION" section in README, in order to make this new setting consistent with how the rest of WhisperBack is configured :)

  • or were you thinking about a GUI element?

No.

#19 Updated by romeopapa about 4 years ago

See the "CONFIGURATION" section in README, in order to make this new setting consistent with how the rest of WhisperBack is configured :)

Should be good, followed the implementation. I believe task is complete.

#20 Updated by romeopapa about 4 years ago

Here's another additional patch, that repacks the validation code of hostname/ipaddress into functions, and now validates the SMTP host and port as well.

The rest of the inputs are not verified.

Patch 0004 attached, and submitted to gitlab at:
https://gitlab.com/romeopapa/whisperback/commits/feature/python3-9412-add-socks-support-to-whisperback

#21 Updated by BitingBird about 4 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 30
  • QA Check changed from Dev Needed to Ready for QA

#22 Updated by alant about 4 years ago

Hi romeopapa

Thanks a lot for your patches! I tested them and they work perfectly. Find below a few comments before I merge them.

  • commit 6d195d8:
    • merge 660a2c4 (do not import os)
  • commit a8cfaea: please rewrite it to:
    • not import os
    • add default values for socks_host and socks_port to the definition of send_message_tls in mail.py
    • do not fix whitespace errors in unrelated commit (# Load the python configuration file "config.py" from different locations)
    • do not move # XXX: Add sanity checks because it still applies to the old lines
    • validateHostnameOrIPv4: see next commit. It would be even more clean if you added all the sanity checks in commit 8257943
    • please add the parameters you add to the "SMTP" section of doc/config.py.sample
    • fixup for docstring It doesn't take any argument. should be in a separate commit in another branch (or be dropped).
  • commit 660a2c4
    • meld into 6d195d8
  • commit 8257943
    • param subject The topic of the feedback; def execute_threaded(self, func, args, progress_callback=None, and next changes; param progress_callback: do not fix whitespace errors in unrelated commit
    • put validateHostnameOrIPv4 and validatePortValue in utils.py, where other validation functions are, and rename them validate_hostname_or_ipv4 and validate_port (according to project coding style ane PEP8).

Congrats again for your contribution!

#23 Updated by romeopapa about 4 years ago

alant, thanks for the review

Having fun rebasing with git, will post cleaned up patches when I'm done.

#24 Updated by intrigeri about 4 years ago

  • QA Check changed from Ready for QA to Dev Needed

#25 Updated by romeopapa about 4 years ago

There we go.

You might want to have a look at the validate_port and validate_hostname_or_ipv4 functions. They return True/False (like the others in utils.py) but I don't use thos values when calling the functions.

#26 Updated by intrigeri about 4 years ago

  • QA Check changed from Dev Needed to Ready for QA

#27 Updated by alant about 4 years ago

romeopapa wrote:

There we go.

Great! I still have a few issues though.

The change in debian/control in 0b4d804 should be in 229e894 which is the commit adding the dependancy (or in a specific commit if you prefern but definately not in "The SOCKS host and port can be configured in whisperback's config.py"

You might want to have a look at the validate_port and validate_hostname_or_ipv4 functions. They return True/False (like the others in utils.py) but I don't use thos values when calling the functions.

Your current proposal doesn't make sense, because the return False statement cannot be reached.

Also, you call the validation functions with the wrong numbers of arguments:

whisperBack.utils.validate_hostname_or_ipv4(self.smtp_host, "smtp_host")
def validate_hostname_or_ipv4(self, hostname_or_ipv4, valuename):

whisperBack.utils is a module, not an object, so there is nothing to reference as self. You are writing pure functions here, not object methods (that's actually why they are best in utils.py).

Finally, I'm not sure of the interest of passing valuename here (see below).

I think there is a choice that needs to be made here:

  • either you use validate_ functions that raise exceptions, as in your 1st patch, and don't set a return value
  • or you use is_valid_ functions that return a boolean, as the others in util.py do, and then raise a whisperBack.exceptions.MisconfigurationException in __check_conf, e.g.:
if not self.socks_host:
    raise whisperBack.exceptions.MisconfigurationException('socks_host')
elif not whisperBack.utils.is_valid_hostname_or_ipv4(self.socks_host):
    raise ValueError("socks_host is not a correct hostname or IPv4 address")

I tend to perfer the 2nd proposal which seems me more coherent with the current code, at the price of having less clear description of the cause of the lack of correctness.

To end with, no need to create patches and attach them here, pushing to your git repository is enough (for others that might want to follow the discussion, you can add an URL to its web interface).

Thanks by advance for your patience.

#28 Updated by romeopapa about 4 years ago

Firstly, I might have rushed things a little. Maybe if I actually tried to run the code I wrote, there'd be less things to change. Sorry about that. Secondly, thanks for putting things so eloquently and taking the time to walk me through all of this, I'm learning some good things!

there is a choice that needs to be made here:
  • either you use validate_ functions that raise exceptions, as in your 1st patch, and don't set a return value
  • or you use is_valid_ functions that return a boolean, as the others in util.py do, and then raise a whisperBack.exceptions.MisconfigurationException in __check_conf

I tend to perfer the 2nd proposal which seems me more coherent with the current code, at the price of having less clear description of the cause of the lack of correctness.

Going with #2.

OK, I've repushed to https://gitlab.com/romeopapa/whisperback/commits/feature/python3-9412-add-socks-support-to-whisperback
All the clean and fixed commits are in there. I have actually tested the code this time ;)

#29 Updated by alant about 4 years ago

  • QA Check changed from Ready for QA to Dev Needed

romeopapa wrote:

All the clean and fixed commits are in there. I have actually tested the code this time ;)

Great! One comment on the last commit:


    try:
        int(candidate)
    except:
        return False

Don't use a generic except but only catch the expected exception; in our case except ValueError: (https://www.python.org/dev/peps/pep-0008/#programming-recommendations).

#30 Updated by romeopapa about 4 years ago

Don't use a generic except but only catch the expected exception; in our case except ValueError: (https://www.python.org/dev/peps/pep-0008/#programming-recommendations).

The excerpt in question: "A bare except: clause will catch SystemExit and KeyboardInterrupt exceptions, making it harder to interrupt a program with Control-C, and can disguise other problems."

I've ammended the last commit and repushed to https://gitlab.com/romeopapa/whisperback/commits/feature/python3-9412-add-socks-support-to-whisperback

#31 Updated by intrigeri about 4 years ago

  • QA Check changed from Dev Needed to Ready for QA

I've ammended the last commit and repushed to
https://gitlab.com/romeopapa/whisperback/commits/feature/python3-9412-add-socks-support-to-whisperback

Cool! Please update the "QA Check" field accordingly next time: https://tails.boum.org/contribute/merge_policy/#submit

#32 Updated by romeopapa about 4 years ago

intrigeri wrote:

Cool! Please update the "QA Check" field accordingly next time: https://tails.boum.org/contribute/merge_policy/#submit

Hi intrigeri.. In case this message was directed towards me: I don't think my user romeopapa on the redmine has the right to do that.. I think I can only add notes (and attachments) to a ticket, not much more (or perhaps I'm not allowed because I haven't created THIS ticket).

Cheers!

#33 Updated by sajolida about 4 years ago

romeopapa: I granted you the "Contributor" role now. You should be able to do more stuff.

#34 Updated by alant about 4 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (alant)
  • % Done changed from 30 to 100
  • QA Check changed from Ready for QA to Pass
  • Feature Branch set to whisperback:feature/9412-add-socks-support-to-whisperback

I merged your branch to whisperback's feature/jessie branch. Thanks for you contribution to Tails!

#35 Updated by romeopapa about 4 years ago

  • Status changed from Fix committed to In Progress
  • Assignee set to alant
  • % Done changed from 100 to 80
  • QA Check changed from Pass to Ready for QA

sajolida wrote:

romeopapa: I granted you the "Contributor" role now. You should be able to do more stuff.

Thanks sajolida.

I've reopened the bug, there's one last commit that must be merged. It is for Tails repository:
Default values in WhisperBack's config file.

Here's the commit (ba8f3ffe) in question.

Thanks,

PS: Sorry if I messed up the ticket's modifications.. Reassigning to alant because he was in charge before.

#38 Updated by alant almost 4 years ago

  • QA Check deleted (Ready for QA)

It should be ready in feature/jessie, but I've got strange build errors with the APT repository, so I can't test yet.

#39 Updated by intrigeri almost 4 years ago

  • QA Check set to Dev Needed

It should be ready in feature/jessie,

I see that 11c68aa introduces socks_port = 9500, and I don't understand how it can work, nor why that number was chosen.

I think we should use the SOCKS port our design doc specifies for Tails-specific applications: https://tails.boum.org/contribute/design/stream_isolation/ (which will need an update as part of merging this branch, by the way, once we use the right port).

but I've got strange build errors with the APT repository, so I can't test yet.

Replied separately, but tl;dr: feature/jessie builds fine for me here, and the problem seems to be in your local build environment.

#40 Updated by alant almost 4 years ago

#41 Updated by alant almost 4 years ago

#42 Updated by intrigeri almost 4 years ago

Looks good, thanks! Sent a report with that WhisperBack and asked frontdesk to forward it to me.

Two small notes, though:

  • I find it sad that the versioning scheme for Jessie-specific releases was suddenly changed with no obvious reason, but well -- I'll try to remember it and not to mess up by assuming it's still like it used to be.
  • Pushed a commit on top to remove trailing whitespace introduced by the branch you merged -- please make sure your Git prevents you from merging such stuff in the future.

#43 Updated by intrigeri almost 4 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 80 to 100
  • QA Check changed from Dev Needed to Pass

Frontdesk got the report, yay!

Also available in: Atom PDF