Project

General

Profile

Bug #8603

Unexpected redirection while sending Host header with wget

Added by sajolida about 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
-
Target version:
Start date:
01/07/2015
Due date:
% Done:

100%

Feature Branch:
bugfix/quote-wrappers-arguments
Type of work:
Code
Blueprint:
Starter:
Affected tool:
check-mirrors

Description

While testing check-mirrors with torsocks in experimental (#8074) I noticed that wget is having a weird behaviour that it didn't had in the past when sending explicit Host headers. It adds an unexpected redirection that bypasses the Host header (and fetches from a different IP address in the context of check-mirrors).

Success without Host header:

amnesia@amnesia:~$ wget http://78.47.150.61/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig --spider --max-redirect=0
Spider mode enabled. Check if remote file exists.
--2015-01-07 20:27:25--  http://78.47.150.61/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig
Connecting to 78.47.150.61:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 836 [application/pgp-signature]
Remote file exists.

Failure with Host header:

amnesia@amnesia:~$ wget http://78.47.150.61/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig --spider --max-redirect=0 --header="Host: dl.amnesia.boum.org" 
Spider mode enabled. Check if remote file exists.
--2015-01-07 20:28:03--  http://78.47.150.61/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig
Connecting to 78.47.150.61:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 836 [application/pgp-signature]
Remote file exists.

Spider mode enabled. Check if remote file exists.
--2015-01-07 20:28:04--  http://dl.amnesia.boum.org/
Resolving dl.amnesia.boum.org (dl.amnesia.boum.org)... 50.254.151.65
Connecting to dl.amnesia.boum.org (dl.amnesia.boum.org)|50.254.151.65|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://cime.net/ [following]
0 redirections exceeded.

Second failure with Host header but no --max-redirect, note that the redirected IP address of the imaginary redirection changes:

amnesia@amnesia:~$ torsocks /usr/bin/wget http://176.9.53.178/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig --spider --header=Host: dl.amnesia.boum.org
Spider mode enabled. Check if remote file exists.
--2015-01-07 20:36:14--  http://176.9.53.178/tails/stable/tails-i386-1.2.2/tails-i386-1.2.2.iso.sig
Connecting to 176.9.53.178:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
Remote file does not exist -- broken link!!!

Spider mode enabled. Check if remote file exists.
--2015-01-07 20:36:15--  http://dl.amnesia.boum.org/
Resolving dl.amnesia.boum.org (dl.amnesia.boum.org)... 96.126.119.95
Connecting to dl.amnesia.boum.org (dl.amnesia.boum.org)|96.126.119.95|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

trace (30.4 KB) sajolida, 01/08/2015 05:34 PM


Related issues

Related to Tails - Feature #8074: Test check-mirrors with torsocks Resolved 10/12/2014
Related to Tails - Feature #6623: Have wget directly use the Tor SOCKS proxy Resolved 01/27/2014
Blocked by Tails - Bug #8830: Buggy passed-through argument quoting in connect-socks, totem, wget and whois wrappers Resolved 02/02/2015

Associated revisions

Revision e0c57b18 (diff)
Added by Tails developers almost 5 years ago

Always pass arguments through wrappers with "$@".

$* doesn't handle arguments with e.g. embedded spaces correctly. The only safe
way to do so is "$@". We know that, but for some reason we've been cargo-culting
and copy-pasting an old buggy wrapper for years, apparently.

Will-Fix: #8603, #8830

Revision f53aec8e
Added by Tails developers almost 5 years ago

Merge branch 'bugfix/quote-wrappers-arguments' into devel

Fix-committed: #8603, #8830

History

#1 Updated by sajolida about 5 years ago

  • Description updated (diff)

#2 Updated by sajolida about 5 years ago

  • Description updated (diff)

#3 Updated by sajolida about 5 years ago

  • Related to Feature #8074: Test check-mirrors with torsocks added

#4 Updated by intrigeri about 5 years ago

  • Assignee changed from intrigeri to sajolida
  • QA Check set to Info Needed
  • Affected tool set to WhisperBack

Looks like a bug in wget: it shouldn't be doing DNS resolution when connecting to a specific IP, even when a Host header is passed. FWIW, I can't reproduce your 2nd test (that fails) on current sid, with a manually torsocks'ified wget. If you can easily test on Tails/experimental with current testing/sid's wget, it would be helpful.

In your 3rd attempt, you explicitly use torsocks, but not in the first two ones. Was that intended? It feels weird, since in the experimental branch we're already wrapping wget with torsocks.

#5 Updated by intrigeri about 5 years ago

  • Affected tool changed from WhisperBack to check-mirrors

#6 Updated by sajolida about 5 years ago

  • Assignee deleted (sajolida)

Note that the same commands in the context of 1.2.2 (same wget package) work fine, so that might as well be a bug in torsocks.

I tried to install wget/testing in experimental and I think that's not possible because it requires an upgrade of libc6:

root@amnesia:~# apt-get install wget=1.16-1 libgnutls-deb0-28=3.3.8-5
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
 libgnutls-deb0-28 : Depends: libc6 (>= 2.17) but 2.13-38+deb7u6 is to be installed
                     Depends: libgmp10 (>= 2:6) but 2:5.0.5+dfsg-2 is to be installed
                     Depends: libhogweed2 (>= 2.7) but 2.4-3 is to be installed
                     Depends: libnettle4 (>= 2.7) but 2.4-3 is to be installed
                     Depends: libp11-kit0 (>= 0.20.7) but 0.12-3 is to be installed
 wget : Depends: libc6 (>= 2.17) but 2.13-38+deb7u6 is to be installed
        Depends: libpsl0 (>= 0.4.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

When I try to install `wget/install` I get a reproducible segfault in at least experiemental and 1.2.1:

root@amnesia:~# apt-get install wget/testing
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '1.16-1' (Debian:testing [i386]) for 'wget'
Selected version '3.3.8-5' (Debian:unstable [i386]) for 'libgnutls-deb0-28' because of 'wget'
Segmentation fault

But maybe that's because `apt-get update` fails with:

Reading package lists... Done
W: Ignoring Provides line with DepCompareOp for package php-psr-log-implementation
W: You may want to run apt-get update to correct these problems

Which might be transient. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774815

On the other hands many other packages install fine.

#7 Updated by sajolida about 5 years ago

Attaching a full trace of the event.

#8 Updated by sajolida about 5 years ago

  • QA Check deleted (Info Needed)

#9 Updated by intrigeri almost 5 years ago

  • Assignee set to sajolida
  • QA Check set to Info Needed

OK, thanks. Please retry in an ISO built from the feature/jessie branch, then: it'll have wget 1.16. I've just merged the fix for #8715 into that branch, hopefully it won't break the build... if it does, get the latest autobuilt Jessie ISO.

#10 Updated by sajolida almost 5 years ago

  • Assignee changed from sajolida to intrigeri
  • QA Check changed from Info Needed to Dev Needed

Ok, I finally did those tests and I think that there is a bug in the wget wrapper.

When I do wget without torsocks I get the same imaginary redirection. This time I tried with bash -x and I think that the arguments are not passed correctly by the wrapper:

amnesia@amnesia:~$ bash -x wget http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig --spider --max-redirect=0 --header="Host: dl.amnesia.boum.org" 
+ unset http_proxy
+ unset HTTP_PROXY
+ unset https_proxy
+ unset HTTPS_PROXY
+ exec torsocks /usr/bin/wget http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig --spider --max-redirect=0 --header=Host: dl.amnesia.boum.org
Spider mode enabled. Check if remote file exists.
--2015-01-31 10:47:44--  http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig
Connecting to 78.47.150.61:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 801 [application/pgp-signature]
Remote file exists.
Spider mode enabled. Check if remote file exists.
--2015-01-31 10:47:44--  http://dl.amnesia.boum.org/
Resolving dl.amnesia.boum.org (dl.amnesia.boum.org)... 195.154.14.189
Connecting to dl.amnesia.boum.org (dl.amnesia.boum.org)|195.154.14.189|:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
Remote file does not exist -- broken link!!!

If I give the correct arguments to torsocks I get the expected result from wget.

amnesia@amnesia:~$ torsocks /usr/bin/wget http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig --spider --max-redirect=0 --header="Host: dl.amnesia.boum.org" 
Spider mode enabled. Check if remote file exists.
--2015-01-31 10:57:45--  http://78.47.150.61/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso.sig
Connecting to 78.47.150.61:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 801 [application/pgp-signature]
Remote file exists.

#11 Updated by sajolida almost 5 years ago

  • Related to Feature #6623: Have wget directly use the Tor SOCKS proxy added

#12 Updated by intrigeri almost 5 years ago

Ok, I finally did those tests and I think that there is a bug in the wget wrapper.

Indeed. It looks like it should be:

exec torsocks /usr/bin/wget "$@" 

instead of

exec torsocks /usr/bin/wget $*

Can you please test that fix?

#13 Updated by Tails almost 5 years ago

  • Status changed from Confirmed to In Progress

Applied in changeset commit:9ee271f5afa99de6adc4e3853907b855e3f5c1dc.

#14 Updated by intrigeri almost 5 years ago

  • Feature Branch set to bugfix/quote-wrappers-arguments

#15 Updated by intrigeri almost 5 years ago

  • Blocked by Bug #8830: Buggy passed-through argument quoting in connect-socks, totem, wget and whois wrappers added

#16 Updated by Tails almost 5 years ago

Applied in changeset commit:009a35821f512ae161b972d2344d7c811722edd1.

#17 Updated by intrigeri almost 5 years ago

  • Assignee changed from intrigeri to bertagaz
  • % Done changed from 0 to 50
  • QA Check changed from Dev Needed to Ready for QA

#18 Updated by intrigeri almost 5 years ago

  • Priority changed from Normal to Elevated

That's a regression, bumping priority.

#19 Updated by sajolida almost 5 years ago

It seems like your patch modifies --header="Host: dl.amnesia.boum.org" into '--header=Host: dl.amnesia.boum.org'. I'm afraid that won't work either. Did you test it?

#20 Updated by sajolida almost 5 years ago

  • Assignee changed from bertagaz to intrigeri
  • QA Check changed from Ready for QA to Dev Needed

#21 Updated by intrigeri almost 5 years ago

It seems like your patch modifies --header="Host: dl.amnesia.boum.org" into '--header=Host: dl.amnesia.boum.org'. I'm afraid that won't work either. Did you test it?

Not by hand, but I wrote an automated test that supposedly tests it.
Did it fail for you in practice?

#22 Updated by intrigeri almost 5 years ago

  • Assignee changed from intrigeri to bertagaz
  • QA Check changed from Dev Needed to Ready for QA

Tested by hand, works fine.

#23 Updated by Tails almost 5 years ago

  • Status changed from In Progress to 11
  • % Done changed from 50 to 100

Applied in changeset commit:c73d1df140c6840ea04c4e992c150b034351c2b5.

#24 Updated by bertagaz almost 5 years ago

  • Assignee deleted (bertagaz)
  • QA Check changed from Ready for QA to Pass

#25 Updated by BitingBird almost 5 years ago

  • Status changed from 11 to Resolved

#26 Updated by sajolida almost 5 years ago

  • Subject changed from Unexpected redirection while send Host header in wget to Unexpected redirection while sending Host header with wget

Also available in: Atom PDF