Project

General

Profile

Bug #10523

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

whois test is fragile

Added by kytv about 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
Start date:
11/09/2015
Due date:
% Done:

100%

Feature Branch:
kytv/test/10523-whois-test-is-fragile
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

Seen in a few recent jobs from jenkins.

calling as amnesia: whois 'torproject.org'
call returned: [0, "WHOIS LIMIT EXCEEDED - SEE WWW.PIR.ORG/WHOIS FOR DETAILS\n", ""]
    When I query the whois directory service for "torproject.org"                  # features/step_definitions/torified_misc.rb:1
    Then the whois command is successful                                           # features/step_definitions/torified_misc.rb:25
    Then the whois standard output contains "The Tor Project"                      # features/step_definitions/torified_misc.rb:34
      The whois standard output does not contain The Tor Project:
      WHOIS LIMIT EXCEEDED - SEE WWW.PIR.ORG/WHOIS FOR DETAILS

      .
      <nil> is not true. (Test::Unit::AssertionFailedError)
      ./features/step_definitions/torified_misc.rb:35:in `/^the (wget|whois) standard output contains "([^"]+)"$/'
      features/torified_misc.feature:24:in `Then the whois standard output contains "The Tor Project"'

This was addressed previously but when the retry_tor code was refactored this regression sneaked in. The problem stems from the fact that whois still "succeeds" because it exits 0 and it exits 0 because whois itself did what it was supposed to; it returned what the server sent.

Associated revisions

Revision 68b5163d (diff)
Added by kytv about 4 years ago

Retry running whois when "LIMIT EXCEEDED" is in its output

Will-fix: #10523

Revision ed01924c
Added by bertagaz about 4 years ago

Merge branch 'test/10523-whois-test-is-fragile' into stable

Fix-committed: #10523

History

#1 Updated by kytv about 4 years ago

  • Parent task set to #10288

#2 Updated by kytv about 4 years ago

  • Description updated (diff)

#3 Updated by kytv about 4 years ago

  • Assignee changed from kytv to anonym
  • % Done changed from 10 to 60
  • QA Check set to Ready for QA
  • Feature Branch set to kytv/test/10523-whois-test-is-fragile

Now:

calling as amnesia: whois 'torproject.org'
call returned: [0, "WHOIS LIMIT EXCEEDED - SEE WWW.PIR.ORG/WHOIS FOR DETAILS\n", ""]
Tor operation failed (Try 1 of 50) with:
RuntimeError: Looking up whois info for torproject.org failed with:
WHOIS LIMIT EXCEEDED - SEE WWW.PIR.ORG/WHOIS FOR DETAILS
Forcing new Tor circuit...
calling as root: . /usr/local/lib/tails-shell-library/tor.sh && tor_control_send "signal NEWNYM" 
call returned: [0, "250 OK\n250 OK\n250 closing connection\n", ""]
calling as amnesia: whois 'torproject.org'
call returned: [0, "Domain Name:TORPROJECT.ORG\nDomain ID: D130971538-LROR\nCreation Date: 2006-10-17T22:02:50Z\nUpdated Date: 2015-08-20T01:06:22Z\nRegistry Expiry Date:
    When I query the whois directory service for "torproject.org"                  # features/step_definitions/torified_misc.rb:1
    Then the whois command is successful                                           # features/step_definitions/torified_misc.rb:25
    Then the whois standard output contains "The Tor Project"                      # features/step_definitions/torified_misc.rb:34

That's more like it. :)

#4 Updated by bertagaz about 4 years ago

Hmm, I wonder if we shouldn't just consider that if we get the "LIMIT EXCEEDED" message, then the test passes too. After all it successfully communicated with the whois server, and it would probably make the test suite more robust.

#5 Updated by intrigeri about 4 years ago

Hmm, I wonder if we shouldn't just consider that if we get the "LIMIT EXCEEDED" message, then the test passes too. After all it successfully communicated with the whois server, and it would probably make the test suite more robust.

Let's please not introduce such mismatches between the Gherkin phrasing, and what the step definitions actually do. If we ever have to downgrade what we're actually testing, then we'll have to update the wording in Gherkin accordingly.

#6 Updated by kytv about 4 years ago

bertagaz wrote:

Hmm, I wonder if we shouldn't just consider that if we get the "LIMIT EXCEEDED" message, then the test passes too. After all it successfully communicated with the whois server, and it would probably make the test suite more robust.

My 2¢: I can understand this argument if we only want to test whether we're stopping whois from working because of what we put into Tails (apparmor profiles, iptables rules, etc.). If we want to test whether whois is actually useful for users, we couldn't do this.

#7 Updated by bertagaz about 4 years ago

  • Assignee changed from anonym to bertagaz

Taking over, want to merge this.

#8 Updated by bertagaz about 4 years ago

  • Status changed from In Progress to 11
  • % Done changed from 60 to 100

#9 Updated by bertagaz about 4 years ago

  • QA Check changed from Ready for QA to Pass

Ran it 200 times, only 6 of them failed, but because of other issues (#10495). Merged then, congrats and sorry for the delay!

#10 Updated by intrigeri about 4 years ago

  • Assignee deleted (bertagaz)

#11 Updated by anonym about 4 years ago

  • Status changed from 11 to Resolved

Also available in: Atom PDF