Project

General

Profile

Feature #5836

Feature #6298: Write more automated tests

Test suite: verify time changes

Added by Tails over 6 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
test/5836-verify-time-changes
Type of work:
Code
Blueprint:
Starter:
No
Affected tool:

Description

After setting the date to something wrong, one should verify that the date change actually took place, else it makes little sense verifying later that our stuff indeed has "fixed" the date: if we don't know for sure that it was broken, we're not really verifying anything. This should be implemented at least in features/step_definitions/time_syncing.rb.

Associated revisions

Revision ead90b15 (diff)
Added by Tails developers about 5 years ago

Verify that the time set by `date` actually is set (Will-fix: #5836).

Revision 2ac540c4
Added by Tails developers almost 5 years ago

Merge remote-tracking branch 'origin/test/5836-verify-time-changes' into testing

Fix-committed: #5836

History

#1 Updated by intrigeri over 6 years ago

  • Category set to Test suite
  • Starter set to No

#2 Updated by BitingBird over 5 years ago

  • Subject changed from test suite: verify time changes to Test suite: verify time changes

#3 Updated by intrigeri over 5 years ago

  • Assignee set to anonym
  • Target version set to Tails_1.2.2
  • Parent task set to #6298

#5 Updated by anonym about 5 years ago

  • Target version changed from Tails_1.2.2 to Tails_1.2.3

#6 Updated by anonym about 5 years ago

  • Target version changed from Tails_1.2.3 to Tails_1.3

#7 Updated by anonym about 5 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 20
  • QA Check set to Dev Needed
  • Feature Branch set to test/5836-verify-time-changes

Just pushing some initial work. I guess we don't only want to verify by checking the exit status of date (done), but also run date again (after waiting a few seconds?) and see that we get an expected value.

#8 Updated by intrigeri about 5 years ago

Just pushing some initial work. I guess we don't only want to verify by checking
the exit status of date (done), but also run date again (after waiting a few
seconds?) and see that we get an expected value.

It would indeed feel safer :)

#9 Updated by Tails about 5 years ago

Applied in changeset commit:0f5dcdc403ed4514f18dafdd858ad513635c13ab.

#10 Updated by anonym about 5 years ago

  • Assignee changed from anonym to kytv
  • % Done changed from 20 to 50
  • QA Check changed from Dev Needed to Ready for QA

#11 Updated by kytv almost 5 years ago

  • Assignee changed from kytv to anonym
  • % Done changed from 50 to 40
  • QA Check changed from Ready for QA to Dev Needed

Unfortunately this is consistently failing for me.

  Scenario: Clock is one day in the past                                 # features/time_syncing.feature:17
    When I bump the system time with "-1 day"                            # features/step_definitions/time_syncing.rb:14
      The guest's time was suppsed to be bumped to '2015-02-07 00:00:00 +0000' but is '2015-02-07 12:17:47 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:24:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:18:in `When I bump the system time with "-1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:104
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:291
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:29
Scenario failed at time 00:00:41

...

  Scenario: Clock is one day in the future                               # features/time_syncing.feature:29
    When I bump the system time with "+1 day"                            # features/step_definitions/time_syncing.rb:14
      The guest's time was suppsed to be bumped to '2015-02-09 00:00:00 +0000' but is '2015-02-09 12:17:47 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:24:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:30:in `When I bump the system time with "+1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:104
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:291
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:29
Scenario failed at time 00:03:17

...

Failing Scenarios:
cucumber features/time_syncing.feature:17 # Scenario: Clock is one day in the past
cucumber features/time_syncing.feature:29 # Scenario: Clock is one day in the future

#12 Updated by kytv almost 5 years ago

With debug info:

calling as root: date
call returned: [0, "Wed Feb 18 14:23:27 UTC 2015\n", ""]
calling as root: date -s 'now -1 day'
call returned: [0, "Tue Feb 17 14:23:27 UTC 2015\n", ""]
calling as root: date
call returned: [0, "Tue Feb 17 14:23:28 UTC 2015\n", ""]
    When I bump the system time with "-1 day"                            # features/step_definitions/time_syncing.rb:14
      The guest's time was supposed to be bumped to '2015-02-17 14:23:27 +0000' but is '2015-02-17 14:23:28 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:24:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:18:in `When I bump the system time with "-1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:110
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:297
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:29
Scenario failed at time 00:03:28
Took screenshot "/tmp/TailsToaster/time_syncing-2015-02-18T14:24:19+00:00.png" 
calling as root: date
call returned: [0, "Wed Feb 18 14:23:27 UTC 2015\n", ""]
calling as root: date -s 'now +1 day'
call returned: [0, "Thu Feb 19 14:23:28 UTC 2015\n", ""]
calling as root: date
call returned: [0, "Thu Feb 19 14:23:28 UTC 2015\n", ""]
    When I bump the system time with "+1 day"                            # features/step_definitions/time_syncing.rb:14
      The guest's time was supposed to be bumped to '2015-02-19 14:23:27 +0000' but is '2015-02-19 14:23:28 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:24:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:30:in `When I bump the system time with "+1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:110
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:297
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:29
Scenario failed at time 00:05:57
Took screenshot "/tmp/TailsToaster/time_syncing-2015-02-18T14:26:48+00:00.png" 

#13 Updated by anonym almost 5 years ago

  • Assignee changed from anonym to kytv
  • QA Check changed from Dev Needed to Ready for QA

kytv wrote in #5836#note-11:

The guest's time was suppsed to be bumped to '2015-02-07 00:00:00 +0000' but is '2015-02-07 12:17:47 +0000' (MiniTest::Assertion)

I find it suspicious that the expected time is exactly 00:00:00. Especially since they're not in you next run. I have no good explanation for this.

kytv wrote in #5836#note-12:

The guest's time was supposed to be bumped to '2015-02-17 14:23:27 +0000' but is '2015-02-17 14:23:28 +0000' (MiniTest::Assertion)

I guess my 1 second upper bound wasn't enough. Let's bump it to 5 seconds.

#14 Updated by kytv almost 5 years ago

  • Assignee changed from kytv to anonym
  • QA Check changed from Ready for QA to Dev Needed

anonym wrote:

kytv wrote in #5836#note-11:

The guest's time was suppsed to be bumped to '2015-02-07 00:00:00 +0000' but is '2015-02-07 12:17:47 +0000' (MiniTest::Assertion)

I find it suspicious that the expected time is exactly 00:00:00. Especially since they're not in you next run. I have no good explanation for this.

Same. That was incredibly peculiar.

kytv wrote in #5836#note-12:

The guest's time was supposed to be bumped to '2015-02-17 14:23:27 +0000' but is '2015-02-17 14:23:28 +0000' (MiniTest::Assertion)

I guess my 1 second upper bound wasn't enough. Let's bump it to 5 seconds.

I was able to complete the test successfully (woohoo) but it's occasionally failing for me during subsequent runs:

calling as root: date
call returned: [0, "Wed Feb 18 16:19:16 UTC 2015\n", ""]
calling as root: date -s 'now -1 day'
call returned: [0, "Tue Feb 17 16:19:17 UTC 2015\n", ""]
calling as root: date
call returned: [0, "Tue Feb 17 16:19:17 UTC 2015\n", ""]
    When I bump the system time with "-1 day"                            # features/step_definitions/time_syncing.rb:21
      The guest's time was supposed to be bumped to '2015-02-17 16:19:16 +0000' but is '2015-02-17 16:19:17 +0000' (MiniTest::Assertion)
      ./features/step_definitions/time_syncing.rb:30:in `/^I bump the system time with "([^"]+)"$/'
      features/time_syncing.feature:18:in `When I bump the system time with "-1 day"'
    And the network is plugged                                           # features/step_definitions/common_steps.rb:110
    And Tor is ready                                                     # features/step_definitions/common_steps.rb:297
    Then Tails clock is less than 5 minutes incorrect                    # features/step_definitions/time_syncing.rb:35
Scenario failed at time 00:04:17

#15 Updated by kytv almost 5 years ago

  • % Done changed from 40 to 50
  • QA Check changed from Dev Needed to Pass

With the latest changes (rev ddadef9) this branch is now working well for me.

#16 Updated by intrigeri almost 5 years ago

  • Assignee changed from anonym to intrigeri

#17 Updated by Tails almost 5 years ago

  • Status changed from In Progress to 11
  • % Done changed from 50 to 100

Applied in changeset commit:ebc19f96f7d64a15467b0852529df94d2b109f6d.

#18 Updated by intrigeri almost 5 years ago

  • Assignee deleted (intrigeri)

#19 Updated by BitingBird almost 5 years ago

  • Status changed from 11 to Resolved

Also available in: Atom PDF