Project

General

Profile

Bug #10774

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

MAC address spoofing failure notifications are not always displayed

Added by intrigeri almost 4 years ago. Updated 6 months ago.

Status:
Confirmed
Priority:
Normal
Assignee:
-
Category:
Spoof MAC
Target version:
-
Start date:
12/18/2015
Due date:
% Done:

0%

Feature Branch:
test/10774-disable-mac-spoofing-notification-tests
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

It might be that the problem does not exist on Jessie, so I would recommend checking how it looks like on the branch that unmarks this scenario as fragile in a month or two.


Related issues

Related to Tails - Bug #11593: Network disabling does not always work Resolved 07/22/2016

Associated revisions

Revision 8a7877a8 (diff)
Added by intrigeri almost 4 years ago

Test suite: mark the "MAC address spoofing fails and macchanger returns false" scenario as fragile.

refs: #10774

Revision dd480a58 (diff)
Added by intrigeri almost 4 years ago

Revert "Test suite: mark the "MAC address spoofing fails and macchanger returns false" scenario as fragile."

This reverts commit 8a7877a8981acbc26d50846b1a67026d2a3958e8.

refs: #10774

Revision d4a1ce7f (diff)
Added by intrigeri almost 4 years ago

Test suite: mark "MAC address spoofing fails and macchanger returns true" scenario as fragile.

It broke too: https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/78/

refs: #10774

Revision 9b174f77 (diff)
Added by intrigeri over 3 years ago

Test suite: mark "Scenario: MAC address spoofing fails and the module is not removed" as fragile.

refs: #10774

Revision 26671c6e (diff)
Added by intrigeri 9 months ago

Test suite: when MAC spoofing fails, test safety-critical properties even if the notification is not displayed.

As said on #10774, it's not clear to me whether these scenarios'
frequent failure is "merely" (sic) a test suite robustness issue,
or an actual bug in Tails.

But anyway, MAC spoofing fails, what matters above all is that the network
interfaces are disabled and that the real MAC address is not leaked, so let's
test this first: this way, even if there's a problem wrt. displaying the
notification or wrt. the test suite catching it, by running these scenarios we
will at least ensure the user is safe — possibly with poor UX, but safe.

refs: #10774

Revision 87663631 (diff)
Added by intrigeri 7 months ago

Test suite: disable tests about notifications in case of MAC spoofing failure (refs: #10774)

These notifications are not always displayed, which seems to be a real bug
and not a test suite robustness issue. This bug has been here for more
than three years. In the last 5 full test suite runs on the stable branch,
these steps failed 5/3×5 = 40% of the time. This does not give us any
useful information, other than confirming the bug, so let's drop these steps,
which incidentally allows us to remove the @fragile tags on the corresponding
scenarios ⇒ greater chances we detect issues in the "fail closed" property,
which is great.

History

#1 Updated by intrigeri almost 4 years ago

  • Feature Branch set to test/10774-fragile-MAC_address_spoofing_fails_and_macchanger_returns_false

#2 Updated by intrigeri almost 4 years ago

I suspect this is "just" a failure of our notification handling (either in Tails itself to show them, or in the test suite to find them on screen). We'll see thanks to https://jenkins.tails.boum.org/view/Raw/job/test_Tails_ISO_test-10774-fragile-mac-address-spoofing-fails-and-macchanger-returns-false/ if Tails/Jessie has the same problem.

#3 Updated by intrigeri over 3 years ago

  • Subject changed from "MAC address spoofing fails and macchanger returns false" scenario is fragile to MAC address spoofing tests are fragile

"Scenario: MAC address spoofing fails and the module is not removed" fails very often as well, I'm going to flag it as fragile.

#4 Updated by anonym over 3 years ago

  • Priority changed from Normal to Elevated

#5 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.3 to Tails_2.4

#6 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.4 to Tails_2.5

#7 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.5 to Tails_2.7

#9 Updated by intrigeri about 3 years ago

  • Feature Branch changed from test/10774-fragile-MAC_address_spoofing_fails_and_macchanger_returns_false to wip/test/10774-fragile-MAC_address_spoofing_fails_and_macchanger_returns_false

#10 Updated by intrigeri about 3 years ago

  • Assignee deleted (anonym)
  • Target version deleted (Tails_2.7)

#11 Updated by intrigeri about 3 years ago

  • Related to Bug #11593: Network disabling does not always work added

#12 Updated by intrigeri about 1 year ago

  • Subject changed from MAC address spoofing tests are fragile to MAC address spoofing failure notifications are not always displayed
  • Category changed from Test suite to Spoof MAC

I've seen a few cases when these tests do fail and looking at the video, indeed the notification was never displayed, so I'm not convinced this is a test suite robustness issue, and I'm starting to think this is a real bug.

#13 Updated by intrigeri 9 months ago

  • Status changed from Confirmed to In Progress

#14 Updated by intrigeri 9 months ago

  • Status changed from In Progress to Confirmed

#15 Updated by anonym 9 months ago

  • Assignee set to anonym
  • Target version set to Tails_3.12

intrigeri wrote:

I've seen a few cases when these tests do fail and looking at the video, indeed the notification was never displayed, so I'm not convinced this is a test suite robustness issue, and I'm starting to think this is a real bug.

Confirmed! I looked at a run in detail, and while panic mode works the notification refuses to show. Explanation:

Jan 20 12:49:40 amnesia spoof-mac[4372]: Failed to unload module virtio_net of NIC eth0.
Jan 20 12:49:44 amnesia spoof-mac[5386]: Networking disabled
Jan 20 12:49:44 amnesia systemd-udevd[3983]: Process '/usr/local/lib/tails-spoof-mac eth0' failed with exit code 1.

I tried running show_notification() in a running session (as root) and that worked fine; it is very unclear to me how and where this error appears.

Hm, but I just realized that for me only the "MAC address spoofing fails and the module is not removed" scenario fails like this, but both "MAC address spoofing fails and macchanger returns true" and the "... false" scenarios manage to see the notification. The only difference is that in the failing case we run stop_and_disable_NM, but from the journal it seems that it ran successfully.

This is all I can afford for now.

#16 Updated by intrigeri 9 months ago

  • Priority changed from Elevated to Normal

#17 Updated by anonym 9 months ago

  • Target version changed from Tails_3.12 to Tails_3.13

#18 Updated by intrigeri 8 months ago

  • Target version deleted (Tails_3.13)

This is all I can afford for now.

Dropping target version, then.

#19 Updated by intrigeri 7 months ago

  • Status changed from Confirmed to In Progress

#20 Updated by intrigeri 7 months ago

  • Assignee changed from anonym to CyrilBrulebois
  • Target version set to Tails_3.14
  • QA Check set to Ready for QA
  • Feature Branch changed from wip/test/10774-fragile-MAC_address_spoofing_fails_and_macchanger_returns_false to test/10774-disable-mac-spoofing-notification-tests

@CyrilBrulebois, I've pushed a branch that removes the test suite steps that do nothing but annoy us, waste our time, and confirm we have a bug here. Please review and merge :) Once merged, set this back to Status = Confirmed, as the bug this is about is still here.

#21 Updated by anonym 7 months ago

  • Assignee changed from CyrilBrulebois to anonym

#22 Updated by anonym 7 months ago

  • Status changed from In Progress to Confirmed
  • Assignee deleted (anonym)
  • QA Check deleted (Ready for QA)

#23 Updated by intrigeri 6 months ago

  • Target version deleted (Tails_3.14)

Also available in: Atom PDF