Bug #10288: Fix newly identified issues to make our test suite more robust and faster
"I can view and print a PDF file" scenarios are fragile
I've seen "Scenario: I can view and print a PDF file stored in /usr/share" and "Scenario: I can view and print a PDF file stored in non-persistent /home/amnesia" fail on Jenkins with a screenshot that looks like it has succeeded, see e.g. the attached file (that comes from https://jenkins.tails.boum.org/job/test_Tails_ISO_experimental/117/).
Scenario: I can view and print a PDF file stored in non-persistent /home/amnesia # features/evince.feature:14 calling as root: echo 'hello?' call returned: [0, "hello?\n", ""] [log] CLICK on (1024,384) calling as root: /sbin/ifconfig eth0 | grep -q 'inet addr' call returned: [1, "", ""] calling as root: date -s '@1451226453' call returned: [0, "Sun Dec 27 14:27:33 UTC 2015\n", ""] Given I have started Tails from DVD without network and logged in # features/step_definitions/snapshots.rb:201 calling as amnesia: cp "/usr/share/cups/data/default-testpage.pdf" "/home/amnesia" call returned: [0, "", ""] And I copy "/usr/share/cups/data/default-testpage.pdf" to "/home/amnesia" as user "amnesia" # features/step_definitions/common_steps.rb:633 calling as root: pidof -x -o '%PPID' gnome-terminal call returned: [1, "", ""] [log] CLICK on (51,16) [log] CLICK on (58,431) [log] CLICK on (264,470) [log] TYPE "evince /home/amnesia/default-testpage.pdf " When I open "/home/amnesia/default-testpage.pdf" with Evince # features/step_definitions/evince.rb:1 Then I see "CupsTestPage.png" after at most 20 seconds # features/step_definitions/common_steps.rb:441 FindFailed: can not find CupsTestPage.png on the screen. Line ?, in File ? (RuntimeError) ./features/step_definitions/common_steps.rb:446:in `rescue in block in <top (required)>' ./features/step_definitions/common_steps.rb:442:in `/^I (do not )?see "([^"]*)" after at most (\d+) seconds$/' features/evince.feature:18:in `Then I see "CupsTestPage.png" after at most 20 seconds' And I can print the current document to "/home/amnesia/output.pdf" # features/step_definitions/evince.rb:5 Scenario failed at time 00:19:35 Screenshot: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10774-fragile-mac-address-spoofing-fails-and-macchanger-returns-false/11/artifact/build-artifacts/00:19:35_I_can_view_and_print_a_PDF_file_stored_in_non-persistent__home_amnesia.png Video: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10774-fragile-mac-address-spoofing-fails-and-macchanger-returns-false/11/artifact/build-artifacts/00:19:35_I_can_view_and_print_a_PDF_file_stored_in_non-persistent__home_amnesia.mkv
Note that this is a different problem from #10775.
Revert "Test suite: mark "Scenario: MAC address spoofing fails and the module is not removed" as fragile."
This reverts commit 9b174f7732d587430df026de39e193ed8ac93248.
#7 Updated by anonym over 2 years ago
- Status changed from Confirmed to In Progress
- % Done changed from 0 to 30
This commit (currently pushed to the
testing branch) might fix it:
commit a8a8992d76cc950ae79cb2b9117ad30d8f7f4b80 Author: anonym <email@example.com> Date: Thu May 26 18:09:45 2016 +0200 Bump the CupsTestPage.png test suite image. Compared to the old image the distance between some of the elements (e.g. the Debian logo and color gradient) is different so let's pick the title only which shouldn't have such issues.
#10 Updated by intrigeri over 2 years ago
- File 02_04_39_I_can_view_and_print_a_PDF_file_stored_in_persistent__home_amnesia_Persistent.mkv added
- File 02_04_39_I_can_view_and_print_a_PDF_file_stored_in_persistent__home_amnesia_Persistent.png View added
This commit (currently pushed to the
testingbranch) might fix it:
Sorry, it doesn't -- see attachments. Corresponding debug log:
When I open "/home/amnesia/Persistent/default-testpage.pdf" with Evince # features/step_definitions/evince.rb:1 [log] TYPE "evince /home/amnesia/Persistent/default-testpage.pdf " Then I see "CupsTestPage.png" after at most 40 seconds # features/step_definitions/common_steps.rb:459 FindFailed: can not find CupsTestPage.png on the screen. Line ?, in File ? (RuntimeError) ./features/step_definitions/common_steps.rb:464:in `rescue in block in <top (required)>' ./features/step_definitions/common_steps.rb:460:in `/^I (do not )?see "([^"]*)" after at most (\d+) seconds$/' features/evince.feature:52:in `Then I see "CupsTestPage.png" after at most 40 seconds' And I can print the current document to "/home/amnesia/Persistent/output.pdf" # features/step_definitions/evince.rb:5