Project

General

Profile

Feature #14814

Feature #12615: Consider basing Tails on quarterly snapshots of Debian testing

Analyze test suite results on Buster: 2017-10 sprint

Added by intrigeri about 2 years ago. Updated almost 2 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
Test suite
Target version:
Start date:
10/09/2017
Due date:
% Done:

20%

Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

Output: test suite updates + tickets to fix actual bugs in Tails


Related issues

Blocks Tails - Feature #13240: Core work 2017Q4: Test suite maintenance Resolved 06/29/2017

History

#1 Updated by intrigeri about 2 years ago

I've analyzed anonym's run-2017-10-08_14:12:48_+0200-git_test-buster-with-more-sikuli-fuzziness_28e814b-SA5xr4:

  • "Install packages using apt": typed only 3 chars in the Greeter's admin password field and 4 chars in the confirmation field passed, let's file a test suite robustness ticket if we see it again
  • "Install packages using Synaptic" fails at 'I install "cowsay" using Synaptic', not clear why without the video ⇒ retry by hand or with --capture passed, let's file a test suite robustness ticket if we see it again
  • "Encryption and decryption using OpenPGP Applet", "Signing and verification using OpenPGP Applet", "Encryption/signing and decryption/verification using OpenPGP Applet", "Symmetric encryption and decryption using OpenPGP Applet": FindFailed: PinEntryPrompt.png, unclear why without a video; on some of the pictures it seems that scenario has worked just fine, so I dunno why the test suite is waiting for a Pinentry prompt ⇒ retried and they fail with FindFailed: GeditNewTab.png or FindFailed: GeditWindow.png, will investigate further fixed those + some more but a few scenarios need more work, let's see after next run
  • USB tests: skipped ⇒ retry after merging the fix for #14809 and #14816
  • "MAC address spoofing is disabled", "MAC address spoofing is successful": fails at "I hotplug a network device and wait for it to be initialized", that I've seen on my sid as well ⇒ retry on Jenkins (that is not affected) works on anonym's system with the fix from #14819
  • "Thunderbird can download the inbox with POP3": I see "Username or password invalid" after clicking the "Done" button, hard to tell what's going on without a video ⇒ retry by hand or with --capture passed, let's file a ticket if we ever see it again.
  • "I can view a file stored in "~/Tor Browser" but not in ~/.gnupg": fails at the step immediately after 'When I open the address "file:///home/amnesia/.gnupg/synaptic.html" in the Tor Browser', i.e. Sikuli finds TorBrowserSynapticManual.png ("Found fuzzy candidate picture for TorBrowserSynapticManual.png with similarity 0.4") while it should not. So if/when we start running with --fuzzy-image-matching it'll break. Looking at the actual picture, that's because we have another tab still open from the previous similar test, with a part of the same text in the tab title, it makes sense ⇒ add more context to TorBrowserSynapticManual.png so the actual web page we're looking for is not confused with similar stuff, or increase the required similarity level for this picture. done
  • "Watching MP4 videos stored on the persistent volume should work as expected given our AppArmor confinement": fail with "Call to virStorageVolCreateXML failed: cannot fork child process: Cannot allocate memory (Guestfs::Error)" ⇒ retry with a host system that works better once #14816 is fixed
  • "VirtualBox guest modules are available": fail with "Call to virStorageVolCreateXML failed: cannot fork child process: Cannot allocate memory (Guestfs::Error)" ⇒ retry with a host system that works better passed
  • "The Unsafe Browser cannot be configured to use Tor and other local proxies" ⇒ update UnsafeBrowserNetworkTabAlreadySelected.png done

#2 Updated by intrigeri about 2 years ago

  • Assignee changed from intrigeri to anonym
  • % Done changed from 10 to 20

Analyzing another, more recent full test suite run:

  • "Watching MP4 videos stored on the persistent volume should work as expected given our AppArmor confinement" now passes, same for most USB tests, so the previous comment can now be ignored. Progress! :)
  • the two "Upgrading an old Tails USB installation from an ISO image" scenarios fail with "FindFailed: can not find GnomeActivitiesOverview.png"; the screenshot shows a "Low Disk Space on tmp.$random" notification, "0 bytes disk space remaining" => investigate if that's a test suite problem or an actual bug)
  • "Electrum Bitcoin client ǂ Using a persistent Electrum configuration" fails with "FindFailed: can not find ElectrumNextButton.png" while I do see a "Next" button on the screenshot => rerun with fuzzy matching in the hope it gives us a good candidate for a picture update
  • 4 of the "Encryption and verification using GnuPG" scenarios fail with "FindFailed: can not find PinEntryPrompt.png":
    • in one case the operation that's supposed to ask a passphrase was not even attempted, so that's clearly a test suite bug
    • in the 3 other cases the operation that's being tested worked just fine, so that's clearly a test suite bug too: looks like the rescue FindFailed part of maybe_deal_with_pinentry does not work anymore? I've had a quick look at 03f0967beec8acea070c35d8908a8f46bc83084b and it seems that some code should be guarded by if ! $config["SIKULI_FUZZY_IMAGE_MATCHING"], because I see "Failed to find fuzzy candidate picture for PinEntryPrompt.png" in the logs => test suite bug, I'll reopen #14817 accordingly.
  • 2 Pidgin scenarios fail with "FindFailed: can not find GtkFileChooserDesktopButton.png", and in the debug log I see "Found fuzzy candidate picture for GtkFileChooserDesktopButton.png with similarity 0.7"; but our Jenkins config does not save these candidates as artifacts, so I can't get them => will add a note on #14817 about it, but anonym's local non-Jenkins run should have a good candidate => please import it to Git
  • "Browsing the web using the Tor Browser ǂ I can view a file stored in "~/Tor Browser" but not in ~/.gnupg" fails with "FindFailed: can not find TorBrowserSynapticManual.png"; possibly I borked things when I updated this picture (FTR I see "Found fuzzy candidate picture for TorBrowserSynapticManual.png with similarity 0.8" => anonym, may you please have a look?
  • "Browsing the web using the Tor Browser ǂ The persistent Tor Browser directory is usable" fails with "FindFailed: can not find GtkTorBrowserPersistentBookmark.png" => updating the picture should be enough
  • "Browsing the web using the Tor Browser ǂ Persistent browser bookmarks" fails with "FindFailed: can not find TorBrowserBookmarkPrompt.png" and indeed the prompt is never shown AFAICT => investigate
  • "Browsing the web using the Unsafe Browser ǂ The Unsafe Browser cannot be configured to use Tor and other local proxies" fails with "FindFailed: can not find UnsafeBrowserProxyRefused.png" but the screenshot looks good => update picture

All in all, that's amazing: not many tests are broken, and not many potentially buggy ones are hidden behind early unrelated failure. I'll let anonym take it from here with his "Test suite maintenance" hat on. This could wait until our next sprint but I'd rather focus on work we're supposed to split fifty-fifty during these sprints, so it would be great if these were addressed before that sprint :)

#3 Updated by intrigeri about 2 years ago

  • Blocks Feature #13240: Core work 2017Q4: Test suite maintenance added

#4 Updated by intrigeri about 2 years ago

  • Target version changed from Tails_4.0 to Tails_3.3

#5 Updated by anonym about 2 years ago

Partial response while I have my finding from #14817 fresh in mind:

intrigeri wrote:

  • 4 of the "Encryption and verification using GnuPG" scenarios fail with "FindFailed: can not find PinEntryPrompt.png":
    • in one case the operation that's supposed to ask a passphrase was not even attempted, so that's clearly a test suite bug

It's not clear what this means to me, please clarify! Also note the "maybe" in maybe_deal_with_pinentry, which is explained by the comment if we fail to find PinEntryPrompt.png:

    # The passphrase was cached or we wasn't prompted at all (e.g. when
    # only encrypting to a public key)

Perhaps that explains why no pin entry prompt appeared, but the operation was in fact performed?

  • in the 3 other cases the operation that's being tested worked just fine, so that's clearly a test suite bug too: looks like the rescue FindFailed part of maybe_deal_with_pinentry does not work anymore? I've had a quick look at 03f0967beec8acea070c35d8908a8f46bc83084b and it seems that some code should be guarded by if ! $config["SIKULI_FUZZY_IMAGE_MATCHING"], because I see "Failed to find fuzzy candidate picture for PinEntryPrompt.png" in the logs => test suite bug, I'll reopen #14817 accordingly.

Should be fixed, #14817 is back on your plate.

#6 Updated by intrigeri about 2 years ago

intrigeri wrote:

  • 4 of the "Encryption and verification using GnuPG" scenarios fail with "FindFailed: can not find PinEntryPrompt.png":
    • in one case the operation that's supposed to ask a passphrase was not even attempted, so that's clearly a test suite bug

It's not clear what this means to me, please clarify!

The test suite keeps did not even try to click on stuff to perform the operation being tested.

Also note the "maybe" in maybe_deal_with_pinentry, which is explained by the comment if we fail to find PinEntryPrompt.png:

>     # The passphrase was cached or we wasn't prompted at all (e.g. when
>     # only encrypting to a public key)
> 

Perhaps that explains why no pin entry prompt appeared, but the operation was in fact performed?

Sure. But then we should not be looking endlessly for a pinentry prompt.

#7 Updated by intrigeri about 2 years ago

intrigeri wrote:

  • "Browsing the web using the Tor Browser ǂ I can view a file stored in "~/Tor Browser" but not in ~/.gnupg" fails with "FindFailed: can not find TorBrowserSynapticManual.png"; possibly I borked things when I updated this picture (FTR I see "Found fuzzy candidate picture for TorBrowserSynapticManual.png with similarity 0.8" => anonym, may you please have a look?

This might be an instance of #14929 (that happens on the devel branch).

#8 Updated by intrigeri about 2 years ago

  • Blocked by Bug #14929: 'I can view a file stored in "~/Tor Browser" but not in ~/.gnupg' scenario is broken added

#9 Updated by anonym about 2 years ago

  • Target version changed from Tails_3.3 to Tails_3.5

#10 Updated by intrigeri about 2 years ago

  • Target version changed from Tails_3.5 to Tails_3.7

#11 Updated by intrigeri almost 2 years ago

  • Blocked by deleted (Bug #14929: 'I can view a file stored in "~/Tor Browser" but not in ~/.gnupg' scenario is broken)

#12 Updated by intrigeri almost 2 years ago

  • Status changed from In Progress to Rejected

Let's come back to this next time we work on Tails/Buster.

Also available in: Atom PDF