Project

General

Profile

Bug #17029

Test suite: findfailed_hook makes Sikuli use fuzzy images even when fuzzy image matching is disabled

Added by intrigeri 16 days ago. Updated 10 days ago.

Status:
Resolved
Priority:
Elevated
Assignee:
-
Category:
Test suite
Target version:
Start date:
Due date:
% Done:

100%

Feature Branch:
https://salsa.debian.org/tails-team/tails/merge_requests/37#note_108671
Type of work:
Code
Blueprint:
Starter:
Affected tool:
OpenPGP Applet

Description

For example, Sikuli clicks on "Untitled document 1" tab while looking for OpenPGP Applet's GpgAppletDecryptVerify.png.


Related issues

Related to Tails - Bug #17030: Missing "Tor Browser" shortcut in the Places menu Resolved
Duplicated by Tails - Bug #17006: Test suite: clicking on the OpenPGP applet icon is fragile Duplicate
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision f7654cc5 (diff)
Added by intrigeri 16 days ago

Test suite: don't attempt to find fuzzy matches with Sikuli unless fuzzy image matching is enabled (refs: #17029).

The theory was that:

- By default, we would try to find fuzzy matches on find failure,
save them, but not use them. This would save time e.g. when font
rendering changes, without any impact on test suite results.
- When fuzzy image matching is enabled, we do the same, except
we do use the fuzzy matches.

But even with fuzzy image matching disabled, FindFailedHookFailure is invoked
and on the second attempt at finding the picture we're looking for, Sikuli uses
the last minimum similarity level that we temporarily set when looking for fuzzy
matches. That is, in practice, we do essentially the same as we would if fuzzy
image matching was enabled, which was not the deal.

So let's do this trade-off:

- We get the behavior we want by default, that is: we use the minimum
similarity level we've configured (0.9) consistently.
This will benefit every reviewer who has to analyze Jenkins results, and
anyone who works on the test suite: no more weird behavior because the test
suite happily proceeded after clicking on a fuzzy match.
- In exchange, we lose some convenience when updating plenty of images at once:
we can't use Jenkins for that. Instead, we have to run the test suite locally
with fuzzy image matching enabled.
This will annoy few of us, rather little, and pretty rarely.

Revision e8895194
Added by segfault 10 days ago

Merge branch 'test/17029+17027+17028+16931+force-all-tests' into devel

Closes: #17029, #17027, #17028, #16931, #16617, #17030, #10708, #16465
Closes: #13469

History

#1 Updated by intrigeri 16 days ago

This happened 4 times in the last 10 runs on Jenkins (devel) branch ⇒ FT.

#2 Updated by intrigeri 16 days ago

#3 Updated by intrigeri 16 days ago

  • Priority changed from Normal to Elevated

#4 Updated by intrigeri 16 days ago

Actually, Sikuli is too fast: on the video, I see the mouse cursor moving towards the "Untitled document 1" tab before OpenPGP Applet had time to open its menu. Now, I think there's a bug in our code. Fuzzy matching is used, that's why Sikuli clicks on the (vaguely similar) tab instead of waiting any further:

03:19:20.989537354: Sikuli: calling wait("GpgAppletDecryptVerify.png")...
03:19:45.108976627: Found fuzzy candidate picture for GpgAppletDecryptVerify.png with similarity 0.4
03:19:45.109665405: FindFailedHookFailure was raised, re-running the failing Sikuli method.
03:19:45.172519400: Sikuli: calling click(M[158,82 176x22]@S(S(0)[0,0 1024x768]) S:0.42 C:246,93 [62 msec])...
03:19:45.902044690: [log] CLICK on L(246,93)@S(0)[0,0 1024x768]

It's not the first time I notice a fuzzy image being used while I don't expect it to be.
I don't see "Using fuzzy candidate picture for #{picture}" in the debug log, which confirms that SIKULI_FUZZY_IMAGE_MATCHING is not true. But FindFailedHookFailure is invoked and on 2nd try, Sikuli is happy to match the fuzzy tab header (I assume "S:0.42" means similarity = 0.42). It looks exactly as if us retrying in findfailed_hook with lower similarity expectations reset the MinSimilarity setting for the following Sikuli action.

I see the same kind of behavior elsewhere, e.g.

03:26:45.644231746: Sikuli: calling wait("Gtk3PrintButton.png")...
03:27:02.733362618: Found fuzzy candidate picture for Gtk3PrintButton.png with similarity 0.6
03:27:02.733999556: FindFailedHookFailure was raised, re-running the failing Sikuli method.
03:27:02.803860495: Sikuli: calling click(M[569,103 81x36]@S(S(0)[0,0 1024x768]) S:0.69 C:609,121 [70 msec])...
03:27:03.527528786: [log] CLICK on L(609,121)@S(0)[0,0 1024x768]

Without diving into the rjb craziness, what we could do is:

  • Call pattern.similar(0.9) at the end of each [0.80, 0.70, 0.60, 0.50, 0.40].each do |similarity| iteration, in the hope we revert the side effect.
  • Skip [0.80, 0.70, 0.60, 0.50, 0.40].each do |similarity| entirely unless fuzzy image matching is enabled. For sure, this will make some of our work more difficult (one will have to run tests locally and enable fuzzy image matching) but IMO it's worth it, in order to fix weird behavior that breaks the test suite in a way that affects every developer, in many runs.

#5 Updated by intrigeri 16 days ago

  • Subject changed from Test suite clicks on "Untitled document 1" tab while looking for OpenPGP Applet's GpgAppletDecryptVerify.png to Test suite: findfailed_hook makes Sikuli use fuzzy images even when fuzzy image matching is disabled
  • Description updated (diff)

#6 Updated by intrigeri 16 days ago

  • Status changed from Confirmed to In Progress
  • Assignee set to intrigeri
  • Feature Branch set to test/17029+17027

#7 Updated by intrigeri 16 days ago

  • Feature Branch changed from test/17029+17027 to test/17029+17027+force-all-tests

#8 Updated by intrigeri 16 days ago

  • Duplicated by Bug #17006: Test suite: clicking on the OpenPGP applet icon is fragile added

#9 Updated by intrigeri 16 days ago

  • Related to Bug #17030: Missing "Tor Browser" shortcut in the Places menu added

#10 Updated by intrigeri 16 days ago

  • Feature Branch changed from test/17029+17027+force-all-tests to test/17029+17027+17028+16931+force-all-tests

#11 Updated by intrigeri 16 days ago

Note that this test suite bug make us miss an actual, user-impacting, bug: #17030.

#12 Updated by intrigeri 14 days ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)
  • Feature Branch changed from test/17029+17027+17028+16931+force-all-tests to https://salsa.debian.org/tails-team/tails/merge_requests/37#note_108671

#13 Updated by intrigeri 14 days ago

  • Target version set to Tails_4.0

#14 Updated by segfault 10 days ago

  • Status changed from Needs Validation to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF