Project

General

Profile

Feature #10721

Improve automated GUI testing robustness using "GUI aware" technologies

Added by anonym over 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
Start date:
12/07/2015
Due date:
% Done:

100%

QA Check:
Pass
Feature Branch:
test/10721-a11y-technologies
Type of work:
Code

Description

With "GUI aware" I mean something like assistive technologies, which for instance the Linux Desktop Testing Project uses. That project is a nice example, because it is in Debian and has Ruby bindings. One issue will be that our target system will run in a VM, and this tool may be designed to run on the same host as the applications to be tested. However, another similar tool, the Robot Framework, explicitly supports the targets to run on a different system (but its not in Debian and I am unsure about Ruby bindings).

The goal would be to get something that is more robust than our current Sikuli-based (hence image driven) approach. Likely OCR (even if we find something better than Sikuli's poor implementation) would not be an improvement.

Associated revisions

Revision 1371722d (diff)
Added by intrigeri about 3 years ago

Install dogtail, to use accessibility technologies in our automated test suite.

We have to stop manually removing python-apt, since python-dogtail Depends:
python-apt.

refs: #10721

Revision 78912d59 (diff)
Added by anonym about 3 years ago

Add wrapper around remote dogtail usage.

This provides a simple interface for generating python code, sending
it to the guest, and executing it.

Refs: #10721

Revision 895c6747 (diff)
Added by anonym about 3 years ago

Use dogtail instead of sikuli for GNOME Applications menu interaction.

Epic simplification++

Refs: #10721

Revision 5e34dc8c
Added by anonym about 3 years ago

Merge branch 'test/10721-a11y-technologies' into devel

Fix-committed: #10721

History

#1 Updated by anonym over 3 years ago

  • Subject changed from Improve automated GUI testing robustness "GUI aware" technologies to Improve automated GUI testing robustness using "GUI aware" technologies

#2 Updated by anonym over 3 years ago

  • Blueprint set to https://tails.boum.org/blueprint/automated_builds_and_tests/testing/

#3 Updated by intrigeri over 3 years ago

  • Feature Branch set to test/10721-a11y-technologies

#4 Updated by anonym about 3 years ago

  • % Done changed from 0 to 30
  • Type of work changed from Research to Discuss

I've force pushed a re-written version of the test/10721-a11y-technologies branch. Currently I've only migrated all application starting (via the Applications menu) and encryption.feature to Dogtail, and the results according to

git diff --stat origin/devel... -- features/*.feature features/step_definitions/ features/images/

is pretty promising: 37 images killed, and we netted 31 lines less than before. Woo! However, looking at the step definitions of encryption.feature, it actually grew 23 lines (net). Most of the loss is explained by killing all the crazy code we had for working around Sikuli not handling the Applications menu navigation very well, and who knows, maybe some of it has to be revived for the Dogtail version -- we'll see.

Any way, I expect that when Sikuli works well, it will require less code than Dogtail ("a picture is worth a kLOC?" :D), but the migration will hopefully be worth it given the eliminated maintenance of these images, and increased robustness, despite this.

BTW, I will have to upstream one new feature to dogtail (see config/chroot_local-patches/python-dogtail_searchShowingOnly.diff) but IMHO it is a brilliant feature that I expect most users will want to enable by default, so I don't think upstreaming it should be a problem.

So I think this concludes the research part, with the conclusion being that Dogtail seems like a great addition to the toolbox we have in our test suite, but not a complete replacement of Sikuli still. I'm unsure what the next step is, given that converting the rest of the test suite is a pretty significant piece of work. I suggest we briefly discuss this during our next CI meeting.

#5 Updated by anonym about 3 years ago

anonym wrote:

BTW, I will have to upstream one new feature to dogtail (see config/chroot_local-patches/python-dogtail_searchShowingOnly.diff) but IMHO it is a brilliant feature that I expect most users will want to enable by default, so I don't think upstreaming it should be a problem.

Upstreaming in progress: https://mail.gnome.org/archives/dogtail-list/2016-April/msg00000.html

#6 Updated by anonym about 3 years ago

anonym wrote:

anonym wrote:

BTW, I will have to upstream one new feature to dogtail (see config/chroot_local-patches/python-dogtail_searchShowingOnly.diff) but IMHO it is a brilliant feature that I expect most users will want to enable by default, so I don't think upstreaming it should be a problem.

Upstreaming in progress: https://mail.gnome.org/archives/dogtail-list/2016-April/msg00000.html

My patch was merged! \o/

In other (good) news, this branch passed without error on jenkins: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/10/

#7 Updated by anonym about 3 years ago

  • Target version set to Tails_2.4
  • % Done changed from 30 to 40
  • Type of work changed from Discuss to Contributors documentation

Here are the next steps, as discussed during April's CI meeting:

1. Update design docs, write minimal guidelines/tips-and-tricks (e.g. use sniff/accerciser/ipython to explore Dogtail's PoV) from test writers, and explain/recommend when to use which.
2. Merge this quickly, so new stuff can change paradigm towards Dogtail when more suitable.
3. Only migrate stuff to Dogtail as we encounter issues, or part of "refactoring"? I.e. no big migration project. However, when working on the Debian migration (currently feature/stretch) we should try to use Dogtail first when an image needs updating, and we do so in devel so we start benefiting from it on current releases as well.

#8 Updated by anonym about 3 years ago

  • Status changed from Confirmed to In Progress
  • Assignee changed from anonym to intrigeri
  • % Done changed from 40 to 50
  • QA Check set to Ready for QA
  • Type of work changed from Contributors documentation to Code

anonym wrote:

Here are the next steps, as discussed during April's CI meeting:

1. Update design docs, write minimal guidelines/tips-and-tricks (e.g. use sniff/accerciser/ipython to explore Dogtail's PoV) from test writers, and explain/recommend when to use which.

Done in 4491b80.

2. Merge this quickly, so new stuff can change paradigm towards Dogtail when more suitable.

Next step! What do you think, intri?

3. Only migrate stuff to Dogtail as we encounter issues, or part of "refactoring"? I.e. no big migration project. However, when working on the Debian migration (currently feature/stretch) we should try to use Dogtail first when an image needs updating, and we do so in devel so we start benefiting from it on current releases as well.

As the Test Automaton Grand General, I'll keep this in mind when commanding my minions.

#9 Updated by intrigeri about 3 years ago

2. Merge this quickly, so new stuff can change paradigm towards Dogtail when more suitable.

Next step! What do you think, intri?

Great job! It's lovely to see this many images go away :)

Code review done, only a few details and minor comments:

  • decrypt_verify_helper and friends: I find the function signature obscure when it's used. How about using named arguments (with whatever tricks Ruby requires to have them)?
  • "This was we can support stuff like" ← I can't parse this sentence.
  • Can't we do python3 instead of python2? (As you can guess, I'm mostly looking ahead to future pain points here.)
  • self.args_to_s might benefit from a comment about its args (I think I got it right by reading the code, but still).
  • Thanks for the comment at the top of def children because oh my, I would have a hard time improving this function if I ever had to.
  • I expected that features/support/helpers/dogtail.rb would be much bigger. I'm glad it's not! :)

Regarding how it fares in practice, I see two recent failures on Jenkins that make me feel a bit uneasy:

I'm now going to do a full test suite run (which might take a while, since my current Internet connection is poor so online tests will be flaky). So for now I'll keep this ticket assigned to me.

#10 Updated by intrigeri about 3 years ago

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Ready for QA to Dev Needed

intrigeri wrote:

I'm now going to do a full test suite run [...]

Done (on an ISO built from the branch). I see a lot of scenarios that rely on /^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/ fail, e.g. all those that start Pidgin, and:

  Scenario: Install packages using Synaptic                                                                        # features/apt.feature:23
    When I start Synaptic                                                                                          # features/step_definitions/apt.rb:52
      Command failed: /usr/bin/python '/tmp/tmp.6yLZBfPfbF'
      error code: 1
      stderr: .
      <false> is not true. (ExecutionFailedInVM)
      ./features/support/helpers/vm_helper.rb:441:in `rescue in execute_successfully'
      ./features/support/helpers/vm_helper.rb:438:in `execute_successfully'
      ./features/support/helpers/dogtail.rb:87:in `run'
      ./features/support/helpers/dogtail.rb:203:in `block (2 levels) in <class:Node>'
      ./features/step_definitions/common_steps.rb:644:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/common_steps.rb:643:in `each'
      ./features/step_definitions/common_steps.rb:643:in `/^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/'
      ./features/step_definitions/apt.rb:53:in `/^I start Synaptic$/'
      features/apt.feature:24:in `When I start Synaptic'

  Scenario: A warning will be displayed if Electrum is not persistent              # features/electrum.feature:9
    Given I have started Tails from DVD and logged in and the network is connected # features/step_definitions/snapshots.rb:208
    When I start Electrum through the GNOME menu                                   # features/step_definitions/electrum.rb:1
      Command failed: /usr/bin/python '/tmp/tmp.ll1lGWeKyd'
      error code: 1
      stderr: .
      <false> is not true. (ExecutionFailedInVM)
      ./features/support/helpers/vm_helper.rb:441:in `rescue in execute_successfully'
      ./features/support/helpers/vm_helper.rb:438:in `execute_successfully'
      ./features/support/helpers/dogtail.rb:87:in `run'
      ./features/support/helpers/dogtail.rb:203:in `block (2 levels) in <class:Node>'
      ./features/step_definitions/common_steps.rb:644:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/common_steps.rb:643:in `each'
      ./features/step_definitions/common_steps.rb:643:in `/^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/'
      ./features/step_definitions/electrum.rb:2:in `/^I start Electrum through the GNOME menu$/'
      features/electrum.feature:11:in `When I start Electrum through the GNOME menu'

  Scenario: Deleting a Tails persistent partition                                                                                       # features/persistence.feature:46
    Given I have started Tails without network from a USB drive with a persistent partition and stopped at Tails Greeter's login screen # features/step_definitions/snapshots.rb:208
    And I log in to a new session                                                                                                       # features/step_definitions/common_steps.rb:280
    Then Tails is running from USB drive "__internal"                                                                                   # features/step_definitions/usb.rb:374
    And the boot device has safe access rights                                                                                          # features/step_definitions/usb.rb:396
    And persistence is disabled                                                                                                         # features/step_definitions/usb.rb:346
    But a Tails persistence partition exists on USB drive "__internal"                                                                  # features/step_definitions/usb.rb:271
    And all notifications have disappeared                                                                                              # features/step_definitions/common_steps.rb:409
    When I delete the persistent partition                                                                                              # features/step_definitions/usb.rb:570
      Command failed: /usr/bin/python '/tmp/tmp.NOurVspCjK'
      error code: 1
      stderr: .
      <false> is not true. (ExecutionFailedInVM)
      ./features/support/helpers/vm_helper.rb:441:in `rescue in execute_successfully'
      ./features/support/helpers/vm_helper.rb:438:in `execute_successfully'
      ./features/support/helpers/dogtail.rb:87:in `run'
      ./features/support/helpers/dogtail.rb:203:in `block (2 levels) in <class:Node>'
      ./features/step_definitions/common_steps.rb:644:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/common_steps.rb:643:in `each'
      ./features/step_definitions/common_steps.rb:643:in `/^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/'
      ./features/step_definitions/usb.rb:571:in `/^I delete the persistent partition$/'
      features/persistence.feature:54:in `When I delete the persistent partition'

  Scenario: Fetching OpenPGP keys using Seahorse should work and be done over Tor. # features/torified_gnupg.feature:21
    When I fetch the "10CC5BC7" OpenPGP key using Seahorse                         # features/step_definitions/torified_gnupg.rb:154
      Command failed: /usr/bin/python '/tmp/tmp.zbDeJjhLOg'
      error code: 1
      stderr: .
      <false> is not true. (ExecutionFailedInVM)
      ./features/support/helpers/vm_helper.rb:441:in `rescue in execute_successfully'
      ./features/support/helpers/vm_helper.rb:438:in `execute_successfully'
      ./features/support/helpers/dogtail.rb:87:in `run'
      ./features/support/helpers/dogtail.rb:203:in `block (2 levels) in <class:Node>'
      ./features/step_definitions/common_steps.rb:644:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/common_steps.rb:643:in `each'
      ./features/step_definitions/common_steps.rb:643:in `/^I start "([^"]+)" via the GNOME "([^"]+)" applications menu$/'
      ./features/step_definitions/torified_gnupg.rb:23:in `start_or_restart_seahorse'
      ./features/step_definitions/torified_gnupg.rb:92:in `/^I start Seahorse( via the OpenPGP Applet)?$/'
      ./features/step_definitions/torified_gnupg.rb:155:in `/^I fetch the "([^"]+)" OpenPGP key using Seahorse( via the OpenPGP Applet)?$/'
      features/torified_gnupg.feature:22:in `When I fetch the "10CC5BC7" OpenPGP key using Seahorse'

Any clue what's happening?

#11 Updated by anonym about 3 years ago

  • Assignee changed from anonym to intrigeri
  • % Done changed from 50 to 60
  • QA Check changed from Dev Needed to Ready for QA

intrigeri wrote:

2. Merge this quickly, so new stuff can change paradigm towards Dogtail when more suitable.

Next step! What do you think, intri?

Great job! It's lovely to see this many images go away :)

Code review done, only a few details and minor comments:

  • decrypt_verify_helper and friends: I find the function signature obscure when it's used. How about using named arguments (with whatever tricks Ruby requires to have them)?

Done in ccff595.

  • "This was we can support stuff like" ← I can't parse this sentence.

"This way we ...". Fixed now!

  • Can't we do python3 instead of python2? (As you can guess, I'm mostly looking ahead to future pain points here.)

Debian ships Dogtail 0.9.0, which is Python2. Dogtail 0.9.9 adds Python3 support, so we'll have to wait until it is available for the Debian version Tails is based on.

  • self.args_to_s might benefit from a comment about its args (I think I got it right by reading the code, but still).

Done in d490bbd.

  • Thanks for the comment at the top of def children because oh my, I would have a hard time improving this function if I ever had to.

You are welcome! :) I know it all got a bit crazy complexity-wise, but I failed to find anything simpler that still would be convenient enough for us.

  • I expected that features/support/helpers/dogtail.rb would be much bigger. I'm glad it's not! :)

Recently I started toying with the idea of extending the remote shell to, essentially, run a python shell loaded with dogtail that we can interact with, so objects persist and can be referenced later. That should simplify the this glue code a lot, but, well, later.

Regarding how it fares in practice, I see two recent failures on Jenkins that make me feel a bit uneasy:

Interesting. So I looked at the debug.log and found this for a bit more context:

[...]
calling as amnesia: echo '#!/usr/bin/python
from dogtail import tree
from dogtail.config import config
config.searchShowingOnly = True
application = tree.root.application('"'"'gedit'"'"')
application.button('"'"'New'"'"').click()' >> '/tmp/tmp.kg0NHEhCeZ'
call returned: [0, "", ""]
calling as amnesia: /usr/bin/python '/tmp/tmp.kg0NHEhCeZ'
call returned: [0, "Creating logfile at /tmp/dogtail-amnesia/logs/tmp.kg0NHEhCeZ_20160425-195535_debug ...\nDogtail: warning: application may be hanging\nClicking on [push button | New]\nMouse button 1 click at (207,70)\n", ""]
calling as root: rm -f '/tmp/tmp.kg0NHEhCeZ'
call returned: [0, "", ""]
calling as amnesia: mktemp
call returned: [0, "/tmp/tmp.qHGZBlkVKv\n", ""]
calling as root: rm -f '/tmp/tmp.qHGZBlkVKv'
call returned: [0, "", ""]
calling as amnesia: echo '#!/usr/bin/python
from dogtail import tree
from dogtail.config import config
config.searchShowingOnly = True
application = tree.root.application('"'"'gedit'"'"')
application.child(roleName='"'"'text'"'"').click(button=3)' >> '/tmp/tmp.qHGZBlkVKv'
call returned: [0, "", ""]
calling as amnesia: /usr/bin/python '/tmp/tmp.qHGZBlkVKv'
call returned: [0, "Creating logfile at /tmp/dogtail-amnesia/logs/tmp.qHGZBlkVKv_20160425-195556_debug ...\nDogtail: warning: application may be hanging\nClicking on [text | ]\nMouse button 3 click at (511,399)\n", ""]
calling as root: rm -f '/tmp/tmp.qHGZBlkVKv'
call returned: [0, "", ""]
calling as amnesia: mktemp
call returned: [0, "/tmp/tmp.dBE66obICt\n", ""]
calling as root: rm -f '/tmp/tmp.dBE66obICt'
call returned: [0, "", ""]
calling as amnesia: echo '#!/usr/bin/python
from dogtail import tree
from dogtail.config import config
config.searchShowingOnly = True
application = tree.root.application('"'"'gedit'"'"')
application.menuItem('"'"'Paste'"'"').click()' >> '/tmp/tmp.dBE66obICt'
call returned: [0, "", ""]
calling as amnesia: /usr/bin/python '/tmp/tmp.dBE66obICt'
call returned: [1, "Creating logfile at /tmp/dogtail-amnesia/logs/tmp.dBE66obICt_20160425-195618_debug ...\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 3)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 4)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 5)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 6)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 7)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 8)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 9)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 10)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 11)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 12)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 13)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 14)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 15)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 16)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 17)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 18)\nsearching for descendent of [application | gedit]: \"Paste\" menuitem (attempt 19)\nTraceback (most recent call last):\n  File \"/tmp/tmp.dBE66obICt\", line 6, in <module>\n    application.menuItem('Paste').click()\n  File \"/usr/lib/python2.7/dist-packages/dogtail/tree.py\", line 1008, in menuItem\n    return self.findChild(predicate.IsAMenuItemNamed(menuItemName=menuItemName), recursive, showingOnly=showingOnly)\n  File \"/usr/lib/python2.7/dist-packages/dogtail/tree.py\", line 897, in findChild\n    raise SearchError(describeSearch(self, pred, recursive, debugName))\nSearchError: descendent of [application | gedit]: \"Paste\" menuitem\n\n", ""]
calling as root: rm -f '/tmp/tmp.dBE66obICt'
call returned: [0, "", ""]
    And I symmetrically encrypt the message with password "asdf"      # features/step_definitions/encryption.rb:143
      Command failed: /usr/bin/python '/tmp/tmp.dBE66obICt'
[...]

The interesting parts are the occurrences of "Dogtail: warning: application may be hanging". I see them nowhere else in the debug.log, and it seems that those two actions (first left-clicking Gedit's "New" button, then right-clicking the text area to get the context menu) fail, which then make the third action (find the "Paste" entry in the context menu) fail.

This is a bit worrying -- it seems like a random source of robustness issues. Luckily I haven't seen it any of the other 10 debug.log:s I've looked at. I feel tempted to add a check for this error and make us abort with a more easily identifiable message, but perhaps it is not always fatal so that is unnecessary. I'll keep an eye out for this.

I see it all over the place in the debug.log so I don't think it is fatal. Looking a bit earlier the issue becomes clearer; fixed in c2db06a.

The causes for the errors you reported in #10721#note-10 are fixed in commits 52069cb and fc77772. I'm really surprised how I didn't notice this during my tests. I cannot possibly have run the full test suite, which I am sorry for.

Massive thanks for your review!

#12 Updated by intrigeri about 3 years ago

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Ready for QA to Info Needed

intrigeri wrote:

  • decrypt_verify_helper and friends: I find the function signature obscure when it's used. How about using named arguments (with whatever tricks Ruby requires to have them)?

Done in ccff595.

Cool. Just curious: why is the Hash approach more suitable here than Ruby 2.x's keyword arguments ?

  • Can't we do python3 instead of python2? (As you can guess, I'm mostly looking ahead to future pain points here.)

Debian ships Dogtail 0.9.0, which is Python2. Dogtail 0.9.9 adds Python3 support, so we'll have to wait until it is available for the Debian version Tails is based on.

OK, sounds good enough!

https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/21/cucumberTestReport/encryption-and-verification-using-gnupg/symmetric-encryption-and-decryption-using-openpgp-applet/ ← this one is quite obscure; can we get more debugging info from the Python script / from dogtail?

Interesting. So I looked at the debug.log and found this for a bit more context: [...]

The interesting parts are the occurrences of "Dogtail: warning: application may be hanging". I see them nowhere else in the debug.log, and it seems that those two actions (first left-clicking Gedit's "New" button, then right-clicking the text area to get the context menu) fail, which then make the third action (find the "Paste" entry in the context menu) fail.

This is a bit worrying -- it seems like a random source of robustness issues. Luckily I haven't seen it any of the other 10 debug.log:s I've looked at. I feel tempted to add a check for this error and make us abort with a more easily identifiable message, but perhaps it is not always fatal so that is unnecessary.

Hmmm. On the one hand, I'm tempted to say that this specific test scenario was fragile anyway, so we're not worse with dogtail. OTOH, I'm concerned that this class of problems may introduce new robustness issues elsewhere, which feels wrong given we're working on this precisely as a mean to improve robustness.

I'll keep an eye out for this.

Note that it happened again already: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/29/cucumberTestReport/encryption-and-verification-using-gnupg/symmetric-encryption-and-decryption-using-openpgp-applet/ -- which makes me seriously wonder if the failure frequency is worrying enough to be a merge blocker. What do you think?

I'm now doing another full test suite run on that branch, but still I'm reassigning to you so you can comment on the above two questions.

#13 Updated by intrigeri about 3 years ago

  • % Done changed from 60 to 70

After a bunch of retries, I got the full testing passing, modulo the Icedove test bug I just reported. Congrats!

#14 Updated by anonym about 3 years ago

intrigeri wrote:

intrigeri wrote:

  • decrypt_verify_helper and friends: I find the function signature obscure when it's used. How about using named arguments (with whatever tricks Ruby requires to have them)?

Done in ccff595.

Cool. Just curious: why is the Hash approach more suitable here than Ruby 2.x's keyword arguments ?

Thanks, I was unaware of this possibility! Fixed in 2c3dbaa.

https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/21/cucumberTestReport/encryption-and-verification-using-gnupg/symmetric-encryption-and-decryption-using-openpgp-applet/ ← this one is quite obscure; can we get more debugging info from the Python script / from dogtail?

Interesting. So I looked at the debug.log and found this for a bit more context: [...]

The interesting parts are the occurrences of "Dogtail: warning: application may be hanging". I see them nowhere else in the debug.log, and it seems that those two actions (first left-clicking Gedit's "New" button, then right-clicking the text area to get the context menu) fail, which then make the third action (find the "Paste" entry in the context menu) fail.

This is a bit worrying -- it seems like a random source of robustness issues. Luckily I haven't seen it any of the other 10 debug.log:s I've looked at. I feel tempted to add a check for this error and make us abort with a more easily identifiable message, but perhaps it is not always fatal so that is unnecessary.

Hmmm. On the one hand, I'm tempted to say that this specific test scenario was fragile anyway, so we're not worse with dogtail. OTOH, I'm concerned that this class of problems may introduce new robustness issues elsewhere, which feels wrong given we're working on this precisely as a mean to improve robustness.

I also worry about this...

I'll keep an eye out for this.

Note that it happened again already: https://jenkins.tails.boum.org/job/test_Tails_ISO_test-10721-a11y-technologies/29/cucumberTestReport/encryption-and-verification-using-gnupg/symmetric-encryption-and-decryption-using-openpgp-applet/

The interesting thing is that it seems to happen in this specific scenario, and not the other similar ones in encryption.feature. I can get that it doesn't happen for the GNOME Applications menu navigation, since it's pretty different. So, unfortunately, it seem we'd need more parts of our test suite using dogtail to understand the scope of this well enough.

which makes me seriously wonder if the failure frequency is worrying enough to be a merge blocker. What do you think?

I'm really at odds here. By now I think it is clear that Sikuli has some fundamental limitations that dogtail would solve much better for us, and there seems to be nothing better around. I'm tempted to take the position that we should migrate to dogtail any way, and instead put a serious effort into fixing the bug we're gitting (or find a way to workaround it), possibly by asking the dogtail maintainer for some serious assistance.

I also worry of postponing this branch merge further -- by not giving it prime time, any effort to get this fixed will be put off and get less attention. Bleh. Let's discuss this at the CI meeting.

#15 Updated by intrigeri about 3 years ago

Thanks, I was unaware of this possibility! Fixed in 2c3dbaa.

Glad to be able to teach you some Ruby sometimes :)

which makes me seriously wonder if the failure frequency is worrying enough to be a merge blocker. What do you think?

I'm really at odds here. By now I think it is clear that Sikuli has some fundamental limitations that dogtail would solve much better for us, and there seems to be nothing better around. I'm tempted to take the position that we should migrate to dogtail any way, and instead put a serious effort into fixing the bug we're gitting (or find a way to workaround it), possibly by asking the dogtail maintainer for some serious assistance.

This could work for me.

I also worry of postponing this branch merge further -- by not giving it prime time, any effort to get this fixed will be put off and get less attention.

Absolutely.

Let's discuss this at the CI meeting.

OK.

#16 Updated by anonym about 3 years ago

  • % Done changed from 70 to 80
  • QA Check changed from Info Needed to Dev Needed

Let's discuss this at the CI meeting.

We decided to revert all dogtail stuff for encryption.feature due to the "application may be hanging" bug, and then switftly merge this. Then dogtail will be available as a tool for other tests we need to write now, and we can continue investigating the "Dogtail: warning: application may be hanging" bug. Any new tests using dogtail will be tested extra to see if we can spot the bug in other contexts.

#17 Updated by anonym about 3 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 80 to 100
  • QA Check changed from Dev Needed to Pass

anonym wrote:

Let's discuss this at the CI meeting.

We decided to revert all dogtail stuff for encryption.feature

Done in 339fced5d1da8c28434a4360eb7acd8860c0e9d3.

due to the "application may be hanging" bug,

I filed #11409 to track it.

and then switftly merge this.

Merged!

#18 Updated by anonym almost 3 years ago

  • Status changed from Fix committed to Resolved

Also available in: Atom PDF