Project

General

Profile

Bug #15321

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

"The Report an Error launcher will…" test suite step is fragile

Added by intrigeri over 1 year ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
Start date:
02/17/2018
Due date:
% Done:

0%

Feature Branch:
test/15321-doc-pages+force-all-tests
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

It caused failures in 4 of the last 10 runs of https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_stable/ so I'm going to mark it as fragile. I don't know if that's a regression since the new tails-documentation was introduced or if it's a test suite bug.


Related issues

Related to Tails - Bug #15514: The "The Tails documentation launcher on the desktop works…" scenarios are fragile Resolved 04/09/2018
Related to Tails - Feature #11717: Drop launchers from the Desktop Rejected 08/25/2016
Related to Tails - Bug #16283: No more icons on desktop in buster Resolved 01/05/2019
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed

Associated revisions

Revision 02428b2f (diff)
Added by intrigeri over 1 year ago

Test suite: mark scenarios that use the "The Report an Error launcher will…" step as fragile (refs: #15321)

Revision 77aa42f0 (diff)
Added by intrigeri about 1 year ago

Use Tor Browser even for offline documentation, drop the custom documentation browser, and adjust the test suite accordingly (refs: #15720).

Incidentally, this brings back test suite robustness workarounds
that 74536c6a04c850db69686588723a9fec5060be7b had removed,
which might help fix another bug (refs: #15321).

Revision b58c6dcc (diff)
Added by intrigeri about 2 months ago

Test suite: make the "the support documentation page opens in Tor Browser" step more robust (refs: #15321)

I've seen this step fail while Tor Browser displays the correct page. That's
because the test was racy to me: after the "has loaded" step has passed, we're
immediately collecting a list of headings; but if Tor Browser is a bit slow, it
may not have exposed the entire page content on the a11y bus yet.

This problem is similar to the one fixed in commit
0d8a2b71774ee1e06788efcf8498a6d3acdd04cf, so let's apply the same solution here.

Revision dd63e1c0 (diff)
Added by intrigeri about 2 months ago

Remove the fragile tag on test that was fixed on this branch (refs: #15321)

History

#1 Updated by intrigeri over 1 year ago

  • Description updated (diff)

#2 Updated by intrigeri over 1 year ago

  • Status changed from Confirmed to In Progress

#3 Updated by intrigeri over 1 year ago

  • Related to Bug #15514: The "The Tails documentation launcher on the desktop works…" scenarios are fragile added

#4 Updated by intrigeri over 1 year ago

  • Description updated (diff)
  • Status changed from In Progress to Confirmed

#5 Updated by intrigeri over 1 year ago

intrigeri wrote:

I don't know if that's a regression since the new tails-documentation was introduced or if it's a test suite bug.

Given the online flavour often fails too, I think this is "just" yet another problem with desktop icons.

#6 Updated by intrigeri over 1 year ago

#7 Updated by intrigeri about 1 year ago

intrigeri wrote:

intrigeri wrote:

I don't know if that's a regression since the new tails-documentation was introduced or if it's a test suite bug.

Given the online flavour often fails too, I think this is "just" yet another problem with desktop icons.

Actually, 74536c6a04c850db69686588723a9fec5060be7b removed some robustnes improvements. I'll reintroduced them via #15720 and we'll see.

#8 Updated by intrigeri about 1 year ago

  • Status changed from Confirmed to In Progress

#9 Updated by intrigeri about 1 year ago

I've found another failure mode for "the support documentation page opens in Tor Browser". The browser does start and open the right page, but then the heading lookup fails:

02:26:58.195719964: executing Python as amnesia: node432 = node428.child('Tails - Hilfe & Support - Tor-Browser', roleName='frame')
02:26:58.247062647: execution complete
02:26:58.247209492: executing Python as amnesia: 
    nodes433 = node432.findChildren(dogtail.predicate.GenericPredicate(roleName='heading'))
    print(len(nodes433))
02:26:59.100786690: execution complete
    Then the support documentation page opens in Tor Browser                                                  # features/step_definitions/checks.rb:97
      <false> is not true. (Test::Unit::AssertionFailedError)
      features/localization.feature:13:in `Then the support documentation page opens in Tor Browser'

Looking at the code:

   step "\"#{expected_title}\" has loaded in the Tor Browser" 
   if @language == 'German'
     browser_name = 'Tor-Browser'
   else
     browser_name = 'Tor Browser'
   end
   headings = @torbrowser
              .child(expected_title + " - #{browser_name}", roleName: 'frame')
              .children(roleName: 'heading')
   assert(
     headings.any? { |heading| heading.text == expected_heading }
   )

… this seems racy to me: after the "has loaded" step has passed, we're immediately collecting a list of headings; but if Tor Browser is a bit slow, it may not have exposed the entire page content on the a11y bus yet. I think we should drop the assert and let dogtail look for the expected heading itself, so we benefit from its internal wait/retry magics. This might require extending our dogtail custom bindings to support the .text selector. If that's not feasible then we should wrap the assert with some retry loop.

#10 Updated by intrigeri 10 months ago

  • Related to Bug #16283: No more icons on desktop in buster added

#11 Updated by intrigeri 7 months ago

  • Status changed from In Progress to Confirmed

#12 Updated by intrigeri 5 months ago

I've seen a case when the test suite actually clicked the "Tails-Dokumentation" icon instead of the "Einen Fehler melden" one: Found fuzzy candidate picture for DesktopReportAnError.png with similarity 0.6. So then obviously, in the next step Dogtail does not find the expected page title.

#13 Updated by intrigeri 2 months ago

intrigeri wrote:

I've seen a case when the test suite actually clicked the "Tails-Dokumentation" icon instead of the "Einen Fehler melden" one: Found fuzzy candidate picture for DesktopReportAnError.png with similarity 0.6. So then obviously, in the next step Dogtail does not find the expected page title.

Seen this failure mode again. That's because the "Einen Fehler melden" icon is half-hidden behind the GNOME Shell top bar. This won't happen anymore on Tails 4.x because the desktop icons there are managed by a GNOME Shell extension instead of Nautilus.

#14 Updated by intrigeri about 2 months ago

  • Status changed from Confirmed to In Progress

#15 Updated by intrigeri about 2 months ago

  • Assignee set to intrigeri
  • Target version set to Tails_4.0
  • Feature Branch set to test/15321-doc-pages+force-all-tests

I've seen the problem reported in #15321#note-9 again. A similar problem was fixed with 0d8a2b71774ee1e06788efcf8498a6d3acdd04cf. I'll do the same here.

#16 Updated by intrigeri about 2 months ago

#17 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Needs Validation
  • Assignee deleted (intrigeri)

#18 Updated by intrigeri about 2 months ago

  • Status changed from Needs Validation to In Progress

#19 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Needs Validation

#20 Updated by hefee about 2 months ago

  • Assignee set to hefee

#21 Updated by hefee about 2 months ago

  • Status changed from Needs Validation to Resolved
  • Assignee deleted (hefee)

Applied in changeset tails|1fa89c94a05aaeb5841c57f13ccd8f04326650eb.

Also available in: Atom PDF