Project

General

Profile

Bug #15321

Bug #10288: Fix newly identified issues to make our test suite more robust and faster

"The Report an Error launcher will…" test suite step is fragile

Added by intrigeri about 1 year ago. Updated 6 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
-
Start date:
02/17/2018
Due date:
% Done:

0%

QA Check:
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

It caused failures in 4 of the last 10 runs of https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_stable/ so I'm going to mark it as fragile. I don't know if that's a regression since the new tails-documentation was introduced or if it's a test suite bug.


Related issues

Related to Tails - Bug #15514: The "The Tails documentation launcher on the desktop works…" scenarios are fragile Confirmed 04/09/2018
Related to Tails - Feature #11717: Drop launchers from the Desktop Rejected 08/25/2016
Related to Tails - Bug #16283: No more icons on desktop in buster Resolved 01/05/2019

Associated revisions

Revision 02428b2f (diff)
Added by intrigeri about 1 year ago

Test suite: mark scenarios that use the "The Report an Error launcher will…" step as fragile (refs: #15321)

Revision 77aa42f0 (diff)
Added by intrigeri 7 months ago

Use Tor Browser even for offline documentation, drop the custom documentation browser, and adjust the test suite accordingly (refs: #15720).

Incidentally, this brings back test suite robustness workarounds
that 74536c6a04c850db69686588723a9fec5060be7b had removed,
which might help fix another bug (refs: #15321).

History

#1 Updated by intrigeri about 1 year ago

  • Description updated (diff)

#2 Updated by intrigeri about 1 year ago

  • Status changed from Confirmed to In Progress

#3 Updated by intrigeri 11 months ago

  • Related to Bug #15514: The "The Tails documentation launcher on the desktop works…" scenarios are fragile added

#4 Updated by intrigeri 11 months ago

  • Description updated (diff)
  • Status changed from In Progress to Confirmed

#5 Updated by intrigeri 9 months ago

intrigeri wrote:

I don't know if that's a regression since the new tails-documentation was introduced or if it's a test suite bug.

Given the online flavour often fails too, I think this is "just" yet another problem with desktop icons.

#6 Updated by intrigeri 9 months ago

#7 Updated by intrigeri 7 months ago

intrigeri wrote:

intrigeri wrote:

I don't know if that's a regression since the new tails-documentation was introduced or if it's a test suite bug.

Given the online flavour often fails too, I think this is "just" yet another problem with desktop icons.

Actually, 74536c6a04c850db69686588723a9fec5060be7b removed some robustnes improvements. I'll reintroduced them via #15720 and we'll see.

#8 Updated by intrigeri 7 months ago

  • Status changed from Confirmed to In Progress

#9 Updated by intrigeri 6 months ago

I've found another failure mode for "the support documentation page opens in Tor Browser". The browser does start and open the right page, but then the heading lookup fails:

02:26:58.195719964: executing Python as amnesia: node432 = node428.child('Tails - Hilfe & Support - Tor-Browser', roleName='frame')
02:26:58.247062647: execution complete
02:26:58.247209492: executing Python as amnesia: 
    nodes433 = node432.findChildren(dogtail.predicate.GenericPredicate(roleName='heading'))
    print(len(nodes433))
02:26:59.100786690: execution complete
    Then the support documentation page opens in Tor Browser                                                  # features/step_definitions/checks.rb:97
      <false> is not true. (Test::Unit::AssertionFailedError)
      features/localization.feature:13:in `Then the support documentation page opens in Tor Browser'

Looking at the code:

   step "\"#{expected_title}\" has loaded in the Tor Browser" 
   if @language == 'German'
     browser_name = 'Tor-Browser'
   else
     browser_name = 'Tor Browser'
   end
   headings = @torbrowser
              .child(expected_title + " - #{browser_name}", roleName: 'frame')
              .children(roleName: 'heading')
   assert(
     headings.any? { |heading| heading.text == expected_heading }
   )

… this seems racy to me: after the "has loaded" step has passed, we're immediately collecting a list of headings; but if Tor Browser is a bit slow, it may not have exposed the entire page content on the a11y bus yet. I think we should drop the assert and let dogtail look for the expected heading itself, so we benefit from its internal wait/retry magics. This might require extending our dogtail custom bindings to support the .text selector. If that's not feasible then we should wrap the assert with some retry loop.

#10 Updated by intrigeri about 2 months ago

  • Related to Bug #16283: No more icons on desktop in buster added

Also available in: Atom PDF