Tails

Actually, regarding rjb at least: this could be a great opportunity for you (anonym) to get involved a bit in Debian: you're one of the few people who understand anything about this Java+Ruby thing and it might be more pleasurable for you to submit a fix than to find someone else to do it.

lamby wrote:

I've fixed the FTBFS and testsuite failures here:

https://bugs.debian.org/874146#28

What are the next steps here?

Should I be pushing this fix in Debian for an easier "merge" or shall I upload to Tails? :)

Just let me know :)

I've fixed the FTBFS and testsuite failures here:

\o/

What are the next steps here?

Announce your intent to NMU, wait a bit (I'll ask Lunar if he's fine with it) and then upload :)

Should I be pushing this fix in Debian for an easier "merge" or shall I upload to Tails? :)

This package is only required for Tails developers who run the test suite on their sid system, so uploading to Debian makes much more sense here that uploading to a Tails-only repo.

Just let me know :)

Pro tip: when asking someone's input here, assign to them and set "QA Check" to "Info Needed" :)

Announce your intent to NMU

Thanks. Done here: https://bugs.debian.org/874146#35

Pro tip: when asking someone's input here, assign to them and set "QA Check" to "Info Needed" :)

Thanks! I guess I didn't know whom to assign to really, but I should have assigned to someone at the very least! :p

I guess I didn't know whom to assign to really, but I should have assigned to someone at the very least! :p

Generally, for Foundations Team work, unless someone else (most likely anonym) is explicitly your team-mate/mentor, assigning to me is best.

(I've filed the FTBFS as https://bugs.debian.org/897664)

Fixed in Debian, uploaded as 1.5.5-2

I meant 1.5.5-3 here, sorry

Great! So we're good wrt. ruby-rjb. Chris, please tell me how much of the 6 hours you've used and I'll validate your work in our accounting :)

The next steps (SikuliX) are documented on #15460#note-17. Let's wait a couple more weeks (until our next FT meeting) to give the maintainers the chance to address https://bugs.debian.org/897215 and then we'll see.

please tell me how much of the 6 hours you've used and I'll validate your work in our accounting :)

Any objection if we do that in one "batch" with the other tickets to keep our overheads low? :)

please tell me how much of the 6 hours you've used and I'll validate your work in our accounting :)

Any objection if we do that in one "batch" with the other tickets to keep our overheads low? :)

OK, let's try!

intrigeri wrote:

The next steps (SikuliX) are documented on #15460#note-17. Let's wait a couple more weeks (until our next FT meeting) to give the maintainers the chance to address https://bugs.debian.org/897215 and then we'll see.

I have:

• Updated the severity of https://bugs.debian.org/897215 to serious.

• Asked the Java maintainers whether they would like me to update it.

Noting RM bug #897333.

WIP SikuliX 1.1.2 packaging (broken at runtime according to the former maintainer): https://salsa.debian.org/java-team/sikuli

During summit we roughly scheduled this as being targetted for Dec/Jan 2018/2019.

Note that Debian will likely be shipping buster with OpenJDK 11.

https://salsa.debian.org/java-team/sikuli/commit/9d14622bb62e7b16e13d55672f8a308820d08c3d :(

A few days ago I stumbled upon some (Python) code that used OpenCV's matchTemplate() which suites perfectly as an "image matching primitive".

Great news, I'm glad you've been looking into this. It's exciting!

So if we want to use OpenCV I suppose we need to call it as a subprocess. In fact, the script as-is would serve for that purpose. A potential issue is that the script has two heavy dependencies, cv2 and numpy, which take almost 300ms to import on my pretty powerful CPU (i7-7700HQ @ 2.8 GHz) which I worry is so much that it might interfere. So perhaps we make it a "server" that we leave running and communicate with over e.g. a unix socket. It will work, but it is a bit ugly...

Interesting. I think that at this point, I'm less worried about the potential ugliness than about reliability: it's one more thing that manages its own state and can get stuck or desynchronized from the callers. The remote shell historical fragility (now thankfully a thing of the past!) comes to mind here.

Wrt. strategy and allocation of our resources, with my "person who manages the FT's budget" hat on: I'm a bit concerned about the costs that come with a NIH solution. I would like us to first look into how OpenQA does images matching, as suggested in the ticket description; and later, fallback on this brand new approach, solve the problems that come with it, and maintain it, if and only if the code that a number of major distros use does not work for us. For example, I suspect that OpenQA's implementation does is not affected by this load time issue. Now, of course, it may be that we quickly realize that the OpenQA implementation is very much non-reusable and we give up on it after spending 1-2h on it; but at this point we don't know this.

intrigeri wrote:

A few days ago I stumbled upon some (Python) code that used OpenCV's matchTemplate() which suites perfectly as an "image matching primitive".

Great news, I'm glad you've been looking into this. It's exciting!

So if we want to use OpenCV I suppose we need to call it as a subprocess. In fact, the script as-is would serve for that purpose. A potential issue is that the script has two heavy dependencies, cv2 and numpy, which take almost 300ms to import on my pretty powerful CPU (i7-7700HQ @ 2.8 GHz) which I worry is so much that it might interfere. So perhaps we make it a "server" that we leave running and communicate with over e.g. a unix socket. It will work, but it is a bit ugly...

Interesting. I think that at this point, I'm less worried about the potential ugliness than about reliability:

Ack. Let's consider this a braindump and forget it for now!

Wrt. strategy and allocation of our resources, with my "person who manages the FT's budget" hat on: I'm a bit concerned about the costs that come with a NIH solution.

Yes, given the history with Sikuli (it was the reason for us initially using JRuby...) this is also my first priority. The thing is, I have (in a very relaxed manner) looked for Sikuli alternatives for years, and OpenCV has been the basically the only answer. Luckily it is a stable project in terms of maintenance, AFAICT! :)

Also, I suspect that the amount of work needed for integrating any matching primitive into our test suite is dwarfed by the work needed for the machinery to around that is shared by all such solutions, e.g. keyboard/mouse stuff, and probably rewriting minimal versions of a few of Sikuli's classes, like Screen and Match, since we are pretty invested in them and they are pretty great abstractions any way. In other words, the NIH-part is small in comparison to the whole work. And it will be compartmentalized so it's little work replacing it with another matching primitive.

I would like us to first look into how OpenQA does images matching, as suggested in the ticket description; and later, fallback on this brand new approach, solve the problems that come with it, and maintain it, if and only if the code that a number of major distros use does not work for us. For example, I suspect that OpenQA's implementation does is not affected by this load time issue. Now, of course, it may be that we quickly realize that the OpenQA implementation is very much non-reusable and we give up on it after spending 1-2h on it; but at this point we don't know this.

OpenQA also employs a straightforward application of OpenCV's matchTemplate(). :) At least we can look for inspiration in their choice of parameters.

I spent another 20 minute searching for image matching alternatives in Ruby-land, and while I can find some things for computer vision in general, none of them provide what we want (fuzzy image matching). I suspect that if more time is needed on this search, anything that comes up will be pretty obscure and hence likely worse in the end.

BTW, everything we need for the OpenCV python solution is available in Debian (python3-opencv python3-numpy python3-pil) so all-in-all it's starting to looks like pretty ok to me.

Wrt. strategy and allocation of our resources, with my "person who manages the FT's budget" hat on: I'm a bit concerned about the costs that come with a NIH solution.

Yes, given the history with Sikuli (it was the reason for us initially using JRuby...) this is also my first priority.

Glad we're on the same page :)

In other words, the NIH-part is small in comparison to the whole work.

Okay. Without having thought about it much, I'm inclined to blindly trust your assessment here.

OpenQA also employs a straightforward application of OpenCV's matchTemplate(). :) At least we can look for inspiration in their choice of parameters.

OK!

I spent another 20 minute searching for image matching alternatives in Ruby-land, and while I can find some things for computer vision in general, none of them provide what we want (fuzzy image matching). I suspect that if more time is needed on this search, anything that comes up will be pretty obscure and hence likely worse in the end.

It's not 100% obvious to me that fuzzy image matching should be an absolute requirement here.

On the one hand, I do see some benefits in it. For example, since you added this feature, we've used it for at least 2 batch-updates: 1f02dda0211b44f0cf0bd59a689ff9af57164bbe, 33fa7c4ba6fe5aac2db07e477fc4307bd7de4d70. That's not nothing. OTOH, it's also not huge. IIRC we expected to get much more benefit out of this feature: you implemented it initially when we were playing with the idea of basing Tails on Debian testing, which would have caused much more churn in images, and in turn would have made fuzzy matching a huge time saver.

So, if there's a way to use Ruby, avoiding the complexity and potential fragility that come from bridging Ruby & Python somehow, at the cost of losing fuzzy matching, then IMO we should seriously consider this option.

Note, in passing, that a bug in our fuzzy matching feature (#17029) has made me, and possibly others, waste substantial amounts of time for 2 years, which greatly outweighed the time we saved thanks to batch-updates so far: I started suspecting something was fishy since ~1 year ago (3ac90f2ed9a544ce235bf7e520aafc3a50208c81, 670bd5f6fcb4b30159fe2cf81a4ffe89b67dca2a) but only tracked down & fixed the problem recently. But that's mostly off-topic because we'll learn from it and ensure that our next implementation of fuzzy matching won't have such problems :)

BTW, everything we need for the OpenCV python solution is available in Debian (python3-opencv python3-numpy python3-pil) so all-in-all it's starting to looks like pretty ok to me.

It's great to have this option if we have to, or decide to, fallback to Python!

intrigeri wrote:

I spent another 20 minute searching for image matching alternatives in Ruby-land, and while I can find some things for computer vision in general, none of them provide what we want (fuzzy image matching). I suspect that if more time is needed on this search, anything that comes up will be pretty obscure and hence likely worse in the end.

It's not 100% obvious to me that fuzzy image matching should be an absolute requirement here.

Sikuli's image matching is always fuzzy, and and we use a default required similarity of 0.9 (sikuli_settings.MinSimilarity = 0.9), that is what I'm talking about. The test suite's --fuzzy-image-matching means that we will try with a couple lower similarity values (i.e. "more fuzz") to get a candidate.

So at the moment we do not know if a pixel-by-pixel perfect matcher is good enough because we have never done that but I guess we could bump the default required similarity to 1.0 and do a full run to get an idea. My intuition is that it will be bad, and I fear it could introduce new fun problems like image compression changing pixels ⇒ false negatives.

That said, I haven't seen any image matcher in Ruby-land except OpenCV. But I guess it would be pretty easy to implement (convert to bitmaps, slide the candidate picture around until all pixels match (success), or all possibilities are exhausted (failure)) and maintain ourselves, so that is not out of the question.

On the one hand, I do see some benefits in it. For example, since you added this feature, we've used it for at least 2 batch-updates: 1f02dda0211b44f0cf0bd59a689ff9af57164bbe, 33fa7c4ba6fe5aac2db07e477fc4307bd7de4d70. That's not nothing. OTOH, it's also not huge. IIRC we expected to get much more benefit out of this feature: you implemented it initially when we were playing with the idea of basing Tails on Debian testing, which would have caused much more churn in images, and in turn would have made fuzzy matching a huge time saver.

To be clear, I would happily sacrifice --fuzzy-image-matching for a native Ruby solution!

So, if there's a way to use Ruby, avoiding the complexity and potential fragility that come from bridging Ruby & Python somehow, at the cost of losing fuzzy matching, then IMO we should seriously consider this option.

Let's seriously consider implementing our own simple pixel-by-pixel equality matcher then!

Note, in passing, that a bug in our fuzzy matching feature (#17029) has made me, and possibly others, waste substantial amounts of time for 2 years, which greatly outweighed the time we saved thanks to batch-updates so far: I started suspecting something was fishy since ~1 year ago (3ac90f2ed9a544ce235bf7e520aafc3a50208c81, 670bd5f6fcb4b30159fe2cf81a4ffe89b67dca2a) but only tracked down & fixed the problem recently.

Eek. Your theory that the default similarity is forgotten in favor of the last explicitly used one could be true. The hack around the comment "Due to bugs in rjb we cannot re-throw Java exceptions" also looks suspicious. Any way...

But that's mostly off-topic because we'll learn from it and ensure that our next implementation of fuzzy matching won't have such problems :)

Indeed. It's worth noting that it is because rjb does an imperfect job at integrating Java into Ruby that implementing the hooks needed for --retry-find and --fuzzy-image-matching have been this error prone. So many hacks... it will be great to get rid of all that mess!

intrigeri:

… and I've just noticed that the very part of our test suite I'm working on today does not behave as intended, precisely because Sikuli matches an image which is not the one we're waiting for. So on my branch for #17056 I've (locally) trying to bump SIKULI_MIN_SIMILARITY to 0.95 to avoid this problem, which might actually fix the painful bug I'm after.

Well, I'm getting sick of Sikuli (boom bada boom tchak, then the rest of the "Sick of Sikuli" song ;)

• With SIKULI_MIN_SIMILARITY = 0.99, Sikuli still matches an image that is not (exactly) on screen AFAICT
• With SIKULI_MIN_SIMILARITY = 1.0, Sikuli does not match an image that is exactly on screen AFAICT (updated it from a failure screenshot I got immediately before)

This suggests that either pixel-perfect image matching won't work for us, or that Sikuli's matching algorithm significantly differs from why my eyes and brain match.
At this point, I'm not sure we can actually check if pixel-perfect image matching can work for us merely by setting SIKULI_MIN_SIMILARITY = 1.0, because I'm losing the faith I had in the fact it would produce the result I understand it should produce :/

Hi @anonym,

(I've sent this comment 5 days ago but just received the rejection message from Redmine, which disliked the facepalm emoji I had included.)

anonym wrote:

intrigeri wrote:

It's not 100% obvious to me that fuzzy image matching should be an absolute requirement here.

Sikuli's image matching is always fuzzy, and and we use a default required similarity of 0.9 (sikuli_settings.MinSimilarity = 0.9), that is what I'm
talking about.

Oops, I had forgotten this.
Looks like my misunderstanding started an interesting discussion and will make us consider another option, so it hopefully it was not only wasted time :)

So at the moment we do not know if a pixel-by-pixel perfect matcher is good enough because we have never done that

Absolutely.

but I guess we could bump the default required similarity to 1.0 and do a full run to get an idea.

I'd be very curious to see the results of such an experiment. Some of our images may be identical across test suite runs but still currently match only thanks to the
current 0.9 number; so we might need to update them to be pixel-perfect; and then the 2nd run and following will give us the answer we want here.

and I fear it could introduce new fun problems like image compression changing pixels ⇒ false negatives.

Just curious: where is image compression involved?

To be clear, I would happily sacrifice --fuzzy-image-matching for a native Ruby solution!

OK, case closed then.

So, if there's a way to use Ruby, avoiding the complexity and potential fragility that come from bridging Ruby & Python somehow, at the cost of losing fuzzy
matching, then IMO we should seriously consider this option.

Let's seriously consider implementing our own simple pixel-by-pixel equality matcher then!

If I understand correctly that the choice is indeed "a shiny new pile of hacks to bridge Python & Ruby, so that we can use OpenCV" vs. "our own simple pixel-by-pixel
equality matcher in Ruby", then yes, I'm leaning towards the latter. Of course, we first need to validate the hypothesis that pixel-perfect matches can work for us: as
you said, at this point we don't know.

I propose this: we put in the work of the machinery needed around the image matching primitive (i.e. remove Sikuli, re-implement some basic versions of classes like Screen, Match, including methods like wait(), findAny() etc (our primitive only gives us the equivalent of exists(), something like @findfailed_hook() etc). Since we want to move away from Sikuli it seems like this is work we must do any way. Once we have all that we have done most of the work (according to my estimation, at least), so it makes sense to do the extra work to evaluate both OpenCV's matchTemplate() and a homegrown pixel-perfect matcher.

Thoughts?

Sounds good to me.

(Also, assigning to me now that we seem to have given up on Sikuli and lamby won't work on this.)

This sounds fair enough to me, given lamby clearly expressed he'd rather not to work directly on our test suite framework, and the current proposal will happen there.
@lamby, I'm sure we can find more exciting Tails work for you than this :)