Project

General

Profile

Feature #8281

Consider replacing Florence with GNOME's own on-screen keyboard

Added by intrigeri over 4 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
11/20/2014
Due date:
% Done:

100%

Feature Branch:
bugfix/8281-replace-florence-with-gnome-osk
Type of work:
Code
Blueprint:
Starter:
Affected tool:
On-screen keyboard

Description

Florence is badly integrated into GNOME Shell. Some tricks can probably save us (see notes about it on the parent ticket) for the time being, but on the long run, using the virtual keyboard shipped by default in GNOME could be the best solution. Let's evaluate it.

  1. Install the last Tails release.
  2. Boot from this ISO on bare metal (some virtualization environment make it hard to display Caribou)
  3. Start caribou : Accessibility menu -> Screen keyboard
  4. Try it out and report back

Related issues

Related to Tails - Bug #8010: Florence icon is badly integrated within GNOME Shell Resolved
Related to Tails - Feature #6064: Handheld Tails Confirmed 07/10/2014
Related to Tails - Feature #6202: Evaluate Caribou Rejected 07/28/2013
Related to Tails - Bug #11062: Florence can't be used when GNOME asks for a password Resolved 02/05/2016
Related to Tails - Bug #12385: The virtual keyboard in the Greeter allows to type only a limited set of keys Resolved 03/19/2017
Related to Tails - Bug #14675: GNOME on-screen keyboard is broken without the X11 XTEST extensions Resolved 09/16/2017
Related to Tails - Feature #14713: Discuss and evaluate feedback coming from the Florence removal Resolved 10/03/2017
Related to Tails - Bug #16628: Remove caribou Resolved
Blocks Tails - Feature #13234: Core work 2017Q3: Foundations Team Resolved 06/29/2017

Associated revisions

Revision d12a824b (diff)
Added by intrigeri almost 2 years ago

Install libcaribou-gtk-module to support GTK+ 2.0 in GNOME's on-screen keyboard (refs: #8281).

This change is relevant even if we decide to keep Florence: it improves
accessibility of GTK+ 2.0 applications for users who need this universal access
technology all the time, and might thus prefer GNOME's on-screen keyboard (that
pops up automatically) to Florence (that one has to toggle on/off manually).

Revision 3e5dc780 (diff)
Added by intrigeri almost 2 years ago

Install libgail-common to support accessibility in GTK+ 2.0 (refs: #8281).

This change is relevant even if we decide to keep Florence: it improves
accessibility of:

Revision f40280c8 (diff)
Added by intrigeri almost 2 years ago

Install qt-at-spi to make Qt applications accessible (refs: #8281).

This change is relevant even if we decide to keep Florence: it makes ORCA work
in KeePassX.

Revision 880b13af (diff)
Added by intrigeri almost 2 years ago

Install caribou, needed to make GNOME's on-screen keyboard work in Tor Browser (refs: #8281).

Revision ddc2daa2 (diff)
Added by intrigeri almost 2 years ago

Drop Florence and the corresponding GNOME Shell extension (refs: #8281).

Revision af436cc5 (diff)
Added by intrigeri almost 2 years ago

Adjust doc to the removal of Florence (refs: #8281).

Non-obvious changes include:

  • Adopt the "GNOME screen keyboard" and "screen keyboard" terminology, (as
    opposed to "virtual keyboard"): that's how this feature is called in the
    Accessibility menu and everywhere else in GNOME. The only exception is the
    doc/encryption_and_privacy/virtual_keyboard page, that I didn't bother
    rename: I bet sajolida would rename it again to give it an even better name
    anyway ;)
  • Drop corresponding doc in the intro to GNOME: we only have to present the
    elements visible on screen when starting Tails, and there is no such element
    anymore for the screen keyboard.
  • Drop the picture as I don't think it's needed: once enabled, the screen
    keyboard is so visible one cannot really miss it.

Revision 1b8f81c6 (diff)
Added by intrigeri almost 2 years ago

Test suite: don't wait for the Florence icon anymore (refs: #8281).

We wait for the icons to appear on the desktop, which should be enough.
If not, then we can reintroduce a similar check for another icon.

Revision 36b46453
Added by anonym almost 2 years ago

Merge remote-tracking branch 'origin/bugfix/8281-replace-florence-with-gnome-osk' into devel

Fix-committed: #8281

History

#1 Updated by intrigeri over 4 years ago

  • Tracker changed from Bug to Feature

#2 Updated by intrigeri over 4 years ago

  • Parent task deleted (#8010)

#3 Updated by intrigeri over 4 years ago

  • Related to Bug #8010: Florence icon is badly integrated within GNOME Shell added

#4 Updated by sajolida over 4 years ago

  • Assignee set to sajolida

#5 Updated by intrigeri over 4 years ago

  • Parent task set to #8312

#6 Updated by alant over 4 years ago

I tested caribou but it is qwerty even when I set another kekboard layout in GNOME. I failed to find how to configure caribou keyboard layout. https://git.gnome.org/browse/caribou/tree/data/layouts/fullscale suggests only qwerty is supported, but there is code that reads X keyboard configuration in https://git.gnome.org/browse/caribou/tree/libcaribou/xadapter.vala. gnome-shell code displaying the virtual keyboard is in https://git.gnome.org/browse/gnome-shell/tree/js/ui/keyboard.js

#7 Updated by sajolida over 4 years ago

  • Assignee deleted (sajolida)

Alan: I'm glad to see that you were faster than me on this one. So I'm deassign it from me. Do you think that there is anything else to be tested here?

#8 Updated by sajolida over 4 years ago

Apparently GNOME has plans to integrate a custom screen keyboard in GNOME Shell but they are not there yet:

https://wiki.gnome.org/Design/OS/ScreenKeyboard

#9 Updated by intrigeri over 4 years ago

  • Assignee set to alant
  • Priority changed from Normal to Low
  • QA Check set to Info Needed
  • Type of work changed from Test to Research

Apparently GNOME has plans to integrate a custom screen keyboard in GNOME Shell but they are not there yet:

In jessie caribou is integrated in GNOME Shell (the UI is Clutter drawn by the shell, but libcaribou is behind the scene)

Alan, do you want to research this any further? (probably not a blocker for Tails/Jessie, now that #8010 is solved)

I don't think it's worth spending more hours reading its code and searchine for its nonexistant documentation. So I see two options:

- we give up and keep current status
- we (possibly me) ask the right GNOME developpers for help

The 1st option is the lazy one. The 2nd, if it succeedes, is more integrated into the desktop and eases the integration of a virtual keyboard in the greeter (it might event git it for free).

What do you think?

#10 Updated by anonym over 4 years ago

  • Assignee deleted (alant)
  • Priority changed from Low to Normal
  • QA Check deleted (Info Needed)
  • Type of work changed from Research to Test

My experience with Caribou wasn't so great. It covers the lower half the screen (at least on screens with small vertical size) and may very well block the text field you are editing. It doesn't switch from bottom to top in that case, which otherwise would have been an improvement, possibly.

#11 Updated by intrigeri over 4 years ago

  • Assignee set to alant
  • Priority changed from Normal to Low
  • QA Check set to Info Needed
  • Type of work changed from Test to Research

#12 Updated by intrigeri over 4 years ago

Indeed, the layout issue seems to be well-known. There's been some work on it recently, but it's still probably not as good as what Florence gives us currently:

#13 Updated by intrigeri over 4 years ago

#14 Updated by alant about 4 years ago

  • Assignee deleted (alant)

Unassigning this ticket from me as I don't plan to work on it shortly.

#15 Updated by BitingBird almost 4 years ago

#16 Updated by intrigeri over 3 years ago

  • Subject changed from Evaluate the Caribou virtual keyboard in Tails/Jessie to Evaluate the Caribou virtual keyboard in Tails/Stretch
  • Priority changed from Low to Normal
  • Target version changed from Tails_2.0 to Tails_3.0
  • QA Check deleted (Info Needed)

3 of us have evaluated it on Jessie and agree that it's not good enough => postponing to Stretch.

#17 Updated by intrigeri over 3 years ago

  • Related to Bug #11062: Florence can't be used when GNOME asks for a password added

#18 Updated by intrigeri almost 3 years ago

  • Description updated (diff)

#19 Updated by intrigeri almost 3 years ago

  • Assignee set to intrigeri
  • Target version changed from Tails_3.0 to Tails_4.0

On the Git side, very little has changed in https://git.gnome.org/browse/gnome-shell/log/js/ui/keyboard.js and in Caribou itself, since last time we checked. No progress on the GNOME bug reports listed above. Still, I've tested it again on feature/stretch. It required installing these packages, so that GNOME's "Screen Keyboard" accessibility feature started working: caribou libcaribou-gtk-module libcaribou-gtk3-module. The keyboard layout used in the screen keyboard still does not reflect the one used elsewhere, except for a very small number of those such as German and French. The positioning problem that anonym reported earlier on this ticket is still present.

So I'm postponing this to Buster.

#20 Updated by intrigeri almost 3 years ago

  • Subject changed from Evaluate the Caribou virtual keyboard in Tails/Stretch to Evaluate the Caribou virtual keyboard in Tails/Buster

#21 Updated by intrigeri almost 3 years ago

  • Assignee deleted (intrigeri)

#22 Updated by intrigeri almost 3 years ago

  • Parent task deleted (#8312)

I'm basically done with #8312 so this ticket can't be a subtask of it anymore.

#23 Updated by intrigeri over 2 years ago

  • Related to Bug #12385: The virtual keyboard in the Greeter allows to type only a limited set of keys added

#24 Updated by intrigeri about 2 years ago

  • Subject changed from Evaluate the Caribou virtual keyboard in Tails/Buster to Evaluate the Caribou virtual keyboard in Tails
  • Assignee set to intrigeri
  • Target version changed from Tails_4.0 to Tails_3.2

I'll give it another try, because #12385#note-14 proved that it's much better than I thought, and we're having lots of issues with Florence itself + its poor integration in GNOME. I'm now confident we could drop Florence entirely in 3.2 :)

#25 Updated by intrigeri about 2 years ago

#26 Updated by intrigeri almost 2 years ago

  • Subject changed from Evaluate the Caribou virtual keyboard in Tails to Consider replacing Florence with GNOME's own on-screen keyboard
  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 30
  • Type of work changed from Research to Discuss

So I've looked into this again.

Advantages of dropping Florence (and relying on GNOME's own on-screen keyboard):

  • We get rid of the many issues we have with Florence and its lack of integration into GNOME, e.g. one can type passwords in GNOME Shell password prompts with the on-screen keyboard (#11062).
  • Better support for touch devices: the GNOME on-screen keyboard is enabled by default on such devices, and it pops up automatically when needed instead of having to toggle it on/off manually.
  • We get rid of Florence bugs that affect us, such as https://bugs.debian.org/805895.
  • One less icon in the top bar => leaner, slicker desktop UI.
  • Smaller delta with other GNOME-based distros.
  • One less third-party application that our friends have to maintain in Debian, and for which we have to deal with a not-very-active upstream.

Drawbacks of GNOME's own on-screen keyboard, compared to Florence:

  • The on-screen keyboard layout does not reflect the physical keyboard layout except for very few languages (but accented and other special chars can be input); that's pretty bad, but if it's good enough for other GNOME-based distros, perhaps it's good enough for us? Actually there are two cases:
    • using the on-screen keyboard as a counter-measure against hardware keyloggers: then this is a serious problem as one has to context-switch between the layout of the physical keyboard and the layout of the on-screen keyboard;
    • using the on-screen keyboard for universal access reasons: in this case, one will simply get used to the QWERTY layout, and no context switch is involved, so it's OK-ish IMO.
  • Toggling the on-screen keyboard on/off requires two clicks instead of one; here again, this is a problem only for those who use this tool only occasionally, as a counter-measure against hardware keyloggers; I think we should file a bug report in GNOME to ask for a keyboard shortcut for toggling the on-screen keyboard accessibility feature.
  • The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it's background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it's not a serious blocker.

So all in all, IMO the UX and maintenance cost benefits of switching to GNOME's own on-screen keyboard outweigh the drawbacks. I think we should do it. Let's make a decision during our next monthly meeting and if we decide to do what I'm proposing, I'll implement it in Tails 3.2.

#27 Updated by intrigeri almost 2 years ago

  • Description updated (diff)

#28 Updated by alant almost 2 years ago

I think it would be great to use GNOME OSK. I've discussed this with GNOME people at GUADEC, and I've looked at it. So here we go:

intrigeri wrote:

Drawbacks of GNOME's own on-screen keyboard, compared to Florence:

  • The on-screen keyboard layout does not reflect the physical keyboard layout except for very few languages (but accented and other special chars can be input); that's pretty bad, but if it's good enough for other GNOME-based distros, perhaps it's good enough for us?

That's more a bug than a choice. Actually, GNOME OSK is drawn by the shell, but still uses Caribou for getting its layout. And caribou is unmaintained. But there are gnome-shell developpers that plan to work on dropping the caribou dependency. I investigated the layout issue, and we should meet to discuss that shortly. So on the long term, this has chances to be fixed.

On the short term, there is a script tools/convert_cldr.py in caribou sources that can import layouts from http://www.unicode.org/repos/cldr/trunk/keyboards/android/. Current version misses a few things, but I've turned to something that's close to actually automatically import layouts. I can work on that if we want to import all these layouts for the Tails before we ship the gnome-shell version including the fixes.

  • Toggling the on-screen keyboard on/off requires two clicks instead of one; here again, this is a problem only for those who use this tool only occasionally, as a counter-measure against hardware keyloggers; I think we should file a bug report in GNOME to ask for a keyboard shortcut for toggling the on-screen keyboard accessibility feature.

I'm gonna ask that, stay tuned.

  • The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it's background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it's not a serious blocker.

I don't think that this will change, it's a design decision.

So all in all, IMO the UX and maintenance cost benefits of switching to GNOME's own on-screen keyboard outweigh the drawbacks. I think we should do it.

I agree with you.

#29 Updated by intrigeri almost 2 years ago

I can work on that if we want to import all these layouts for the Tails before we ship the gnome-shell version including the fixes.

Sounds great to me. I wouldn't call it a blocker personally, but it's reassuring that we can rely on this to be fixed some day :)

  • The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it's background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it's not a serious blocker.

I don't think that this will change, it's a design decision.

If possible, it would be nice to link to the reasoning that lead to this decision (I fear that otherwise we're gonna do this reasoning from scratch ourselves, only with less well-thought arguments and thus a potentially sub-optimal conclusion).

#30 Updated by sajolida almost 2 years ago

I agree with all your reasoning and I'm fine with replacing Florence with GNOME Screen Keyboard. I tried to test it myself but it fails to load on 3.0.1.

I get in journalctl -f:

Aug 02 10:38:50 amnesia keepassx[20197]: Failed to load module "gail" 
Aug 02 10:38:50 amnesia Web Content[20130]: Failed to load module "caribou-gtk-module" 
Aug 02 10:38:50 amnesia firefox[14797]: Failed to load module "gail" 
Aug 02 10:38:50 amnesia Web Content[20130]: Failed to load module "gail" 
Aug 02 10:38:50 amnesia gnome-shell[11251]: caribou_group_model_create_group_name: assertion 'group != NULL' failed
Aug 02 10:38:50 amnesia gnome-shell[11251]: caribou_keyboard_model_populate_group: assertion 'group != NULL' failed

How did you test it?

#31 Updated by sajolida almost 2 years ago

  • Affected tool set to On-screen keyboard

#32 Updated by sajolida almost 2 years ago

  • QA Check set to Info Needed

Actually, it's there but only shows up or works in rare places.

  • Places it shows up (expected):
    • Activities overview
    • SSH key password prompt
  • Places it doesn't show up (unexpected):
  • Places where it shows up but I cannot type (unexpected):
    • OpenPGP Applet password prompt, pinentry
    • Password prompt from GPG command line

All these ones involving password prompts are clear blockers to me. I'm fine doing more tests but first I want to check whether I'm missing something obvious here.

#33 Updated by alant almost 2 years ago

sajolida wrote:

  • Places it doesn't show up (unexpected):

Both works on fresh debian/stretch

  • Places where it shows up but I cannot type (unexpected):

[...]

  • Password prompt from GPG command line

Works in debian/stretch with pinentry-gnome3

All these ones involving password prompts are clear blockers to me. I'm fine doing more tests but first I want to check whether I'm missing something obvious here.

Tails seems to be missing libcaribou-gtk-module and pinentry-gnome3.

#34 Updated by alant almost 2 years ago

  • The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it's background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it's not a serious blocker.

I don't think that this will change, it's a design decision.

If possible, it would be nice to link to the reasoning that lead to this decision (I fear that otherwise we're gonna do this reasoning from scratch ourselves, only with less well-thought arguments and thus a potentially sub-optimal conclusion).

See https://wiki.gnome.org/Design/OS/ScreenKeyboard

#35 Updated by sajolida almost 2 years ago

Good that it work in Debian/Stretch, so that's a problem in Tails and can be solved.

I tried to install libcaribou-gtk-module and restart GNOME Shell and the Screen Keyboard would still not show up in all those places.

#36 Updated by intrigeri almost 2 years ago

All these ones involving password prompts are clear blockers to me.

Note that some other password prompts work better than with Florence, so IMO we should weigh them against each other and not look only at regressions in isolation.

#37 Updated by intrigeri almost 2 years ago

Tails seems to be missing libcaribou-gtk-module and pinentry-gnome3.

We made a collective decision (against my opinion but whatever) to ship another pinentry implementation. IIRC the main reason was supporting copy'n'pasting from KeePassX after having been requested for the OpenPGP passphrase. We can certainly revisit this, but it's more complicated than one missing package.

#38 Updated by emmapeel almost 2 years ago

We receive many bug reports of Florence, is a feature peple uses and it is quite buggy...

I think the change will be a good idea.

#39 Updated by nodens almost 2 years ago

course of action decided on last monthly meeting:
  1. intrigeri tries to prepare a branch without florence that doesn't introduce too many regressions, and let people test it early enough before the 3.2 freeze.
  2. intrigeri ensures we discuss the remaining regressions (e.g. pinentry) and what solutions we should implement later on (not blocking on that to drop Florence though.)

#40 Updated by sajolida almost 2 years ago

  • QA Check changed from Info Needed to Dev Needed
  • Type of work changed from Discuss to Code

Discussed during the August 2017 monthly meeting:

https://tails.boum.org/contribute/meetings/201708/

  • emmapeel has seen several reports on buggy Florence
  • It's mostly unmaintained
  • does the pros outwight the cons ?
    • some issues with GNOME OSK, but according to intrigeri, fixable appart from pinentry issue
    • pinentry issue might be worked around by revisiting previous project decision to ship pinentry-gtk2 and not pinentry-gnome3, after weighting the consequences carefully.
  • Course of action:
    • intrigeri tries to prepare a branch without florence that doesn't introduce too many regressions, and let people test it early enough before the 3.2 freeze.
    • intrigeri ensures we discuss the remaining regressions (e.g. pinentry) and what solutions we should implement later on (not blocking on that to drop Florence though.)

#41 Updated by alant almost 2 years ago

There is developpment in GNOME Shell based on the unicode.org CLDR layouts we worked on at GUADEC : https://git.gnome.org//browse/gnome-shell/log/?h=wip/carlosg/osk-cldr

Let me know if you want me to backport these layouts to be used with current Tails.

#42 Updated by intrigeri almost 2 years ago

There is developpment in GNOME Shell based on the unicode.org CLDR layouts we worked on at GUADEC : https://git.gnome.org//browse/gnome-shell/log/?h=wip/carlosg/osk-cldr

Thanks!

Let me know if you want me to backport these layouts to be used with current Tails.

I want to focus on blockers, and I think the layouts issue is not one. If you have time to help in time for 3.2, I suggest you instead (explicitly) pick one of the serious issues mentioned above. I'm not sure when exactly I'll come back to this (probably at some point in the 1st half of September); let's try to coordinate and avoid duplicating work :)

#43 Updated by intrigeri almost 2 years ago

Alan wrote:

  • The GNOME on-screen keyboard uses lots of space when open and cannot be moved, so it can partially hide the GUI one wants to use it for. Thankfully it's background is transparent, and one can still move the window one wants to use the on-screen keyboard for, so IMO it's not a serious blocker.

I don't think that this will change, it's a design decision.

If possible, it would be nice to link to the reasoning that lead to this decision (I fear that otherwise we're gonna do this reasoning from scratch ourselves, only with less well-thought arguments and thus a potentially sub-optimal conclusion).

See https://wiki.gnome.org/Design/OS/ScreenKeyboard

Sorry, I was not able to find what I was asking for there :/ Could you please point me to the exact bits you were referring to?

The only related thing I've seen is When the OSK is displayed, the view should be shifted in order to keep the text cursor visible". So at least they're aware that there is a real-world problem that's not addressed by the current design :)

FWIW the https://github.com/schuhumi/gnome-shell-extension-caribou-resize-workspace extension is intended to workaround this problem. It could be an option for us on the short term.

#44 Updated by intrigeri almost 2 years ago

  • Feature Branch set to bugfix/8281-replace-florence-with-gnome-osk

The topic branch repairs GNOME's on-screen keyboard in:

  • pinentry-gtk2, OpenPGP Applet password prompt, password prompt from gpg command line: installing some missing libraries was enough to fix it, and now GNOME's OSK can read the passphrase prompts.
  • Password fields in Firefox (eg. https://mail.riseup.net/): installing the caribou package did the trick
  • KeePassX

The only other regression that was reported here was about gedit, which works for me flawlessly on Tails 3.1. I expect this core GNOME app to work fine with a11n so I'm surprised it has failed => sajolida, please reproduce and send me the Journal, if you want. Anyway, I don't think this is a blocker.

So we've reached the state when no serious regression is known. I'll call for testing, will update the test suite and documentation, and hopefully will submit for QA in time for 3.2 :)

#45 Updated by intrigeri almost 2 years ago

  • QA Check changed from Dev Needed to Ready for QA
  • Starter deleted (Yes)

Sent call for testing. I'll wait a few days and will send to anonym for review if no serious issue is reported.

Meanwhile:

  • anonym: you might want to do a first quick code review.
  • sajolida: you might want to have a look at the doc update.

#46 Updated by intrigeri almost 2 years ago

The branch passed a full test suite run (modulo #14586 of course) on my personal Jenkins.

#47 Updated by intrigeri almost 2 years ago

  • Assignee changed from intrigeri to anonym

No test report (not very surprising), let's go ahead! Please reassign to sajolida once merged so he can review the doc (not a blocker for the merge IMO as I think my draft is not that bad that it justifies delaying this change by N more months).

#48 Updated by anonym almost 2 years ago

  • Status changed from In Progress to Fix committed
  • Assignee deleted (anonym)
  • % Done changed from 30 to 100

#49 Updated by intrigeri almost 2 years ago

  • Related to Bug #14675: GNOME on-screen keyboard is broken without the X11 XTEST extensions added

#50 Updated by intrigeri almost 2 years ago

  • Related to Feature #14713: Discuss and evaluate feedback coming from the Florence removal added

#51 Updated by anonym almost 2 years ago

  • Status changed from Fix committed to Resolved

#52 Updated by intrigeri over 1 year ago

  • QA Check deleted (Ready for QA)

#53 Updated by intrigeri about 2 months ago

Also available in: Atom PDF