No char events for characters entered via IM without releasing Ctrl+Shift

[johannesv]

Unicode charcters entered in a TextCtrl using an
alternate input method (e.g. cyrillic or chinese) are
reported in wx.EVT_CHAR event's GetUnicodeKey () with a
wrong value.
Example: While GetString() in wx.EVT_TEXT returns a
u'\u0430' in the corresponding wx.EVT_CHAR
GetUnicodeKey() returns 97 (instead of the proper value
for that unicode symbol).

This is an essential bug if someone must use
wx.EVT_CHAR like validators.

[roebling]

What version of wxGTK?

[johannesv]

Oh, sorry for leaving that out. Here is missing this
information:

Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
wx.VERSION: (2, 6, 1, 1, 'pre')
wx.PlatformInfo: ('WXGTK', 'wxGTK', 'unicode', 'gtk2',
'wx-assertions-off')

[roebling]

There were quite a few changes (also in that area) since
wxWidgets 2.6.1. so I'd suggest to wait for wxPython 2.6.3
to be soon and report further if the problem persists.
Ideally reproduce in C++.

[johannesv]

Is there a release plan to be viewed somewhere on the web,
so we can have a look at when 2.6.3 gets released ?

[robind]

Robert, this also happens in the current 2.6 branch CVS and
also is in HEAD. I think that it should be easily seen in
the text sample if the log messages for the KEY/CHAR events
are changed to show a bit more info from the event.

Johannes, 2.6.3 has been in "real soon" mode for a couple
weeks...

[roebling]

I will look into that, probably next week. How have you
reproduced that? I didn't know you speak Russian or Chinese..

[robind]

I don't, (so I have no idea if I typed anything that makes
sense or is not offensive. ;-) )

I tested in two ways. First, right-click on a wxTextCrtl
and select Input Methods --> Cyrillic. Secondly I also
tried by using the KDE Keyboard Layout tool to set a Russian
layout.

[roebling]

Sorry, I didn't think far enough when responding last time.
The point is that if we intercept events from the native
controls, such as most importantly wxTextCrl, we (illegally)
synthesize the wxEVT_CHAR event from the GDK key_press
event. This results in non-sense for Chinese etc. because it
ignores the input methods (IM). We do use the IM when using
our own controls as we then intercept the commit_cb event
from GDK and that event contains the final character after
IM filtering. In short, your problem cannot be fixed right now.

[johannesv]

Hm, that are bad news. Can you see a workaround for this
problem ? I'm a bit confused, since it works on wxMSW and
wxMac. And on GTK using pyGTK for instance it is working
too. I'm still wondering how to capture symbols like the
Euro-Symbol (€), which is a unicode-character too, using a
wx.EVT_CHAR. (that latter one also works on wxMSW and
wxMac). How can I implement a Validator if no reliable
wx.EVT_CHAR is available ?

Thanks,
Johannes

[wojdyr]

#4584 was marked as a duplicate of this ticket

[vadz]

IMO this is a really serious problem. I didn't know about it before but now that I do it seems to me that it makes EVT_CHAR in wxTextCtrl (which is the class it's used with most often probably) totally useless in non-English programs.

I understand nothing in IM stuff but considering that someone did write IM support code in src/gtk/window.cpp I don't quite understand why can't it work for the standard widgets (do we need to intercept some signals before they reach them?).

It could be also interesting to know how does PyGTK implement this if anybody can look at it (I won't do it myself in case I have to work at this code later, I'm not sure about the licencing issues).

[vaclavslavik]

Replying to vadz:

EVT_CHAR in wxTextCtrl (which is the class it's used with most often probably)

Strange, I would think it's the class where it makes the least sense to EVT_CHAR, because wxTextCtrl already handles text input...

I understand nothing in IM stuff but considering that someone did write IM support code in src/gtk/window.cpp I don't quite understand why can't it work for the standard widgets (do we need to intercept some signals before they reach them?).

As Robert said, it's because the text control already uses IM. IM works by buffering keystrokes until they make sense in the particular input method used, then it emits Unicode character. As far as I can tell, the only way to make EVT_CHAR work with wxTextCtrl would be to intercept all inputs in our IM and then "replay" it to GTK+ control's IM. That's not quite simple, because we don't have access to the latter and it would be lot of code for keeping track of things.

And it would only work assuming that a) no IM method is time-dependent (e.g. timeout of internal state) and b) we know all IM-eaten keystrokes (because we must not delay anything else). I don't know if a) holds, but b) certainly doesn't. And asking our IM if it has eaten the keystroke is of no help, because wxTextCtrl can use multiple user-switchable IMs. In short, this route is a receipt for disaster if you care about native feel at all.

IMHO, it should instead be asked why would you ever need to override EVT_CHAR in wxTextCtrl and provide a better, higher-level mechanism for doing it that could actually be implemented on top of GTK+ reasonably easily.

It could be also interesting to know how does PyGTK implement this

GTK+ (and so PyGTK) doesn't make the distinction between EVT_KEYDOWN and EVT_CHAR, so I don't understand the claim that this works in PyGTK.

[vadz]

I re-checked this and, unsurprisingly, it's still broken with 2.9.2. Unfortunately I still have no idea what to do about it, we don't have access to the GtkIMContext used by GtkTreeView (or GtkEntry) internally so we can't connect to its "commit" signal and I don't see how can we get the character(s) being inserted otherwise.

BTW, I'm somewhat surprised that IM input in wxTextCtrl works at all correctly as according to the docs we should be calling  gtk_text_view_im_context_filter_keypress() from our "key_press_event" handler but we don't do it currently.

Still, even though IM does work, generating wrong EVT_CHAR events for it remains a serious problem. Any hints about how to solve it would be welcome.

[HansR]

I have just come across this in wxMSW too, so it is not just a GTK issue. The problem appears to be that GetUnicodeKey() returns exactly the same value as GetRawKey(). For example, pushing F1 returns 112, which equals 'p'. This is the raw key code for F1 on the windows platform. In fact, wxKeyEvent's m_Unicode and m_rawCode fields contain exactly the same value, regardless of the key that was pressed.

Apart from also having this problem in Windows, my code gets keyboard events from a wxPanel, and not a TreeView or other control. This suggests that it is a general key handling problem, and not something that is specific to certain native controls.

One final note, this being broken means that the wxKeyEvent::GetKeyCode() documentation's example just doesn't work. I have had to resort to reading the raw keycode and flags, which makes the code platform specific.

[vadz]

Replying to HansR:

I have just come across this in wxMSW too, so it is not just a GTK issue. The problem appears to be that GetUnicodeKey() returns exactly the same value as GetRawKey().

How exactly can this be reproduced under MSW? I.e. you don't seem to be using any kind of IM but pressing e.g. F12 in the text sample clear shows that it works as expected (code is WXK_F12, Unicode code is 0). So what do you do?

[HansR]

Replying to vadz:

How exactly can this be reproduced under MSW? I.e. you don't seem to be using any kind of IM but pressing e.g. F12 in the text sample clear shows that it works as expected (code is WXK_F12, Unicode code is 0). So what do you do?

In my code, I'm simply getting the key-down and key-up events from a wxPanel child-class. You can see this happen in the keyboard sample. In Visual studio, set a break-point in the MyFrame::OnKeyDown() method. Run it and push F2 (F1 is intercepted elsewhere), look at the contents of the event variable, you will get:
- event {m_x=41 m_y=24 m_keyCode=341 ...} wxKeyEvent &

wxEvent {m_eventObject=0x023d2e48 m_eventType=10045 m_timeStamp=23189046 ...} wxEvent
wxKeyboardState {m_controlDown=false m_shiftDown=false m_altDown=false ...} wxKeyboardState
m_x 41 int
m_y 24 int
m_keyCode 341 long
m_uniChar 113 L'q' wchar_t
m_rawCode 113 unsigned int
m_rawFlags 3932161 unsigned int
ms_classInfo {m_className=0x0077a874 "wxKeyEvent" m_objectSize=68 m_objectConstructor=0x004225b0 ...} wxClassInfo

Note that m_uniChar == m_rawCode, and that its value is the 'q' character.

Actually, I get this in the text sample too, with the method. Setting a breakpoint in the MyTextCtrl::OnKeyDown() method, the value of m_uniChar is not zero, but matches m_rawCode.

I'm using an SVN version of wxWidgets, but I haven't updated the code in several months. My version is somewhere between 2.9.0 and 2.9.2.

[HansR]

Replying to vadz:

Replying to HansR:

I have just come across this in wxMSW too, so it is not just a GTK issue. The problem appears to be that GetUnicodeKey() returns exactly the same value as GetRawKey().

How exactly can this be reproduced under MSW? I.e. you don't seem to be using any kind of IM but pressing e.g. F12 in the text sample clear shows that it works as expected (code is WXK_F12, Unicode code is 0). So what do you do?

I have just updated to the latest SVN code, and it now works as it should. There was an SVN problem that I had to fix before I could update, which is why I didn't do it earlier. Anyway, the bug no longer exists in the MSW version. Maybe someone should check if the problem still exists in the GTK version with the latest cutting edge code.

[mmarsan]

Please, read my comments at:
 http://article.gmane.org/gmane.comp.lib.wxwidgets.devel/142391

[mmarsan]

This patch fixes EVT_CHAR event in GTK+.

wxTextCtrl (single and multiline) and wxComboBox (and other wx controls that use these two) use native GTK controls. With this patch now EVT_CHAR event for these controls is emitted, and can be vetoed.
It takes care of IM context, so not only ANSI characters, but also Unicode characters typed using Shift+Control+U+val are also handled.

It also fixes max-length check before allowing new text.

It needs gtk 2.22 (sep 2010).

It should also work for wx2.8. I didn't tested it.

Note:
When the ANSI keyval is deduced from its Unicode value (at GTKSendIMCharEvent()),
wxConvLocal is used. Perhaps it's better to use the same method as in gtk_wxwindow_commit_cb()

[vadz]

Thanks, I've finally found time to look at this patch and, as usual, I have some remarks:

1. This is the obvious one: the patch breaks compilation for GTK+ < 2.22. This is not surprising because while it adds a test to configure (which is incorrect, as we still need the check for 2.18 too), it doesn't test for it anywhere in the code. This would have to be fixed in any case.
2. I'd prefer to avoid adding a relatively big wxKeyEvent object to each and every wxWindow, of which there can be quite a few. Couldn't we have a global object for this, considering that there is only one keyboard input going on at any time? Adding m_IM_Finished is probably not such a big deal but I'd rather stash it into the same global struct too if we use it anyhow.
3. OTOH I wonder if we shouldn't use a virtual function instead of GTK_IS_ENTRY() and GTK_IS_TEXT_VIEW() tests, IMHO it would be cleaner. Although it would increase wxWindow vtbl size...
4. Why do we call GTKSetIMFinished() in wxComboBox::OnPaste() but in wxTextCtrl::Paste()? Shouldn't we use the same method for both?

I can address the first point if you have troubles with it but I'd be grateful if you could please look at the other ones.

OTOH I will apply the max-length-related part of the patch as it looks fine to me, thanks!

[VZ]

(In [72778]) Prevent pasting too much text into limited length wxTextCtrl in wxGTK.

Improve insert-text signal handler to block pasting text into the control,
which may overflow the specified max length in one action, and not only
entering individual characters.

See #3158.

[mmarsan]

The patch breaks compilation for GTK+ < 2.22, yes. I don't think there are many people using GTK+ < 2.24. Should we care? Instead IMHO configure.in should be updated to require GTK+ > 2.22

I keep the current design (m_imData->lastKeyEvent) adding m_lastKeyEvent also for windows that haven't m_imData. Test it in the keyboard sample.
Replacing per-window with a global var may be a good idea, but let's leave it for a future patch, there are so many better improvements...

GTK_XXX_YYY() macros, do you think those 12 new lines used just once deserve a virtual function? And taking account they are in a C (not C++) signal callback function?

I set GTKSetIMFinished() in wxComboBox::OnPaste() simply because it has no Paste(), and I didn't thought more on it.

For configure.in, yes, I prefer you to deal with it. I don't feel yet confortamble with it. TIA.

[vadz]

Replying to mmarsan:

The patch breaks compilation for GTK+ < 2.22, yes. I don't think there are many people using GTK+ < 2.24.

At first I didn't realize you were serious but apparently you're so let me notice that currently we require GTK+ 2.6 only and we only dropped support for 2.4 relatively recently.

FWIW Debian (which I use) has 2.20.

Should we care?

Yes, especially considering how simple it is to conditionally compile the new code for 2.22+ only.

I keep the current design (m_imData->lastKeyEvent) adding m_lastKeyEvent also for windows that haven't m_imData. Test it in the keyboard sample.
Replacing per-window with a global var may be a good idea, but let's leave it for a future patch, there are so many better improvements...

I'm really reluctant to add that much data to all windows in the system. Chances are we're never going to undo this later...

GTK_XXX_YYY() macros, do you think those 12 new lines used just once deserve a virtual function?

I think it would be more clear. Even if it's a relatively minor point.

And taking account they are in a C (not C++) signal callback function?

It's C++, of course, even if it's extern "C".

I set GTKSetIMFinished() in wxComboBox::OnPaste() simply because it has no Paste(), and I didn't thought more on it.

Such inconsistencies are surprising so I think it should override Paste() if this is the right thing to do.

Thanks!

[mmarsan]

FWIW Debian (which I use) has 2.20.

I answer you with some frequent words from you: "please, update to..."

Should we care?

Yes, especially considering how simple it is to conditionally compile the new code for 2.22+ only.

If you insist in keeping 2.20 compatibility then the patch needs quite rework.

GTK_XXX_YYY() macros, do you think those 12 new lines used just once deserve a virtual function?

I think it would be more clear. Even if it's a relatively minor point.

I strongly disagree. A virtual function implementation would require much more than 12 lines. And will *not* be more clear. Well, perhaps you can cheat your eyes to see two distant parts of code (the function call and its implementation) at once ;)

I set GTKSetIMFinished() in wxComboBox::OnPaste() simply because it has no Paste(), and I didn't thought more on it.

Such inconsistencies are surprising so I think it should override Paste() if this is the right thing to do.

Or move GTKSetIMFinished() from wxTextCtrl::Paste() to OnPaste(), like in wxComboBox.

[vadz]

Replying to mmarsan:

FWIW Debian (which I use) has 2.20.

I answer you with some frequent words from you: "please, update to..."

Updating from one svn revision to the next one is hardly the same thing as updating to a different OS. Anyhow, there is really no question that we're not going to start requiring 2.22 right now, especially not just for this.

Should we care?

Yes, especially considering how simple it is to conditionally compile the new code for 2.22+ only.

If you insist in keeping 2.20 compatibility then the patch needs quite rework.

All its changes should simply be taken inside #ifdef __WXGTK222__. Is this really so difficult?

GTK_XXX_YYY() macros, do you think those 12 new lines used just once deserve a virtual function?

I think it would be more clear. Even if it's a relatively minor point.

I strongly disagree. A virtual function implementation would require much more than 12 lines. And will *not* be more clear. Well, perhaps you can cheat your eyes to see two distant parts of code (the function call and its implementation) at once ;)

The point is that it's easier for a class itself to be responsible for everything it does instead of having to remember that you need to implement part of its behaviour in a completely different place. But ok, forget about this one.

I set GTKSetIMFinished() in wxComboBox::OnPaste() simply because it has no Paste(), and I didn't thought more on it.

Such inconsistencies are surprising so I think it should override Paste() if this is the right thing to do.

Or move GTKSetIMFinished() from wxTextCtrl::Paste() to OnPaste(), like in wxComboBox.

Whatever you think is better, but consistency is important (unless there is a good reason to not have it, of course).

[vadz]

I've tried applying this patch but while, AFAICS, I've made only insignificant modifications, using Ctrl-Shift-U IM doesn't work for me for some reason -- the IME gets cancelled after the first digit. Do you have any idea what could I have broken?

I'm still committing my version as it at least allows to get events for non-ASCII characters from wxTextCtrl and wxComboBox and so (finally!) closes this bug, but it would be nice to make Ctrl-Shift-U work too, of course...

[VZ]

(In [73693]) Refactor wxGTK IM-related code to allow future modifications.

No real changes, just make it possible to use a different IM than the one
allocated in wxWindow for input handling. This will be used in the upcoming
changes to wxTextEntry and the related classes.

See #3158.

[VZ]

(In [73695]) Add IM and full wxEVT_CHAR support to wxTextCtrl and wxComboBox in wxGTK.

Generate wxEVT_CHAR events for non-ASCII characters entered in these controls
by intercepting their insert-text signal.

Also try to use GtkEntry/GtkTextView internal IM objects but unsuccessfully so
far.

Closes #3158.

[mmarsan]

Replying to vadz:

I've tried applying this patch but while, AFAICS, I've made only insignificant modifications, using Ctrl-Shift-U IM doesn't work for me for some reason -- the IME gets cancelled after the first digit. Do you have any idea what could I have broken?

window.cpp:1007 win->m_imKeyEvent = NULL;
This 'NULL' is too premature.
During IM processing, window.cpp:1120 func wxWindowGTK::GTKOnInsertText is called (from textentry or textctrl). Due to that NULL, the caller stops signal emission for "insert-text".
Solution: make imKeyEvent = NULL at wxWindowGTK::GTKOnInsertText().
Be aware that pasting also is a caller that breaks C-S-U.

I'm still committing my version as it at least allows to get events for non-ASCII characters from wxTextCtrl and wxComboBox and so (finally!) closes this bug

Your commit doesn't handle pasting. Are you sure to close this bug?

[vadz]

Thanks for looking at this!

Replying to mmarsan:

Replying to vadz:

I've tried applying this patch but while, AFAICS, I've made only insignificant modifications, using Ctrl-Shift-U IM doesn't work for me for some reason -- the IME gets cancelled after the first digit. Do you have any idea what could I have broken?

window.cpp:1007 win->m_imKeyEvent = NULL;
This 'NULL' is too premature.
During IM processing, window.cpp:1120 func wxWindowGTK::GTKOnInsertText is called (from textentry or textctrl). Due to that NULL, the caller stops signal emission for "insert-text".
Solution: make imKeyEvent = NULL at wxWindowGTK::GTKOnInsertText().

I'm not sure I understand this explanation: GTKOnInsertText() is called from inside GTKIMFilterKeypress() so m_imKeyEvent is non-NULL inside it. In any case, the solution doesn't work for me, if I apply this patch:

src/gtk/window.cpp

@@ -1004 +1004 @@
         // we should send the key_down event anyway.
         const int intercepted_by_IM = win->GTKIMFilterKeypress(gdk_event);
 
-        win->m_imKeyEvent = NULL;
-
         if ( intercepted_by_IM )
         {
             wxLogTrace(TRACE_KEYS, wxT("Key event intercepted by IM"));
@@ -1123 +1121 @@
         return false;
     }
 
-    return GTKDoInsertTextFromIM(text);
+    bool rc = GTKDoInsertTextFromIM(text);
+
+    m_imKeyEvent = NULL;
+
+    return rc;
 }
 
 

nothing changes and it still doesn't work.

Be aware that pasting also is a caller that breaks C-S-U.

I'll worry about it when I can make it work...

I'm still committing my version as it at least allows to get events for non-ASCII characters from wxTextCtrl and wxComboBox and so (finally!) closes this bug

Your commit doesn't handle pasting.

I don't actually understand the problem with pasting because for me pressing Shift+Ins while this input mode is active in pure GTK+ applications just results in a beep and nothing is pasted, i.e. it looks like it simply shouldn't work. But, again, right now this doesn't work at all in wxGTK so I'm not there yet...

Are you sure to close this bug?

Well, the originally reported bug is fixed. Ctrl+Shift+U is a different problem -- although still important one, of course. I'll probably open a new bug for it if I can't fix it soon.

[mmarsan]

GTK allows two kinds of IM:
(a) Press and hold 'Shift' 'Ctrl' and 'U'. Then release only 'U' and type the hexadecimal code for the desired character. Then release 'Shift' and 'Ctrl'.
(b) Press and hold 'Shift' 'Ctrl' and 'U'. Then release every key and type the hexadecimal code for the desired character. Then press 'Enter' or type a space.

If you add a wxTextCtrl to minimal sample, everything works as expected, both (a) and (b) are fine.
With the keyboard sample, only (a) works.
This patch solves it:

Index: samples/keyboard/keyboard.cpp
===================================================================
--- samples/keyboard/keyboard.cpp	(revisión: 73707)
+++ samples/keyboard/keyboard.cpp	(copia de trabajo)
@@ -70,7 +70,7 @@
         if ( m_skipDown )
             event.Skip();
     }
-    void OnKeyUp(wxKeyEvent& event) { LogEvent("KeyUp", event); }
+    void OnKeyUp(wxKeyEvent& event) { LogEvent("KeyUp", event); event.Skip(); }
     void OnChar(wxKeyEvent& event) { LogEvent("Char", event); event.Skip(); }
     void OnCharHook(wxKeyEvent& event)
     {

I've done some more testing and found that pasting also works almost (*) as expected.
Now I think that my original patch may be "overcoded". Perhaps not so many booleans for pasting case. I don't know. If I set them, I had a reason I don't see now. Some more thinking may be needed.

(*) Have some chars in the clipboard. Press S-C-U, release them. Press Ctrl-V. The result is not what I would expect, because the IM keeps alive.

Some cleaning may be needed. Not only for my "overcoded", but also for leftovers such as 'wxGtkIMData' (found twice in all of SVN).

[vadz]

I didn't know about (a), all my testing was done with (b) and you're right, of course, it is indeed due to not skipping the event in OnKeyUp(). However the problem with IM is still not fixed in this case... So let me finally reopen this bug with the new summary.

To be precise, when you use the approach (b), i.e. press Shift+Ctrl+U, 4, 1, Return consequently, you get char event for 'A', as expected. But when you press Shift+Ctrl+, u, 4, 1, i.e. keep Shift and Ctrl pressed while you enter the subsequent keys, you don't get any char events at all even though the same 'A' key does appear in the control.

So while things do work correctly if you use (b), for (a) they're still broken. At least when using GTK+ 2.24.10.

P.S. As for the pasting, I honestly don't see any problems with it not working in the middle of the IM input. Who does this anyhow?

[VZ]

(In [73716]) No changes, just remove wxGtkIMData forward declarations.

This class doesn't exist any more, so clean up references to it.

See #3158.

[vadz]

This is not that high priority any more because we have at least one way of entering Unicode characters now.

Any help with fixing the other way would still be very welcome, of course.