Opened 10 years ago

Last modified 13 months ago

#11594 confirmed defect

Fix wxLocale::GetSystemLanguage() and enhance it for Vista and later

Reported by: nielsm Owned by:
Priority: normal Milestone:
Component: wxMSW Version: 2.8.10
Keywords: wxLocale i18n MUI Cc: vaclavslavik, drschlosser@…
Blocked By: Blocking:
Patch: no

Description

Wanting to collect statistics on user preferred languages, I looked into what wx has to offer for detecting the system default language.

I discovered that the wxLocale::GetSystemLanguage() function has an incorrect implementation on Win32. It uses the Win32 API GetUserDefaultLCID(), which returns the user's preference for time and number formats, nothing related to user interface language.

The correct implementation should use GetSystemDefaultUILanguage() in general, and dynamically check for the availability of either the GetThreadUILanguage() function or the GetSystemPreferredUILanguages() function. Both of those were introduced with Windows Vista and return user preferences for MUI in Windows, while GetSystemDefaultUILanguage() exists since Windows 2000 at least, and returns the system install language.

I am currently working on an implementation using these functions in my software and will create a patch for wx once I have it working.

I have confirmed this in 2.8.10 and 2.9.0.

Change History (9)

comment:1 Changed 10 years ago by vaclavslavik

  • Cc vaclavslavik added

See also high-level overview.

The underlying problem is that we combine two things in wxLocale: the locale (for which GetUserDefaultLCID() is appropriate) and the language. This model is appropriate for Unix, but it doesn't match Vista+ or OS X.

For the record, Vista seems to follow OS X model: the user chooses ordered preference for languages and localized resources are used based on it, without respect to the locale (which, on OS X, is often just "C").

comment:2 Changed 10 years ago by vadz

  • Keywords i18n MUI added
  • Status changed from new to confirmed
  • Summary changed from wxLocale::GetSystemLanguage() uses wrong function on Win32 to Fix wxLocale::GetSystemLanguage() and enhance it for Vista and later

I think there are 2 issues here:

  1. The UI language may not correspond to the current locale. This is presumably rare but clearly may happen and should be fixed just as you propose.
  1. wxLocale only can return a single language and not an (ordered) list of languages. Ideally we'd add a new function to get array of all languages.

Patches for either/both of these would be appreciated, of course, but there is no need to solve both of them at the same time AFAICS.

Note that according to MSDN GetSystemDefaultUILanguage() is not present in Win 9x so we'd need to load it dynamically too unless we officially decide to drop support for those finally.

comment:3 Changed 13 months ago by kgschlosser

  • Cc drschlosser@… added

Since this problem is already reported I figured I would put some code in to be able to replicate the issue at hand. and give explicit instruction on how to replicate.

the code below is in Python and will require wxPython to be used. the issue actually lies in wxWidgets \src\common\intl.cpp line 789

What I have personally tested and does not work properly
Windows Versions: 7, 10
Python Versions: 2.7, 3.5
wxPython Versions: 3.0.2, 4.0.3

from __future__ import print_function
import wx
import ctypes
import locale
from ctypes.wintypes import (
    DWORD,
    WORD,
    INT,
    WCHAR,
    HANDLE
)

LOCALE_NAME_MAX_LENGTH = 85

LOCALE_INVARIANT = 0x007F
LOCALE_USER_DEFAULT =- 0x0400
LOCALE_SYSTEM_DEFAULT = 0x0800

KL_NAMELENGTH = 9

kernel32 = ctypes.windll.Kernel32
user32 = ctypes.windll.User32


if ctypes.sizeof(ctypes.c_long) == ctypes.sizeof(ctypes.c_void_p):
    ULONG_PTR = ctypes.c_ulong
elif ctypes.sizeof(ctypes.c_longlong) == ctypes.sizeof(ctypes.c_void_p):
    ULONG_PTR = ctypes.c_ulonglong
else:
    ULONG_PTR =ctypes. c_ulong


LCID = DWORD
LANGID = WORD
GEOTYPE = DWORD
HKL = HANDLE
DWORD_PTR = ULONG_PTR
PWSTR = ctypes.POINTER(WCHAR)


# LCID GetUserDefaultLCID();
GetUserDefaultLCID = kernel32.GetUserDefaultLCID
GetUserDefaultLCID.restype = LCID

user_default_lcid = GetUserDefaultLCID()
print('kernel32.GetUserDefaultLCID:', user_default_lcid)
if user_default_lcid in locale.windows_locale:
    print(
        'kernel32.GetUserDefaultLCID canonical name:',
        locale.windows_locale[user_default_lcid]
    )

else:
    print('kernel32.GetUserDefaultLCID canonical name: None')
# LCID GetSystemDefaultLCID();
GetSystemDefaultLCID = kernel32.GetSystemDefaultLCID
GetSystemDefaultLCID.restype = LCID

system_default_lcid = GetSystemDefaultLCID()
print()
print('kernel32.GetSystemDefaultLCID:', system_default_lcid)
if system_default_lcid in locale.windows_locale:
    print(
        'kernel32.GetSystemDefaultLCID canonical name:',
        locale.windows_locale[system_default_lcid]
    )

else:
    print('kernel32.GetSystemDefaultLCID canonical name: None')
# int GetUserDefaultLocaleName(
#   LPWSTR lpLocaleName,
#   int    cchLocaleName
# );
GetUserDefaultLocaleName = kernel32.GetUserDefaultLocaleName
GetUserDefaultLocaleName.restype = INT

user_locale_name = (WCHAR * LOCALE_NAME_MAX_LENGTH)()
GetUserDefaultLocaleName(
    ctypes.byref(user_locale_name),
    LOCALE_NAME_MAX_LENGTH
)

print()

print('kernel32.GetUserDefaultLocaleName:', user_locale_name.value)

# int GetSystemDefaultLocaleName(
#   LPWSTR lpLocaleName,
#   int    cchLocaleName
# );
GetSystemDefaultLocaleName = kernel32.GetSystemDefaultLocaleName
GetSystemDefaultLocaleName.restype = INT

system_locale_name = (WCHAR * LOCALE_NAME_MAX_LENGTH)()
GetSystemDefaultLocaleName(
    ctypes.byref(system_locale_name),
    LOCALE_NAME_MAX_LENGTH
)

print('kernel32.GetSystemDefaultLocaleName:', system_locale_name.value)

# LANGID GetUserDefaultUILanguage();
GetUserDefaultUILanguage = kernel32.GetUserDefaultUILanguage
GetUserDefaultUILanguage.restype = LANGID

user_default_ui_language = GetUserDefaultUILanguage()

print()
print('kernel32.GetUserDefaultUILanguage:', user_default_ui_language)
if user_default_ui_language in locale.windows_locale:
    print(
        'kernel32.GetUserDefaultUILanguage canonical name:',
        locale.windows_locale[user_default_ui_language]
    )

else:
    print('kernel32.GetUserDefaultUILanguage canonical name: None')


print()
# LANGID GetSystemDefaultUILanguage();
GetSystemDefaultUILanguage = kernel32.GetSystemDefaultUILanguage
GetSystemDefaultUILanguage.restype = LANGID

system_default_ui_language = GetSystemDefaultUILanguage()

print('kernel32.GetSystemDefaultUILanguage:', system_default_ui_language)

if system_default_ui_language in locale.windows_locale:
    print(
        'kernel32.GetSystemDefaultUILanguage canonical name:',
        locale.windows_locale[system_default_ui_language]
    )
else:
    print('kernel32.GetSystemDefaultUILanguage canonical name: None')


# LANGID GetUserDefaultLangID();
GetUserDefaultLangID = kernel32.GetUserDefaultLangID
GetUserDefaultLangID.restype = LANGID

user_default_langid = GetUserDefaultLangID()

print()
print('kernel32.GetUserDefaultLangID:', user_default_langid)

if user_default_langid in locale.windows_locale:
    print(
        'kernel32.GetUserDefaultLangID canonical name:',
        locale.windows_locale[user_default_langid]
    )

else:
    print('kernel32.GetUserDefaultLangID canonical name: None')

# LANGID GetSystemDefaultLangID();
GetSystemDefaultLangID = kernel32.GetSystemDefaultLangID
GetSystemDefaultLangID.restype = LANGID

system_default_langid = GetSystemDefaultLangID()
print()
print('kernel32.GetSystemDefaultLangID:', system_default_langid)

if system_default_langid in locale.windows_locale:
    print(
        'kernel32.GetSystemDefaultLangID canonical name:',
        locale.windows_locale[system_default_langid]
    )
else:
    print('kernel32.GetSystemDefaultLangID canonical name: None')


# HKL GetKeyboardLayout(
#   DWORD idThread
# );

GetKeyboardLayout = user32.GetKeyboardLayout
GetKeyboardLayout.restype = HKL

def LOWORD(l):
    return WORD(DWORD_PTR(l).value & 0xffff)


keyboard_layout = GetKeyboardLayout(DWORD(0))
keyboard_langid = LANGID(LOWORD(keyboard_layout).value)
print()
print('user32.GetKeyboardLayout:', keyboard_langid.value)
# BOOL GetKeyboardLayoutNameW(
#   LPWSTR pwszKLID
# );

if keyboard_langid.value in locale.windows_locale:
    print(
        'user32.GetKeyboardLayout canonical name:',
        locale.windows_locale[keyboard_langid.value]
    )

else:
    print('user32.GetKeyboardLayout canonical name: None')


wx_default = wx.Locale.GetSystemLanguage()
print('\n')
print('wx.Locale.GetSystemLanguage: ' + str(wx_default))

lang_info = wx.Locale.GetLanguageInfo(wx_default)

print('wx.LanguageInfo.LocaleName:', lang_info.GetLocaleName())
print('wx.LanguageInfo.CanonicalName:', lang_info.CanonicalName)
print('wx.LanguageInfo.Description:', lang_info.Description)
print('wx.LanguageInfo.Language:', lang_info.Language)

The code above leverages ctypes to access the Windows specific API calls to grab the language. Windows has a very complex language/locale setup. I believe this is due to decades of adding and patching.

In order to replicate the issue.
Start -> Control Panel -> Region and Language
Keyboards and Languages tab. Display langiage section. Install and enable
English (United States).
Formats tab. Format drop down. Select English (United States)
Administrative tab. (Now here is something very misleading) Change System Locale button. Set to English (United States). This may require a reboot.
Click the apply button. Run the script above the output will be

kernel32.GetUserDefaultLCID: 1033
kernel32.GetUserDefaultLCID canonical name: en_US

kernel32.GetSystemDefaultLCID: 1031
kernel32.GetSystemDefaultLCID canonical name: de_DE

kernel32.GetUserDefaultLocaleName: en-US
kernel32.GetSystemDefaultLocaleName: de-DE

kernel32.GetUserDefaultUILanguage: 1033
kernel32.GetUserDefaultUILanguage canonical name: en_US

kernel32.GetSystemDefaultUILanguage: 1033
kernel32.GetSystemDefaultUILanguage canonical name: en_US

kernel32.GetUserDefaultLangID: 1033
kernel32.GetUserDefaultLangID canonical name: en_US

kernel32.GetSystemDefaultLangID: 1031
kernel32.GetSystemDefaultLangID canonical name: de_DE

user32.GetKeyboardLayout: 1033
user32.GetKeyboardLayout canonical name: en_US


wx.Locale.GetSystemLanguage: 60
wx.LanguageInfo.LocaleName: English_United States.1252
wx.LanguageInfo.CanonicalName: en_US
wx.LanguageInfo.Description: English (U.S.)
wx.LanguageInfo.Language: 60

Now repeat the steps above making only one change. Change the format
to Hebrew (Israel). Take note this is not changing the the displayed language
It is changing the language format which would be display of date/time, currency things of that nature. Which as best as i figure would be the locale.

rerun the script and the output will be

kernel32.GetUserDefaultLCID: 1037
kernel32.GetUserDefaultLCID canonical name: he_IL

kernel32.GetSystemDefaultLCID: 1031
kernel32.GetSystemDefaultLCID canonical name: de_DE

kernel32.GetUserDefaultLocaleName: he-IL
kernel32.GetSystemDefaultLocaleName: de-DE

kernel32.GetUserDefaultUILanguage: 1033
kernel32.GetUserDefaultUILanguage canonical name: en_US

kernel32.GetSystemDefaultUILanguage: 1033
kernel32.GetSystemDefaultUILanguage canonical name: en_US

kernel32.GetUserDefaultLangID: 1037
kernel32.GetUserDefaultLangID canonical name: he_IL

kernel32.GetSystemDefaultLangID: 1031
kernel32.GetSystemDefaultLangID canonical name: de_DE

user32.GetKeyboardLayout: 1033
user32.GetKeyboardLayout canonical name: en_US


wx.Locale.GetSystemLanguage: 100
wx.LanguageInfo.LocaleName: Hebrew_Israel.1255
wx.LanguageInfo.CanonicalName: he_IL
wx.LanguageInfo.Description: Hebrew
wx.LanguageInfo.Language: 100

now if you notice the returned languages from the Windows API has not changed at all. But the language code from wxWidgets has. This is incorrect behavior

In Windows You have the ability to set the locale, displayed language, keyboard layout and finally the system language all separately. this adds quite a bit of complexity in deciding what to use in order to display the GUI components.

wxLocale.GetSystemLanguage is incorrect in 2 ways. first is it does not return the system language it actually returns the user locale. second is it should not return anything user based it should return the system language as the method name implies. But with respect to returning the default system language this is not something that should be needed. it is for use in non unicode applications as well as being the System "install" language, the defaulted language that windows uses when it is getting installed. It can be change via the Administrative tab and clicking on the Change System Locale button. and the button being labeled as such is very misleading because not only does setting that change the system locale. but it also changes the system language.

Now I specifically used the languages above for a reason. if you set wxLocale
using wx.Locale(wx.Locale.GetSystemLanguage()) and you have the format field set to Hebrew (Israel) the GUI gets completely flipped. including the written text gets displayed right to left. English (United States) is not a language that gets read from right to left and should not be displayed in this manner. But the whole GUI is actually flipped horizontally including the close and min/max buttons in the caption bar. Also if you open a dialog setting the current frame as a parent the displayed text in the dialog is in Hebrew and not the default user displayed language.

I can submit a patch for repairing the single method. But it appears that possibly other parts of wxWidgets might possibly use that method. I am unfamiliar with the code and it would take me a long while to locate any potential issues. As a quick and dirty band-aide I have build a cross reference from Windows LANDID's to wx.LANGUAGE_* codes. Use the ctypes Windows API call to GetUserDefaultUILanguage to get the LANGID feed it into the cross reference and use that output to set the language properly.

I believe the whole of wxLocale as well as wxLanguageInfo needs a code rewrite. I feel this should be done because when dealing with Windows the locale, displayed language, keyboard layout are separate things and using the locale to set the language is incorrect. and using the language to set the locale is also incorrect. I do not know how other operating systems function in respect to this.

The script above has the methods that can be used in determining the keyboard/language/locale.

Thank You for taking the time to go over this problem.

Last edited 13 months ago by kgschlosser (previous) (diff)

comment:4 Changed 13 months ago by vadz

Just to be clear: it's called GetSystemLanguage() but it is supposed to return the language that should be used for the user and has nothing to do with the system default language, whatever this is.

Also, I think we need to take a step back and think about what exactly is this function useful for. AFAIK it should only be used to determine the language to load translations for and from this point of view, it should indeed use ::GetThreadUILanguage(), just as mentioned in the original bug report, as it's more appropriate than ::GetUserDefaultLCID() if the two don't match (and in general they don't, although very often they do, of course).

And ideally we'd use ::GetThreadPreferredUILanguages() to allow selecting the first language for which we have translations available. Again, this is something mentioned in the original bug report and I still think it would be great to implement this.

I'm not sure if there are any other conclusions to draw from the comment:3. In particular I don't think we need to totally rewrite this code.

comment:6 Changed 13 months ago by vaclavslavik

Note that in the past 9 years, something changed: there's now wxTranslations API for loading translations in the way comment:4 and the original report describe. The flipping could be addressed by using RTL information from that, not the locale (and ideally deprecating anything language- related in wxLocale and keeping it for locale information only).

comment:7 Changed 13 months ago by vadz

To be honest, I'm not sure how is wxTranslations supposed to be used if you just want the application to appear in the user's language. It would be nice to modernize samples/internat to show it. Also, presumably, wxLocale still needs to be used if you want to show dates/times etc according to user's preferences, so I'm not completely convinced that language-dependent parts of wxLocale need to be deprecated as this would mean that you would have to explicitly use both it and wxTranslations in the most common case, instead of using just a single class (and forwarding the translations stuff to wxTranslations), which doesn't really look like a big gain for me.

Again, if anybody has any time to look into this (I really don't right now, sorry), I'd recommend starting by replacing ::GetUserDefaultLCID() with ::GetThreadUILanguage() and seeing what problems are left after doing this.

comment:8 Changed 13 months ago by vaclavslavik

To be honest, I'm not sure how is wxTranslations supposed to be used if you just want the application to appear in the user's language.

Parts of wxLocale are implemented in it, answering that… The API is mostly the same (except the addition of GetBestTranslation, which is used internally and which uses OS priority list to determine suitable language for given domain).

lso, presumably, wxLocale still needs to be used if you want to show dates/times etc according to user's preferences, so I'm not completely convinced that language-dependent parts

Locale formatting is not language-dependent. That's the whole point: while UI language handling doesn't map to locale well, has multiple prioritized languages and is disconnected from locale in modern OSes, locale is one, (user-)globally configured and handles formatting etc.

which doesn't really look like a big gain for me.

Clarity would be the gain - separating independent things (UI language, locale) into independent classes. It would fix the long-standing confusion of concepts in wx API that does lead to problems. The very flipping problem above illustrates this, it's a consequence of confusing locale with UI language even among core devs - not in the sense that we're unclear on the concepts (but many users are), but in the sense that there are all too handy, but wrong, tools at our disposal to do the wrong thing automatically. Clearly separating the two concepts would help avoid that.

comment:9 Changed 13 months ago by vadz

Vaclav, I don't really disagree with anything in the comment:8, but I'm not sure what exactly do you suggest should be done. AFAICS we still need to do what I wrote in the comment:7 and even though I don't have time to do anything anyhow right now, I still think it would be useful if you could please explain what (if anything) do you think should be done in more details. TIA!

Note: See TracTickets for help on using tickets.