summaryrefslogtreecommitdiffstats
path: root/util/locale_database
Commit message (Collapse)AuthorAgeFilesLines
* Use char16_t in favor of ushort for locale data tablesEdward Welbourne2020-02-171-1/+1
| | | | | | Change-Id: I890dd2b52c1b786db1081744c8ca343baba93de4 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* Allow surrogate pairs for various "single character" locale dataEdward Welbourne2020-02-173-83/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Extract the character in its proper unicode form and encode it in a new single_character_data table of locale data. Record each entry as the range within that table that encodes it. Also added an assertion in the generator script to check that the digits CLDR gives us are a contiguous sequence in increasing order, as has been assumed by the C++ code for some time. Lots of number-formatting code now has to take account of how wide the digits are. This leaves nowhere for updateSystemPrivate() to record values read from sys_locale->query(), so we must always consult that function when accessing these members of the systemData() object. Various internal users of these single-character fields need the system-or-CLDR value rather than the raw CLDR value, so move QLocalePrivate's methods to supply them down to QLocaleData and ensure they check for system values, where appropriate first. This allows us to finally support the Chakma language and script, for whose number system UTF-16 needs surrogate pairs. Costs 10.8 kB in added data, much of it due to adding two new locales that need surrogates to represent digits. [ChangeLog][QtCore][QLocale] Various QLocale methods that returned single QChar values now return QString values to accommodate those locales which need a surrogate pair to represent the (single character) return value. Fixes: QTBUG-69324 Fixes: QTBUG-81053 Change-Id: I481722d6f5ee266164f09031679a851dfa6e7839 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Merge remote-tracking branch 'origin/5.15' into devLiang Qi2020-02-131-0/+1
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: examples/widgets/graphicsview/boxes/scene.h src/corelib/Qt5CoreMacros.cmake src/corelib/Qt6CoreMacros.cmake src/network/ssl/qsslsocket.cpp src/network/ssl/qsslsocket.h src/platformsupport/fontdatabases/windows/qwindowsfontenginedirectwrite.cpp src/testlib/CMakeLists.txt src/testlib/.prev_CMakeLists.txt tests/auto/corelib/tools/qscopeguard/tst_qscopeguard.cpp Disabled building manual tests with CMake for now, because qmake doesn't do it, and it confuses people. Done-With: Alexandru Croitor <alexandru.croitor@qt.io> Done-With: Volker Hilsheimer <volker.hilsheimer@qt.io> Change-Id: I865ae347bd01f4e59f16d007b66d175a52f1f152
| * Enable system locale to skip digit-grouping if configured to do soEdward Welbourne2020-02-031-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On macOS it's possible to configure the system locale to not do digit grouping (separating "thousands", in most western locales); it then returns an empty string when asked for the grouping character, which QLocale's system-configuration then ignored, falling back on using the base UI locale's grouping separator. This could lead to the same separator being used for decimal and grouping, which should never happen, least of all when configured to not group at all. In order to notice when this happens, query() must take care to return an empty QString (as a QVariant, which is then non-null) when it *has* a value for the locale property, and that value is empty, as opposed to a null QVariant when it doesn't find a configured value. The caller can then distinguish the two cases. Furthermore, the group and decimal separators need to be distinct, so we need to take care to avoid cases where the system overrides one with what the CLDR has given for the other and doesn't over-ride that other. Only presently implemented for macOS and MS-Win, since the (other) Unix implementation of the system locale returns single QChar values for the numeric tokens - see QTBUG-69324, QTBUG-81053. Fixes: QTBUG-80459 Change-Id: Ic3fbb0fb86e974604a60781378b09abc13bab15d Reviewed-by: Ulf Hermann <ulf.hermann@qt.io>
* | Separate offsets from sizes in QLocale's dataEdward Welbourne2020-01-303-108/+132
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This enables us to make the sizes quint8 and benefit from the resulting packing, making the locale data smaller. The sizes for long month-name lists (which concatenate twelve names with semicolon as separator) can overflow an 8-bit member, so use quint16 where needed. Re-ordered the data in QLocaleData and QCalendarLocale. Now all long-short(-narrow) families arise in that order; and any standalone is grouped with the one of the same length. (This cost 20 bytes in the date-format table, which optimises out more duplication if short is before long, but the saving in the (smaller) time-format table more than make up for it; and 20 bytes isn't worth the confusion that being inconsistent in ordering might cause.) At the same time, drop trailing semicolons from list entries (which join various names with semicolon) as they're not needed: we know where the end of the list is, because we know the size of the string that results from concatenation. The code that parses such lists can even correctly handle empty entries at the end. Saves 26 kB of data in the compiled binaries. Task-number: QTBUG-81053 Change-Id: If6ccc96a6910828817aa605d10fd814f567ae1e8 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* | Deduplicate locale data tablesEdward Welbourne2020-01-301-22/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some entries in tables were sub-strings (e.g. prefixes) of others. Since we store start-index and length (with no need for terminators), any entry that appears as a sub-string of an earlier entry can be recorded without making a separate copy of its content, just by recording where it appeared as a sub-string of an earlier entry. (Sadly this doesn't apply to month- or day-names and their short-forms: for those, we store ';'-joined lists. Thus, although each short-form is a prefix of its long-form, the short-form is stored in a list with other short-forms; and this is not a prefix of the list of matching long-forms.) The savings are modest (780 bytes at present), but cost us nothing except when running the python script that generates the data files (it takes a little longer now), which usually only happens at a CLDR update. Change-Id: I05bdaa9283365707bac0190ae983b31f074dd6ed Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* | Minor tidy-up in qlocalexml2cpp.pyEdward Welbourne2020-01-301-10/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Split a long line. Use pythonic chained comparison to save some repetition. Comment on a field not currently in actual use. Say "zeros" rather than "0s" in one comment to match another. Added a .h suffix to the main locale data tempfile to match the naming of the tempfiles used for calendar data. Simplify generation of the blank line between Language and Script; and include a matching blank between Script and Country. This adds one blank line to qlocale.h Removed a stray space that misaligned locale data lines. This produces a space-only change in the generated *_data_p.h files. Change-Id: I974a9e8923c3dfd2178855d2cf1d6a5074e130b3 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* | Preserve the case of the exponent separator CLDR suppliesEdward Welbourne2020-01-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We have long (since 4.5.1) coerced it to lower-case, for no readily apparent, much less documented, reason. CLDR says most locales use an upper-case E for this - let's actually use what CLDR says we should use. The code that matches the exponent separator was doing so case-insensitively in any case; that needed adaptation now that the separator's case isn't pre-determined; and, in any case, should have been done using case-folding rather than upper-casing. In the process, removed some spurious checks for "'e' or 'E'" in the result, since the exponent separator is always represented by 'e' (and an 'e' might also be present for the separate reason of its use as a beyond-decimal digit representing fourteen). [ChangeLog][QtCore][QLocale] QLocale::exponential() now preserves the case of the CLDR source, where previously it was lower-cased. Change-Id: Ic9ac02136cff79cb9f136d72141b5dbf54d9e0a6 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* | Ensure we use UTF-8 for the emitted QLocaleXML data fileEdward Welbourne2020-01-291-1/+5
|/ | | | | | | | | | | | Python helpfully uses a sensible locale when stdout is a tty but uses the system (not the filesystem) default encoding, which may be ascii and unable to encode some of the data we need to save. So brute force kludge it to ensure sys.stdout.encoding is UTF-8 when writing the output we'll read as UTF-8 anyway. Task-number: QTBUG-79902 Change-Id: I218dc0ec4c71a6b1b7181db55b018266d803bc58 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* Update CLDR to v36Edward Welbourne2019-10-252-3/+8
| | | | | | | | | | | | | | | | | | | | | | Released on October 4th. Adds Windows names for two time zones, Qyzylorda and Volgograd. Added languages Chickasaw (cic), Muscogee (mus) and Silesian (szl). Norwegian number formatting has flipped back to using colon rather than dot as time separator; it's flipped back and forth over the last several CLDR releases. The dot form is present as a variant, the colon form was long given as the normal pattern, then went away; but now it's back as a contributed draft and that's what we pick up. The MS-Win time-zone ID script was iterating a dict, causing random reshuffling when new entries are added. Fixed that by doing the critical iteration in sorted order. Omitted locales ccp_BD and ccp_IN due to QTBUG-69324. Task-number: QTBUG-79418 Change-Id: I43869ee1810ecc1fe876523947ddcbcddf4e550a Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* Correct some references to corelib/tools/ to say corelib/text/Edward Welbourne2019-10-251-1/+1
| | | | | | | | | | The Unicode data tables moved with QString and friends. So did the locale data generated from CLDR. This amends commit a9aa206b7b8ac4e69f8c46233b4080e00e845ff5. Change-Id: If12f0420b559dcb78993adc00e9f39751bca684a Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Add support for the Islamic Civil calendarSoroush Rabiei2019-08-223-3/+18
| | | | | | | | | | | | | | | | This has its own locale data, extracted from CLDR. This data may potentially be shared with other variants on the Islamic calendar, so is handled by a separate base-class, QHijriCalendar, on which such variants may base their implementations. [ChangeLog][QtCore][QCalendar] Added support for the Islamic Civil calendar, controlled by feature islamiccivilcalendar, with locale data that can be shared with other implementations, controlled by feature hijricalendar. Fixes: QTBUG-56675 Change-Id: Idf32d3da7034baa8ec5e66ef847e59a8a2f31cbd Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Add support for the Jalali (Solar Hijri or Persian) calendarSoroush Rabiei2019-08-213-2/+7
| | | | | | | | | | | | This has its own locale data, extracted from CLDR. [ChangeLog][QtCore][QCalendar] Added support for the Jalali (Persian or Solar Hijri) calendar, controlled by feature jalalicalendar. Fixes: QTBUG-58404 Change-Id: Id5c56a10db05a4fd612aafc01615273db81ec743 Reviewed-by: Paul Wicking <paul.wicking@qt.io> Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Add support for calendars beside GregorianSoroush Rabiei2019-08-203-54/+138
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add QCalendarBackend as a base class for calendar implementations and QCalendar as a facade via which to access it. QDate's implicit implementation of the Gregorian calendar becomes QGregorianCalendar and QDate methods now support choice of calendar. Convert QLocale's CLDR data for month names to a locale-data component of each supported calendar and relevant QLocale methods now support choice of calendar. Adapt Python scripts for locale data generation to extract month name data from CLDR (keeping on version v35.1) into the new calendar-locale files. The locale data for the Gregorian calendar is held in a Roman calendar base, for sharing with other calendars. Add tests for basic uses of the new API. [ChangeLog][QtCore][QCalendar] Added QCalendar to support diverse calendars, supported by implementing QCalendarBackend. [ChangeLog][QtCore][QDate] Allow choice of calendar in various operations, with Gregorian remaining the default. Done-with: Lars Knoll <lars.knoll@qt.io> Done-with: Edward Welbourne <edward.welbourne@qt.io> Fixes: QTBUG-17110 Fixes: QTBUG-950 Change-Id: I9d6278f394269a183aee8156e990cec4d5198ab8 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Extract a large format string as a module constant valueSoroush Rabiei2019-08-081-14/+15
| | | | | | | | | The template for the "This is a generated file" notice made a clumsy intrusion in the code in which it appeared, so split it out as a constant of the module and access it by name where it's used. Change-Id: Ic4dfb8e873078c54410b191654d6c21d082c9016 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* Move text-related code out of corelib/tools/ to corelib/text/Edward Welbourne2019-07-102-11/+11
| | | | | | | | This includes byte array, string, char, unicode, locale, collation and regular expressions. Change-Id: I8b125fa52c8c513eb57a0f1298b91910e5a0d786 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Add data for Windows Time-Zone IDs added in the last two yearsEdward Welbourne2019-07-011-0/+29
| | | | | | | | | | | | | | | | | | | We've not run util/locale_database/cldr2qtimezone.py for a while, so CLDR has had time to add several more zones. Catch up, inserting the new entries in order. Change-Id: I8625548b0f7775958230eccbd89b897d7afed9e9 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Tidy up in cldr2qtimezone.py and document the need to run itEdward Welbourne2019-07-012-12/+11
| | | | | | | | | | | | | | | | | | | It wasn't mentioned in cldr2qlocalexml.py's instructions, so I didn't know to run it. The data it used in an illustration was out of date. Two tests could be combined with no loss. Change-Id: I26e619e6210ea5b1258326fc4bc2b6aee9d6a999 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* cldr2qtimezone.py: report all missing zones, rather than just the firstEdward Welbourne2019-07-011-1/+6
| | | | | | | | | | | | | | | | | | | | | When scanning the CLDR data, the script raised an exception if it didn't recognize a zone ID. Instead, collect up such unrecognized IDs in a list and report them all at the end, so that whoever runs this can do them all in one go, rather than doing one, running the script, doing the next, running the script, ad nauseam. Change-Id: Ia659f1d1c7e1c1b4ccb87cc23828a0588a5bf958 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* Use simpler data structures in cldr2qtimezone.pyEdward Welbourne2019-07-011-169/+165
| | | | | | | | | | | | | | | | | | | | | | Use tuples for the fixed data. The numbering of rows in the data tables isn't part of any public API, so we can change it freely; it is thus unnecessary, as we can just enumerate a tuple of the data values to generate sequential indices on the fly. (Updates to the data shall no longer need to renumber in order to insert entries.) Restore ordering of the data tables, and remove wanton spacing from inside parens, in the process. Change-Id: I59956cfb6191fe729300b57070671b7e66bd0379 Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Separate out the time, zone, date code from corelib/tools/Edward Welbourne2019-06-061-2/+2
| | | | | | | | | | We'll be adding calendar code here as well, and tools/ was getting rather crowded, so it looks like time to move out a reasonably coherent sub-bundle of it all. Change-Id: I7e8030f38c31aa307f519dd918a43fc44baa6aa1 Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* Add locale support for Cebuano and Erzya languages (new in CLDR v35.1)Edward Welbourne2019-05-201-0/+2
| | | | | | | Change-Id: I5d0ee7bc27eeca1c046d442b0410128ea5abbdb3 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
* Suggest name, when available, for unknown codesEdward Welbourne2019-05-202-5/+57
| | | | | | | | | | | | | | | | When parsing the CLDR data, we only handle language, script and territory (which we call country) codes if they are known to our enumdata.py tables. When reporting the rest as unknown, in the content of an actual locale definition (not the likely subtag data), check whether en.xml can resolve the code for us; if it can, report the full name it provides, as a hint to whoever's running the script that an update to enumdata.py may be in order. Change-Id: I9ca1d6922a91d45bc436f4b622e5557261897d7f Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
* Rename util/locale_database/ to include the e that was missingEdward Welbourne2019-05-2015-0/+4137
It was misnamed local_database, quite missing the point of its name. Change-Id: I73a4fdf24f53daac12304de1f443636d89afacb2 Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-22 04:51:18 +0000