The Unicode Blog

Tuesday, May 23, 2023

Unicode 15.1 Beta Review Open

The beta review period for Unicode 15.1 has started, and is open until July 4, 2023. The beta is intended primarily for review of character property data and changes to algorithm specifications (Unicode Standard Annexes).

Normally at this phase of a release, the character repertoire is considered stable and very unlikely to change. Also, the plan for Unicode 15.1 had been for a minor release with only a very limited set of new characters.

Recent developments have led to a tentative change in those plans, however.

China has a very urgent need for encoding of certain CJK ideographs used in public services databases. To accommodate this urgent need, the Unicode Technical Committee (UTC) decided at its April 2023 meeting to encode 603 new characters in Unicode 15.1 as CJK Unified Ideographs Extension I. This new block is included in the delta charts for the Unicode 15.1 beta. However, inclusion of these characters in Unicode 15.1 is contingent on support for this addition from China, and on support for this addition in the corresponding ISO/IEC 10646 standard from ISO/IEC JTC 1/SC 2 at their upcoming meeting in June. While support for the new block is anticipated, there is a small chance that minor changes to this repertoire will be made after the beta, or that UTC will pull this block entirely from the 15.1 release.

Several of the Unicode Standard Annexes have significant modifications and associated data changes for version 15.1. For example, UAX #14, Unicode Line Breaking Algorithm has significant enhancements to support line breaking at orthographic syllable boundaries in several South and Southeast Asian scripts. Also, in conjunction with the parallel development of a new standard, UTS #55, Unicode Source Code Handling (see Public Review Issue #474), there are significant revisions to UAX #31, Unicode Identifiers and Syntax that will provide better specifications and guidance related to security, and also improved guidance for applications that define identifier systems using Unicode.

While draft content for the beta has been published as of May 23rd, the work groups preparing updates to the content could continue to make changes to data or specs during the Beta review period. Any substantive changes for the beta will be frozen by June 5th.

Please review the documentation, adjust your code, test the data files, and report errors and other issues to the Unicode Consortium by July 4, 2023. The review period will only be for six weeks, so prompt feedback is appreciated. Feedback instructions are on the beta page.

See https://www.unicode.org/versions/beta-15.1.0.html for more information about testing and providing feedback on the 15.1.0 beta.

See https://www.unicode.org/versions/Unicode15.1.0/ for the current draft summary of Unicode Version 15.1.0.

Support Unicode
To support Unicode’s mission to ensure everyone can communicate in their languages across all devices, please consider adopting a character, making a gift of stock, or making a donation. As Unicode, Inc. is a US-based open source, open standards, non-profit, 501(c)3 organization, your contribution may be eligible for a tax deduction. Please consult with a tax advisor for details.

Tuesday, May 16, 2023

LDML (UTS#35) Part 7: Keyboards

CLDR-TC has authorized a new Public Review Issue, #476, for a major revision of LDML (UTS#35) Part 7: Keyboards. CLDR-TC and CLDR Keyboard-SC would appreciate feedback on whether there are specific changes or enhancements that should be made in the proposed specification.

Today, every platform must independently evaluate, prioritize, and implement all new or updated keyboard layouts, leading to major inconsistencies and delays especially where digitally disadvantaged languages are concerned. Consequently, language communities and other keyboard authors must see their designs developed independently for every platform/operating system, resulting in unnecessary duplication of technical and organizational effort.

“Keyboard 3.0” is designed from the ground up to be usable as a solution to support both hardware and on-screen (touch) layouts for all platforms in a single source file for each language.

With Keyboard 3.0, leading members of the language communities will be able to submit their layout once to CLDR, and it will be available to all platforms as part of the latest version of CLDR, making adoption much easier for platforms. Platform vendors will not need to develop and maintain their own keyboard layout data, especially for languages that they don’t yet support.

This work contributes to the goals of the United Nations International Decade of Indigenous Languages by improving the path for Digitally Disadvantaged Language communities to develop platform support for their languages. Users should see improvements in consistency between platforms, as layouts can be shared.

Tuesday, May 2, 2023

UTC #175 Highlights

by Peter Constable, UTC Chair

We had another productive Unicode Technical Committee (UTC) meeting last week,hosted at Adobe headquarters in downtown San Jose, California. Here are some highlights from the meeting.

Unicode 15.1 Beta

UTC has authorized the Beta release for Unicode 15.1. There were various, relatively minor technical changes to be made based on feedback during the Alpha review period, plus one major change that I’ll describe below. The Beta is scheduled for release on May 23, for a six week public review period to end July 4. That closing date will provide time for working groups to review feedback and provide recommendations for the next UTC meeting July 25 – 27.

CJK Extension I & GB 18030

A major change for Unicode 15.1 that was decided on was to encode 603 characters in a new CJK Unified Ideographs Extension I block. (See L2/23-106.) This was part of long discussions about GB 18030-2022 and Amendment 1 of that standard which China is currently developing. China has an urgent need for these characters, and the draft of their amendment has them allocated in reserved code positions of Unicode and ISO/IEC 10646, which is not viable from the perspective of the international standards. So, UTC has taken initiative to have China's need accommodated in a standards-conforming manner.

There was discussion as to whether the new characters should be added to Unicode 15.1 or to Unicode 16.0: it was generally preferred to wait for 16.0, but 15.1 was tentatively chosen in case that makes a significant difference for China’s process.

UTC recommended the addition of CJK Extension I to the INCITS/CS&I committee (mirror for JTC 1/SC 2—also met last week) who agreed to recommend to SC 2 the addition of that block in Amendment 2 of ISO/IEC 10646. See L2/23-114 and L2/23-115 for more information.

Orthographic syllable support in UAX #14

Another significant addition for Unicode 15.1 is that UTC approved extending UAX #14 Unicode Line Breaking Algorithm to support breaking of various South and Southeast Asian scripts at orthographic syllable boundaries. The algorithm for this is based on a proposal from Norbert Lindenberg and others (see L2/22-086), with details for incorporation into UAX #14 provided by Robin Leroy (see L2/23-072). A prototype implementation had been created as a public review issue (see PRI #472), and feedback had been positive. This will be a very significant enhancement in Unicode 15.1 providing important improvements in support for several South and Southeast Asian scripts.

Unicode display in text terminals

A new UTC project was initiated at this meeting to develop specifications for supporting display of scripts that require complex shaping in text terminals. This was introduced with a presentation by Renzhi Li and Dustin Howett of Microsoft (see L2/23-107). Even though the majority of computing device usage today is via GUIs, text terminals are still used in many scenarios. Thus, there was considerable interest among UTC participants in this proposal. An ad-hoc working group, chaired by Dustin Howett, will be formed to develop specifications. If interested in participating, let me know and I’ll connect you with Dustin.

Full details on these and other outcomes will be provided in the draft minutes that will be available soon (as L2/23-076 in the document registry).

Monday, April 17, 2023

ICU4X 1.2: Now With Text Segmentation and More on Low-Resource Devices

By Shane Carr, Chair of the ICUX Subcommittee

Across the globe, people are coming online with smaller and more varied devices including smartphones, smart watches, and gadgets. An offshoot of the International Components for Unicode (ICU) Committee, the ICU4X Committee is responsible for enabling these next-generation devices to communicate with each other in thousands of languages. Written in Rust, ICU4X brings lightweight, modular, and secure internationalization libraries to low-resource devices and many programming languages.

Since our first big release in September 2022, the ICU4X team has been busy building additional features and infrastructure. Today, the team is excited to announce ICU4X 1.2, featuring the first stable release of the Segmenter component, more Unicode properties, property names, a technology preview of language and script display names, HarfBuzz bindings, CLDR 43, full compliance with the Unicode Bidirectional Algorithm (UAX #9), and many smaller features and improvements to the ICU4X components.

Text segmentation is the process of dividing strings into meaningful units, such as words, sentences, or grapheme clusters (characters). It is a fundamental task in a wide range of applications, including cursor movement, highlighting spans of text, evaluating text for spelling and grammatical correctness, information retrieval, and text layout.

ICU4X 1.2 supports the two standards Unicode Text Segmentation (UAX #29) for word, sentence, and grapheme cluster segmentation and Unicode Line Breaking Algorithm (UAX #14) for line segmentation.

Given ICU4X's focus on being lightweight for deployment in resource-constrained environments, the team focused on ways to reduce data size versus ICU4C. The highest-impact differences come from the use of runtime tailoring (reducing the number of rule tables) and machine learning models (eliminating the need for Southeast Asian word dictionaries). Overall, ICU4X data for segmentation is 20.1% smaller than the equivalent data in ICU4C, and 60.7% smaller for line break segmentation.

In addition to being smaller in size, ICU4X's line and word segmenters are 19.1% and 52.2% faster in non-complex scripts and 46.9% and 32.1% faster in Chinese than the equivalents in ICU4C, respectively.

The machine learning models in ICU4X are used for word and line breaking in Southeast Asian languages including Thai, Lao, Khmer, and Myanmar. The models use an LSTM, are trained on large datasets, and achieve high accuracy while retaining small model size. By leveraging modern computer architecture features such as SIMD, the team optimized the performance of the LSTM inference to be about 3× faster than the naive implementation. However, the dictionary model remains the fastest, about two orders of magnitude faster than the LSTM. ICU4X offers both types of models for clients to choose.

Another focus of ICU4X 1.2 has been to support your text layout stack. A text layout engine requires more than the scope of either ICU4C and ICU4X, but any layout engine requires at least two ICU features: line break segmentation and the ability to correctly order bidirectional text. ICU4X 1.2 supports the segmentation and bidirectional text needs of Skia’s SkParagraph and HarfBuzz.

Finally, ICU4X 1.2 brings a number of smaller features to other components. The experimental Display Names component now supports language and script display names, in addition to region display names; the Properties component supports converting UCD property and value enum discriminants to their long and short names, and vice-versa; and all components have been upgraded to support CLDR 43.

Read the full ICU4X 1.2 release notes and then the ICU4X tutorial to start using ICU4X in your project.

To learn more about the latest release, be sure to attend our ICU4X Virtual Open House this Wednesday, April 19th at 9am PT.

Thursday, April 13, 2023

ICU 73 Released

Unicode® ICU 73 has just been released. ICU is the premier library for software internationalization, used by a wide array of companies and organizations to support the world's languages, implementing both the latest version of the Unicode Standard and of the Unicode locale data (CLDR). ICU 73 updates to CLDR 43 locale data with various additions and corrections.

ICU 73 improves Japanese and Korean short-text line breaking, reduces C++ memory use in date formatting, and promotes the Java person name formatter from tech preview to draft.

ICU 73 and CLDR 43 are minor releases, mostly focused on bug fixes and small enhancements. (The fall CLDR/ICU releases will update to Unicode 15.1 which is planned for September.)

ICU 73 updates to the time zone data version 2023c (March 2023). Note that pre-1970 data for a number of time zones has been removed, as has been the case in the upstream tzdata release since 2021b.

For details, please see https://icu.unicode.org/download/73.

Wednesday, April 12, 2023

Unicode CLDR v43 released

Formatting Person Names
- Completing the data for formatting person names, allowing it to advance out of “tech preview”. For more information on the benefits of this feature, see Background.
Locales
- Adding substantially to the LikelySubtags data: This is used to find the likely writing system and country for a given language, used in normalizing locale identifiers and inheritance. The data has been contributed by SIL.
- Inheritance: Adding components to parentLocales, and documenting the different inheritance for rgScope data, which inherits primarily by region.
Other data updates
- In English, Türkiye is now the primary country name for the country code TR, and Turkey is available as an alternate. Other locales have been reviewed to see whether similar changes would be appropriate.
- Name for the new timezone Ciudad Juárez.
Structure
- Adding some structure and data needed for ICU4X & JavaScript, for calendar eras and parentLocales.
Collation & Searching
- Treat various quote marks as equivalent at a Primary strength, also including Geresh and Gershayim.

To find out more about these and other changes, see the CLDR v43 release page.

Thursday, March 30, 2023

The Unicode CLDR v43 Beta is now available for integration testing

CLDR provides key building blocks for software to support the world's languages (dates, times, numbers, sort-order, etc.). For example, all major browsers and all modern mobile phones use CLDR for language support. (See Who uses CLDR?)

Via the online Survey Tool, contributors supply data for their languages — data that is widely used to support much of the world’s software. This data is also a factor in determining which languages are supported on mobile phones and computer operating systems.

It is important to review the Migration section for changes that might require action by implementations using CLDR directly or indirectly (eg, via ICU), and the Specification changes, since those are new since the Alpha.

We appreciate feedback from both ICU and non-ICU consumers of CLDR data. (The Beta has already been integrated into the development version of ICU.) Feedback can be filed at CLDR Tickets. Any tickets should be filed as soon as possible, because the target release date is 2023 Apr 12, Wed.

CLDR 43 is a limited-submission release, focusing on just a few areas:

Formatting Person Names
- Completing the data for formatting person names, allowing it to advance out of “tech preview”. For more information on the benefits of this feature, see Background.
Locales
- Adding substantially to the LikelySubtags data: This is used to find the likely writing system and country for a given language, used in normalizing locale identifiers and inheritance. The data has been contributed by SIL
- Inheritance: Adding components to parentLocales, and documenting the different inheritance for rgScope data, which inherits primarily by region
Other data updates
- Alternate names for Turkey / Türkiye
- Name for the new timezone Ciudad Juárez
Structure
- Adding some structure and data needed for ICU4X & JavaScript, for calendar eras and parentLocales.
Collation & Searching
- Treat various quote marks as equivalent at a Primary strength, also including Geresh and Gershayim.

To find out more about these and other changes, see the draft CLDR v43 release page, which has information on accessing the date, reviewing charts of the changes, and — importantly — Migration issues.

Wednesday, March 22, 2023

Remembering John H. Jenkins (井作恆)

The Unicode community is greatly saddened and affected by the recent and sudden loss of John H. Jenkins, a long-time colleague and friend. John was most recently the Vice-Chair of the Unicode CJK & Unihan Group. The vast majority of characters in the Unicode Standard are Chinese, Japanese, and Korean (aka Han) ideographs, which are historically used with a broader range of languages. These have been challenging characters to deal with in script encoding, because of significant regional drift over hundreds of years. As an expert in Han ideographs, John contributed a non-trivial amount of work and effort, sometimes needing to make difficult character encoding decisions for the benefit of the large user community.

Many people have worked with John and appreciated his substantial contributions. Here are some reflections from two people who worked with him most closely.

From Lee Collins:

I met John when he joined our team at Apple in 1991. He came from an internship in Apple's Advanced Technology Group (ATG), having graduated in math and ancient Greek at UC Berkeley. In addition to his technical skills, he could read, write and speak Cantonese. All in all, he was a perfect addition to the team, since one of our main tasks was completion of the first version of the Unicode standard, in particular the Unified Han character set. A key component was the database we had built to track all the different Han character encodings, beginning with Xerox, later adding Mac OS version of JIS, GB, Big5, and KSC, then the unified simplified and traditional mappings provided by Mr Zhang Zhoucai of China. The database was a Hypercard stack that ran on a version of Mac OS I cobbled together to allow Chinese, Japanese and Korean text to be edited and displayed simultaneously. John took over management of that system and database and began to learn the arcane art of Chinese character encoding. He also found time to write a Risk-like game based on the classical world. I don't remember the name of that game, but it was a nice diversion from work.

I had been the primary Unicode representative at the first meetings of international experts to refine what became the ISO 10646 Unified Repertoire and Ordering / Unicode V1.0. The group, initially known as the CJK-JRG (Chinese Japanese, Korean Joint Research Group) later became the current IRG. Hoping he would take over my work, I invited John to join one of the early meetings in Hong Kong, November 1991, and he later became the primary representative. John continued to contribute to the IRG and the Unihan database for the rest of his career.

We both joined the ill-fated Taligent effort, where we developed the internationalization classes that later became the foundation for ICU. Those designs were probably one of the few things of value that came out of Taligent. I left Taligent and went back to Apple. John came back sometime later after IBM took it over completely. I was manager of the team charged with developing Apple's first Unicode-based text library, which we called ATSUI (Apple Type Services for Unicode Imaging). It was largely based on the model of text layout developed for Quickdraw GX. John was the engineer charged with developing the library. That role was not a good fit for John's talents, so he moved to the Typography group where he was responsible for the font tools Apple used to develop our Truetype fonts. My team also developed support for complex scripts like Hindi and Thai, so I often used John's tools to create fonts with the required layout tables.

I moved on to other areas of Apple, ceased to work directly with John, and eventually left Apple. But, since 2015 or so, I again became involved in the IRG as the representative for Vietnam. That allowed me to work with John once more in his various capacities on the Unicode Technical Committee, especially his responsibility for the Unihan database and participation in the IRG. I enjoyed being able to work with him again. Knowing the size and complexity of the work he did for Unicode, he will not be easily replaced.

While we had our differences on technical and work issues at times, he was always a kind and thoughtful person. The world is a lesser place without him.

John was much more familiar with Cantonese than Mandarin due to his missionary work in Hong Kong. I think John’s characters, 井作恆, satisfied two criteria: they are close to his name phonetically (zeng2 zok3 hang4) and look like an actual Chinese name. Purely phonetic transcriptions often use a limited set of characters that look obviously foreign. These don't.

From Ken Lunde:

Nothing brought more joy to John than attending IRG (Ideographic Research Group) meetings, particularly when they took place in Chinese-speaking regions, especially Hong Kong, which held a special place in John’s heart. For those who are unaware, the IRG is responsible for reviewing and preparing the thousands of characters in the growing number of CJK Unified Ideographs blocks, which comprise approximately one-third of the total number of characters in the Unicode Standard.

Fun fact: John and I had an unwritten and informal agreement that he would attend these one-week IRG meetings when they took place in Chinese-speaking regions, and I would attend those hosted elsewhere, in a quasi yin and yang relationship. This would completely explain why I have never attended an IRG meeting in a Chinese-speaking region. This relationship was also evident in John’s focus on all things Chinese and my focus on all things Japanese, though both of us performed sufficiently dangerous dabbling in the other language.

John and I began working much more closely together as a result of COVID-19, which necessitated the formation of the Unicode CJK & Unihan Group, with me serving as the Chair, and John serving as the Vice-Chair. This group, which was formed in early 2020, pre-digests proposals and public feedback, interacts with the IRG, and provides its recommendations to the UTC.

[Photo of Ken Lunde and John Jenkins, October 2022]

Please visit John’s obituary to read more about his extraordinary life, or to express condolences to John’s family:

https://www.larkinmortuary.com/obituary/view/john-howard-jenkins/

Tuesday, May 23, 2023

Unicode 15.1 Beta Review Open

Tuesday, May 16, 2023

LDML (UTS#35) Part 7: Keyboards

Tuesday, May 2, 2023