Strings

Overview

This section explains how to handle Unicode strings with ICU in C and C++.

Sample code is available in the ICU source code library at icu/source/samples/ustring/ustring.cpp .

Text Access Overview

Strings are the most common and fundamental form of handling text in software. Logically, and often physically, they contain contiguous arrays (vectors) of basic units. Most of the ICU API functions work directly with simple strings, and where possible, this is preferred.

Sometimes, text needs to be accessed via more powerful and complicated methods. For example, text may be stored in discontiguous chunks in order to deal with frequent modification (like typing) and large amounts, or it may not be stored in the internal encoding, or it may have associated attributes like bold or italic styles.

Guidance

ICU provides multiple text access interfaces which were added over time. If simple strings cannot be used, then consider the following:

  1. UText: Intended to be the strategic text access API for use with ICU. C API, high performance, writable, supports native indexes for efficient non-UTF-16 text storage.

  2. Replaceable (Java & C++) and UReplaceable (C): Writable, designed for use with Transliterator.

  3. CharacterIterator (Java JDK & C++): Read-only, used in many APIs. Large differences between the JDK and C++ versions.

  4. UCharacterIterator (Java): Back-port of the C++ CharacterIterator to ICU4J for support of supplementary code points and post-increment iteration.

  5. UCharIterator (C): Read-only, C interface used mostly in incremental normalization and collation.

The following provides some historical perspective and comparison between the interfaces.

CharacterIterator

ICU has long provided the CharacterIterator interface for some services. It allows for abstract text access, but has limitations:

  1. It has a per-character function call overhead.

  2. Originally, it was designed for UCS-2 operation and did not support direct handling of supplementary Unicode code points. Such support was later added.

  3. Its pre-increment iteration semantics are uncommon, and are inefficient when used with a variable-width encoding form (UTF-16). Functions for post-increment iteration were added later.

  4. The C++ version added iteration start/limit boundaries only because the C++ UnicodeString copies string contents during substringing; the Java CharacterIterator does not have these extra boundaries – substringing is more efficient in Java.

  5. CharacterIterator is not available for use in C.

  6. CharacterIterator is a read-only interface.

  7. It uses UTF-16 indexes into the text, which is not efficient for other encoding forms.

  8. With the additions to the API over time, the number of methods that have to be overridden by subclasses has become rather large.

The core Java adopted an early version of CharacterIterator; later functionality, like support for supplementary code points, was back-ported from ICU4C to ICU4J to form the UCharacterIterator class.

The UCharIterator C interface was added to allow for incremental normalization and collation in C. It is entirely code unit (UChar)-oriented, uses only post-increment iteration and has a smaller number of overridable methods.

Replaceable

The Replaceable (Java & C++) and UReplaceable (C) interfaces are designed for, and used in, Transliterator. They are random-access interfaces, not iterators.

UText

The UText text access interface was designed as a possible replacement for all previous interfaces listed above, with additional functionality. It allows for high-performance operation through the use of storage-native indexes (for efficient use of non-UTF-16 text) and through accessing multiple characters per function call. Code point iteration is available with functions as well as with C macros, for maximum performance. UText is also writable, mostly patterned after Replaceable. For details see the UText chaper.

Strings in ICU

Strings in Java

In Java, ICU uses the standard String and StringBuffer classes, char[], etc. See the Java documentation for details.

Strings in C/C++

Strings in C and C++ are, at the lowest level, arrays of some particular base type. In most cases, the base type is a char, which is an 8-bit byte in modern compilers. Some APIs use a “wide character” type wchar_t that is typically 8, 16, or 32 bits wide and upwards compatible with char. C code passes char * or wchar_t pointers to the first element of an array. C++ enables you to create a class for encapsulating these kinds of character arrays in handy and safe objects.

The interpretation of the byte or wchar_t values depends on the platform, the compiler, the signed state of both char and wchar_t, and the width of wchar_t. These characteristics are not specified in the language standards. When using internationalized text, the encoding often uses multiple chars for most characters and a wchar_t that is wide enough to hold exactly one character code point value each. Some APIs, especially in the standard library (stdlib), assume that wchar_t strings use a fixed-width encoding with exactly one character code point per wchar_t.

ICU: 16-bit Unicode strings

In order to take advantage of Unicode with its large character repertoire and its well-defined properties, there must be types with consistent definitions and semantics. The Unicode standard defines a default encoding based on 16-bit code units. This is supported in ICU by the definition of the UChar to be an unsigned 16-bit integer type. This is the base type for character arrays for strings in ICU.

:point_right: Note: Endianness is not an issue on this level because the interpretation of an integer is fixed within any given platform.

With the UTF-16 encoding form, a single Unicode code point is encoded with either one or two 16-bit UChar code units (unambiguously). “Supplementary” code points, which are encoded with pairs of code units, are rare in most texts. The two code units are called “surrogates”, and their unit value ranges are distinct from each other and from single-unit value ranges. Code should be generally optimized for the common, single-unit case.

16-bit Unicode strings in internal processing contain sequences of 16-bit code units that may not always be well-formed UTF-16. ICU treats single, unpaired surrogates as surrogate code points, i.e., they are returned in per-code point iteration, they are included in the number of code points of a string, and they are generally treated much like normal, unassigned code points in most APIs. Surrogate code points have Unicode properties although they cannot be assigned an actual character.

ICU string handling functions (including append, substring, etc.) do not automatically protect against producing malformed UTF-16 strings. Most of the time, indexes into strings are naturally at code point boundaries because they result from other functions that always produce such indexes. If necessary, the user can test for proper boundaries by checking the code unit values, or adjust arbitrary indexes to code point boundaries by using the C macros U16_SET_CP_START() and U16_SET_CP_LIMIT() (see utf.h) and the UnicodeString functions getChar32Start() and getChar32Limit().

UTF-8 and UTF-32 are supported with converters (ucnv.h), macros (utf.h), and convenience functions (ustring.h), but only a subset of APIs works with UTF-8 directly as string encoding form.

See the UTF-8 subpage for details about working with UTF-8. Some of the following sections apply to UTF-8 APIs as well; for example sections about handling lengths and overflows.

Separate type for single code points

A Unicode code point is an integer with a value from 0 to 0x10FFFF. ICU 2.4 and later defines the UChar32 type for single code point values as a 32 bits wide signed integer (int32_t). This allows the use of easily testable negative values as sentinels, to indicate errors, exceptions or “done” conditions. All negative values and positive values greater than 0x10FFFF are illegal as Unicode code points.

ICU 2.2 and earlier defined UChar32 depending on the platform: If the compiler’s wchar_t was 32 bits wide, then UChar32 was defined to be the same as wchar_t. Otherwise, it was defined to be an unsigned 32-bit integer. This means that UChar32 was either a signed or unsigned integer type depending on the compiler. This was meant for better interoperability with existing libraries, but was of little use because ICU does not process 32-bit strings — UChar32 is only used for single code points. The platform dependence of UChar32 could cause problems with C++ function overloading.

Compiler-dependent definitions

The compiler’s and the runtime character set’s codepage encodings are not specified by the C/C++ language standards and are usually not a Unicode encoding form. They typically depend on the settings of the individual system, process, or thread. Therefore, it is not possible to instantiate a Unicode character or string variable directly with C/C++ character or string literals. The only safe way is to use numeric values. It is not an issue for User Interface (UI) strings that are translated. These UI strings are loaded from a resource bundle, which is generated from a text file that can be in Unicode or in any other ICU-provided codepage. The binary form of the genrb tool generates UTF-16 strings that are ready for direct use.

There is a useful exception to this for program-internal strings and test strings. Within each “family” of character encodings, there is a set of characters that have the same numeric code values. Such characters include Latin letters, the basic digits, the space, and some punctuation. Most of the ASCII graphic characters are invariant characters. The same set, with different but again consistent numeric values, is invariant among almost all EBCDIC codepages. For details, see icu4c/source/common/unicode/utypes.h . With strings that contain only these invariant characters, it is possible to use efficient ICU constructs to write a C/C++ string literal and use it to initialize Unicode strings.

In some APIs, ICU uses char * strings. This is either for file system paths or for strings that contain invariant characters only (such as locale identifiers). These strings are in the platform-specific encoding of either ASCII or EBCDIC. All other codepage differences do not matter for invariant characters and are manipulated by the C stdlib functions like strcpy().

In some APIs where identifiers are used, ICU uses char * strings with invariant characters. Such strings do not require the full Unicode repertoire and are easier to handle in C and C++ with char * string literals and standard C library functions. Their useful character repertoire is actually smaller than the set of graphic ASCII characters; for details, see utypes.h . Examples of char * identifier uses are converter names, locale IDs, and resource bundle table keys.

There is another, less efficient way to have human-readable Unicode string literals in C and C++ code. ICU provides a small number of functions that allow any Unicode characters to be inserted into a string with escape sequences similar to the one that is used in the C and C++ language. In addition to the familiar \n and \xhh etc., ICU also provides the \uhhhh syntax with four hex digits and the \Uhhhhhhhh syntax with eight hex digits for hexadecimal Unicode code point values. This is very similar to the newer escape sequences used in Java and defined in the latest C and C++ standards. Since ICU is not a compiler extension, the “unescaping” is done at runtime and the backslash itself must be escaped (duplicated) so that the compiler does not attempt to “unescape” the sequence itself.

Handling Lengths, Indexes, and Offsets in Strings

The length of a string and all indexes and offsets related to the string are always counted in terms of UChar code units, not in terms of UChar32 code points. (This is the same as in common C library functions that use char * strings with multi-byte encodings.)

Often, a user thinks of a “character” as a complete unit in a language, like an ‘Ä’, while it may be represented with multiple Unicode code points including a base character and combining marks. (See the Unicode standard for details.) This often requires users to index and pass strings (UnicodeString or UChar *) with multiple code units or code points. It cannot be done with single-integer character types. Indexing of such “characters” is done with the BreakIterator class (in C: ubrk_ functions).

Even with such “higher-level” indexing functions, the actual index values will be expressed in terms of UChar code units. When more than one code unit is used at a time, the index value changes by more than one at a time.

ICU uses signed 32-bit integers (int32_t) for lengths and offsets. Because of internal computations, strings (and arrays in general) are limited to 1G base units or 2G bytes, whichever is smaller.

Using C Strings: NUL-Terminated vs. Length Parameters

Strings are either terminated with a NUL character (code point 0, U+0000) or their length is specified. In the latter case, it is possible to have one or more NUL characters inside the string.

Input string arguments are typically passed with two parameters: The (const) UChar * pointer and an int32_t length argument. If the length is -1 then the string must be NUL-terminated and the ICU function will call the u_strlen() method or treat it equivalently. If the input string contains embedded NUL characters, then the length must be specified.

Output string arguments are typically passed with a destination UChar * pointer and an int32_t capacity argument and the function returns the length of the output as an int32_t. There is also almost always a UErrorCode argument. Essentially, a UChar[] array is passed in with its start and the number of available UChars. The array is filled with the output and if space permits the output will be NUL-terminated. The length of the output string is returned. In all cases the length of the output string does not include the terminating NUL. This is the same behavior found in most ICU and non-ICU string APIs, for example u_strlen(). The output string may contain NUL characters as part of its actual contents, depending on the input and the operation. Note that the UErrorCode parameter is used to indicate both errors and warnings (non-errors). The following describes some of the situations in which the UErrorCode will be set to a non-zero value:

  1. If the output length is greater than the output array capacity, then the UErrorCode will be set to U_BUFFER_OVERFLOW_ERROR and the contents of the output array is undefined.

  2. If the output length is equal to the capacity, then the output has been completely written minus the terminating NUL. This is also indicated by setting the UErrorCode to U_STRING_NOT_TERMINATED_WARNING. Note that U_STRING_NOT_TERMINATED_WARNING does not indicate failure (it passes the U_SUCCESS() macro). Note also that it is more reliable to check the output length against the capacity, rather than checking for the warning code, because warning codes do not cause the early termination of a function and may subsequently be overwritten.

  3. If neither of these two conditions apply, the error code will indicate success and not a U_STRING_NOT_TERMINATED_WARNING. (If a U_STRING_NOT_TERMINATED_WARNING code had been set in the UErrorCode parameter before the function call, then it is reset to a U_ZERO_ERROR.)

Preflighting: The returned length is always the full output length even if the output buffer is too small. It is possible to pass in a capacity of 0 (and an output array pointer of NUL) for “pure preflighting” to determine the necessary output buffer size. Add one to make the output string NUL-terminated.

Note that — whether the caller intends to “preflight” or not — if the output length is equal to or greater than the capacity, then the UErrorCode is set to U_STRING_NOT_TERMINATED_WARNING or U_BUFFER_OVERFLOW_ERROR respectively, as described above.

However, “pure preflighting” is very expensive because the operation has to be processed twice — once for calculating the output length, and a second time to actually generate the output. It is much more efficient to always provide an output buffer that is expected to be large enough for most cases, and to reallocate and repeat the operation only when an overflow occurred. (Remember to reset the UErrorCode to U_ZERO_ERROR before calling the function again.) In C/C++, the initial output buffer can be a stack buffer. In case of a reallocation, it may be possible and useful to cache and reuse the new, larger buffer.

:point_right: Note:The exception to these rules are the ANSI-C-style functions like u_strcpy(), which generally require NUL-terminated strings, forbid embedded NULs, and do not take capacity arguments for buffer overflow checking.

Using Unicode Strings in C

In C, Unicode strings are similar to standard char * strings. Unicode strings are arrays of UChar and most APIs take a UChar * pointer to the first element and an input length and/or output capacity, see above. ICU has a number of functions that provide the Unicode equivalent of the stdlib functions such as strcpy(), strstr(), etc. Compared with their C standard counterparts, their function names begin with u_. Otherwise, their semantics are equivalent. These functions are defined in icu/source/common/unicode/ustring.h.

Code Point Access

Sometimes, Unicode code points need to be accessed in C for iteration, movement forward, or movement backward in a string. A string might also need to be written from code points values. ICU provides a number of macros that are defined in the icu/source/common/unicode/utf.h and utf8.h/utf16.h headers that it includes (utf.h is in turn included with utypes.h).

Macros for 16-bit Unicode strings have a U16_ prefix. For example:

U16_NEXT(s, i, length, c)
U16_PREV(s, start, i, c)
U16_APPEND(s, i, length, c, isError)

There are also macros with a U_ prefix for code point range checks (e.g., test for non-character code point), and U8_ macros for 8-bit (UTF-8) strings. See the header files and the API References for more details.

UTF Macros before ICU 2.4

In ICU 2.4, the utf*.h macros have been revamped, improved, simplified, and renamed. The old macros continue to be available. They are in utf_old.h, together with an explanation of the change. utf.h, utf8.h and utf16.h contain the new macros instead. The new macros are intended to be more consistent, more useful, and less confusing. Some macros were simply renamed for consistency with a new naming scheme.

The documentation of the old macros has been removed. If you need it, see a User Guide version from ICU 4.2 or earlier (see the download page).

C Unicode String Literals

There is a pair of macros that together enable users to instantiate a Unicode string in C — a UChar [] array — from a C string literal:

/*
* In C, we need two macros: one to declare the UChar[] array, and
* one to populate it; the second one is a noop on platforms where
* wchar_t is compatible with UChar and ASCII-based.
* The length of the string literal must be counted for both macros.
*/
/* declare the invString array for the string */
U_STRING_DECL(invString, "such characters are safe 123 %-.", 32);
/* populate it with the characters */
U_STRING_INIT(invString, "such characters are safe 123 %-.", 32);

With invariant characters, it is also possible to efficiently convert char * strings to and from UChar \ strings:

static const char *cs1="such characters are safe 123 %-.";
static UChar us1[40];
static char cs2[40];
u_charsToUChars(cs1, us1, 33); /* include the terminating NUL */
u_UCharsToChars(us1, cs2, 33);

Testing for well-formed UTF-16 strings

It is sometimes useful to test if a 16-bit Unicode string is well-formed UTF-16, that is, that it does not contain unpaired surrogate code units. For a boolean test, call a function like u_strToUTF8() which sets an error code if the input string is malformed. (Provide a zero-capacity destination buffer and treat the buffer overflow error as “is well-formed”.) If you need to know the position of the unpaired surrogate, you can iterate through the string with U16_NEXT() and U_IS_SURROGATE().

Using Unicode Strings in C++

UnicodeString is a C++ string class that wraps a UChar array and associated bookkeeping. It provides a rich set of string handling functions.

UnicodeString combines elements of both the Java String and StringBuffer classes. Many UnicodeString functions are named and work similar to Java String methods but modify the object (UnicodeString is “mutable”).

UnicodeString provides functions for random access and use (insert/append/find etc.) of both code units and code points. For each non-iterative string/code point macro in utf.h there is at least one UnicodeString member function. The names of most of these functions contain “32” to indicate the use of a UChar32.

Code point and code unit iteration is provided by the CharacterIterator abstract class and its subclasses. There are concrete iterator implementations for UnicodeString objects and plain UChar [] arrays.

Most UnicodeString constructors and functions do not have a UErrorCode parameter. Instead, if the construction of a UnicodeString fails, for example when it is constructed from a NULL UChar * pointer, then the UnicodeString object becomes “bogus”. This can be tested with the isBogus() function. A UnicodeString can be put into the “bogus” state explicitly with the setToBogus() function. This is different from an empty string (although a “bogus” string also returns true from isEmpty()) and may be used equivalently to NULL in UChar * C APIs (or null references in Java, or NULL values in SQL). A string remains “bogus” until a non-bogus string value is assigned to it. For complete details of the behavior of “bogus” strings see the description of the setToBogus() function.

Some APIs work with the Replaceable abstract class. It defines a simple interface for random access and text modification and is useful for operations on text that may have associated meta-data (e.g., styled text), especially in the Transliterator API. UnicodeString implements Replaceable.

C++ Unicode String Literals

Like in C, there are macros that enable users to instantiate a UnicodeString from a C string literal. One macro requires the length of the string as in the C macros, the other one implies a strlen().

UnicodeString s1=UNICODE_STRING("such characters are safe 123 %-.", 32);
UnicodeString s1=UNICODE_STRING_SIMPLE("such characters are safe 123 %-.");

It is possible to efficiently convert between invariant-character strings and UnicodeStrings by using constructor, setTo() or extract() overloads that take codepage data (const char *) and specifying an empty string (“”) as the codepage name.

Using C++ Strings in C APIs

The internal buffer of UnicodeString objects is available for direct handling in C (or C-style) APIs that take UChar * arguments. It is possible but usually not necessary to copy the string contents with one of the extract functions. The following describes several direct buffer access methods.

The UnicodeString function getBuffer() const returns a readonly const UChar *. The length of the string is indicated by UnicodeString’s length() function. Generally, UnicodeString does not NUL-terminate the contents of its internal buffer. However, it is possible to check for a NUL character if the length of the string is less than the capacity of the buffer. The following code is an example of how to check the capacity of the buffer: (s.length()<s.getCapacity() && buffer[s.length()]==0)

An easier way to NUL-terminate the buffer and get a const UChar * pointer to it is the getTerminatedBuffer() function. Unlike getBuffer() const, getTerminatedBuffer() is not a const function because it may have to (reallocate and) modify the buffer to append a terminating NUL. Therefore, use getBuffer() const if you do not need a NUL-terminated buffer.

There is also a pair of functions that allow controlled write access to the buffer of a UnicodeString: UChar *getBuffer(int32_t minCapacity) and releaseBuffer(int32_t newLength). UChar *getBuffer(int32_t minCapacity) provides a writeable buffer of at least the requested capacity and returns a pointer to it. The actual capacity of the buffer after the getBuffer(minCapacity) call may be larger than the requested capacity and can be determined with getCapacity().

Once the buffer contents are modified, the buffer must be released with the releaseBuffer(int32_t newLength) function, which sets the new length of the UnicodeString (newLength=-1 can be passed to determine the length of NUL-terminated contents like u_strlen()).

Between the getBuffer(minCapacity) and releaseBuffer(newLength) function calls, the contents of the UnicodeString is unknown and the object behaves like it contains an empty string. A nested getBuffer(minCapacity), getBuffer() const or getTerminatedBuffer() will fail (return NULL) and modifications of the string via UnicodeString member functions will have no effect. Copying a string with an “open buffer” yields an empty copy. The move constructor, move assignment operator and Return Value Optimization (RVO) transfer the state, including the open buffer.

See the UnicodeString API documentation for more information.

Using C Strings in C++ APIs

There are efficient ways to wrap C-style strings in C++ UnicodeString objects without copying the string contents. In order to use C strings in C++ APIs, the UChar * pointer and length need to be wrapped into a UnicodeString. This can be done efficiently in two ways: With a readonly alias and a writable alias. The UnicodeString object that is constructed actually uses the UChar * pointer as its internal buffer pointer instead of allocating a new buffer and copying the string contents.

If the original string is a readonly const UChar *, then the UnicodeString must be constructed with a read only alias. If the original string is a writable (non-const) UChar * and is to be modified (e.g., if the UChar * buffer is an output buffer) then the UnicodeString should be constructed with a writeable alias. For more details see the section “Maximizing Performance with the UnicodeString Storage Model” and search the unistr.h header file for “alias”.

Maximizing Performance with the UnicodeString Storage Model

UnicodeString uses four storage methods to maximize performance and minimize memory consumption:

  1. Short strings are normally stored inside the UnicodeString object. The object has fields for the “bookkeeping” and a small UChar array. When the object is copied, the internal characters are copied into the destination object.
  2. Longer strings are normally stored in allocated memory. The allocated UChar array is preceded by a reference counter. When the string object is copied, the allocated buffer is shared by incrementing the reference counter. If any of the objects that share the same string buffer are modified, they receive their own copy of the buffer and decrement the reference counter of the previously co-used buffer.
  3. A UnicodeString can be constructed (or set with a setTo() function) so that it aliases a readonly buffer instead of copying the characters. In this case, the string object uses this aliased buffer for as long as the object is not modified and it will never attempt to modify or release the buffer. This model has copy-on-write semantics. For example, when the string object is modified, the buffer contents are first copied into writable memory (inside the object for short strings or the allocated buffer for longer strings). When a UnicodeString with a readonly setting is copied to another UnicodeString using the fastCopyFrom() function, then both string objects share the same readonly setting and point to the same storage. Copying a string with the normal assignment operator or copy constructor will copy the buffer. This prevents accidental misuse of readonly-aliased strings. (This is new in ICU 2.4; earlier, the assignment operator and copy constructor behaved like the new fastCopyFrom() does now.) Important:
    1. The aliased buffer must remain valid for as long as any UnicodeString object aliases it. This includes unmodified fastCopyFrom()and movedFrom() copies of the object (including moves via the move constructor and move assignment operator), and when the compiler uses Return Value Optimization (RVO) where a function returns a UnicodeString by value.
    2. Be prepared that return-by-value may either make a copy (which does not preserve aliasing), or moves the value or uses RVO (which do preserve aliasing).
    3. It is an error to readonly-alias temporary buffers and then pass the resulting UnicodeString objects (or references/pointers to them) to APIs that store them for longer than the buffers are valid.
    4. If it is necessary to make sure that a string is not a readonly alias, then use any modifying function without actually changing the contents (for example, s.setCharAt(0, s.charAt(0))).
    5. In ICU 2.4 and later, a simple assignment or copy construction will also copy the buffer.
  4. A UnicodeString can be constructed (or set with a setTo() function) so that it aliases a writable buffer instead of copying the characters. The difference from the above is that the string object writes through to this aliased buffer for write operations. A new buffer is allocated and the contents are copied only when the capacity of the buffer is not sufficient. An efficient way to get the string contents into the original buffer is to use the extract(..., UChar *dst, ...) function. The extract(..., UChar *dst, ...) function copies the string contents only if the dst buffer is different from the buffer of the string object itself. If a string grows and shrinks during a sequence of operations, then it will not use the same buffer, even if the string would fit. When a UnicodeString with a writeable alias is assigned to another UnicodeString, the contents are always copied. The destination string will not point to the buffer that the source string aliases point to. However, a move constructor, move assignment operator, and Return Value Optimization (RVO) do preserve aliasing.

In general, UnicodeString objects have “copy-on-write” semantics. Several objects may share the same string buffer, but a modification only affects the object that is modified itself. This is achieved by copying the string contents if it is not owned exclusively by this one object. Only after that is the object modified.

Even though it is fairly efficient to copy UnicodeString objects, it is even more efficient, if possible, to work with references or pointers. Functions that output strings can be faster by appending their results to a UnicodeString that is passed in by reference, compared with returning a UnicodeString object or just setting the local results alone into a string reference.

:point_right: Note: UnicodeStrings can be copied in a thread-safe manner by just using their standard copy constructors and assignment operators. fastCopyFrom() is also thread-safe, but if the original string is a readonly alias, then the copy shares the same aliased buffer.

Using UTF-8 strings with ICU

As mentioned in the overview of this chapter, ICU and most other Unicode-supporting software uses 16-bit Unicode for internal processing. However, there are circumstances where UTF-8 is used instead. This is usually the case for software that does little or no processing of non-ASCII characters, and/or for APIs that predate Unicode, use byte-based strings, and cannot be changed or replaced for various reasons.

A common perception is that UTF-8 has an advantage because it was designed for compatibility with byte-based, ASCII-based systems, although it was designed for string storage (of Unicode characters in Unix file names) rather than for processing performance.

While ICU mostly does not natively use UTF-8 strings, there are many ways to work with UTF-8 strings and ICU. For more information see the newer UTF-8 subpage.

Using UTF-32 strings with ICU

It is even rarer to use UTF-32 for string processing than UTF-8. While 32-bit Unicode is convenient because it is the only fixed-width UTF, there are few or no legacy systems with 32-bit string processing that would benefit from a compatible format, and the memory bandwidth requirements of UTF-32 diminish the performance and handling advantage of the fixed-width format.

Over time, the wchar_t type of some C/C++ compilers became a 32-bit integer, and some C libraries do use it for Unicode processing. However, application software with good Unicode support tends to have little use for the rudimentary Unicode and Internationalization support of the standard C/C++ libraries and often uses custom types (like ICU’s) and UTF-16 or UTF-8.

For those systems where 32-bit Unicode strings are used, ICU offers some convenience functions.

  1. Conversion of whole strings: u_strFromUTF32() and u_strFromUTF32() in ustring.h.

  2. Access to code points is trivial and does not require any macros.

  3. Using a UTF-32 converter with all of the ICU conversion APIs in ucnv.h, including ones with an “Algorithmic” suffix.

  4. UnicodeString has fromUTF32() and toUTF32() methods.

  5. For conversion directly between UTF-32 and another charset use ucnv_convertEx(). However, since ICU converters work with byte streams in external charsets on the non-“Unicode” side, the UTF-32 string will be treated as a byte stream (UTF-32 Character Encoding Scheme) rather than a sequence of 32-bit code units (UTF-32 Character Encoding Form). The correct converter must be used: UTF-32BE or UTF-32LE according to the platform endianness (U_IS_BIG_ENDIAN). Treating the string like a byte stream also makes a difference in data types (char *), lengths and indexes (counting bytes), and NUL-termination handling (input NUL-termination not possible, output writes only a NUL byte, not a NUL 32-bit code unit). For the difference between internal encoding forms and external encoding schemes see the Unicode Standard.

  6. Some ICU APIs work with a CharacterIterator, a UText or a UCharIterator instead of directly with a C/C++ string parameter. There is currently no ICU instance of any of these interfaces that reads UTF-32, although an application could provide one.

Changes in ICU 2.0

Beginning with ICU release 2.0, there are a few changes to the ICU string facilities compared with earlier ICU releases.

Some of the NUL-termination behavior was inconsistent across the ICU API functions. In particular, the following functions used to count the terminating NUL character in their output length (counted one more before ICU 2.0 than now): ucnv_toUChars, ucnv_fromUChars, uloc_getLanguage, uloc_getCountry, uloc_getVariant, uloc_getName, uloc_getDisplayLanguage, uloc_getDisplayCountry, uloc_getDisplayVariant, uloc_getDisplayName

Some functions used to set an overflow error code even when only the terminating NUL did not fit into the output buffer. These functions now set UErrorCode to U_STRING_NOT_TERMINATED_WARNING rather than to U_BUFFER_OVERFLOW_ERROR.

The aliasing UnicodeString constructors and most extract functions have existed for several releases prior to ICU 2.0. There is now an additional extract function with a UErrorCode parameter. Also, the getBuffer, releaseBuffer and getCapacity functions are new to ICU 2.0.

For more information about these changes, please consult the old and new API documentation.


Table of contents