ICU 74.1 74.1
Public Member Functions | Static Public Member Functions
icu::Normalizer2 Class Referenceabstract

Unicode normalization functionality for standard Unicode normalization or for using custom mapping tables. More...

#include <normalizer2.h>

Inheritance diagram for icu::Normalizer2:
icu::UObject icu::UMemory icu::FilteredNormalizer2

Public Member Functions

 ~Normalizer2 ()
 Destructor. More...
 
UnicodeString normalize (const UnicodeString &src, UErrorCode &errorCode) const
 Returns the normalized form of the source string. More...
 
virtual UnicodeStringnormalize (const UnicodeString &src, UnicodeString &dest, UErrorCode &errorCode) const =0
 Writes the normalized form of the source string to the destination string (replacing its contents) and returns the destination string. More...
 
virtual void normalizeUTF8 (uint32_t options, StringPiece src, ByteSink &sink, Edits *edits, UErrorCode &errorCode) const
 Normalizes a UTF-8 string and optionally records how source substrings relate to changed and unchanged result substrings. More...
 
virtual UnicodeStringnormalizeSecondAndAppend (UnicodeString &first, const UnicodeString &second, UErrorCode &errorCode) const =0
 Appends the normalized form of the second string to the first string (merging them at the boundary) and returns the first string. More...
 
virtual UnicodeStringappend (UnicodeString &first, const UnicodeString &second, UErrorCode &errorCode) const =0
 Appends the second string to the first string (merging them at the boundary) and returns the first string. More...
 
virtual UBool getDecomposition (UChar32 c, UnicodeString &decomposition) const =0
 Gets the decomposition mapping of c. More...
 
virtual UBool getRawDecomposition (UChar32 c, UnicodeString &decomposition) const
 Gets the raw decomposition mapping of c. More...
 
virtual UChar32 composePair (UChar32 a, UChar32 b) const
 Performs pairwise composition of a & b and returns the composite if there is one. More...
 
virtual uint8_t getCombiningClass (UChar32 c) const
 Gets the combining class of c. More...
 
virtual UBool isNormalized (const UnicodeString &s, UErrorCode &errorCode) const =0
 Tests if the string is normalized. More...
 
virtual UBool isNormalizedUTF8 (StringPiece s, UErrorCode &errorCode) const
 Tests if the UTF-8 string is normalized. More...
 
virtual UNormalizationCheckResult quickCheck (const UnicodeString &s, UErrorCode &errorCode) const =0
 Tests if the string is normalized. More...
 
virtual int32_t spanQuickCheckYes (const UnicodeString &s, UErrorCode &errorCode) const =0
 Returns the end of the normalized substring of the input string. More...
 
virtual UBool hasBoundaryBefore (UChar32 c) const =0
 Tests if the character always has a normalization boundary before it, regardless of context. More...
 
virtual UBool hasBoundaryAfter (UChar32 c) const =0
 Tests if the character always has a normalization boundary after it, regardless of context. More...
 
virtual UBool isInert (UChar32 c) const =0
 Tests if the character is normalization-inert. More...
 
- Public Member Functions inherited from icu::UObject
virtual ~UObject ()
 Destructor. More...
 
virtual UClassID getDynamicClassID () const
 ICU4C "poor man's RTTI", returns a UClassID for the actual ICU class. More...
 

Static Public Member Functions

static const Normalizer2getNFCInstance (UErrorCode &errorCode)
 Returns a Normalizer2 instance for Unicode NFC normalization. More...
 
static const Normalizer2getNFDInstance (UErrorCode &errorCode)
 Returns a Normalizer2 instance for Unicode NFD normalization. More...
 
static const Normalizer2getNFKCInstance (UErrorCode &errorCode)
 Returns a Normalizer2 instance for Unicode NFKC normalization. More...
 
static const Normalizer2getNFKDInstance (UErrorCode &errorCode)
 Returns a Normalizer2 instance for Unicode NFKD normalization. More...
 
static const Normalizer2getNFKCCasefoldInstance (UErrorCode &errorCode)
 Returns a Normalizer2 instance for Unicode toNFKC_Casefold() normalization which is equivalent to applying the NFKC_Casefold mappings and then NFC. More...
 
static const Normalizer2getNFKCSimpleCasefoldInstance (UErrorCode &errorCode)
 Returns a Normalizer2 instance for a variant of Unicode toNFKC_Casefold() normalization which is equivalent to applying the NFKC_Simple_Casefold mappings and then NFC. More...
 
static const Normalizer2getInstance (const char *packageName, const char *name, UNormalization2Mode mode, UErrorCode &errorCode)
 Returns a Normalizer2 instance which uses the specified data file (packageName/name similar to ucnv_openPackage() and ures_open()/ResourceBundle) and which composes or decomposes text according to the specified mode. More...
 

Detailed Description

Unicode normalization functionality for standard Unicode normalization or for using custom mapping tables.

All instances of this class are unmodifiable/immutable. Instances returned by getInstance() are singletons that must not be deleted by the caller. The Normalizer2 class is not intended for public subclassing.

The primary functions are to produce a normalized string and to detect whether a string is already normalized. The most commonly used normalization forms are those defined in http://www.unicode.org/unicode/reports/tr15/ However, this API supports additional normalization forms for specialized purposes. For example, NFKC_Casefold is provided via getInstance("nfkc_cf", COMPOSE) and can be used in implementations of UTS #46.

Not only are the standard compose and decompose modes supplied, but additional modes are provided as documented in the Mode enum.

Some of the functions in this class identify normalization boundaries. At a normalization boundary, the portions of the string before it and starting from it do not interact and can be handled independently.

The spanQuickCheckYes() stops at a normalization boundary. When the goal is a normalized string, then the text before the boundary can be copied, and the remainder can be processed with normalizeSecondAndAppend().

The hasBoundaryBefore(), hasBoundaryAfter() and isInert() functions test whether a character is guaranteed to be at a normalization boundary, regardless of context. This is used for moving from one normalization boundary to the next or preceding boundary, and for performing iterative normalization.

Iterative normalization is useful when only a small portion of a longer string needs to be processed. For example, in ICU, iterative normalization is used by the NormalizationTransliterator (to avoid replacing already-normalized text) and ucol_nextSortKeyPart() (to process only the substring for which sort key bytes are computed).

The set of normalization boundaries returned by these functions may not be complete: There may be more boundaries that could be returned. Different functions may return different boundaries.

Stable:
ICU 4.4

Definition at line 85 of file normalizer2.h.

Constructor & Destructor Documentation

◆ ~Normalizer2()

icu::Normalizer2::~Normalizer2 ( )

Destructor.

Stable:
ICU 4.4

Member Function Documentation

◆ append()

virtual UnicodeString & icu::Normalizer2::append ( UnicodeString first,
const UnicodeString second,
UErrorCode errorCode 
) const
pure virtual

Appends the second string to the first string (merging them at the boundary) and returns the first string.

The result is normalized if both the strings were normalized. The first and second strings must be different objects.

Parameters
firststring, should be normalized
secondstring, should be normalized
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
first
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.

◆ composePair()

virtual UChar32 icu::Normalizer2::composePair ( UChar32  a,
UChar32  b 
) const
virtual

Performs pairwise composition of a & b and returns the composite if there is one.

Returns a composite code point c only if c has a two-way mapping to a+b. In standard Unicode normalization, this means that c has a canonical decomposition to a+b and c does not have the Full_Composition_Exclusion property.

This function is independent of the mode of the Normalizer2. The default implementation returns a negative value.

Parameters
aA (normalization starter) code point.
bAnother code point.
Returns
The non-negative composite code point if there is one; otherwise a negative value.
Stable:
ICU 49

Reimplemented in icu::FilteredNormalizer2.

◆ getCombiningClass()

virtual uint8_t icu::Normalizer2::getCombiningClass ( UChar32  c) const
virtual

Gets the combining class of c.

The default implementation returns 0 but all standard implementations return the Unicode Canonical_Combining_Class value.

Parameters
ccode point
Returns
c's combining class
Stable:
ICU 49

Reimplemented in icu::FilteredNormalizer2.

◆ getDecomposition()

virtual UBool icu::Normalizer2::getDecomposition ( UChar32  c,
UnicodeString decomposition 
) const
pure virtual

Gets the decomposition mapping of c.

Roughly equivalent to normalizing the String form of c on a UNORM2_DECOMPOSE Normalizer2 instance, but much faster, and except that this function returns false and does not write a string if c does not have a decomposition mapping in this instance's data. This function is independent of the mode of the Normalizer2.

Parameters
ccode point
decompositionString object which will be set to c's decomposition mapping, if there is one.
Returns
true if c has a decomposition, otherwise false
Stable:
ICU 4.6

Implemented in icu::FilteredNormalizer2.

◆ getInstance()

static const Normalizer2 * icu::Normalizer2::getInstance ( const char *  packageName,
const char *  name,
UNormalization2Mode  mode,
UErrorCode errorCode 
)
static

Returns a Normalizer2 instance which uses the specified data file (packageName/name similar to ucnv_openPackage() and ures_open()/ResourceBundle) and which composes or decomposes text according to the specified mode.

Returns an unmodifiable singleton instance. Do not delete it.

Use packageName=nullptr for data files that are part of ICU's own data. Use name="nfc" and UNORM2_COMPOSE/UNORM2_DECOMPOSE for Unicode standard NFC/NFD. Use name="nfkc" and UNORM2_COMPOSE/UNORM2_DECOMPOSE for Unicode standard NFKC/NFKD. Use name="nfkc_cf" and UNORM2_COMPOSE for Unicode standard NFKC_CF=NFKC_Casefold.

Parameters
packageNamenullptr for ICU built-in data, otherwise application data package name
name"nfc" or "nfkc" or "nfkc_cf" or "nfkc_scf" or name of custom data file
modenormalization mode (compose or decompose etc.)
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
the requested Normalizer2, if successful
Stable:
ICU 4.4

◆ getNFCInstance()

static const Normalizer2 * icu::Normalizer2::getNFCInstance ( UErrorCode errorCode)
static

Returns a Normalizer2 instance for Unicode NFC normalization.

Same as getInstance(nullptr, "nfc", UNORM2_COMPOSE, errorCode). Returns an unmodifiable singleton instance. Do not delete it.

Parameters
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
the requested Normalizer2, if successful
Stable:
ICU 49

◆ getNFDInstance()

static const Normalizer2 * icu::Normalizer2::getNFDInstance ( UErrorCode errorCode)
static

Returns a Normalizer2 instance for Unicode NFD normalization.

Same as getInstance(nullptr, "nfc", UNORM2_DECOMPOSE, errorCode). Returns an unmodifiable singleton instance. Do not delete it.

Parameters
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
the requested Normalizer2, if successful
Stable:
ICU 49

◆ getNFKCCasefoldInstance()

static const Normalizer2 * icu::Normalizer2::getNFKCCasefoldInstance ( UErrorCode errorCode)
static

Returns a Normalizer2 instance for Unicode toNFKC_Casefold() normalization which is equivalent to applying the NFKC_Casefold mappings and then NFC.

See https://www.unicode.org/reports/tr44/#NFKC_Casefold

Same as getInstance(nullptr, "nfkc_cf", UNORM2_COMPOSE, errorCode). Returns an unmodifiable singleton instance. Do not delete it.

Parameters
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
the requested Normalizer2, if successful
Stable:
ICU 49

◆ getNFKCInstance()

static const Normalizer2 * icu::Normalizer2::getNFKCInstance ( UErrorCode errorCode)
static

Returns a Normalizer2 instance for Unicode NFKC normalization.

Same as getInstance(nullptr, "nfkc", UNORM2_COMPOSE, errorCode). Returns an unmodifiable singleton instance. Do not delete it.

Parameters
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
the requested Normalizer2, if successful
Stable:
ICU 49

◆ getNFKCSimpleCasefoldInstance()

static const Normalizer2 * icu::Normalizer2::getNFKCSimpleCasefoldInstance ( UErrorCode errorCode)
static

Returns a Normalizer2 instance for a variant of Unicode toNFKC_Casefold() normalization which is equivalent to applying the NFKC_Simple_Casefold mappings and then NFC.

See https://www.unicode.org/reports/tr44/#NFKC_Simple_Casefold

Same as getInstance(nullptr, "nfkc_scf", UNORM2_COMPOSE, errorCode). Returns an unmodifiable singleton instance. Do not delete it.

Parameters
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
the requested Normalizer2, if successful
Draft:
This API may be changed in the future versions and was introduced in ICU 74

◆ getNFKDInstance()

static const Normalizer2 * icu::Normalizer2::getNFKDInstance ( UErrorCode errorCode)
static

Returns a Normalizer2 instance for Unicode NFKD normalization.

Same as getInstance(nullptr, "nfkc", UNORM2_DECOMPOSE, errorCode). Returns an unmodifiable singleton instance. Do not delete it.

Parameters
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
the requested Normalizer2, if successful
Stable:
ICU 49

◆ getRawDecomposition()

virtual UBool icu::Normalizer2::getRawDecomposition ( UChar32  c,
UnicodeString decomposition 
) const
virtual

Gets the raw decomposition mapping of c.

This is similar to the getDecomposition() method but returns the raw decomposition mapping as specified in UnicodeData.txt or (for custom data) in the mapping files processed by the gennorm2 tool. By contrast, getDecomposition() returns the processed, recursively-decomposed version of this mapping.

When used on a standard NFKC Normalizer2 instance, getRawDecomposition() returns the Unicode Decomposition_Mapping (dm) property.

When used on a standard NFC Normalizer2 instance, it returns the Decomposition_Mapping only if the Decomposition_Type (dt) is Canonical (Can); in this case, the result contains either one or two code points (=1..4 char16_ts).

This function is independent of the mode of the Normalizer2. The default implementation returns false.

Parameters
ccode point
decompositionString object which will be set to c's raw decomposition mapping, if there is one.
Returns
true if c has a decomposition, otherwise false
Stable:
ICU 49

Reimplemented in icu::FilteredNormalizer2.

◆ hasBoundaryAfter()

virtual UBool icu::Normalizer2::hasBoundaryAfter ( UChar32  c) const
pure virtual

Tests if the character always has a normalization boundary after it, regardless of context.

If true, then the character does not normalization-interact with following characters. In other words, a string containing this character can be normalized by processing portions up to this character and after this character independently. This is used for iterative normalization. See the class documentation for details. Note that this operation may be significantly slower than hasBoundaryBefore().

Parameters
ccharacter to test
Returns
true if c has a normalization boundary after it
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.

◆ hasBoundaryBefore()

virtual UBool icu::Normalizer2::hasBoundaryBefore ( UChar32  c) const
pure virtual

Tests if the character always has a normalization boundary before it, regardless of context.

If true, then the character does not normalization-interact with preceding characters. In other words, a string containing this character can be normalized by processing portions before this character and starting from this character independently. This is used for iterative normalization. See the class documentation for details.

Parameters
ccharacter to test
Returns
true if c has a normalization boundary before it
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.

◆ isInert()

virtual UBool icu::Normalizer2::isInert ( UChar32  c) const
pure virtual

Tests if the character is normalization-inert.

If true, then the character does not change, nor normalization-interact with preceding or following characters. In other words, a string containing this character can be normalized by processing portions before this character and after this character independently. This is used for iterative normalization. See the class documentation for details. Note that this operation may be significantly slower than hasBoundaryBefore().

Parameters
ccharacter to test
Returns
true if c is normalization-inert
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.

◆ isNormalized()

virtual UBool icu::Normalizer2::isNormalized ( const UnicodeString s,
UErrorCode errorCode 
) const
pure virtual

Tests if the string is normalized.

Internally, in cases where the quickCheck() method would return "maybe" (which is only possible for the two COMPOSE modes) this method resolves to "yes" or "no" to provide a definitive result, at the cost of doing more work in those cases.

Parameters
sinput string
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
true if s is normalized
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.

◆ isNormalizedUTF8()

virtual UBool icu::Normalizer2::isNormalizedUTF8 ( StringPiece  s,
UErrorCode errorCode 
) const
virtual

Tests if the UTF-8 string is normalized.

Internally, in cases where the quickCheck() method would return "maybe" (which is only possible for the two COMPOSE modes) this method resolves to "yes" or "no" to provide a definitive result, at the cost of doing more work in those cases.

This works for all normalization modes. It is optimized for UTF-8 for all built-in modes except for FCD. The base class implementation converts to UTF-16 and calls isNormalized().

Parameters
sUTF-8 input string
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
true if s is normalized
Stable:
ICU 60

Reimplemented in icu::FilteredNormalizer2.

◆ normalize() [1/2]

UnicodeString icu::Normalizer2::normalize ( const UnicodeString src,
UErrorCode errorCode 
) const
inline

Returns the normalized form of the source string.

Parameters
srcsource string
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
normalized src
Stable:
ICU 4.4

Definition at line 223 of file normalizer2.h.

◆ normalize() [2/2]

virtual UnicodeString & icu::Normalizer2::normalize ( const UnicodeString src,
UnicodeString dest,
UErrorCode errorCode 
) const
pure virtual

Writes the normalized form of the source string to the destination string (replacing its contents) and returns the destination string.

The source and destination strings must be different objects.

Parameters
srcsource string
destdestination string; its contents is replaced with normalized src
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
dest
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.

◆ normalizeSecondAndAppend()

virtual UnicodeString & icu::Normalizer2::normalizeSecondAndAppend ( UnicodeString first,
const UnicodeString second,
UErrorCode errorCode 
) const
pure virtual

Appends the normalized form of the second string to the first string (merging them at the boundary) and returns the first string.

The result is normalized if the first string was normalized. The first and second strings must be different objects.

Parameters
firststring, should be normalized
secondstring, will be normalized
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
first
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.

◆ normalizeUTF8()

virtual void icu::Normalizer2::normalizeUTF8 ( uint32_t  options,
StringPiece  src,
ByteSink sink,
Edits edits,
UErrorCode errorCode 
) const
virtual

Normalizes a UTF-8 string and optionally records how source substrings relate to changed and unchanged result substrings.

Implemented completely for all built-in modes except for FCD. The base class implementation converts to & from UTF-16 and does not support edits.

Parameters
optionsOptions bit set, usually 0. See U_OMIT_UNCHANGED_TEXT and U_EDITS_NO_RESET.
srcSource UTF-8 string.
sinkA ByteSink to which the normalized UTF-8 result string is written. sink.Flush() is called at the end.
editsRecords edits for index mapping, working with styled text, and getting only changes (if any). The Edits contents is undefined if any error occurs. This function calls edits->reset() first unless options includes U_EDITS_NO_RESET. edits can be nullptr.
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Stable:
ICU 60

Reimplemented in icu::FilteredNormalizer2.

◆ quickCheck()

virtual UNormalizationCheckResult icu::Normalizer2::quickCheck ( const UnicodeString s,
UErrorCode errorCode 
) const
pure virtual

Tests if the string is normalized.

For the two COMPOSE modes, the result could be "maybe" in cases that would take a little more work to resolve definitively. Use spanQuickCheckYes() and normalizeSecondAndAppend() for a faster combination of quick check + normalization, to avoid re-checking the "yes" prefix.

Parameters
sinput string
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
UNormalizationCheckResult
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.

◆ spanQuickCheckYes()

virtual int32_t icu::Normalizer2::spanQuickCheckYes ( const UnicodeString s,
UErrorCode errorCode 
) const
pure virtual

Returns the end of the normalized substring of the input string.

In other words, with end=spanQuickCheckYes(s, ec); the substring UnicodeString(s, 0, end) will pass the quick check with a "yes" result.

The returned end index is usually one or more characters before the "no" or "maybe" character: The end index is at a normalization boundary. (See the class documentation for more about normalization boundaries.)

When the goal is a normalized string and most input strings are expected to be normalized already, then call this method, and if it returns a prefix shorter than the input string, copy that prefix and use normalizeSecondAndAppend() for the remainder.

Parameters
sinput string
errorCodeStandard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.)
Returns
"yes" span end index
Stable:
ICU 4.4

Implemented in icu::FilteredNormalizer2.


The documentation for this class was generated from the following file: