Understanding unicode

Computers store and process information using numbers. In order to display this information as human-readable text, a character encoding is used to map the numbers to specific characters. Unicode is a standard for encoding, representing, and processing text in multiple languages.

How does unicode work?

The unicode standard assigns a code point, or number, to every character. For example, the code point for the letter A is U+0041 (hexadecimal), while the code point for the emoji grinning face with smiling eyes is U+1F601.

When a unicode-compliant program needs to display a character, it looks up the character's code point and displays the corresponding glyph. For example, the code point U+1F601 might be displayed as a grinning face emoji on one system, while on another system it might be displayed as a series of squares or a question mark if the system does not have a glyph for that code point.

Terminology

When talking about unicode, it's important to understand the difference between characters, code points, code units, graphemes, and glyphs.

  • A character is a abstract unit of text. It is an overloaded term that can mean different things in different contexts.
  • A code point is a number that uniquely identifies a character. For example, the code point for the letter 'A' is U+0041. Code points are written using the "U+" notation.
  • A code unit is the smallest logical piece of a character encoding. In UTF-8, code units are bytes. In UTF-16, code units are 16-bit words.
  • A grapheme, or grapheme cluster is the smallest visual unit of a character. For example, "a", "á" and "🌟” are all graphemes. A grapheme can have multiple code points and also multiple representations. For example, "á" can be encoded as two code points: U+0061 U+0301 (LATIN SMALL LETTER A + COMBINING ACUTE ACCENT) or as a single code point: U+00E1 (LATIN SMALL LETTER A WITH ACUTE).
  • A glyph is a specific graphical representation of a grapheme. The same grapheme can be represented by different glyphs on different systems, which explains why emojis can look different on different devices.

Unicode Transformation Formats (UTF)

There are four unicode text formats: UTF-8, UTF-16, UTF-32, and UCS-2.

  • UTF-8 is the most common character encoding on the web. It uses one byte for most ASCII characters, and up to four bytes for other characters.
  • UTF-16 is used by Java, JavaScript, and Windows. It uses two bytes for most characters, and four bytes for others.
  • UTF-32 uses four bytes for all characters. It is not as widely used as the other two formats.

Why you would chooose one format over another depends on the needs of your application. For example, UTF-16 is more efficient for applications that heavily use characters from the Basic Multilingual Plane (BMP), while UTF-8 is more efficient for applications that use a lot of ASCII characters or that need to save space. UTF-32, on the other hand, is a good choice if you need to support all Unicode characters or if you need random access to individual characters.

UTF-8

UTF-8 is the most common character encoding on the web. It uses one byte for most ASCII characters, and up to four bytes for other characters.

Number of Bytes Bits Min. Code Point Max. Code Point Byte 1 Byte 2 Byte 3 Byte 4
1 7 U+0000 U+007F 0xxxxxxx
2 11 U+0080 U+07FF 110xxxxx 10xxxxxx
3 16 U+0800 U+FFFF 1110xxxx 10xxxxxx 10xxxxxx
4 21 U+10000 U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Here are some examples:

Character Code Point Byte 1 Byte 2 Byte 3 Byte 4
A U+0041 01000001
U+20AC 00100000 10101100 00100000
🌟 U+1F31F 00111110 00011111 00010010 11110000

The number of bytes is determined by the code point of the character. The most significant bit (MSB) of the first byte is always set to 0, while the MSB of subsequent bytes is set to 1. The remaining bits of each byte are used to store the code point data.

UTF-8 has the nice property that ASCII characters are encoded using a single byte, while non-ASCII characters are encoded using multiple bytes, so there is no ambiguity about where one character starts and another character ends.

It also has the advantage of being backwards compatible with ASCII, which means that any ASCII string is also a valid UTF-8 string.

UTF-8 has become the de facto standard for character encoding on the web. When in doubt, use UTF-8.

UTF-16

UTF-16 is used by Java, JavaScript, and Windows. It uses two bytes for most characters, and four bytes for others.

Number of Bytes Bits Min. Code Point Max. Code Point Byte 1 Byte 2 Byte 3 Byte 4
2 16 U+0000 U+FFFF xxxxxxxx xxxxxxxx
4 21 U+10000 U+10FFFF 110110xx xxxxxxxxx 110111xx xxxxxxxxx

Here are some examples:

Character Code Point Byte 1 Byte 2 Byte 3 Byte 4
A U+0041 00000000 01000001
U+20AC 00100000 10101100
🌟 U+1F31F 11110000 10011111 00100000 11110001

UTF-16 is a variable-width encoding, which means that the number of bytes used for each character can vary. The first byte of a UTF-16 code unit is called the lead byte, while the second byte is called the trail byte.

If a code point is between U+0000 and U+FFFF, it is encoded as two bytes, with the lead byte being 0x00 and the trail byte being the code point itself. For example, the code point U+0041 (LATIN SMALL LETTER A) is encoded as 0x0041.

If a code point is between U+10000 and U+10FFFF, it is encoded as four bytes. The first two bytes are the high surrogate, while the last two bytes are the low surrogate. The high surrogate is in the range 0xD800-0xDBFF, while the low surrogate is in the range 0xDC00-0xDFFF.

To decode a UTF-16 code unit, we need to check if it is a high surrogate or low surrogate. If it is a high surrogate, we need to wait for the next code unit and check if it is a low surrogate. If it is, we can decode the two code units as a single code.

To calculate the code point, we use the following formula:

codePoint = (highSurrogate - 0xD800) * 0x400 + (lowSurrogate - 0xDC00) + 0x10000

Here is an example of how to decode the UTF-16 code units 0xD83D 0xDE00:

  • The first code unit, 0xD83D, is in the range 0xD800-0xDBFF, so we know it is a high surrogate.
  • The second code unit, 0xDE00, is in the range 0xDC00-0xDFFF, so we know it is a low surrogate.
  • We know that the two code units form a valid UTF-16 code point because the high surrogate is less than 0xDBFF and the low surrogate is greater than 0xDCFF.
  • We can calculate the code point using the formula, which gets us 0x1F600.

UTF-16 is a good choice for applications that heavily use characters from the Basic Multilingual Plane (BMP), such as most European languages.

UTF-32

UTF-32 uses four bytes for all characters. It is not as widely used as the other two formats.

Number of Bytes Bits Min. Code Point Max. Code Point Byte 1 Byte 2 Byte 3 Byte 4
4 32 U+00000000 U+7FFFFFFF xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx

Here are some examples:

Character Code Point Byte 1 Byte 2 Byte 3 Byte 4
A U+0041 00000000 00000000 00000000 01000001
U+20AC 00100000 00000000 00000000 10101100
🌟 U+1F31F 00000000 00011111 00010001 11110011

The number of bytes is determined by the code point of the character. The most significant bit (MSB) of the first byte is always set to 0, while the MSB of subsequent bytes is set to 1. The remaining bits of each byte are used to store the code point data.

UTF-32 is a good choice if you need to support all Unicode characters or if you need random access to individual characters. The disadvantage of UTF-32 is that it is not as efficient as other encodings when it comes to space.

Unicode Normalization

One issue you may encounter when working with unicode text is that there can be multiple ways to represent the same character.

In order to compare or process text, we need to normalize it so that these different representations are treated as equivalent. Unicode defines equivalence by two criteria: compatibility and canonical equivalence.

  • Compatibility equivalence is when two characters have the same meaning but not necessarily similar visually. For example, "ff" (LATIN SMALL LIGATURE FF) is treated as compatible with "ff" (LATIN SMALL LETTER F + LATIN SMALL LETTER F) although they do not look the same.
  • Canonical equivalence is when two characters are visually similar. For example, "á" can be represented as two code points: U+0061 U+0301 (LATIN SMALL LETTER A + COMBINING ACUTE ACCENT) or as a single code point: U+00E1 (LATIN SMALL LETTER A WITH ACUTE).

Unicode defines four normalization forms: NFC, NFD, NFKC, and NFKD.

  • NFC (Canonical Composition) combines code points into a single code point if possible. For example, 'á' would be represented as U+00E1 (LATIN SMALL LETTER A WITH ACUTE).
  • NFD (Canonical Decomposition) splits code points into multiple code points if necessary. For example, 'á' would be represented as U+0061 U+0301 (LATIN SMALL LETTER A + COMBINING ACUTE ACCENT).
  • NFKC (Compatibility Composition) is similar to NFC but also applies compatibility decomposition. For example, "ff" (LATIN SMALL LIGATURE FF) will be normalized to "ff" (LATIN SMALL LETTER F + LATIN SMALL LETTER F).
  • NFKD (Compatibility Decomposition) is similar to NFD but also applies compatibility decomposition.

Which normalization form you choose depends on your needs. For example, if you need to compare two pieces of text for equality, you would use NFC. If you need to process a piece of text for compatibility, you would use NFKC or NFKD.

String Length

Calculation of the length of a string in unicode is one of the more confusing aspects. As we have discussed, there are various terminologies such as bytes, code points, graphemes and so on. Based on the terminology and context, the length of a string changes.

The length of a string in code points would vary depending on the encoding format used, and whether normalization is applied. For instance, the length of the emoji "🤦🏼‍♂️” would be 17, 7 and 5 in UTF-8, UTF-16 and UTF-32 respectively.

The length of a string in grapheme also depends on the version of unicode standard used, as the standard continually updates what consists as a grapheme.

References

https://hsivonen.fi/string-length/

https://www.unicode.org/reports/