Bytes, code points and graphemes: why string length lies

A user signs up with the display name 👩‍👩‍👧‍👦. Your validation says it is 11 characters, your database column is VARCHAR(10), and the insert fails. You raise the limit to 20, the insert succeeds, and now the profile page shows a name truncated into two broken symbols and a replacement character.

Both bugs come from the same place: "length" is not one number. A string has at least four, and they disagree.

The four counts

Take the family emoji above. It is a single thing a reader perceives, built from four person emoji joined by zero width joiners.

Every one of those numbers is correct for some purpose. The bug is using the wrong one.

Latin text hides the problem because all four numbers are equal for ASCII. Everything past that diverges, and it diverges for ordinary text long before you reach emoji. The letter é is one grapheme, but it can be one code point (U+00E9) or two (U+0065 followed by U+0301, a combining acute accent). Both render identically. Neither is wrong.

Which count each job needs

UTF-8 bytes for anything with a physical limit: database column sizes in some engines, index key limits, HTTP header budgets, network frames, filesystem name limits. A VARCHAR(255) in Postgres counts characters, but the btree index behind it has a byte limit, and MySQL's older utf8 was a three byte encoding that could not store emoji at all. If your MySQL column is not utf8mb4, emoji either error or get silently mangled depending on the strict mode setting.

Code points for almost nothing directly, but it is the unit that regular expressions, case conversion and normalization operate on. It is the right internal unit, and the wrong unit to show a user.

UTF-16 code units for no product decision at all. It is an implementation detail of JavaScript, Java, C# and Windows APIs that leaks into String.length. Treat any limit expressed in it as accidental.

Grapheme clusters for anything a person counts: a character limit in a form, a truncation for a preview, cursor movement, a "name too long" message. This is the count users mean, and the one most codebases never compute.

Getting graphemes

Modern JavaScript has a segmenter built in:

const segmenter = new Intl.Segmenter('en', { granularity: 'grapheme' })
const count = [...segmenter.segment(name)].length
// 👩‍👩‍👧‍👦 => 1

Splitting by spread ([...str]) gets you code points, which is closer than .length but still splits the family into seven pieces. Python's len() also counts code points, so len('👩‍👩‍👧‍👦') is 7 there; the regex module or grapheme package gives clusters. Swift is the outlier that gets it right by default: Character is a grapheme cluster, so .count returns 1.

Truncation is where the distinction stops being academic. Slicing a JavaScript string at an arbitrary index can cut between the two halves of a surrogate pair, producing a lone surrogate. That is not valid UTF-8, so it cannot be encoded, and it surfaces as U+FFFD, the replacement character, or as a JSON encoding error three services downstream. Truncate on cluster boundaries, then check the byte length of the result, then trim further if the byte budget is still exceeded.

Normalization: the same text, twice

Because é has two valid encodings, two strings can look identical, print identically, and compare unequal.

'café' === 'café'   // false, one is NFC and one is NFD

Unicode defines four normalization forms. Two matter:

• NFC composes where possible, so é becomes the single code point. Shorter, and the right default for storage and transport. The W3C recommends NFC for text on the web.
• NFD decomposes into base plus combining marks. Useful as an intermediate step, notably for stripping accents: normalize to NFD, remove the combining mark range, and café becomes cafe.

const slug = input.normalize('NFD').replace(/\p{M}/gu, '')

Normalize on input, once, at the boundary. Do it before you hash, before you compare, before you store, and before you use a string as a uniqueness key. macOS returns filenames in NFD while Linux returns whatever was written, which is why a file list synced between the two can contain what look like duplicates.

Note what normalization does not do. It does not fold case, it does not remove invisible formatting characters, and it does not stop two different scripts from producing visually identical text. A Cyrillic а and a Latin a are separate letters that no normalization form will merge, which is the basis of homograph domain attacks.

Case conversion is locale dependent

toUpperCase is not a per character mapping.

• ß uppercases to SS, so the string gets longer.
• In Turkish, i uppercases to İ (dotted) and I lowercases to ı (dotless). The default mapping gives the wrong letter, which is why "I".toLowerCase() producing i breaks Turkish text and why locale insensitive comparison of the identifier ID has bitten real systems.
• Greek final sigma ς and medial sigma σ both uppercase to Σ, and the reverse depends on position.

For display, pass the locale: str.toLocaleUpperCase('tr'). For comparison, do not use case conversion at all. Use case folding, which exists precisely to make comparison stable, or compare with Intl.Collator and the sensitivity you actually want. The case converter applies the standard mappings, which is what you want for identifiers and slugs rather than for user visible prose in a specific locale.

Validating input

A few rules cover most of the damage:

1. Reject invalid UTF-8 at the edge. Unpaired surrogates, overlong encodings and truncated sequences should fail fast with a clear error rather than being replaced with U+FFFD and stored.
2. Normalize to NFC. One line, at the same edge.
3. Strip or reject control and invisible characters. Zero width space, right to left override and the bidirectional control characters are the ones abused to disguise text. Allow the zero width joiner only if you are allowing emoji.
4. Set limits in graphemes for the user, and check bytes for the storage. Show the user "40 of 50", and separately guarantee the result fits the column.
5. Do not use a character limit as a security control. It is a UX affordance. The database constraint is the guarantee.

Counting a real string

To see the four numbers for text you have in hand, paste it into the text statistics tool, which counts characters, words and bytes locally in the browser without sending the text anywhere. For inspecting the individual code points behind a suspicious looking string, the HTML entities encoder will show you the numeric references, which is a fast way to spot a combining mark or an invisible character that should not be there.

The short version: store UTF-8, normalize to NFC on input, count graphemes when a human is looking, count bytes when hardware is, and never use .length for a rule you intend to explain to a user.

Related reading: URL encoding explained covers what happens to these bytes in a URL, HTML entities encode and decode covers the same problem in markup, and word count, reading time and readability covers counting at the other end of the scale.