toolhq.io

All posts
5 min readby Jameel Haider

Unicode property escapes in regex

A pattern like \w+ looks like it means "a word," but feed it café or naïve and it stops at the accented letter. The shorthand classes were defined for ASCII, and most of the world's text is not ASCII. Unicode property escapes are how you match letters and digits as they actually exist across languages.

Why \w is ASCII only

In JavaScript and many other engines, \w is defined as exactly [A-Za-z0-9_]. That is sixty three specific characters. It does not include é, ñ, ü, Greek, Cyrillic, Arabic, or any CJK character. The same limitation applies to \d, which by default means only 0 to 9, and \b, whose notion of a word boundary follows the ASCII \w.

So \w+ against café matches caf and leaves the é behind. This is not a bug; it is the historical definition. The fix is to say what you mean in Unicode terms.

Property escapes

With the u flag enabled, \p{...} matches characters by their Unicode property. The ones you reach for most:

  • \p{L} any kind of letter, in any script.
  • \p{N} any kind of numeric character, including digits in other scripts.
  • \p{Lu} and \p{Ll} uppercase and lowercase letters specifically.
  • \p{P} punctuation, \p{Z} separators like spaces.
  • \P{...} the negation, meaning any character not having that property.

So the Unicode aware way to match a word is:

\p{L}+

With the u flag, this matches café in full, and naïve, and words in Greek or Japanese, because every one of those characters has the letter property.

/\p{L}+/u.test("café")   // true, matches "café"
/\w+/.exec("café")[0]     // "caf"

Property escapes require the u flag. Without it, \p{L} is not treated as a property escape at all. For the full tour of what u and the other modifiers do, see regex flags explained.

Matching emoji

Emoji make the point sharply. Many emoji are single code points, but plenty are built from several joined with zero width joiners, and some carry skin tone modifiers. A plain . under the u flag matches one code point, which can be half of such a sequence.

To match emoji as whole units, use the dedicated properties:

  • \p{Emoji} matches emoji characters, though it also matches some digits that have emoji presentation.
  • \p{Extended_Pictographic} is a more reliable base for pictographic symbols.

For composed sequences like a family emoji or a flag, the robust approach is to match grapheme clusters rather than code points, because a user perceived character can span several code points. This connects to a deeper topic: the difference between bytes, code points, and graphemes.

Checking it on real text

The quickest way to see \w fail and \p{L} succeed is to paste multilingual text and toggle the u flag. In the Regex Tester you can watch an accented word match in full once you switch from \w to \p{L} with u enabled. The matching runs in your browser, so the sample text never leaves your device.

The takeaway: treat \w and \d as ASCII only shortcuts. The moment your input includes accents, other scripts, or emoji, switch to \p{L}, \p{N}, and the pictographic properties with the u flag, and test against text from the languages you actually support.