Unicode and Why Text Breaks

Unicode assigns numbers; encodings store bytes

The Unicode Standard assigns a code point—a number, written like U+0041 for Latin capital A—to each abstract character in its catalog. The catalog is not the same as a font: fonts map code points to glyphs. UTF-8, UTF-16, and UTF-32 are encoding schemes that turn code point sequences into bytes for files, HTTP, and databases. Bugs appear when software treats UTF-8 bytes as if they were Latin-1 characters, or when two systems assume different encodings without declaring them.

UTF-8 in practice

UTF-8 is variable-width: ASCII letters use one byte; many European scripts use two; most emoji and rare scripts use four. It is the dominant encoding on the web (HTTP, JSON, HTML). When a system says “Unicode” in 2026, it usually means “UTF-8 on the wire unless stated otherwise.”

Character: é
UTF-8 bytes: 0xC3 0xA9
Mis-decoded as Latin-1: à ©  → often shown as "é" (mojibake)

UTF-16 and JavaScript strings

JavaScript strings are sequences of UTF-16 code units, not Unicode code points. Characters outside the Basic Multilingual Plane (code points above U+FFFF)—many emoji, rare Han characters—take two UTF-16 code units called a surrogate pair. That means String.length counts code units, not user-visible characters, and indexing with str[i] can split a surrogate pair and produce garbage.

const smile = "😀";
console.log(smile.length); // 2 (surrogate pair)
console.log(smile[0] === smile[1]); // false
console.log([...smile].length); // 1 (iterates code points)
console.log(Array.from(smile).length); // 1

Spread and Array.from use Unicode code point iteration. For production limits, prefer Intl.Segmenter or a grapheme-aware library when counting what users see.

Grapheme clusters: what users count

A user-visible character may combine multiple code points. A base letter plus combining acute accent (e + ◌́) can render like é but remain two code points. Emoji flags are pairs of regional-indicator letters; skin-tone modifiers attach to person emoji; ZWJ sequences join family emoji. Unicode defines extended grapheme cluster boundaries for “what feels like one character” in editing and cursor movement.

const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" });
const text = "e\u0301"; // e + combining acute
console.log(text.length); // 2
console.log([...segmenter.segment(text)].length); // 1 grapheme

const bio = "Hi 👨‍👩‍👧‍👦!"; // family ZWJ sequence
console.log(bio.length); // 11 UTF-16 code units
console.log([...bio].length); // 8 Unicode code points
console.log([...new Intl.Segmenter("en", { granularity: "grapheme" }).segment(bio)].length); // 7 graphemes (letters, space, family emoji, !)
// A "max 10 characters" rule using .length rejects this bio even though it looks short.

Mojibake and fixing encoding mistakes

Mojibake (garbled text) usually means bytes were interpreted with the wrong decoder. Classic case: UTF-8 bytes read as Windows-1252 or ISO-8859-1. The fix is to re-read the original byte stream with the encoding that produced it, then transcode to UTF-8—not to manually replace “é” with “é” in a corrupted string that lost information.

Emoji and length limits

Product limits—“140 characters”, SMS segments, database VARCHAR—pick a counting rule. Twitter-style counts historically used weighted code units; SMS uses GSM-7 or UCS-2 segments; databases may count bytes. An emoji-heavy message can exceed a byte budget while looking short visually. Document which metric your counter exposes: bytes, UTF-16 units, code points, or graphemes.

• Storage limits (PostgreSQL bytea vs char length) often care about bytes in UTF-8.
• JavaScript maxlength on inputs often counts UTF-16 code units.
• Regulations or billing may count graphemes or legacy code pages.
• Base64 expands binary UTF-8—encoding text for URLs is not the same as character count.

Normalization: when equality fails

The same visual string can have multiple Unicode representations. NFC and NFD normalization forms pick canonical precomposed vs decomposed sequences. Passwords and usernames should be compared after agreed normalization—or you lock out users whose IME produces a different but equivalent sequence.

Case folding and locale

Turkish dotted and dotless I, German ß, and Greek sigma variants show why toLowerCase() alone is a weak equality strategy. Unicode case folding (case-fold) and locale-aware collation belong in search indexes and username policies, not ad hoc string hacks in application code.

Surrogates and invalid sequences

UTF-8 allows only well-formed byte sequences; lone surrogates are illegal in UTF-8 but can appear when UTF-16 data is mishandled. Parsers that silently substitute replacement characters (U+FFFD) may hide data corruption—log and reject invalid input at trust boundaries when integrity matters.

Checklist for developers

1. Declare charset=utf-8 on HTML and JSON responses.
2. Never assume one byte per character in buffer sizes.
3. Do not use string length alone for user-facing limits.
4. Use Intl.Segmenter or ICU for grapheme-aware editing limits.
5. Test with combining marks, emoji flags, and ZWJ sequences in QA fixtures.
6. When integrating legacy CSV or Windows exports, ask which encoding created the file.

SmartKitio character and word counters should be read alongside their documented counting mode—compare results to your platform’s rules before shipping limits to production.

Binary data is not text

Base64 and hex encodings represent bytes, not Unicode strings. Decoding Base64 yields bytes that you must still decode with UTF-8 (or the correct charset) before treating the result as text. Piping binary through string APIs without an explicit encoding step recreates mojibake at a different layer—common in upload pipelines and email attachments.

What to remember

• Unicode names characters as code points; UTF-8 (and other encodings) store those points as bytes.
• JavaScript .length counts UTF-16 code units—surrogate pairs and combining sequences break naive limits.
• Users count grapheme clusters; use Intl.Segmenter or ICU when product rules say “characters.”
• Mojibake is usually wrong decoding at a boundary—fix the pipeline, not the string with search-and-replace.
• Normalize (usually NFC) before comparing passwords and usernames that may come from different IMEs.