Character Encoding Explained: ASCII, UTF-8, and Why Text Breaks

Every developer eventually opens a file and sees café rendered as café, or watches a Python script blow up with UnicodeDecodeError, or finds a JWT-adjacent JSON blob that JSON.parse refuses to touch even though it "looks fine" in the editor. None of this is random. Text is just bytes, and bytes only mean something once you know which rulebook was used to turn characters into them — and which rulebook you are using to turn them back. When those two rulebooks disagree, you get mojibake, split emoji, and parsers that die on byte zero. This guide goes through the actual bit patterns so the failures stop looking mysterious.

ASCII: 128 slots, and only 128

ASCII (American Standard Code for Information Interchange) is a 7-bit encoding. Seven bits give you 2^7 = 128 possible values, numbered 0 through 127, and ASCII assigns every single one of them a meaning. Codes 0-31 and 127 are control characters (tab, newline, carriage return, null, escape, and so on — not printable text at all). Codes 32-126 cover the printable set: space, digits 0-9, uppercase A-Z, lowercase a-z, and the punctuation on a US keyboard. That is the entire alphabet. There is no slot for é, ñ, ü, ø, ß, or anything Cyrillic, Greek, Hebrew, Arabic, Devanagari, or Han. ASCII was built by and for a specific slice of the English-speaking computing world in the 1960s, and it physically cannot represent more than 128 distinct symbols — the 7th bit simply is not there. Every byte in memory has 8 bits, so ASCII text always leaves the top bit set to 0 and wastes the other 128 possible byte values (128-255).

The code-page era: same bytes, different alphabets

Once computers needed to handle French, German, Russian, Greek, and everything else, vendors reused that unused top bit. An 8-bit byte can represent 256 values, so codes 128-255 became fair game — and every region, vendor, and OS filled them in differently. Latin-1 (ISO-8859-1) put Western European accented letters there. Windows-1252 did something similar but not identical (it swapped a handful of Latin-1's control-range slots for smart quotes and the euro sign). Separate code pages covered Cyrillic (Windows-1251, KOI8-R), Greek (ISO-8859-7), Hebrew, Turkish, and dozens more. None of these encodings knew about each other, and a plain byte stream carries no label saying which one was used.

That is the actual root cause of classic mojibake: the byte value 0xE9 (233 decimal) is é under Windows-1252, but the same byte is a completely different character under a Cyrillic code page, and undefined under plain ASCII. If a French document gets saved as Windows-1252 and then opened by a program that assumes Latin-1 (or the other way around), most bytes match up by luck because the two overlap heavily, but the ones that do not produce garbage — recognizably wrong, but not obviously broken, which is what makes it insidious. Multiply this by dozens of incompatible 8-bit code pages and you get the pre-Unicode mess: no single encoding could hold text from more than one script family, and exchanging files across regions was a gamble.

Unicode's actual idea: separate the number from the bytes

Unicode fixes this with one conceptual move that most encoding confusion comes from missing: Unicode defines what a character is (a codepoint — an abstract number) completely separately from how that number gets turned into bytes (an encoding). The codepoint for é is U+00E9. The codepoint for the grinning-face emoji is U+1F600. These numbers do not change no matter what encoding you use to store or transmit the text. What changes is the byte representation: UTF-8, UTF-16, and UTF-32 are three different, interchangeable ways of encoding the exact same set of codepoints. "Convert this file to Unicode" is a category error — Unicode is not a byte format, it is the numbering system. What you actually convert to is a specific encoding, almost always UTF-8 today.

UTF-8: variable width, self-describing bytes

UTF-8 encodes each codepoint using 1 to 4 bytes, and it is designed so you can tell how many bytes a character occupies just by looking at the leading bits of its first byte:

1 byte:  0xxxxxxx                                  (codepoints U+0000   - U+007F)
2 bytes: 110xxxxx 10xxxxxx                         (codepoints U+0080   - U+07FF)
3 bytes: 1110xxxx 10xxxxxx 10xxxxxx                (codepoints U+0800   - U+FFFF)
4 bytes: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx        (codepoints U+10000  - U+10FFFF)

Every continuation byte (the 2nd, 3rd, or 4th byte of a multi-byte character) starts with the bit pattern 10, which is exactly how a decoder knows a byte is a continuation rather than the start of a new character. The 1-byte form starting with 0 is why UTF-8 is byte-for-byte identical to ASCII for codepoints 0-127 — an ASCII file is already a valid UTF-8 file, no conversion needed. That backward compatibility is the whole reason UTF-8 won over UTF-16 and UTF-32 for text on the web and in most file formats.

Here is the worked example for é, codepoint U+00E9 (decimal 233):

U+00E9 in binary (11 bits needed, right-padded to fit the 2-byte pattern):
  00E9  ->  000 11101001

Split across the 2-byte template  110xxxxx 10xxxxxx  (11 x's total):
  110 00011  10 101001

Which gives the two bytes:
  11000011  10101001
  =  0xC3     0xA9

So é (U+00E9) is encoded in UTF-8 as the two bytes: C3 A9

That is exactly the pair of bytes a Latin-1 or Windows-1252 decoder misreads as two separate characters (Ã then ©), which is why café becomes café when a UTF-8 file gets opened under the wrong assumed encoding. Same bytes, two different rulebooks, two different results.

UTF-16 and the surrogate-pair bug you have probably shipped

UTF-16 stores most common characters (the Basic Multilingual Plane, roughly U+0000 to U+FFFF) as a single 16-bit unit. But Unicode has room for codepoints up to U+10FFFF, well past what one 16-bit unit can hold. For those — most emoji included, like 😀 at U+1F600 — UTF-16 uses a surrogate pair: two 16-bit code units, a "high surrogate" followed by a "low surrogate," which only mean anything together. Neither half is a valid character on its own.

JavaScript strings are UTF-16 internally, and .length counts 16-bit code units, not actual characters. 😀 has a .length of 2, not 1. That means naive code like str.slice(0, 5) or a character-by-character loop using indices can cut a string exactly between the two halves of a surrogate pair, leaving one dangling unpaired surrogate at the boundary. The result renders as a broken tofu box or the replacement character, and re-encoding it can even throw. This is why string truncation, byte-length limits (SMS-style character counters, database column limits), and naive substring logic are a recurring source of "the emoji got mangled" bug reports — the fix is to iterate by codepoint (for example with for...of or Array.from(str) in JavaScript) instead of by raw index.

The BOM: a marker that outstays its welcome

The byte order mark is the codepoint U+FEFF, which some tools write at the very start of a file to declare its encoding and, for UTF-16/UTF-32, its byte order (little-endian vs big-endian). In UTF-8 a BOM has no byte-order purpose at all — UTF-8 has no byte order to disambiguate — but some Windows tools (Notepad, Excel's "CSV UTF-8" export) still prepend the three bytes EF BB BF to signal "this is UTF-8" anyway. Most modern parsers strip a leading BOM silently. Strict ones do not: a strict JSON.parse or a CSV parser that expects the first character to be { or a header field will choke on those three invisible bytes and report a syntax error at position 0, even though the file "looks empty at the start" in a text editor that hides the marker.

Try it

If you want to see codepoint-versus-byte behavior firsthand, run some accented text or an emoji through the DevBox Base64 Encoder / Decoder and inspect the resulting bytes, or drop a file with a leading BOM into the JSON Formatter & Validator to see exactly where strict parsing draws the line.