How JSON Parsers Work: Tokenizing, Validating, and Recovering from Errors

Most explanations of "invalid JSON" stop at the symptom: a trailing comma is bad, single quotes are bad, unquoted keys are bad. That's true, but it treats the parser as a black box that mysteriously rejects certain inputs. It isn't a black box — it's two small, deterministic pieces of code, a tokenizer and a recursive-descent parser, each enforcing a formal grammar defined by RFC 8259 and its sibling standard ECMA-404. Once you see what those two pieces actually do character by character, every "Unexpected token" error stops looking arbitrary and starts looking like exactly what it is: a state machine refusing an input that doesn't match its rules.

Two passes: tokenizing, then parsing

No JSON parser works directly on the raw character stream when building structure. It first runs a lexer (tokenizer) that groups characters into a flat sequence of tokens, discarding insignificant whitespace along the way. Only then does a second stage — the parser proper — consume that token stream and assemble it into nested objects, arrays, and scalar values. Even implementations that interleave the two for performance (pulling one token at a time on demand rather than materializing the whole list up front) are still logically doing both jobs.

The token set for JSON is small and fixed: structural characters { } [ ] : ,, and four value-token kinds — STRING, NUMBER, and the two literal keywords true/false (often unified as a BOOLEAN token) and null. Take this input:

{"id":42,"active":true}

The tokenizer walks it left to right and emits:

LBRACE        {
STRING        "id"
COLON         :
NUMBER        42
COMMA         ,
STRING        "active"
COLON         :
TRUE          true
RBRACE        }

Notice what's already been decided at this point, before any object has been "built": the tokenizer had to decide where the string "id" ends (at the closing unescaped quote), where the number 42 ends (at the first character that isn't a digit — here, the comma), and that true is a complete keyword token, not the start of some longer identifier. Each of those decisions is a small grammar of its own — the NUMBER token alone has rules for an optional leading minus, no leading zeros before other digits, an optional fractional part, and an optional exponent. The parser that runs afterward never looks at a single raw character; it only ever sees this flat list of typed tokens.

Recursive descent: why the parser calls itself

JSON's grammar is recursive by definition: a value can be an object, and an object's members can have values that are themselves objects or arrays, to arbitrary depth. The natural implementation of a recursive grammar is a set of mutually recursive functions, one per grammar production, each calling back into the others (and ultimately into itself) as deeply as the input requires. This is called recursive-descent parsing, and it's how the overwhelming majority of JSON parsers — including the C++ parser behind V8's JSON.parse — are structured:

function parseValue(tokens) {
  const t = tokens.peek();
  switch (t.type) {
    case "LBRACE":  return parseObject(tokens);
    case "LBRACKET":return parseArray(tokens);
    case "STRING":  return parseString(tokens);
    case "NUMBER":  return parseNumber(tokens);
    case "TRUE":    tokens.next(); return true;
    case "FALSE":   tokens.next(); return false;
    case "NULL":    tokens.next(); return null;
    default:
      throw new SyntaxError(
        `Unexpected token ${t.type} at line ${t.line}, column ${t.col}`
      );
  }
}

function parseObject(tokens) {
  tokens.expect("LBRACE");
  const obj = {};
  if (tokens.peek().type === "RBRACE") { tokens.next(); return obj; }
  while (true) {
    const keyTok = tokens.expect("STRING");
    tokens.expect("COLON");
    obj[keyTok.value] = parseValue(tokens); // <-- recursion happens here
    const sep = tokens.next();
    if (sep.type === "RBRACE") return obj;
    if (sep.type !== "COMMA") {
      throw new SyntaxError(
        `Unexpected token ${sep.type} at line ${sep.line}, column ${sep.col}`
      );
    }
    // loop again: a comma must be followed by another member, never '}'
  }
}

function parseArray(tokens) {
  tokens.expect("LBRACKET");
  const arr = [];
  if (tokens.peek().type === "RBRACKET") { tokens.next(); return arr; }
  while (true) {
    arr.push(parseValue(tokens)); // <-- recursion happens here too
    const sep = tokens.next();
    if (sep.type === "RBRACKET") return arr;
    if (sep.type !== "COMMA") {
      throw new SyntaxError(
        `Unexpected token ${sep.type} at line ${sep.line}, column ${sep.col}`
      );
    }
  }
}

The load-bearing detail is the pair of recursive calls marked above: parseObject calls parseValue to read each member's value, and parseValue calls parseObject or parseArray right back when it sees { or [. There's no separate code path for "an object nested three levels deep" — it's the same four functions calling each other, with the JavaScript call stack itself tracking how deep the nesting goes. (This is also why extremely deeply nested JSON — tens of thousands of levels — can blow the stack in a naive recursive-descent implementation; some hardened parsers use an explicit stack instead of native recursion specifically to avoid that.)

Walking the descent through a real example

Take {"a":[1,2,{"b":3}]} and trace what actually happens, call by call:

1. parseValue peeks LBRACE and dispatches to parseObject.
2. parseObject consumes {, reads key "a", consumes :, then calls parseValue for the value — one stack frame deep.
3. That call peeks LBRACKET and dispatches to parseArray — two frames deep.
4. parseArray consumes [ and loops: parseValue for element 0 sees NUMBER and returns 1 directly (no further recursion — numbers are a leaf production); it consumes ,, loops, and does the same for element 1, returning 2.
5. On the third iteration parseArray calls parseValue again, which this time peeks LBRACE and opens a brand-new parseObject call — three frames deep now (outer object → array → inner object). That call reads key "b", calls parseValue for its value (NUMBER 3, a leaf), consumes the closing }, and returns { b: 3 } up to the array.
6. parseArray consumes the next token, ], and returns [1, 2, { b: 3 }] up to the outer parseValue from step 3.
7. The outermost parseObject now has that array as the value for "a", consumes the final }, and returns the finished object.

Every closing bracket corresponds to exactly one recursive call returning back up the stack — the nesting in the text and the nesting of function calls are the same shape.

Why specific "invalid JSON" mistakes fail — and where

These aren't stylistic complaints; each one is a concrete grammar violation caught at a specific stage.

Unquoted keys and single-quoted strings

Both fail inside the tokenizer, before the parser is even involved. The STRING token production in RFC 8259 is explicit: a string is a sequence of Unicode characters wrapped in " (U+0022), full stop — there is no alternate production for a bare identifier or for '-delimited text. When the tokenizer encounters a bare letter like n in {name:, no token rule matches it — not STRING, not a brace, not the start of null — so it fails immediately as an unrecognized character, without ever producing a token for the parser to reject. The mistake is caught at the lexical level, not the structural one.

Trailing commas

This one fails inside the parser, one level up. Look again at the loop in parseObject: after consuming a member's value, the parser reads the next token and branches — } means the object is done; , means loop back expecting another member, so the next thing it demands is a STRING key, not a closing brace. Given {"a":1,}, the comma is consumed, the loop repeats, and tokens.expect("STRING") is called — but the next token is RBRACE. That's the exact failure point: the comma was accepted as valid, and the state machine then found nothing satisfying what it demanded next. It isn't "sloppy-intolerance" — the grammar for a JSON object is literally member (',' member)*, a comma is a separator between two members, never a trailing terminator, so ,} has no production that accounts for it.

Comments

Same failure mode as unquoted keys: / doesn't begin any JSON token, so the tokenizer rejects it on sight.

How "line X, column Y" gets computed

This requires the tokenizer to do a small amount of bookkeeping as it scans, independent of tokenizing itself: it keeps a running line counter (starting at 1, incremented every time it consumes a \n) and a running column counter (incremented per character consumed, reset to 1 right after each newline). Every time the tokenizer emits a token, it stamps that token with the (line, column) of its first character. When the parser later rejects a token — say, it got RBRACE where it expected STRING — it doesn't need to re-scan anything; it just reads the position already stamped on that token object and interpolates it into the thrown SyntaxError message. The position you see in an error is always the position of the token the parser choked on, not the position of whatever came before it — which is why the reported column sometimes looks one character later than where you'd instinctively look for the "actual" mistake (a missing comma is reported at the token after where the comma should have been, because that's the token that broke the expectation).

Strict failure vs. lenient recovery

JSON.parse and most standard-library JSON parsers are strict: the grammar defines no error-recovery production, so the first violation throws and the entire parse aborts. There is no partial result — you get an exception or a fully valid value, never something in between. This is deliberate (RFC 8259 defines no syntax for recovering from errors), and it's what makes JSON safe to trust: if JSON.parse doesn't throw, the interpretation is unambiguous.

Lenient tools that "fix" pasted JSON (auto-correcting a trailing comma, upgrading single quotes, quoting a bare key) aren't running a more forgiving version of the same parser — they layer something in front of or instead of it: either a pattern-based pre-pass that rewrites known near-miss text into valid JSON before handing it to a real strict parser, or a hand-rolled parser with explicit recovery rules — e.g., when the "expect STRING, got RBRACE" state from the trailing-comma example is hit, silently treat the dangling comma as if it were never there and record a diagnostic instead of throwing. Either way, recovery is bolted on beyond the JSON grammar, which is why such tools converge on documenting their own dialect (often close to JSON5) rather than claiming to still parse "JSON."

A detail worth knowing: the reviver argument

JSON.parse(text, reviver) takes a second argument most code never uses. After parsing completes and the full value tree already exists, the reviver runs once per key, walking the tree bottom-up — deepest nested values first, then their parents, ending with the root value under the empty-string key. Each call gets (key, value) and whatever it returns replaces that entry (returning undefined deletes the key); a common use is reviving ISO date strings into Date objects. Structurally, it runs strictly after tokenizing and parsing are both done — it operates on an already-valid tree and has no way to influence or rescue a syntax error.

Try it

The tokenizer/parser split, the recursive descent, and the exact error positions described above are all things you can watch happen by feeding malformed input into a real implementation. The DevBox JSON Formatter runs a strict parse in your browser and surfaces the same line/column-tagged error a hand-rolled recursive-descent parser would produce, so you can pair the mental model above with the actual failure point on your own data.