Tracing Byte-Exact Json Bugs With A Deterministic Diff Harness

I got bitten by a bug that looked “nondeterministic” at first: sometimes my API returned the exact same data but the downstream system rejected it anyway. No stack traces, no obvious validation errors—just a polite “invalid payload.”

What finally cracked it wasn’t deeper logging or more retries. It was a byte-exact diff harness that compared the raw JSON bytes (including whitespace and key order) between “expected” and “actual.” Once I could see the differences at the byte level, the culprit became obvious: the JSON encoder was changing formatting and key ordering in a way the downstream system treated as significant.

Below is the small deterministic harness I built to make those issues impossible to miss.

The core problem: “valid JSON” can still be rejected

JSON validity is about syntax (brackets/quotes/commas), but some systems also do strict checks on the exact serialized representation—commonly:

• Key ordering: some consumers expect keys in a particular order.
• Whitespace: some legacy parsers normalize incorrectly (or even compare raw strings).
• Escaping rules: Unicode escaping differences can matter.
• Number formatting: 1 vs 1.0, exponent notation, etc.

My payloads were valid JSON, but the string form wasn’t what the consumer expected.

What I built: a byte-exact diff harness

The idea is simple:

1. Generate an “expected” JSON string.
2. Generate an “actual” JSON string from the code under test.
3. Compare the raw bytes and show the first byte where they differ.
4. Also parse into a stable structure (so I can distinguish “semantic difference” from “serialization difference”).

A tiny Python script that pinpoints the first differing byte

# json_byte_diff.py
import json
import difflib

def to_bytes(s: str) -> bytes:
    # Use UTF-8 since JSON strings are typically UTF-8 on the wire.
    return s.encode("utf-8")

def describe_first_difference(expected: bytes, actual: bytes) -> str:
    max_len = min(len(expected), len(actual))
    for i in range(max_len):
        if expected[i] != actual[i]:
            # Show a small window around the mismatch for context.
            exp_slice = expected[max(0, i - 10): i + 10]
            act_slice = actual[max(0, i - 10): i + 10]
            return (
                f"First mismatch at byte index {i}\n"
                f"Expected byte: {expected[i]} (0x{expected[i]:02x})\n"
                f"Actual byte:   {actual[i]} (0x{actual[i]:02x})\n"
                f"Expected context (utf-8 with replacement): {exp_slice.decode('utf-8', 'replace')}\n"
                f"Actual context   (utf-8 with replacement): {act_slice.decode('utf-8', 'replace')}\n"
            )
    if len(expected) != len(actual):
        return (
            f"No mismatch in first {max_len} bytes, but length differs:\n"
            f"Expected length: {len(expected)}\n"
            f"Actual length:   {len(actual)}\n"
        )
    return "Byte-for-byte identical."

def semantic_compare(expected_json: str, actual_json: str) -> str:
    # Parse JSON into Python objects. This ignores formatting differences
    # and normalizes key order at the object level.
    expected_obj = json.loads(expected_json)
    actual_obj = json.loads(actual_json)

    if expected_obj == actual_obj:
        return "Semantic equality: parsed objects match (differences are purely serialization/formatting)."
    return "Semantic inequality: parsed objects differ."

def show_unified_diff(expected_json: str, actual_json: str) -> None:
    expected_lines = expected_json.splitlines(keepends=True)
    actual_lines = actual_json.splitlines(keepends=True)
    diff = difflib.unified_diff(
        expected_lines,
        actual_lines,
        fromfile="expected.json",
        tofile="actual.json",
    )
    print("---- Unified diff (line-based) ----")
    for line in diff:
        print(line.rstrip("\n"))

def main():
    # This simulates the “expected” payload the consumer strictly wants.
    # Note key order and whitespace are part of the raw bytes.
    expected_json = '{"a":1,"b":2,"nested":{"x":true,"y":"ok"}}'

    # This simulates a different serialization produced by the code under test.
    # json.dumps() with sort_keys=False preserves insertion order of dicts,
    # but if you build dicts differently, key order can change.
    actual_obj = {
        "b": 2,
        "a": 1,
        "nested": {"y": "ok", "x": True},
    }

    # Example of a common “looks fine but bytes differ” issue:
    # - separators add spaces
    # - ensure_ascii changes escaping for non-ascii (not shown here)
    actual_json = json.dumps(actual_obj, ensure_ascii=False, separators=(", ", ": "))

    expected_bytes = to_bytes(expected_json)
    actual_bytes = to_bytes(actual_json)

    print("---- Byte-level report ----")
    print(describe_first_difference(expected_bytes, actual_bytes))

    print("---- Semantic report ----")
    print(semantic_compare(expected_json, actual_json))

    show_unified_diff(expected_json, actual_json)

if __name__ == "__main__":
    main()

Run it:

python json_byte_diff.py

You’ll see:

• The first mismatching byte index
• A small context window around that mismatch
• A semantic comparison result that usually says the objects match (meaning the consumer is sensitive to serialization, not meaning)

In the common case above, the semantic objects match, but the bytes differ due to whitespace and key order.

A step-by-step “root cause” workflow

I used this exact flow when the consumer rejected my payloads:

1) Confirm whether it’s semantic or serialization mismatch

The semantic comparison uses json.loads(), which parses JSON into objects (dicts/lists). That ignores whitespace, and key order generally won’t matter for equality of parsed objects.

• If semantic equality is true, the bug is almost certainly about encoding/serialization format.
• If semantic equality is false, then the data itself differs.

2) Use the first differing byte index as a compass

Instead of staring at entire payloads, I focused on:

• The first byte where they differ
• The decoded context around it

That made it obvious whether the mismatch was at:

• a comma or quote (syntax/structure)
• a key name boundary (ordering or missing/extra fields)
• a space character (whitespace normalization)
• a digit (number formatting)

3) Lock down stable serialization (when semantics match)

In practice, if the consumer expects a canonical form, you’ll typically fix it by generating a stable serialization. Here’s a deterministic serialization helper:

# json_canonical.py
import json
from typing import Any

def canonical_json_bytes(obj: Any) -> bytes:
    # ensure_ascii=False keeps Unicode as UTF-8 characters (not \uXXXX escapes).
    # sort_keys=True guarantees key ordering within objects.
    # separators=(',', ':') removes unnecessary whitespace to stabilize output.
    text = json.dumps(
        obj,
        ensure_ascii=False,
        sort_keys=True,
        separators=(",", ":"),
    )
    return text.encode("utf-8")

def canonical_json_text(obj: Any) -> str:
    return canonical_json_bytes(obj).decode("utf-8")

Using sort_keys=True plus compact separators is one of the most common “make bytes deterministic” moves.

Example: comparing two real serializations safely

Here’s a small harness showing both raw bytes and semantic equality:

# json_compare_example.py
import json
from json_byte_diff import describe_first_difference, to_bytes
from json_canonical import canonical_json_text

expected_obj = {"a": 1, "b": 2, "nested": {"x": True, "y": "ok"}}

# Different insertion order and formatting:
actual_obj = {"b": 2, "a": 1, "nested": {"y": "ok", "x": True}}

expected_json = json.dumps(expected_obj, separators=(",", ":"))
actual_json = json.dumps(actual_obj, separators=(", ", ": "))

expected_bytes = to_bytes(expected_json)
actual_bytes = to_bytes(actual_json)

print("Raw JSON expected:", expected_json)
print("Raw JSON actual:  ", actual_json)

print("\n---- First mismatch ----")
print(describe_first_difference(expected_bytes, actual_bytes))

print("\n---- Semantic equality ----")
print(json.loads(expected_json) == json.loads(actual_json))

print("\n---- Canonical form ----")
print("Expected canonical:", canonical_json_text(expected_obj))
print("Actual canonical:  ", canonical_json_text(actual_obj))

Run:

python json_compare_example.py

You get a quick answer:

• Are semantics equal? (data correctness)
• Are bytes different? (format/canonicalization)
• Do canonical forms match? (fix direction)

Why byte-exact diffs are more reliable than “print debugging”

“Print debugging” often fails because it hides formatting differences:

• Developers copy-paste JSON into logs that reformat it.
• JSON pretty printers add indentation and line breaks.
• Some logging systems escape strings differently than actual network bytes.

The harness I used compares the exact bytes in the serialized payload. That’s what the downstream system actually receives.

Closing thoughts

I learned to treat serialization as a first-class part of correctness: even when JSON is syntactically valid and semantically identical, a consumer may enforce byte-level expectations. Building a byte-exact diff harness made the invisible differences visible—pinpointing the first mismatching byte and separating semantic mismatches from formatting/key-order issues—so the fix became deterministic rather than guesswork.