Why Your difflib SequenceMatcher Ratio Suddenly Dropped: Invisible NO-BREAK SPACE (U+00A0) from OpenAI Transcripts

Similarity score dropped with difflib SequenceMatcher? NO-BREAK SPACE (U+00A0) from the OpenAI API skews ratios; normalize inputs to restore stable results.

When a similarity score suddenly changes without any code or environment updates, it feels like the ground shifted under your feet. This guide breaks down a real-world case where Python’s difflib.SequenceMatcher started returning a lower ratio for the same strings, and the entire issue came down to one invisible character. There’s no mystery in the algorithm. The inputs changed.

Context and symptom

A system compared two short strings using difflib.SequenceMatcher. The same comparison had consistently passed an internal threshold until a specific point in time. After that, across both development and production, on Windows and Unix hosts, the ratio dropped and the check began to fail. No deployments, no dependency bumps, no config tweaks. The only externality: one of the strings originated from the OpenAI API.

Baseline: the comparison snippet

The core logic is minimal and predictable. It computes a ratio based purely on the sequences provided.

from difflib import SequenceMatcher as SM
similarity_score = SM(
    None,
    transcript_clean,
    expected_clean
).ratio()

What actually happened

The algorithm didn’t change. The environment didn’t change. The inputs did. The string coming from the OpenAI transcription API began to include NO-BREAK SPACE (U+00A0) instead of a regular space (U+0020). These characters are visually indistinguishable in many contexts, but they are not the same code point. Because SequenceMatcher is deterministic and operates on the exact sequences it receives, a different byte is a real difference, and the similarity ratio will reflect that.

“It’s entirely self-contained and purely functional (the results depend solely on the sequences passed to it). It knows nothing about time, which platform it’s running on, or anything in its environment …”

This is precisely why the ratio dropped. The inputs diverged at the Unicode code point level even though they looked identical to humans. As soon as the transcription API started returning U+00A0, the matcher correctly saw more mismatches and the score decreased.

Reproducing the effect

You can observe the same behavior with a minimal example by contrasting strings that differ only by the space character.

from difflib import SequenceMatcher as SM
left = "Hello\u00A0world"   # NO-BREAK SPACE between words
right = "Hello world"       # regular space
print(SM(None, left, right).ratio())

Even though both strings look identical, the ratio is lower than if both used the same kind of space.

Fix: normalize the input that contains U+00A0

If your pipeline relies on a stable similarity threshold, sanitize the strings before comparison by mapping U+00A0 to U+0020. This preserves human-visible intent while aligning the underlying code points.

from difflib import SequenceMatcher as SM
def strip_nbsp(x: str) -> str:
    return x.replace("\u00A0", " ")
left_norm = strip_nbsp(transcript_clean)
right_norm = strip_nbsp(expected_clean)
similarity_score = SM(None, left_norm, right_norm).ratio()

This change targets exactly the observed cause: a NO-BREAK SPACE leaking into the comparison string. The rest of the logic remains the same.

Why this matters

Unicode is a double-edged sword in production text processing. Two strings that look identical to users may differ in code points, normalization forms, or whitespace semantics. When downstream logic depends on deterministic similarity scores, seemingly innocuous differences like U+00A0 versus U+0020 can flip a decision boundary. In this case, the algorithm did its job perfectly; the inputs were no longer what they used to be. The way out is to standardize the inputs that matter to your business rules.

Practical takeaways

Treat similarity metrics as pure functions of their inputs and assume nothing about external sources. If a score shifts without a code change, verify the exact code points of the strings being compared. When your source is an API, be prepared for subtle changes that preserve human readability but alter machine interpretation. A small, deliberate normalization step—like converting U+00A0 to a regular space before comparison—can restore stability without diluting the meaning of the text.

Conclusion

difflib.SequenceMatcher didn’t regress and doesn’t vary by time, platform, or environment; it only reflects the sequences it receives. The immediate fix is to replace NO-BREAK SPACE (U+00A0) with a regular space (U+0020) in the input derived from the OpenAI transcription API before calling .ratio(). Keep your thresholds, keep your comparisons, and make the invisible differences visible in code where it counts.