AI HELP · DUPLICATE TEXT 2026-10-14

AI left two identical-looking labels. Are their characters actually the same?

Compare the underlying characters and the agreed normalization rule. Text that looks alike can have different Unicode sequences.

Unicode distinguishes canonical equivalence from compatibility equivalence. NFC normalization can give canonically equivalent text a common representation. That is a text-comparison rule, not proof that two records describe the same person or item.

Two local examples we checked

Our first pair was é as U+00E9 and e plus U+0301. Our second was Korean 가 as U+AC00 and U+1100 plus U+1161. Each pair had two distinct raw strings but one distinct string after NFC. We tested the character sequences, not an AI-generated list.

Do not erase meaningful differences

In our test, plain e remained different from é after NFC. Circled ① remained different from 1 under NFC but became 1 under NFKC, a compatibility form. A broader normalization can remove distinctions your data needs; agree the rule before changing originals.

Ask for a comparison column

Suggested input: “Preserve the original text and record IDs. Add an NFC comparison value and show suspected duplicate labels. Do not merge records solely because their labels match.” Review the candidates against the task’s actual identifiers.

A matching label is a limited result

The same normalized spelling does not establish matching dates, accounts or ownership. Keep the original strings for review, record the applied rule and decide separately whether the records should be combined.

Published October 14, 2026. We tested invented strings using the existing Python runtime’s Unicode 15.0 database; the current Unicode reference is 18.0. We did not run a full conformance test, an AI service or deduplication of personal records.

Sources: Unicode · Normalization · Python · unicodedata.