CJK

The problem with stock wdiff on CJK

Stock wdiff (and any tool using isspace() for word boundaries) treats CJK strings without spaces as one giant "word". So a simplified-vs-traditional diff gets reported as a single delete+add of the whole line:

$ printf '体育老师无奈，总被佔课现象困扰。\n' > a.txt
$ printf '体育老师无奈，总被占课现象困扰。\n' > b.txt
$ wdiff a.txt b.txt
[-体育老师无奈，总被佔课现象困扰。-]{+体育老师无奈，总被占课现象困扰。+}

The only character that actually changed is 佔 → 占. Stock wdiff can't tell.

What dwdiff -d '，' gives you

dwdiff uses the bundled ICU 78.3 to do UAX #29 grapheme cluster segmentation. Combined with the -d '，' flag (Chinese comma as a delimiter), it produces:

$ dwdiff -d '，' a.txt b.txt
体育老师无奈，[-总被佔课现象困扰-]{+总被占课现象困扰+}。

The 佔课 → 占课 change is localised inside the markers. The rest of the line passes through as common text.

Choosing delimiters for CJK

dwdiff treats whitespace as a delimiter by default. For CJK, you almost always want to add CJK punctuation as additional delimiters so the diff can find word boundaries inside sentences:

# Traditional Chinese / Japanese punctuation
$ dwdiff -d '，。、；：？！（）' a.txt b.txt

# Simplified Chinese punctuation
$ dwdiff -d '，。、；：？！（）' a.txt b.txt

# Japanese kana (hiragana + katakana common chars)
$ dwdiff -d '、。・「」『』' a.txt b.txt

# Korean
$ dwdiff -d '.,?!' ko-a.txt ko-b.txt   # Korean doesn't use CJK punctuation

Or use -P (or --punctuation) to treat all Unicode punctuation as delimiters.

What ICU contributes

ICU 78.3 (statically linked into the dwdiff binary) provides:

• UTF-8 validation + grapheme cluster segmentation (UAX #29). Without this, dwdiff would treat each UTF-8 byte as a "character" and produce garbage on multi-byte CJK text.
• Casefold (UAX #21). The -i flag uses ICU's u_strFoldCase, which handles Turkish dotted-i, German sharp-s, and Greek final sigma correctly — not just naive ASCII tolower().
• Whitespace classification. ICU's u_isUWhiteSpace includes U+3000 (CJK ideographic space), U+00A0 (non-breaking space), and other non-ASCII whitespace — not just ASCII isspace().
• Normalisation (UAX #15). dwdiff normalises characters to NFD before comparing, so composed and decomposed forms (e.g. é vs e + combining acute) compare equal.

CJK acceptance test (build gate)

The build script's smoke.sh runs a CJK round-trip fixture as the release gate. The assertion is documented in security §"CJK acceptance test". A correctly-built dwdiff must produce the localised marker, not the line-level marker. Failure means ICU is not actually linked in.