Description
Three related Unicode string gaps (all in test_gap_string_methods):
-
NFC/NFD normalization: "é".normalize("NFC").length returns 2 instead of 1; "é".normalize("NFD").length returns 3 instead of 2. Perry stores strings as UTF-8 bytes and .length returns byte length, not codepoint/grapheme count.
-
isWellFormed() / toWellFormed(): Lone surrogate detection returns wrong results. "\uD800".isWellFormed() returns true (should be false).
-
UTF-16 length semantics: JS .length counts UTF-16 code units, Perry counts bytes. Diverges on any string with characters outside BMP.
These are all manifestations of the same root issue: Perry uses UTF-8 byte-oriented string representation while JS uses UTF-16.
Description
Three related Unicode string gaps (all in
test_gap_string_methods):NFC/NFD normalization:
"é".normalize("NFC").lengthreturns 2 instead of 1;"é".normalize("NFD").lengthreturns 3 instead of 2. Perry stores strings as UTF-8 bytes and.lengthreturns byte length, not codepoint/grapheme count.isWellFormed()/toWellFormed(): Lone surrogate detection returns wrong results."\uD800".isWellFormed()returnstrue(should befalse).UTF-16 length semantics: JS
.lengthcounts UTF-16 code units, Perry counts bytes. Diverges on any string with characters outside BMP.These are all manifestations of the same root issue: Perry uses UTF-8 byte-oriented string representation while JS uses UTF-16.