Skip to content

String NFC/NFD normalization and lone surrogate handling #18

Description

@proggeramlug

Description

Three related Unicode string gaps (all in test_gap_string_methods):

  1. NFC/NFD normalization: "é".normalize("NFC").length returns 2 instead of 1; "é".normalize("NFD").length returns 3 instead of 2. Perry stores strings as UTF-8 bytes and .length returns byte length, not codepoint/grapheme count.

  2. isWellFormed() / toWellFormed(): Lone surrogate detection returns wrong results. "\uD800".isWellFormed() returns true (should be false).

  3. UTF-16 length semantics: JS .length counts UTF-16 code units, Perry counts bytes. Diverges on any string with characters outside BMP.

These are all manifestations of the same root issue: Perry uses UTF-8 byte-oriented string representation while JS uses UTF-16.

Metadata

Metadata

Assignees

No one assigned

    Labels

    parityCompatibility gap with Node.js, ECMAScript, or the supported ecosystem

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions