fix(analyzer): keep Stanza multi-word tokens as surface tokens to preserve text and entity offsets - #2253
Open
ManoharPaturi wants to merge 1 commit into
Open
Conversation
…serve text and entity offsets Stanza's mwt processor expands German contractions (im/am/zum/zur/beim/ vom/ins/ans) into multiple words. StanzaTokenizer.__get_tokens_with_heads flattened the expanded words, which broke alignment with the original text: _convert_doc then replaced the doc text with space-separated expanded tokens, silently dropped every NER entity (offsets no longer mapped to tokens), and, on indented text, let tokens_indices end before pattern matches near the end of the text, raising 'Did not find word ...' in LemmaContextAwareEnhancer (HTTP 500). Keep a multi-word token as a single surface token (MultiWordTokenSurface: text/lemma are the surface form, other annotations come from the first expanded word) and remap 1-based word heads to the collapsed token list so dependency heads around the collapsed token stay correct. Fixes data-privacy-stack#2249
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2249.
StanzaTokenizer.__get_tokens_with_headsflattened stanza multi-word-token expansions (im->in+dem), so token texts no longer matched the source text._convert_docthen rebuilt the doc text from those tokens, which corrupted the document text (see the issue repro), made Stanza NER spans miss every token boundary so entities were silently dropped, and could raiseValueErrorfrom the lemma enhancer on indented text.Multi-word tokens now stay a single surface token (a small wrapper carrying surface text/lemma plus the first expanded word's annotations), with a word-index to token-index map that remaps dependency heads across the collapsed token. Single-word tokens keep the previous code path byte-for-byte.
Verified with the real German model:
doc.textis preserved,PERandIP_ADDRESSentities are detected where pristine drops them. 11 new tests (pure-unit head remap + German model tests over all 8 contractions) fail on main and pass here; existing stanza suites pass unchanged (12 + 13 + 16), ruff clean vs main baseline.