Skip to content

Omitted deny_list_score yields 0.0 through RecognizerRegistryProvider but 1.0 through PatternRecognizer and add_pattern_recognizer_from_dict #2242

Description

@omri374

Describe the bug

A custom deny-list recognizer that omits deny_list_score gets a different score depending on how it is loaded:

Path deny_list_score Deny-list pattern score
PatternRecognizer(supported_entity=..., deny_list=[...]) 1.0 1.0
RecognizerRegistry.add_pattern_recognizer_from_dict({...}) 1.0 1.0
RecognizerRegistryProvider(registry_configuration={...}) (YAML / dict registry config) 0.0 0.0

Cause: CustomRecognizerConfig.deny_list_score in presidio_analyzer/input_validation/yaml_recognizer_models.py declares default=0.0, and the validated model is dumped with that default present, so the loader passes deny_list_score=0.0 to PatternRecognizer even when the YAML said nothing. PatternRecognizer.__init__ defaults to 1.0.

A deny-list match with score 0.0 is filtered out by any positive score_threshold, so a YAML deny list without an explicit score effectively never fires through the provider path.

To Reproduce

from presidio_analyzer import PatternRecognizer
from presidio_analyzer.recognizer_registry import RecognizerRegistry, RecognizerRegistryProvider

conf = {
    "supported_languages": ["en"],
    "global_regex_flags": 26,
    "recognizers": [
        {"name": "Titles", "supported_entity": "TITLE", "deny_list": ["Mr.", "Mrs."]}
    ],
}
reg = RecognizerRegistryProvider(registry_configuration=conf).create_recognizer_registry()
r = next(x for x in reg.recognizers if x.name == "Titles")
print(r.deny_list_score, [p.score for p in r.patterns])   # 0.0 [0.0]

print(PatternRecognizer(supported_entity="TITLE", deny_list=["Mr."]).deny_list_score)  # 1.0

reg2 = RecognizerRegistry()
reg2.add_pattern_recognizer_from_dict(
    {"name": "T2", "supported_language": "en", "supported_entity": "TITLE", "deny_list": ["Mr."]}
)
print(reg2.recognizers[-1].deny_list_score)  # 1.0

Expected behavior

Omitting deny_list_score means the PatternRecognizer default (1.0) on every path. An explicit value is honored as today.

Proposed fix

Change CustomRecognizerConfig.deny_list_score to Optional[float] = Field(default=None, ge=0.0, le=1.0) and dump recognizer entries with exclude_unset=True, so an omitted key never reaches the constructor.

Behavior change

Deny-list matches from YAML registry configs without an explicit deny_list_score change from 0.0 to 1.0. Anyone relying on the 0.0 default should set deny_list_score explicitly. The change must be called out in the PR description and the docs (docs/analyzer/recognizer_registry_provider.md).

Environment

main at 5e2fcea, presidio-analyzer.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions