fix(analyzer): match full TLDs in UrlRecognizer instead of a prefix by stephencattaneo · Pull Request #2245 · data-privacy-stack/presidio

stephencattaneo · 2026-09-09T19:43:01Z

The URL patterns never asserted that the matched TLD ends the host, so the engine stopped at the first alternative that fit. chase.bank matched only chase.ba, home.barclays only home.bar, and www.americanexpress only www.am - silently truncating the host and dropping the path, so anonymizing chase.bank/login left nk/login in the output.

Add a (?![a-z0-9\-]) boundary after the TLD. Ordering alone cannot fix this: bar is itself a gTLD, so home.barclays truncates regardless of how the alternation is sorted.

Replace the hand-maintained TLD list with the full IANA root zone (1438 entries, vendored as tlds.txt with update_tlds.py to refresh it), so financial TLDs such as .bank, .barclays, .americanexpress, .insurance and .creditunion are covered.

Compress the alternation into a prefix tree rather than a flat (?:aaa)|(?:aarp)|.... Despite covering twice as many TLDs, matching is ~30% faster (32.9ms -> 22.8ms per 26KB document) since the engine can discard whole subtrees on a single character; compiled patterns cost +4.4MB, once per process, as PATTERNS is class-level.

Keeping the TLD set inside the regex, rather than validating the last label in Python, avoids false negatives: the host part is greedy, so Visit www.example.com.Then call matches www.example.com.Then, and post-hoc validation can only reject the whole match. Backtracking finds the correct boundary and reports www.example.com.

Note www.microsoft is now detected, as MICROSOFT is a delegated brand TLD - the same reason BARCLAYS and AMERICANEXPRESS are matched.

Change Description

Describe your changes

Issue reference

Fixes #2244

Checklist

I have reviewed the contribution guidelines
I agree to follow this project's Code of Conduct
I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
My code includes unit tests
All unit tests and lint checks pass locally
My PR contains documentation updates / additions if required

The URL patterns never asserted that the matched TLD ends the host, so the engine stopped at the first alternative that fit. `chase.bank` matched only `chase.ba`, `home.barclays` only `home.bar`, and `www.americanexpress` only `www.am` - silently truncating the host and dropping the path, so anonymizing `chase.bank/login` left `nk/login` in the output. Add a `(?![a-z0-9\-])` boundary after the TLD. Ordering alone cannot fix this: `bar` is itself a gTLD, so `home.barclays` truncates regardless of how the alternation is sorted. Replace the hand-maintained TLD list with the full IANA root zone (1438 entries, vendored as tlds.txt with update_tlds.py to refresh it), so financial TLDs such as .bank, .barclays, .americanexpress, .insurance and .creditunion are covered. Compress the alternation into a prefix tree rather than a flat `(?:aaa)|(?:aarp)|...`. Despite covering twice as many TLDs, matching is ~30% faster (32.9ms -> 22.8ms per 26KB document) since the engine can discard whole subtrees on a single character; compiled patterns cost +4.4MB, once per process, as PATTERNS is class-level. Keeping the TLD set inside the regex, rather than validating the last label in Python, avoids false negatives: the host part is greedy, so `Visit www.example.com.Then call` matches `www.example.com.Then`, and post-hoc validation can only reject the whole match. Backtracking finds the correct boundary and reports `www.example.com`. Note `www.microsoft` is now detected, as MICROSOFT is a delegated brand TLD - the same reason BARCLAYS and AMERICANEXPRESS are matched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

🟡 Changes recommended

It drops previously supported legacy TLDs and lacks configuration-path detection coverage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes truncated URL recognition by enforcing TLD boundaries and using the complete IANA TLD list.

Changes:

Generates an optimized trie-based TLD regex.
Adds an IANA TLD update utility and vendored dataset.
Expands URL boundary and brand-TLD tests.

File summaries

File	Description
`test_url_recognizer.py`	Tests full TLD matching and trie correctness.
`pyproject.toml`	Includes the vendored TLD file.
`url_recognizer.py`	Loads TLDs and generates the URL regex.
`update_tlds.py`	Refreshes the IANA dataset.
`tlds.txt`	Provides the current IANA root-zone list.

Review details

Files reviewed: 5/5 changed files
Comments generated: 2
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

    """

-    BASE_URL_REGEX = r"((www\d{0,3}[.])?[a-z0-9.\-]{1,253}[.](?:(?:com)|(?:edu)|(?:gov)|(?:int)|(?:mil)|(?:net)|(?:onl)|(?:org)|(?:pro)|(?:red)|(?:tel)|(?:uno)|(?:xxx)|(?:academy)|(?:accountant)|(?:accountants)|(?:actor)|(?:adult)|(?:africa)|(?:agency)|(?:airforce)|(?:apartments)|(?:app)|(?:archi)|(?:army)|(?:art)|(?:asia)|(?:associates)|(?:attorney)|(?:auction)|(?:audio)|(?:auto)|(?:autos)|(?:baby)|(?:band)|(?:bar)|(?:bargains)|(?:beer)|(?:berlin)|(?:best)|(?:bet)|(?:bid)|(?:bike)|(?:bio)|(?:black)|(?:blackfriday)|(?:blog)|(?:blue)|(?:boats)|(?:bond)|(?:boo)|(?:boston)|(?:bot)|(?:boutique)|(?:build)|(?:builders)|(?:business)|(?:buzz)|(?:cab)|(?:cafe)|(?:cam)|(?:camera)|(?:camp)|(?:capital)|(?:car)|(?:cards)|(?:care)|(?:careers)|(?:cars)|(?:casa)|(?:cash)|(?:casino)|(?:catering)|(?:center)|(?:ceo)|(?:cfd)|(?:charity)|(?:chat)|(?:cheap)|(?:christmas)|(?:church)|(?:city)|(?:claims)|(?:cleaning)|(?:click)|(?:clinic)|(?:clothing)|(?:cloud)|(?:club)|(?:codes)|(?:coffee)|(?:college)|(?:com)|(?:community)|(?:company)|(?:computer)|(?:condos)|(?:construction)|(?:consulting)|(?:contact)|(?:contractors)|(?:cooking)|(?:cool)|(?:coupons)|(?:courses)|(?:credit)|(?:creditcard)|(?:cricket)|(?:cruises)|(?:cyou)|(?:dad)|(?:dance)|(?:date)|(?:dating)|(?:day)|(?:degree)|(?:delivery)|(?:democrat)|(?:dental)|(?:dentist)|(?:desi)|(?:design)|(?:dev)|(?:diamonds)|(?:diet)|(?:digital)|(?:direct)|(?:directory)|(?:discount)|(?:doctor)|(?:dog)|(?:domains)|(?:download)|(?:earth)|(?:eco)|(?:education)|(?:email)|(?:energy)|(?:engineer)|(?:engineering)|(?:enterprises)|(?:equipment)|(?:esq)|(?:estate)|(?:events)|(?:exchange)|(?:expert)|(?:exposed)|(?:express)|(?:fail)|(?:faith)|(?:family)|(?:fans)|(?:farm)|(?:fashion)|(?:feedback)|(?:film)|(?:finance)|(?:financial)|(?:fish)|(?:fishing)|(?:fit)|(?:fitness)|(?:flights)|(?:florist)|(?:flowers)|(?:football)|(?:forsale)|(?:foundation)|(?:fun)|(?:fund)|(?:furniture)|(?:futbol)|(?:fyi)|(?:gallery)|(?:game)|(?:games)|(?:garden)|(?:gay)|(?:gdn)|(?:gifts)|(?:gives)|(?:giving)|(?:glass)|(?:global)|(?:gmbh)|(?:gold)|(?:golf)|(?:graphics)|(?:gratis)|(?:green)|(?:gripe)|(?:group)|(?:guide)|(?:guitars)|(?:guru)|(?:hair)|(?:hamburg)|(?:haus)|(?:health)|(?:healthcare)|(?:help)|(?:hiphop)|(?:hockey)|(?:holdings)|(?:holiday)|(?:homes)|(?:horse)|(?:hospital)|(?:host)|(?:hosting)|(?:house)|(?:how)|(?:icu)|(?:info)|(?:ink)|(?:institute)|(?:insure)|(?:international)|(?:investments)|(?:irish)|(?:jewelry)|(?:jetzt)|(?:juegos)|(?:kaufen)|(?:kids)|(?:kitchen)|(?:kiwi)|(?:krd)|(?:kyoto)|(?:land)|(?:lat)|(?:law)|(?:lawyer)|(?:lease)|(?:legal)|(?:lgbt)|(?:life)|(?:lighting)|(?:limited)|(?:limo)|(?:link)|(?:live)|(?:loan)|(?:loans)|(?:lol)|(?:london)|(?:love)|(?:ltd)|(?:ltda)|(?:luxury)|(?:maison)|(?:management)|(?:market)|(?:marketing)|(?:markets)|(?:mba)|(?:media)|(?:melbourne)|(?:meme)|(?:memorial)|(?:men)|(?:miami)|(?:mobi)|(?:moda)|(?:moe)|(?:mom)|(?:money)|(?:monster)|(?:mortgage)|(?:motorcycles)|(?:mov)|(?:movie)|(?:nagoya)|(?:name)|(?:navy)|(?:network)|(?:new)|(?:news)|(?:ngo)|(?:ninja)|(?:now)|(?:nyc)|(?:observer)|(?:okinawa)|(?:one)|(?:ong)|(?:onl)|(?:online)|(?:organic)|(?:osaka)|(?:page)|(?:paris)|(?:partners)|(?:parts)|(?:party)|(?:pet)|(?:phd)|(?:photo)|(?:photography)|(?:photos)|(?:pics)|(?:pictures)|(?:pink)|(?:pizza)|(?:place)|(?:plumbing)|(?:plus)|(?:poker)|(?:porn)|(?:press)|(?:pro)|(?:productions)|(?:prof)|(?:promo)|(?:properties)|(?:property)|(?:protection)|(?:pub)|(?:quest)|(?:racing)|(?:recipes)|(?:red)|(?:rehab)|(?:reise)|(?:reisen)|(?:rent)|(?:rentals)|(?:repair)|(?:report)|(?:republican)|(?:rest)|(?:restaurant)|(?:review)|(?:reviews)|(?:rip)|(?:rocks)|(?:rodeo)|(?:rsvp)|(?:run)|(?:saarland)|(?:sale)|(?:salon)|(?:sarl)|(?:sbs)|(?:school)|(?:schule)|(?:science)|(?:services)|(?:sex)|(?:sexy)|(?:sh)|(?:shoes)|(?:shop)|(?:shopping)|(?:show)|(?:singles)|(?:site)|(?:skin)|(?:soccer)|(?:social)|(?:software)|(?:solar)|(?:solutions)|(?:soy)|(?:space)|(?:spiegel)|(?:study)|(?:style)|(?:sucks)|(?:supply)|(?:support)|(?:surf)|(?:surgery)|(?:systems)|(?:tax)|(?:taxi)|(?:team)|(?:tech)|(?:technology)|(?:tel)|(?:theater)|(?:tips)|(?:tires)|(?:today)|(?:tools)|(?:top)|(?:tours)|(?:town)|(?:toys)|(?:trade)|(?:training)|(?:tube)|(?:uk)|(?:university)|(?:uno)|(?:vacations)|(?:ventures)|(?:vet)|(?:video)|(?:villas)|(?:vin)|(?:vip)|(?:vision)|(?:vlaanderen)|(?:vodka)|(?:vote)|(?:voting)|(?:voyage)|(?:wales)|(?:wang)|(?:watch)|(?:webcam)|(?:website)|(?:wedding)|(?:wiki)|(?:wine)|(?:work)|(?:works)|(?:world)|(?:wtf)|(?:xyz)|(?:yoga)|(?:yokohama)|(?:you)|(?:zone)|(?:ac)|(?:ad)|(?:ae)|(?:af)|(?:ag)|(?:ai)|(?:al)|(?:am)|(?:an)|(?:ao)|(?:aq)|(?:ar)|(?:as)|(?:at)|(?:au)|(?:aw)|(?:ax)|(?:az)|(?:ba)|(?:bb)|(?:bd)|(?:be)|(?:bf)|(?:bg)|(?:bh)|(?:bi)|(?:bj)|(?:bm)|(?:bn)|(?:bo)|(?:br)|(?:bs)|(?:bt)|(?:bv)|(?:bw)|(?:by)|(?:bz)|(?:ca)|(?:cc)|(?:cd)|(?:cf)|(?:cg)|(?:ch)|(?:ci)|(?:ck)|(?:cl)|(?:cm)|(?:cn)|(?:co)|(?:cr)|(?:cu)|(?:cv)|(?:cw)|(?:cx)|(?:cy)|(?:cz)|(?:de)|(?:dj)|(?:dk)|(?:dm)|(?:do)|(?:dz)|(?:ec)|(?:ee)|(?:eg)|(?:er)|(?:es)|(?:et)|(?:eu)|(?:fi)|(?:fj)|(?:fk)|(?:fm)|(?:fo)|(?:fr)|(?:ga)|(?:gb)|(?:gd)|(?:ge)|(?:gf)|(?:gg)|(?:gh)|(?:gi)|(?:gl)|(?:gm)|(?:gn)|(?:gp)|(?:gq)|(?:gr)|(?:gs)|(?:gt)|(?:gu)|(?:gw)|(?:gy)|(?:hk)|(?:hm)|(?:hn)|(?:hr)|(?:ht)|(?:hu)|(?:id)|(?:ie)|(?:il)|(?:im)|(?:in)|(?:io)|(?:iq)|(?:ir)|(?:is)|(?:it)|(?:je)|(?:jm)|(?:jo)|(?:jp)|(?:ke)|(?:kg)|(?:kh)|(?:ki)|(?:km)|(?:kn)|(?:kp)|(?:kr)|(?:kw)|(?:ky)|(?:kz)|(?:la)|(?:lb)|(?:lc)|(?:li)|(?:lk)|(?:lr)|(?:ls)|(?:lt)|(?:lu)|(?:lv)|(?:ly)|(?:ma)|(?:mc)|(?:md)|(?:me)|(?:mg)|(?:mh)|(?:mk)|(?:ml)|(?:mm)|(?:mn)|(?:mo)|(?:mp)|(?:mq)|(?:mr)|(?:ms)|(?:mt)|(?:mu)|(?:mv)|(?:mw)|(?:mx)|(?:my)|(?:mz)|(?:na)|(?:nc)|(?:ne)|(?:nf)|(?:ng)|(?:ni)|(?:nl)|(?:no)|(?:np)|(?:nr)|(?:nu)|(?:nz)|(?:om)|(?:pa)|(?:pe)|(?:pf)|(?:pg)|(?:ph)|(?:pk)|(?:pl)|(?:pm)|(?:pn)|(?:pr)|(?:ps)|(?:pt)|(?:pw)|(?:py)|(?:qa)|(?:re)|(?:ro)|(?:rs)|(?:ru)|(?:rw)|(?:sa)|(?:sb)|(?:sc)|(?:sd)|(?:se)|(?:sg)|(?:sh)|(?:si)|(?:sj)|(?:sk)|(?:sl)|(?:sm)|(?:sn)|(?:so)|(?:sr)|(?:st)|(?:su)|(?:sv)|(?:sx)|(?:sy)|(?:sz)|(?:tc)|(?:td)|(?:tf)|(?:tg)|(?:th)|(?:tj)|(?:tk)|(?:tl)|(?:tm)|(?:tn)|(?:to)|(?:tp)|(?:tr)|(?:tt)|(?:tv)|(?:tw)|(?:tz)|(?:ua)|(?:ug)|(?:uk)|(?:us)|(?:uy)|(?:uz)|(?:va)|(?:vc)|(?:ve)|(?:vg)|(?:vi)|(?:vn)|(?:vu)|(?:wf)|(?:ws)|(?:ye)|(?:yt)|(?:za)|(?:zm)|(?:zw))(?:/[^\s()<>\"']*)?)"  # noqa: E501
+    TLDS = load_tlds()



+        # Brand TLDs, incl. ones that share a prefix with a shorter TLD
+        # (.bank/.ba, .barclays/.bar, .americanexpress/.am)
+        ("https://www.chase.bank/login", 1, ((0, 28),), 0.6,),


omri374 · 2026-09-10T08:48:00Z

+        + r"(?:/[^\s()<>\"']*)?)"
+    )

    PATTERNS = [


Please split the no-scheme pattern by TLD case so confidence reflects the false-positive risk:

Standard Url with scheme: unchanged, 0.6, case-insensitive. https://Example.Com is a URL regardless of case.

Non schema URL: 0.5, TLD must be all-lowercase or all-uppercase: (?-i: + lowercase trie + | + uppercase trie + ) inside the existing (?i) pattern. The regex module supports scoped flags.

Non schema URL, mixed-case TLD: 0.3, the current case-insensitive pattern.

remove_duplicates already keeps the higher-scoring result for an identical span, so no analyze override is needed, and the pattern name lands in AnalysisExplanation for free. I prototyped this and got to.microsoft / TO.MICROSOFT / www.chase.bank at 0.5, Done.Next / to.Microsoft / Example.Com at 0.3, https://Example.Com at 0.6.

Please add those as test cases. Note Example.Com already matched at 0.5 on main, so this is a behavior change to declare in the description.

omri374 · 2026-09-10T08:48:00Z

+    root: Dict = {}
+    for tld in tlds:
+        node = root
+        for char in tld.lower():


Gotcha for the uppercase trie above: this lowercases the input, so tlds_to_regex(t.upper() for t in tlds) returns the same lowercase fragment. Either drop the lowercasing here (the file is already lowercase) or .upper() the generated fragment, which is safe since it only contains letters, digits and regex punctuation.

omri374 · 2026-09-10T08:48:00Z

 urls = {Homepage = "https://github.com/data-privacy-stack/presidio"}
 readme = "README.md"
-include = ["conf/*",]
+include = ["conf/*", "predefined_recognizers/generic/tlds.txt"]


This is a no-op, please revert. include under [project] is not a PEP 621 field, so poetry-core ignores it, and the path would be wrong relative to the project root anyway. I built the wheel with and without this line: tlds.txt ships either way, because poetry-core packages every file under the package directory.

omri374 · 2026-09-10T08:48:00Z

@@ -0,0 +1,41 @@
+"""Refresh the vendored IANA root zone TLD list used by ``UrlRecognizer``."""


This is a maintenance script that does network I/O, and it currently ships in the wheel (I checked the built artifact). Please move it out of the package and we'll periodically update it manually

github-actions Bot added external PII recognizers analyzer dependencies Pull requests that update a dependency file test labels Sep 9, 2026

omri374 requested a balanced review from Copilot September 10, 2026 06:49

Copilot started reviewing on behalf of omri374 September 10, 2026 06:50 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

omri374 reviewed Sep 10, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix(analyzer): match full TLDs in UrlRecognizer instead of a prefix - #2245

fix(analyzer): match full TLDs in UrlRecognizer instead of a prefix#2245
stephencattaneo wants to merge 1 commit into
data-privacy-stack:mainfrom
multimodelai:fix/url-recognizer-tld-boundary

stephencattaneo commented Sep 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

omri374 Sep 10, 2026

Uh oh!

omri374 Sep 10, 2026

Uh oh!

omri374 Sep 10, 2026

Uh oh!

omri374 Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,41 @@
		"""Refresh the vendored IANA root zone TLD list used by ``UrlRecognizer``."""

Uh oh!

Conversation

stephencattaneo commented Sep 9, 2026

Change Description

Issue reference

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

omri374 Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

omri374 Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

omri374 Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

omri374 Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants