feat(page-cluster): landmark-based chrome extraction and clustering pipeline - #907
Merged
Merged
Conversation
Add resolvePageClusterKeys to connect resolveBlockingGroupKeys and
resolveStructuralClusterKeys into one entry point, composing cross-block-safe
cluster keys via JSON.stringify. Add subpath exports for every function built
in prior PRs (derivePathGroupKey, deriveStylesheetGroupKey,
resolveBlockingGroupKeys, resolveStructuralClusterKeys,
computeDocumentFrequency, splitTokensByFrequency, arrayEditDistance,
jaccardSimilarity, resolvePageClusterKeys) — package.json's exports field
previously mapped only tokenize(), leaving everything else unreachable from
outside the package.
Fix confirmed bugs found via real-data validation and an xhigh code review:
- tokenize() no longer folds <body>'s own class into every leaf path (CMS
body_class() conventions inject per-page-category labels there, which
collapsed near-duplicate pages to zero similarity); the class is returned
separately as bodyClassList instead of being discarded
- resolveStructuralClusterKeys now narrows each page to its page-specific
content via splitTokensByFrequency before comparing, so chrome no longer
inflates similarity between pages with little shared content, with a
documented floor (n >= 10) and a raw-token fallback for a page that
narrows to an empty set (two empty sets compare as similarity 1, which
would otherwise force-merge two structurally unrelated all-chrome pages)
- noise-class-patterns.ts's sc-/css-/generic-hex patterns no longer flag
real English words that happen to fit a hash's character shape
(sc-header, css-editor, section-facade, ...)
- a pre-existing literal NUL byte in tokenize.production-scale.spec.ts that
made git treat the whole file as binary, hiding every diff to it
Fix resolveBlockingGroupKeys' JSDoc, which claimed no later refinement step
exists to justify its exclusive css:/path: key assignment — that premise
predates resolveStructuralClusterKeys and is no longer accurate; the
exclusivity itself is deliberately left unchanged pending a dedicated
follow-up (an OR-merge redesign needs its own validation cycle).
BREAKING CHANGE: tokenize() now returns { tokens, bodyClassList } instead of
a bare string[]. Package is 0.x, so no migration guide is required.
…clustering The frequency-based chrome/content split alone still left real crawl data over-fragmented (302 pages into 192 clusters). Explicit landmark decomposition removes shared site chrome by structural markers (<header>/<footer>/<nav>/<aside> tags or the matching ARIA landmark roles) instead of statistical inference, avoiding both the classify-then-refine circularity and the naive iterative chrome- broadening approach already shown to backfire (removing a shared token subset can only lower Jaccard similarity, never raise it). - add extractLandmarks(): carves out the single shallowest instance of each landmark type per page, returning both the landmark HTML and the remaining content-only HTML - add resolveLandmarkVariantKeys(): classifies which landmark variant (e.g. "which header design") a page has, as metadata independent of the overall per-page template key - resolvePageClusterKeys() now takes each page's raw html instead of a pre-tokenized Set, so it can exclude landmarks before comparing (breaking change, package is 0.x); excludeLandmarks defaults to true and can be disabled to fall back to the previous behavior - share opaque-tag (script/style/noscript/svg) detection between run-tokenizer.ts and the new extract-landmarks.ts via opaque-tags.ts instead of duplicating the tag list and type guard Validated against two real crawl corpora throughout implementation: header/footer/nav present on 99%+ of pages, extractLandmarks resolves malformed/unclosed landmark tags safely (discards the candidate rather than corrupting the remainder), and resolveLandmarkVariantKeys classifies an 800-page real sample's headers into 6 clean variants versus the ~2,000 clusters a full-page comparison produces.
…raction Code review flagged that upgrading to excludeLandmarks:true (the new default) without re-tuning similarityThreshold can silently split or merge pages differently than before, with only prose in the JSDoc warning about it. Add a regression test reproducing the same shape as the JSDoc's real-crawl-data example (shared header inflates raw similarity above 0.8, but is at 0.6 once landmarks are excluded) so the claim is enforced by the test suite, not just documented.
… regex
htmlparser2 accepts tag names containing regex metacharacters (e.g.
"<div(foo role=\"banner\">" parses with tag name "div(foo"). Building
isGenuineClose's RegExp directly from an unescaped tag name let such
markup either throw ("Unterminated group") or silently change what
the regex matched. Escape the tag name before interpolating it.
…actLandmarks ExtractLandmarksResult's JSDoc said a field is absent only "if the page has none", but a field is also left absent when the only candidate found was malformed markup extractLandmarks declined to trust. Document that on extractLandmarks itself (where the discard happens) and cross-reference it from the result type. Add a regression test pinning that a discarded malformed candidate still falls back to another well-formed candidate of the same type when one exists, rather than giving up on that type entirely.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
resolveStructuralClusterKeys, which clusters pages already grouped into oneblocking key (e.g. from
resolveBlockingGroupKeys) by structural similarity, usingcomplete-linkage hierarchical clustering computed via the NN-chain algorithm (O(n²)).
first but rejected: its "chaining" failure mode lets one unrepresentative page
transitively merge two otherwise-unrelated templates, defeating template detection.
Complete-linkage requires every pair across two clusters to clear the threshold,
ruling that out, at no extra asymptotic cost since the pairwise similarity matrix is
computed either way.
the exact clustering in O(n²), and real-data validation (see test plan) confirmed
O(n²) is fast enough at the largest real block size found (~1,400 pages, ~685ms).
Test plan
yarn build/yarn lint/yarn testpass/code-review xhighrun; 4 findings (floating-point threshold boundary epsilon,unicorn/prefer-math-trunc, Prettier formatting, cspell) all fixed/qa-engineerreview added a differential test against a naive brute-forcecomplete-linkage reference (caught one real algorithm bug during development —
an early-exit-on-threshold that's unsound when multiple disjoint NN-chains
exist), plus a regression test for the epsilon fix and threshold boundary tests
/product-managerand/docreviews passed with one JSDoc gap fixedperformance at the largest observed real block size, and found (as a documented,
non-blocking finding for a future iteration) that per-page body classes on some
real sites can zero out structural similarity between same-template pages —
a caller-side normalization concern, not a bug in this function