fix(co-occurrence): exclude import declarations from clone detection at the tokenizer

[norvalbv]

What

jscpd now runs with --ignore-pattern regexes that remove import/re-export declarations from the token stream before clone detection. The gate's semantic becomes: clones are measured over non-import code. Two modules using the same shared libs are forced into identical, linter-sorted import preambles — the module system working, not copy-paste — and post-#305 the repo scan showed two such blocks big enough to clear the 50-token floor on their own, which would false-block any commit touching those sibling files once a release wires the gate into hooks.

Why tokenizer-level, not fragment classification

The first design classified reported fragments post-hoc (isImportPreamble). The commit-gate correctness reviewer killed six drafts across seven rounds, each with a verified exploit: a ratio threshold hid a duplicated statement behind imports; /^import\b/ matched dynamic import(...) whose unterminated state absorbed callbacks; an incomplete-statement excusal let a truncated call ride; a floorless excusal swallowed zero-import JSX molecules; the interior state accepted any quote-terminated line (SECRET_KEY = "x"); import-attributes stuck the state open. The lesson is structural: classifying jscpd's token-boundary fragments after the fact is unfixable — every heuristic boundary is exploitable. Removing imports before tokenization eliminates the attack surface: nothing can ride a preamble over the token floor, and nothing real can hide behind one. The adversarial cases live on as end-to-end gate-contract tests (31 total, all green; full suite passed pre-push).

The four patterns are bounded (single-line forms forbid \n; the multi-line form crosses lines only inside {...} and requires the } from '...' closer) and comma-free (the CLI comma-splits the option). Dynamic import(...) deliberately matches none. Unmatched exotic shapes fall through to normal tokenization — the safe direction: worst case a clone reports and the allowlist is the remedy.

Live scan: 62 → 60. The dropped sync-agents fragment's residual LegacyAssetManifest dup is sub-floor on its own merits — it only crossed 50 tokens by riding the imports (it's still on the inventory below as dedup-worthy).

The remaining 60 clones — inventory + recommendation (your call, not in this PR)

• 43 eval-harness clones (review/eval vs review/eval/conventions bench+matcher family, plus critique/eval): recommend guard-clone json | guard-dup-allowlist baseline-clones rather than dedup — each suite hashes its runner/scorer implementation, so deduping the copied harness drifts published checkpoint hashes for 2–3 suites with zero behavior change (the overrides.mts precedent). Dedupe opportunistically when those harnesses next change substantively.
• 1 deliberate: run-review.mts ↔ reviewers/bench.mts (bench drives the real cascade by design; also implementation-hash-sensitive). Baseline it.
• 16 real product-code clones worth actual dedup, biggest clusters: cli/lib/ship/review/ (repository/state.mts is a 4-clone hub), sync-agents/sync-skills residuals (incl. the twice-declared LegacyAssetManifest interface), sync-manifest↔install-hooks, critique/ lifecycle pair, decisions/check-alignment↔detect. Happy to take these as a follow-up branch when you're out of the repo.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved duplicate-code detection by excluding import and re-export statements from clone analysis.
  • Import-only repetition no longer incorrectly blocks validation.
  • Duplicate executable code, including code following imports and dynamic-import logic, continues to be detected and reported.

• Tests

  • Added coverage for import-only duplication, duplicated code after imports, and duplicated dynamic-import runtime code.

[coderabbitai]

📝 Walkthrough

Walkthrough

The clone detector now passes bounded import and re-export ignore patterns to jscpd. Integration tests verify that import-only duplication is ignored, while duplicated executable and dynamic-import code still blocks.

Changes

Clone filtering

Layer / File(s)	Summary
Import pattern filtering gate-engine/co-occurrence/clone-detector.mts	IMPORT_IGNORE_PATTERNS covers named, namespace, default, side-effect, and re-export declarations. detectClones passes these patterns to jscpd.
Clone filtering validation gate-engine/co-occurrence/__tests__/clone-detector.test.mts	Tests verify that identical import preambles do not block, while duplicated executable code and dynamic-import blocks produce exit status 1.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes excluding import declarations from tokenizer-level clone detection.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/clone-gate-import-preamble

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@gate-engine/co-occurrence/clone-detector.mts`:
- Around line 239-240: Extend the re-export pattern in IMPORT_IGNORE_PATTERNS to
match `export type { ... } from ...` alongside existing named and wildcard
re-exports, while preserving current matching behavior; add contract cases
covering type-only re-exports so duplicated type-only preambles are ignored.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 186c329f-93a9-42e3-88ef-63de126ef444

📥 Commits

Reviewing files that changed from the base of the PR and between 6e9949c and 8ce4663.

📒 Files selected for processing (2)

• gate-engine/co-occurrence/__tests__/clone-detector.test.mts
• gate-engine/co-occurrence/clone-detector.mts

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C2 --glob '*.{ts,tsx,mts}' \
  '^\s*export\s+(type\s+)?\*\s+as\s+|^\s*export\s+type\s+\{' .

Repository: norvalbv/devkit

Length of output: 2263

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== targeted source =="
sed -n '220,250p' gate-engine/co-occurrence/clone-detector.mts

echo
echo "== relevant static re-export declarations =="
rg -n --glob '*.{ts,tsx,mts}' '^\s*export\s+(\*|\{[^}]*\})\s+as\s+|\bexport\s+type\s+\{' . || true

echo
echo "== read-only regex behavior probe =="
python3 - <<'PY'
import re
pattern = re.compile(r"export\s*(\{[^}]*\}|\*)\s*from\s*['"][^'"]+['"];?", re.I)
cases = [
    "export { Widget } from './widgets';",
    "export * from './widgets';",
    "export {} from './widgets';",
    "export type { Widget } from './widgets';",
    "export * as widgets from './widgets';",
    "export * as $widgets from './widgets';",
]
for case in cases:
    print(case, "-->", "matches" if pattern.search(case) else "no match")
PY

Repository: norvalbv/devkit

Length of output: 2945

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== pattern definitions =="
rg -n --context 8 "IMPORT_IGNORE_PATTERNS" gate-engine/co-occurrence/clone-detector.mts || true

echo
echo "== broad source form =="
rg -n "Re-export barrels|export\\s\\(\\{\\[^\\}\\]\\*\\}\\)|import_ignore_patterns|IMPORT_IGNORE_PATTERNS" gate-engine -S || true

echo
echo "== existing export declarations =="
rg -n --glob '*.{ts,tsx,mts}' '^\s*export\s+(type\s+)?\*(\s+as\s+[\w$]+)?\s+from\s+|^\s*export\s+type\s+\{' . || true

echo
echo "== read-only regex behavior probe =="
python3 - <<'PY'
import re

literal_source = r"""String.raw`export\s*(\{[^}]*\}|\*)\s*from\s*['"][^'"]+['"];?`,"""
current = re.compile(r"export\s*(\{[^}]*\}|\*)\s*from\s*['"][^'"]+['"];?")
proposed = re.compile(r"export\s*(type\s+)?(\{[^}]*\}|\*(\s+as\s+[\w$]+)?)\s*from\s*['"][^'"]+['"];?")

cases = [
    "export { Widget } from './widgets';",
    "export * from './widgets';",
    "export {} from './widgets';",
    "export type { Widget } from './widgets';",
    "export * as widgets from './widgets';",
    "export * as $widgets from './widgets';",
]

for case in cases:
    print(case)
    print(" current matches", current.search(case) is not None)
    print(" proposed matches", proposed.search(case) is not None)
PY

Repository: norvalbv/devkit

Length of output: 3483

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 <<'PY'
import re

current = re.compile(r"export\s*(\{[^}]*\}|\*)\s*from\s*['\"].*?['\"];\??")
namespace = re.compile(r"export\s*(type\s+)?(\{[^}]*\}|\*(\s+as\s+[\w$]+)?)\s*from\s*['\"].*?['\"];\??")

cases = [
    "export { Widget } from './widgets';",
    "export * from './widgets';",
    "export type { Widget } from './widgets';",
    "export * as widgets from './widgets';",
    "export * as $widgets from './widgets';",
]

for case in cases:
    print(case)
    print(" current matches", bool(current.search(case)))
    print(" proposed matches", bool(namespace.search(case)))
PY

Repository: norvalbv/devkit

Length of output: 562

---

Match type-only re-exports.

Type-only exports are valid module declarations, and export type { ... } from 'module'; is currently present in the repo. It falls outside the current IMPORT_IGNORE_PATTERNS re-export regex, so duplicated type-only re-export preambles can still block the gate. Extend the pattern for export type { ... } from ... and add matching contract cases.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@gate-engine/co-occurrence/clone-detector.mts` around lines 239 - 240, Extend
the re-export pattern in IMPORT_IGNORE_PATTERNS to match `export type { ... }
from ...` alongside existing named and wildcard re-exports, while preserving
current matching behavior; add contract cases covering type-only re-exports so
duplicated type-only preambles are ignored.