site-migrator: extractPages/migrate hold the full page corpus in memory (OOM risk at ~100k pages)

[YusukeHirao]

Problem

extractPages (and migrate, which calls it) holds the entire page corpus's intermediate state in memory across every pipeline stage, with no chunking. At ~100k-page archive scale this risks OOM.

Concretely, in packages/@d-zero/site-migrator/src/page-extractor/extract-pages.ts:

• extractedByUrl accumulates every matched page's originalHtml + extractedHtml before the next stage can start.
• layoutResultsByUrl accumulates every page's LayoutAnalysisResult (anatomist's LayoutBlock tree, including innerHTML per leaf) before block conversion can start.
• blockOutcomeByUrl / blocksByUrl accumulate every page's converted TBlocks before the final render/write stage.

None of these are released until the whole extractPages() call returns — the corpus is processed as one giant batch, not in bounded windows.

Root cause

resolvePageLayouts is deliberately called once with every matched URL in a single batch, specifically to avoid launching a separate Puppeteer browser instance per page (see its JSDoc). That design choice requires every matched page's extracted HTML to already be in memory before the batch call can begin, which is what forces the whole-corpus-resident shape of the rest of the pipeline.

Note: resource downloads (downloadResources/downloadBlockFiles) do not have this problem — they write each file to disk immediately inside their @d-zero/dealer worker and never accumulate results in memory. Only the page-conversion pipeline in extract-pages.ts is affected.

Suggested direction

Chunk the page corpus (e.g. process N pages at a time, config­urable) through the full fetch → layout-resolve → classify → render → write pipeline, releasing each chunk's intermediate Maps before starting the next. The Puppeteer browser instance in resolve-page-layout.ts is already a lazily-launched singleton (browserPromise), so chunking should be able to reuse it across chunks without losing the "launch once" optimization — the browser lifecycle would just need to be lifted out of a single resolvePageLayouts call into something the caller manages across chunks.

Scope

This is adapter-agnostic — it affects extractPages/migrate regardless of which BlockTargetAdapter is used, so it should be fixed once in the shared pipeline rather than per adapter.

Follow-up from #1007.