feat: add a WASM fallback backend - #89
Open
figulusproject wants to merge 12 commits into
Open
Conversation
- update lockfile from v1 to v3 - update glob from 7 to ^12 (fixes 2 high severity vulnerabilities) - update underscore from ^1.12.1 to ^1.13.7 (fixes high severity vulnerability) - update postinstall.js to use globSync (required after updating glob version) - version++
- pin node-addon-api to ^2.0.0 (was "*", which resolved to a version requiring C++17's <string_view>, unsupported by the VS2015/MSBuild 14.0 toolchain used in CI) - update package-lock.json
- move NODE_API_MODULE out of the NodeCld namespace (VS2015's non-conformant name lookup misresolves the macro's internal Napi::RegisterModule call as a member of NodeCld::Napi, causing error C2039) - add using NodeCld::Init declaration since NODE_API_MODULE token-pastes its argument and can't take a qualified name
- no glob release fixes the brace-expansion DoS advisory (GHSA-mh99-v99m-4gvg) while supporting Node <20, and glob@^12 itself requires "20 || >=22" - drop Node 8/10/12 from the AppVeyor test matrix - bump os to Visual Studio 2022 (VS2015/MSBuild 14.0 lacks the C++17 support Node 20/22 addon builds need) - bump package.json engines to "20 || >=22" - update package-lock.json
- move CLDInput/CLDOutput and DetectLanguage() out of src/cld.cc into new src/cld_core.h/.cc, with zero Napi:: dependency - src/cld.cc now only does Napi-specific arg unpacking/repacking and calls the shared function - add src/cld_core.cc to binding.gyp, no behavior change (native test suite passes unchanged)
- new wasm/glue.cc + scripts/build-wasm.sh: emcc build of the same CLD2 sources as binding.gyp's cld-c target, targeting both Node (CJS) and browser (ESM). includes the built wasm/dist/* artifacts since requiring Emscripten on every install would defeat the point
- new lib/backend.js + lib/wasm-wrap.js: try the native addon first, fall back to instantiating the WASM module on require() failure. wasm-wrap.js is dependency-free so bundlers never see the native require path through it
- new lib/detect-shape.js: extracts index.js's validation/defaults/error-mapping into a shared, backend-agnostic detect() builder used by both the Node and browser entry points
- new scripts/generate-metadata.js + lib/metadata.json: snapshot LANGUAGES/DETECTED_LANGUAGES/ENCODINGS as checked-in JSON, since they're static regardless of backend and shouldn't depend on either backend's (possibly async) load
- new bin/build-native.js, wired as the "install" script: runs node-gyp rebuild but always exits 0, so a broken C++ toolchain no longer fails npm install before the fallback is ever reachable
- new wasm/browser-entry.js + package.json "exports" "browser" condition: WASM-only entry point for bundlers, kept separate from index.js since bundlers choke on require('.node') even in unreachable branches
- new test/runner-wasm.js, wired into `npm test`: confirms WASM output is identical to native across every test/data.js fixture
- index.js now composes lib/backend.js + lib/detect-shape.js + lib/metadata.json instead of hardcoding the native addon
- add a "files" field to package.json -- previously there was no .npmignore or "files" field, so publishing silently fell back to .gitignore rules; now explicitly includes wasm/dist/ and excludes dev-only files - new .github/workflows/ci.yml: installs a pinned emsdk, builds both backends, runs npm test across Node 20/22 -- .travis.yml/appveyor.yml have no path to an Emscripten toolchain
- exclude "score" from the strict WASM/native comparison in test/runner-wasm.js -- verified on Linux that native (gcc/clang) and this WASM build (also clang, via emcc) match byte-for-byte including score across all 184 fixtures; on Windows/MSVC, native disagrees with both, which is a pre-existing MSVC-vs-Clang numerical quirk in CLD2's own scoring code (scoreonescriptspan.cc), not something the WASM port introduced - still strictly compare name/code/percent/reliable/textBytes/chunks, the fields that actually define whether detection is correct - fix wasm/glue.cc's JSON serializer to set ostringstream precision to 17 (was defaulting to 6 significant digits, which would silently truncate any score needing more precision than that) so it round-trips a double exactly, matching what Napi::Number::New already does on the native side - rebuild wasm/dist/* with this fix
- excluding just "score" (previous fix) wasn't enough -- AppVeyor's next run showed native (MSVC) diverging from WASM on chunk boundaries too (offset/bytes), for a different fixture (CEBUANO); confirmed on Linux that native (gcc/clang) and WASM agree with each other exactly on that same fixture, so it's native-Windows/MSVC that's the outlier, not the WASM port - comparing WASM against "whatever native build happens to be on this CI runner" was the actual problem: native builds on different compilers can legitimately disagree with each other on this pre-existing CLD2 algorithm, so no single live native build is reliable ground truth across platforms - new scripts/generate-wasm-fixtures.js + test/wasm-fixtures.json: a fixed, checked-in snapshot of expected detect() output captured once from the native build (Linux/gcc-clang, already proven to match this WASM build exactly across all 184 fixtures) - rewrite test/runner-wasm.js to compare the WASM backend against this snapshot instead of a live native require -- makes the test give the same answer on every platform, and drops the native build as a runtime dependency of this test entirely - restore full strict equality (score included) now that the snapshot's source of truth is trustworthy, replacing the earlier score-field exclusion
… cld.web.wasm from:
- wasm/browser-entry.js: export setWasmModuleOptions({ locateFile }), stored module-level and passed to createCldModule() -- Emscripten's own glue already checks Module["locateFile"] before falling back to its bundled-relative-URL default, so this is purely additive
- index.d.ts: types for the new export
- README.md: document the override for bundlers that serve cld.web.wasm from a non-default path
- fix browser-entry.js's metadata.json import to use the `with { type: 'json' }` attribute Node's ESM loader requires -- without it the real browser entry point couldn't be loaded via plain Node import() at all, which is what made it impossible to test
- test/runner-wasm-browser.js: confirms the override is actually honored, not just accepted -- spawns the real browser-entry.js in a child process with no override (fails, since the default WASM URL resolves to a file:// URL that Node's fetch() can't handle) and again with locateFile pointed at a local HTTP server serving the real wasm binary (succeeds with a correct detection), so success in the second case can only come from the override taking effect
- package.json: add the new test to npm test
- rename to wasm/browser-entry.mjs -- Node 20 lacks the fallback that reparses typeless .js files as ESM, so import() hit them through the CJS loader and errored on the `import` statement. .mjs avoids the ambiguity
- update the filename in package.json ("browser"/"files"), index.d.ts, lib/detect-shape.js, lib/wasm-wrap.js, and test/runner-wasm-browser.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Heads up: