Native package ecosystem v1: open, extensible, with zero developer-facing change

[proggeramlug]

Problem

Requests for native bindings to Rust libraries (most recently #424 tursodb, #425 iroh; before that mongodb, lettre, etc.) keep landing in this repo. Each one is a judgement call: bake it into perry-stdlib (bloat the default build, slow every user's compile, couple another upstream's release cadence to ours) or decline (which loses a real use case Perry is well-suited for).

The mechanism to do this externally already exists — perry.nativeLibrary in a package.json (see crates/perry/src/commands/compile/resolve.rs:75-231 and link.rs:1547-1709). Bloom Engine ships ~230 FFI functions (wgpu renderer + Jolt physics + per-platform AppKit/UIKit/DirectX/Vulkan/WebGPU code) entirely outside this repo using exactly that mechanism.

What's missing is the contract + tooling that makes writing one of these cheap, sustainable, and not coupled to perry's internals. Today a wrapper has to know the layout of StringHeader, the meaning of NaN-boxing tags, etc. — internals that change. We need to extract a stable ABI and tooling around it, then dogfood by moving the wrapper-style code currently in perry-stdlib out.

Goals

1. Zero developer-facing change. import { createConnection } from 'mysql2' keeps working byte-identically. So do ioredis, ws, ethers, dotenv, pg, mongodb, fastify, etc. The user never installs anything new to get what works today. They never see @perry/mysql2 if they don't want to. Naming is unenforced — packages can be @perry/tursodb, tursodb-perry, whatever-they-want.
2. Anyone can publish a wrapper for any Rust crate without a PR to this repo, and it Just Works after npm install.
3. Big maintained wrappers live in the perryts GitHub org as their own repos (perryts/tursodb, perryts/iroh, eventually perryts/mysql2-bindings, etc.) with their own release cadence.
4. Wrapper authors don't depend on perry-runtime internals. They depend on a small, versioned perry-ffi crate. Breaking changes to perry's runtime layout don't break the ecosystem.
5. First-compile UX stays fast. Pulling in iroh shouldn't put the user in front of a 4-minute cargo build; prebuilt artifacts should be supported.

Non-Goals

• Replacing the existing perry-stdlib-as-staticlib link model. Binaries still statically link only the bindings they use.
• A package registry / hosting infrastructure. We use npm + GitHub.
• Forcing renames or migrations on existing user code.
• Dynamic loading / plugin dlopen — these are still build-time linked.

Architecture

Three layers, from most stable to most flexible:

┌─────────────────────────────────────────────────────────────────┐
│ Layer 3: Bindings packages (anyone can publish)                  │
│   perryts/tursodb          @perry/iroh        community/foo      │
│   ──────────────           ───────────         ─────────────     │
│   package.json with perry.nativeLibrary manifest                  │
│   src/index.ts (TS surface)                                       │
│   native/<plat>/ Rust crate (wraps upstream)                      │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ depends on
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Layer 2: perry-ffi crate (published to crates.io, versioned)     │
│   read_string, alloc_string, read_array, alloc_array,             │
│   nanbox_pointer, etc. ~15 helpers — the only stable surface      │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ implementation detail of
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│ Layer 1: perry-runtime internals (free to change)                │
│   StringHeader layout, NaN-boxing tags, GC, arena, ...            │
└─────────────────────────────────────────────────────────────────┘

Resolution flow (preserves bare names):

import X from 'mysql2'
       │
       ▼
┌────────────────────────────────────────────────────┐
│ 1. node_modules/mysql2/ exists with a              │
│    perry.nativeLibrary manifest? → use it          │
│ 2. node_modules/mysql2/ exists without manifest?   │
│    → fall through (V8 / error as today)             │
│ 3. \`well_known_bindings.toml\` (shipped in perry    │
│    install) maps 'mysql2' → bundled wrapper         │
│    crate path + prebuilt .a → use it                │
│ 4. None match → error                               │
└────────────────────────────────────────────────────┘

The well-known table is what makes `import 'mysql2'` keep working with zero `npm install`. The bundled wrappers ship inside the perry install (as crates in this repo, prebuilt during release). Step 1 means a user can override any well-known by dropping their own wrapper in node_modules.

---

Phase plan

Five phases, each independently shippable. The dogfood migration in Phase 5 is the load-bearing one — if the contract from Phases 1-3 can't sustain the wrappers we already maintain, it can't sustain third-party ones.

Phase 1 — Extract `perry-ffi` crate (the ABI contract)

Why: Single biggest risk. Today wrappers reach into `perry-runtime` internals (StringHeader layout, NaN-boxing tag values, etc.) that change between releases. Without a stable surface, every refactor breaks every external package silently.

Scope:

• New `crates/perry-ffi/` workspace member, published to crates.io as `perry-ffi = "0.5"`.
• Audit every `#[no_mangle] extern "C"` callsite in `perry-stdlib` and Bloom (`../../bloom/engine`) for what they need from the runtime: string read/write, array read/write, pointer NaN-boxing, JSValue tag extraction.
• Distill into a minimal helper set (~15 functions). Initial sketch:

  pub fn read_string(handle: i64) -> &'static str
  pub fn alloc_string(s: &str) -> i64
  pub fn read_array(value: f64) -> Vec<JsValue>
  pub fn alloc_array(items: &[JsValue]) -> f64
  pub fn read_object_field(obj: f64, key: &str) -> JsValue
  pub fn write_object_field(obj: f64, key: &str, value: JsValue)
  pub fn nanbox_pointer(ptr: *mut c_void) -> f64
  pub fn nanbox_string(s: i64) -> f64
  pub fn unbox_pointer(value: f64) -> *mut c_void
  pub fn js_undefined() -> f64
  pub fn js_null() -> f64
  pub fn js_true() -> f64
  pub fn js_false() -> f64
  pub fn is_truthy(value: f64) -> bool
  // + JsValue enum for typed inspection

• `perry-ffi` is a thin re-export layer around `perry-runtime` (same process, same arena, same GC). It exists for API stability, not isolation.
• Set semver: `0.5.x` matches Perry's current minor; bump major when the surface breaks.

Deliverables:

• `crates/perry-ffi/` published to crates.io
• `docs/native-libraries/abi.md` documenting every function with bit-level semantics
• `perry-runtime` stops being a public dependency for wrapper authors (still internal to perry itself)

Validation: A throwaway external crate (`/tmp/perry-ffi-smoke/`) that uses only `perry-ffi` builds, links, and round-trips a string + array + object through a Perry binary.

Risks:

• Choosing the wrong surface granularity. Mitigation: do the Phase 5 dogfood port of `dotenv` (smallest binding) before shipping perry-ffi 0.5.0 publicly. If dotenv needs a helper not in perry-ffi, add it before publishing.
• Bloom may need helpers we didn't anticipate (it has its own audit). Loop them in before freezing.

---

Phase 2 — Freeze manifest spec + add `abiVersion`

Why: Today the manifest schema is implicit (whatever `resolve.rs:109-231` parses). External authors have no spec, no version field, and no way to declare compatibility. When perry's ABI changes, external packages segfault instead of failing loudly.

Scope:

• Write `docs/native-libraries/manifest-v1.md` documenting every field, every `returns` type (`string`/`ptr`/`i64`/`i64_str`/`f64`/`void`), every `params` type, every per-target field (`crate`, `lib`, `frameworks`, `libs`, `pkgConfig`, `swift_sources`, `metal_sources`).
• Add JSON schema at `docs/native-libraries/manifest.schema.json` for editor validation.
• Add required field: `perry.nativeLibrary.abiVersion: "0.5"` (semver range).
• `resolve.rs` rejects manifests whose `abiVersion` doesn't satisfy the perry binary's range, with an actionable error pointing at the package.json line.
• Backwards-compat: missing `abiVersion` is allowed for the v0.5.x cycle (warning), required from v0.6.0.

Deliverables:

• Manifest spec doc + JSON schema
• `abiVersion` parsing + validation in `resolve.rs` / `parse_native_library_manifest`
• Bloom's manifest updated to declare `abiVersion: "0.5"` (proves the spec matches reality)
• Deprecation path documented for any field that's currently undocumented and we want to remove

Validation: `cargo run -- foo.ts` against a wrapper declaring `abiVersion: "99.0"` errors with a clear message naming the package.

---

Phase 3 — `perry native` CLI subcommand (scaffold + validate + prebuild)

Why: Writing a wrapper today means reading Bloom's source. Tooling drops the barrier from "hours of archaeology" to "minutes".

Scope:

• `perry native init ` — scaffolds a new package:

  <name>/
    package.json          (with perry.nativeLibrary template, abiVersion filled)
    src/index.ts          (TS declarations matching FFI surface)
    native/macos/
      Cargo.toml          (perry-ffi dep, the upstream crate dep, staticlib)
      src/lib.rs          (one example #[no_mangle] fn)
    .github/workflows/
      release.yml         (prebuild artifacts for all targets, attach to GH releases)
    README.md
  

  Asks the user for: package name, upstream crate name, target list (defaults: macos, ios, linux, windows, android, web).
• `perry native validate` — run from the wrapper's root:
  1. Parses the manifest.
  2. Runs `cargo build --release` on the native crate.
  3. Runs `nm` on the resulting `.a` and diffs symbols against the `functions[]` array.
  4. Type-checks `src/index.ts`'s declarations against the manifest signatures.
  5. Reports drift (missing symbols, extra symbols, signature mismatches) with file:line.
• `perry native prebuild --target ` — builds the staticlib + writes it to `prebuilt//lib.a`. Manifest gains optional `targets..prebuilt: "./prebuilt//lib.a"` field; `link.rs` prefers prebuilt over cargo-build when present.

Deliverables:

• `perry native init/validate/prebuild` subcommands
• Template lives in `crates/perry/templates/native-package/` (rendered with handlebars or similar)
• `link.rs` reads `prebuilt:` field, falls back to cargo-build
• Prebuilt artifacts work on at least macos + linux (other targets land incrementally)

Validation: `perry native init tursodb-bindings && cd tursodb-bindings && perry native validate` produces a green build that another perry project can import via `file:../tursodb-bindings`.

---

Phase 4 — Well-known bindings registry (preserves bare-name imports)

Why: The whole point — `import 'mysql2'` must keep working. With Phase 5 about to move bindings out of `perry-stdlib`, we need a resolution layer that still maps the bare name to the bindings code.

Scope:

• New file in this repo: `crates/perry/well_known_bindings.toml`:

  [\"mysql2\"]
  repo = \"perryts/mysql2-bindings\"
  bundled_crate = \"perry-ext-mysql2\"   # crate path inside this workspace
  
  [\"ioredis\"]
  repo = \"perryts/ioredis-bindings\"
  bundled_crate = \"perry-ext-ioredis\"
  # ... etc for ws, ethers, dotenv, pg, mongodb, fastify, axios, etc.

• Bundled wrapper crates (`crates/perry-ext-*`) are built during `cargo build --release` like any other workspace member, and their `.a` files are bundled with releases (Homebrew bottle, install tarball).
• `resolve.rs` resolution order:
  1. `node_modules//` with a `perry.nativeLibrary` manifest → use it (user override wins).
  2. `node_modules//` without a manifest → fall through to V8 (existing behavior).
  3. `well_known_bindings.toml` has `` → use the bundled crate's prebuilt `.a`.
  4. None match → error.
• The hardcoded `PERRY_NATIVE_EXTENSION_PACKAGES` list in `resolve.rs:236` is deleted — replaced entirely by the well-known table.

Deliverables:

• `well_known_bindings.toml` schema + parser
• Resolution order updated in `resolve.rs`
• Each well-known entry validated at perry startup (errors at install time, not user-import time, if a bundled crate is missing)
• User can override any well-known by installing their own wrapper in node_modules — covered by a test

Validation: Build perry, build a TS file with `import { createConnection } from 'mysql2'`, confirm output is byte-identical to today's behavior.

---

Phase 5 — Dogfood: migrate `perry-stdlib` wrappers out

Why: This is the test of whether Phases 1-4 actually work. Real wrapper authors will hit every gap in the contract; if our own wrappers can't survive the move, theirs definitely can't.

Scope (in migration order, smallest first):

1. `dotenv` (~80 LOC, no deps) — proves Phase 1 (perry-ffi surface) and Phase 2 (manifest spec). New crate `crates/perry-ext-dotenv/`, registered in well-known table. Code physically moves; user TS imports unchanged.
2. `uuid`, `nanoid`, `slugify`, `bcrypt`, `argon2` (small, single-crate wrappers) — batched proof.
3. `ws` — proves async-runtime sharing works between bindings and core.
4. `mysql2`, `pg`, `sqlite`, `ioredis`, `mongodb` (database batch) — biggest wrappers, most likely to surface contract gaps. Each gets its own crate.
5. `reqwest`-backed wrappers (`fetch`, `axios`) — proves shared transitive deps are tolerable (each crate pulls reqwest; cargo dedupes at workspace level).
6. `fastify`, `hyper` server pieces, `tokio-tungstenite` — biggest dep tree, last to move.

What stays in `perry-stdlib`: anything genuinely coupled to runtime internals — timers, threads, GC-aware bits, the framework hooks that aren't "wrapper around a Rust crate". Target end state: `perry-stdlib` shrinks to maybe 10 files, all touching arena/GC/promise machinery directly.

Two-phase migration per wrapper to keep main green:

1. Add new `crates/perry-ext-/` crate, register in well-known table, leave old code in `perry-stdlib` (gated off via feature). CI builds both, well-known resolution prefers the new crate. Validates the move.
2. Once a release ships and no regressions surface, delete the old code from `perry-stdlib`.

Deliverables:

• All 6 batches migrated
• `perry-stdlib` defaults shrunk; build time measured before/after (target: 30%+ reduction)
• Each migrated wrapper bumps independently — first one to do so without a perry release proves the loop closed
• At least one wrapper (`perryts/mysql2-bindings`) lives as a separate repo end-to-end, including its own release CI

Validation: Full parity sweep (`./run_parity_tests.sh`) and full `cargo test --workspace` stay green at every step. End state: any test that worked before works after, byte-identical output.

---

Migration strategy / risk surface

For users: Zero action required. `npm install` of an existing project continues to work. Compiled binaries continue to work. The only visible change is faster default builds (because perry-stdlib is smaller).

For maintainer (Ralph): Each Phase 5 batch is one commit/PR cycle. Migration order is risk-ordered (smallest first); if dotenv reveals a perry-ffi surface gap, we fix Phase 1 before continuing.

For external contributors: After Phase 3 ships, "add support for X" issues get a one-line answer: "Run `perry native init x-bindings`, fill in the FFI, publish to npm. Happy to link from the awesome list once it works." Issues #424 and #425 become tractable for anyone, not just Ralph.

Rollback: Each phase is independently revertable. The dogfood migration's two-step approach (add new crate, then later delete old code) means we can always point well-known resolution back at the old in-stdlib code if something breaks in production.

---

Open questions

1. Should perry-ffi re-export the entire JSValue type or just opaque accessors? Re-export is more ergonomic for wrappers; opaque accessors give us more refactor freedom. I lean re-export with #[non_exhaustive] to thread the needle.

2. Prebuilt artifact distribution — once a wrapper repo's release CI publishes prebuilt `.a`s as GitHub release assets, how does `npm install` get them onto the user's disk? Options: (a) the npm package's `postinstall` downloads them; (b) the package vendors prebuilt artifacts directly in the npm tarball (simpler but bigger). Lean (b) for now; revisit if package sizes get painful.

3. Cross-target prebuilds — the wrapper author may not have iOS/tvOS/Windows machines. CI matrix on GitHub Actions covers it but adds release-time complexity. The `perry native init` template should ship a working multi-target release.yml so authors don't have to figure this out.

4. Do we want a CLI command to list well-known bindings? `perry native list` printing what's available out-of-the-box would help discovery. Cheap addition; punt to Phase 4 implementation.

5. Async runtime sharing — most wrappers need tokio. perry-stdlib currently exposes a shared runtime via internal helpers. perry-ffi needs an equivalent so external wrappers don't each spawn their own runtime (would deadlock spectacularly). Phase 1 must include a `perry_ffi::spawn_async` / `block_on` surface.

---

What I'd start with

Phase 1 (perry-ffi extraction) blocks everything else and is the highest-risk piece (getting the surface right). Realistic first deliverable: extract perry-ffi + port `dotenv` (Phase 5 step 1) as a single PR, treating dotenv as the perry-ffi acceptance test. If perry-ffi 0.5.0 can't ship dotenv, it's not done.

[ansarizafar]

Why not generate bindings automatically by using Rust crate doc.