feat(migration): seed Exa pre-install from EXA_API_KEY env var#19993
feat(migration): seed Exa pre-install from EXA_API_KEY env var#19993FelixMalfait wants to merge 26 commits intomainfrom
Conversation
Adds the plumbing for server admins to declare a list of npm app packages that are auto-installed on every new workspace and backfillable onto existing workspaces. The follow-up PR migrates Exa onto this path. What this PR introduces: - **Server-level variable injection into logic function execution.** LogicFunctionExecutorService now resolves env vars as: (registration-level ApplicationRegistrationVariable) overridden by (workspace-level ApplicationVariable). The manifest already declares serverVariables; this closes the loop so apps can read those values at runtime without storing them per-workspace. - **PRE_INSTALLED_APPS config variable.** Comma-separated list of npm package names. Default empty — opt-in by server admin. - **PreInstalledAppsService.** On application bootstrap, fetches each package's manifest from the app registry CDN, upserts an ApplicationRegistration, and seeds declared server variables from matching env vars (EXA_API_KEY env -> encrypted registration variable). Exposes installOnWorkspace(workspaceId) for hooks and backfill. - **Auto-install on new workspace activation.** WorkspaceService invokes PreInstalledAppsService.installOnWorkspace in prefillCreatedWorkspaceRecords. Failures are non-blocking — the admin can backfill later. - **install-pre-installed-apps CLI command.** Idempotent backfill that iterates active and suspended workspaces. Run after changing PRE_INSTALLED_APPS to roll the change out to existing tenants. - **POST /app/billing/charge endpoint.** Authenticated via APPLICATION_ACCESS token (DEFAULT_APP_ACCESS_TOKEN already injected into logic function execution env). Emits USAGE_RECORDED workspace events with applicationId as resourceId. Generic — future apps (call recorder, etc.) use the same endpoint. - **Tool name prefix change `logic_function_` -> `app_`.** Shorter, signals that these tools come from installed apps. Only affects tools sourced from LogicFunctionToolProvider; other providers unchanged. No user-visible change yet — PRE_INSTALLED_APPS defaults to empty. The follow-up PR ships the Exa app package, sets the default, and removes the current WebSearchTool / WebSearchService / ExaDriver. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Parallelize per-app installs in PreInstalledAppsService.installOnWorkspace
via Promise.allSettled. ApplicationInstallService acquires a per-app
cache lock internally, so installs are safe to run concurrently. 3x
speedup for multi-app pre-install lists.
- Make the auto-install hook in WorkspaceService.prefillCreatedWorkspaceRecords
fire-and-forget. Npm downloads + migrations could add several seconds
of latency per app to workspace activation; users shouldn't wait for
non-critical installs to complete before seeing a ready workspace.
- Optimize the server-variable query in LogicFunctionExecutorService:
filter unfilled variables server-side with `encryptedValue: Not('')`
so the hot path ships only populated rows. Added a comment pointing
future reviewers at WorkspaceApplicationVariableMapCacheService as the
pattern to adopt if this becomes a measurable bottleneck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five findings from code review: - UsageResourceType.APP (not AI) for app billing events. App-emitted charges are misclassified as AI usage otherwise. - Always decrypt ApplicationRegistrationVariable.encryptedValue in the logic function executor. The storage contract (set by ApplicationRegistrationVariableService.createVariable and .updateVariable) always encrypts regardless of isSecret — isSecret is display metadata only. The previous conditional decryption would inject ciphertext as env vars for non-secret values. - Symmetric fix in PreInstalledAppsService.seedServerVariablesFromEnv: always encrypt on write, matching the storage contract. - Retry registration resolution in installOnWorkspace when some pre-installed packages have no registration row. If bootstrap failed transiently (CDN outage, cold-start race), the first new workspace used to silently skip those apps until an admin ran the backfill command. - Return 403 ForbiddenException from /app/billing/charge when the JWT is valid but not an APPLICATION_ACCESS context. 400 misrepresents an auth problem as a malformed body. - Add @IsString() to ChargeDto.resourceContext so class-validator enforces the declared type on provided values. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The repo's controller-guard lint rule requires every REST controller to
declare both an auth guard and a permission guard. AppBillingController
had AuthGuard('jwt') but no permission guard — the endpoint doesn't map
to workspace permissions (the app token is the authorization), so
NoPermissionGuard is the correct pairing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The controller-guard lint rule checks guards per-method, not per-class. Moving `@UseGuards(NoPermissionGuard)` onto the `charge` method matches the pattern used by other REST controllers (e.g. PageLayoutWidgetController) and unblocks server-lint-typecheck. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two findings from remote code review: - AppBillingController was missing a ValidationPipe, so the ChargeDto class-validator decorators (@IsInt/@IsEnum/@IsString/@min) were silently no-ops at runtime. Any caller with a valid APPLICATION_ACCESS token could submit malformed payloads (negative credits, wrong-type fields, unknown enum values) that flowed unchanged into USAGE_RECORDED events and polluted downstream billing/cap-enforcement/analytics. Added `@UsePipes(new ValidationPipe({ whitelist: true, forbidNonWhitelisted: true }))` on the charge method, mirroring the OAuth REST controllers in this feature area. - PreInstalledAppsService resolved package versions via MarketplaceService.fetchAppsFromRegistry(), which hits the npm search endpoint with a hard-coded size=250 and no pagination. A pre-installed package ranked outside that window would be silently dropped — fine today while the twenty-app ecosystem is small, but wrong for exact-name resolution. Added `fetchLatestVersionFromRegistry(packageName)` that fetches the package document directly (`GET {registry}/{name}`) and reads `dist-tags.latest`. PreInstalledAppsService now uses it instead of the bounded search. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The repo's oxlint rule `rest-api-methods-should-be-guarded` only
recognizes a specific set of auth-guard class names (UserAuthGuard,
WorkspaceAuthGuard, PublicEndpointGuard, FilePathGuard, FileByIdGuard) —
passport's `AuthGuard('jwt')` is a CallExpression, not an Identifier,
and the rule matches by identifier name.
Switch to the idiomatic Twenty REST-controller stack:
@UseGuards(JwtAuthGuard, WorkspaceAuthGuard, NoPermissionGuard)
JwtAuthGuard validates the bearer token (including APPLICATION_ACCESS
tokens — see jwt-auth.guard.ts line 38) and binds `application`,
`workspace`, and optional user fields onto the request object.
WorkspaceAuthGuard asserts the workspace was populated; NoPermissionGuard
marks this as not gated by role permissions (the app token itself is
the authorization).
AppBillingModule now imports AuthModule and WorkspaceCacheStorageModule
so JwtAuthGuard's dependencies are resolvable. Reading `application`/
`workspace` from `request` directly (not `request.user`) since
JwtAuthGuard's bindDataToRequestObject populates the request root.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `@ValidateIf` callback on EXA_API_KEY in config-variables.ts (added in the stacked PR) reimplements the same comma-split-trim-filter logic as PreInstalledAppsService.getPreInstalledPackageNames. Extract the pure parser once so the startup-validator and the runtime service can share the invariant (whitespace, empty handling) — if the parsing rule ever changes, both sites update together. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- workspace.service.spec.ts now mocks PreInstalledAppsService. The
service was added to WorkspaceService as a dependency in this PR but
the unit-test module wasn't updated, causing NestJS DI resolution to
fail ("argument PreInstalledAppsService at index [15]").
- Move PreInstalledAppsModule and AppBillingModule to the end of
CoreEngineModule's imports list. NestJS walks modules in
declaration order for GraphQL code-first schema emission; inserting
these two mid-list shifted MarketplaceModule's resolvers forward,
producing a spurious reorder in the generated client-sdk schema.
Appending the new modules keeps the emission order of existing
modules stable, so no regeneration is needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Module-ordering tweaks in core-engine.module.ts shifted the GraphQL code-first emission order, producing a pure-reordering diff in the SDK's generated schema. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves "pre-installed apps" from an env-var + npm-resolve model to a flag on ApplicationRegistration. Admin registers an app once (via admin UI / API), flips `autoInstallOnNewWorkspaces`, and the new-workspace hook + backfill command install it everywhere. This mirrors how AppExchange, GitHub Apps, and similar ecosystems distribute apps — registration is the install intent, not a deployment-time config. Removed: - `PRE_INSTALLED_APPS` env var + `parsePreInstalledApps` util - `PreInstalledAppsService.ensureRegistrationsExist` (CDN fetch, manifest resolution, upsert-from-catalog at boot) - `seedServerVariablesFromEnv` (server variables are managed per- registration, same as any other app) - `MarketplaceService.fetchLatestVersionFromRegistry` (dead code with env-var path gone; addresses review feedback to fix the root cause rather than branching the registry-fetch flow) Added: - `autoInstallOnNewWorkspaces` column on ApplicationRegistration (fast instance migration, default false) Also: - Moves `app-billing/` from `application/` to `billing/` (per review feedback — it's a billing submodule, not an application submodule) - Drops obvious comments on the billing controller - Documents why `unit` and `operationType` on ChargeDto are not redundant (per review question) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses findings from the security review of the app-billing surface added in the pre-installed-apps refactor: - **Per-charge amount caps**: `ChargeDto` now enforces `creditsUsedMicro <= 1_000_000_000` (= \$1000) and `quantity <= 10_000`. An exfiltrated application-access token or a buggy app could otherwise submit `Number.MAX_SAFE_INTEGER` and corrupt accounting / drain the workspace's credits before downstream caps fire. - **Per-(workspace, application) rate limit**: 1000 charges / 60s via the existing `ThrottlerService`. The logic-function executor already throttles *executions*, but application-access tokens are plain JWTs usable outside the runtime — this is the belt-and-suspenders layer. - **IS_BILLING_ENABLED gate**: return 404 when billing is disabled so Community-instance apps fail fast instead of silently discarding charges that no listener consumes. - **Enterprise license headers**: added `/* @license Enterprise */` to app-billing controller / service / module / DTO. The billing pipeline it feeds is Enterprise-licensed; keep the upstream aligned. - Switched the charge response to 204 No Content — the POST has no meaningful body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `Pre-installed only` toggle on the right of the Admin Panel → Apps search bar via the existing `SearchInput.filterDropdown` slot (same pattern as the marketplace `Available` tab). Reuses `Dropdown` + `DropdownContent` + `MenuItemToggle` — no new UI components. The fragment picks up `isPreInstalled` so both admin and non-admin application-registration queries surface the flag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop `unit` from `ChargeDto`: every `UsageOperationType` has one
canonical unit (mirrors how `ai-billing.service.ts` emits native
events). The service now derives `unit` from `operationType` via a
typed map; apps no longer have to repeat the same pairing.
- Replace `void this.preInstalledAppsService.installOnWorkspace(...)` in
workspace activation with sequential `try { await ... } catch`,
matching the `prefillWorkflowCommandMenuItems` block right above.
`void this.*` was unique to this spot in the whole server codebase.
- Trim block comments on the billing controller, service, DTO, entity,
pre-installed-apps service + backfill command.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Exa moves from a hardcoded action tool to a standalone app at `packages/twenty-apps/community/exa`, installable via the PRE_INSTALLED_APPS infrastructure landed in the prior PR. The tool surfaces to the model as `app_exa_web_search` (logic-function-sourced, hence the `app_` prefix). New app package: - `application.config.ts` declares the Exa app with one server variable (EXA_API_KEY, required, secret). Server admins set EXA_API_KEY as an env var; PreInstalledAppsService seeds it into the app registration at bootstrap. - `logic-functions/exa-web-search.ts` is the runtime handler. Reads EXA_API_KEY from its injected execution env, calls Exa via the official SDK, records usage by POSTing to the generic /app/billing/charge endpoint (with the injected TWENTY_APP_ACCESS_TOKEN), returns the structured results. - Tool input schema mirrors the previous WebSearchTool: query, optional category, optional numResults (1-30). Removed (now provided by the app): - packages/twenty-server/src/engine/core-modules/web-search/ — the entire module, drivers, types, and interface - packages/twenty-server/src/engine/core-modules/tool/tools/ web-search-tool/ — WebSearchTool, its schema, and input type - WebSearchTool injection + toolMap entry + descriptor in ActionToolProvider - WebSearchService injection from ActionToolProvider - WebSearchModule from CoreEngineModule imports - WEB_SEARCH_DRIVER config variable (no longer needed — Exa is an app) - Custom driver-toggle wiring in chat preload: replaced with plain `app_exa_web_search` preload EXA_API_KEY config variable stays but its description now says it seeds the Exa app's server variables. Chat and the frontend display already use the new `app_exa_web_search` name. Deployment coordination: server admins must publish @twenty-apps/exa to the app registry and set PRE_INSTALLED_APPS=@twenty-apps/exa (plus EXA_API_KEY) post-merge. Existing workspaces backfill via the `install-pre-installed-apps` CLI command added in the prior PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add `.oxlintrc.json` to the Exa app package; its `package.json` lint scripts reference the file, matching the convention in packages/twenty-apps/examples/postcard. - Reinstate conditional validation for EXA_API_KEY. Before the migration it was required only when WEB_SEARCH_DRIVER=EXA; the migration made it unconditionally optional, so an admin listing @twenty-apps/exa in PRE_INSTALLED_APPS without setting EXA_API_KEY would only fail at tool-invocation time. Now validated when PRE_INSTALLED_APPS contains @twenty-apps/exa. - Check response.ok after the billing fetch in the Exa handler. fetch() only rejects on network errors; a 4xx/5xx from /app/billing/charge would otherwise be silently swallowed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ValidateIf alone gates other validators but isn't itself a validator, so without @IsString() the conditional was a no-op — admins listing @twenty-apps/exa in PRE_INSTALLED_APPS without setting EXA_API_KEY would still pass startup validation. Add @IsString() alongside the existing @ValidateIf so the conditional branch actually enforces presence (and type). Import IsString from class-validator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review cleanup: - Exa handler now imports DEFAULT_API_URL_NAME and DEFAULT_APP_ACCESS_TOKEN_NAME from twenty-shared/application instead of hardcoding the env var names as string literals. Keeps the app in sync with whatever the execution env provides. - EXA_API_KEY's @ValidateIf callback now uses parsePreInstalledApps (extracted in the prior commit on this stack). One parser, one invariant, no drift between the startup validator and the runtime PreInstalledAppsService. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes the last coupling of Exa to deployment-time config after the infrastructure PR moved pre-installation to a registration flag: - Drops the `EXA_API_KEY` env var. Exa is now registered as an `ApplicationRegistration` and its API key lives on the registration's `ApplicationRegistrationVariable` row, same as any other app. - Updates the Exa app manifest + README to describe the admin flow (register app, flip `autoInstallOnNewWorkspaces`, set EXA_API_KEY server variable) instead of env-var seeding. - Updates the AI chat preload TODO comment to reflect the flag-based install model. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Security-review follow-up: add inner timeouts so a slow upstream Exa API or app-billing endpoint can't consume the full 30s runtime budget. - `AbortSignal.timeout(5_000)` on the fire-and-forget billing charge fetch. - Promise.race with a 25s inner deadline on `exa.search` (exa-js has no built-in abort/timeout). Leaves 5s for the billing charge + a graceful return before the outer runtime kill switch fires. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`ChargeDto` no longer accepts `unit`; the server derives it from `operationType`. With `forbidNonWhitelisted: true` on the validation pipe, sending it now returns 400. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- **Move to `internal/`**: The Exa app is Twenty-built, not a community contribution, so it belongs next to `call-recording`, `self-hosting`, `twenty-for-twenty` under `packages/twenty-apps/internal/`. - **Drop `@twenty-apps/` scope**: Every other internal + example app uses an unscoped package name (`self-hosting`, `hello-world`, `postcard`). Renamed to `exa`. - **Add `defaultRoleUniversalIdentifier`**: `defineApplication` validates it, so the previous config would have failed at manifest parse time. Introduces a minimal no-op role (Exa only reads the `EXA_API_KEY` server variable — no workspace data). - **Extract `chargeCredits` helper** (`src/utils/charge-credits.ts`): generic billing call, no more per-app fetch boilerplate. - **Split schema file** (one export per file, per Twenty convention): - `logic-functions/constants/default-num-results.constant.ts` - `logic-functions/constants/exa-categories.constant.ts` - `logic-functions/schemas/exa-web-search-input.schema.ts` - `logic-functions/types/exa-web-search-input.type.ts` - **Extend `InputJsonSchema`** (twenty-shared): add `minimum` / `maximum` so the tool schema can type-check `numResults` bounds properly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- **Package name**: `exa` is taken on npm by an unrelated package. Renamed to `twenty-exa` — matches the `twenty-sdk` / `twenty-shared` / `twenty-client-sdk` publish convention. Discovery still works via the `twenty-app` keyword (marketplace cron queries `npm search text=keywords:twenty-app`, not by name). - **Bump SDK deps** to `2.0.0` — the `0.9.0` release on npm doesn't export `twenty-sdk/define`, so the build couldn't resolve the imports. - **Align with exa-js 1.10 API**: `exa.search(...)` moved content options to a separate `exa.searchAndContents(...)` method. Updated the handler. Also aligned `EXA_CATEGORIES` with what exa-js types accept (dropped `people`, added `github` / `tweet` / `linkedin profile`). - **Add brand assets**: Exa logomark + full logo SVGs from Exa's official brand kit, wired via `logoUrl: 'public/exa-logomark.svg'`. - **Add scaffolding**: `.yarnrc.yml`, `.nvmrc`, `.gitignore`, empty `yarn.lock` — makes the package installable/buildable as a standalone project (matching `examples/postcard`). Local `yarn twenty build` + `npm publish --dry-run` both succeed. Ready to publish with `npm publish` from `.twenty/output/` once the maintainer runs `npm login`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Exposes the generic app-billing helper as a first-class SDK subpath so any Twenty app can charge credits without copy-pasting the fetch/retry boilerplate. twenty-sdk changes: - New subpath: `twenty-sdk/billing` exporting `chargeCredits` and `ChargeCreditsParams`. Source: `src/sdk/billing/`. - New `vite.config.billing.ts` (mirrors `vite.config.define.ts`). - Added billing entry to `rollup.config.sdk-dts.mjs` + build/dev targets in `project.json` + `"./billing"` export in `package.json`. - Bumped `twenty-sdk` version to `2.1.0`. Exa app changes: - Depend on `twenty-sdk@2.1.0` and import `chargeCredits` from `twenty-sdk/billing`. - Deleted `src/utils/charge-credits.ts` (no longer needed). - Bumped `exa-js` to `^2.12.1` — the 2.x release ships accurate types for `category` (includes `people`, drops the stale `github`/`tweet`/ `linkedin profile`), so the earlier type cast goes away and our `EXA_CATEGORIES` list aligns with the published Exa API. - Switched `tsconfig.moduleResolution` from `node` to `bundler` so TypeScript honors `twenty-sdk`'s exports map (otherwise `twenty-sdk/billing` can't resolve against the published package). Publish workflow: publish `twenty-sdk@2.1.0` first, then republish `twenty-exa` (its lockfile will then resolve the new subpath). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-shot bridge for instances that set `EXA_API_KEY` as an env var under the old native `web-search` driver (removed in the Exa migration PR). Runs at deploy time: 1. Skips if `EXA_API_KEY` is unset. 2. Fetches the `twenty-exa` manifest from the app registry CDN and upserts the `ApplicationRegistration` (via the existing catalog-sync code path — no new registration plumbing). 3. Seeds `EXA_API_KEY` onto the registration's server variable, encrypted via `SecretEncryptionService`. Never overwrites a value already edited through the admin UI. 4. Flips `isPreInstalled=true` so new workspaces auto-install. Existing workspaces are backfilled with `install-pre-installed-apps`. Idempotent — each step no-ops when its target already exists. Safe to rerun. After a successful deploy, the env var can be dropped from infra since the key now lives on the registration row. The slow instance command hook gives us full DI, so this reuses `MarketplaceService`, `ApplicationRegistrationService`, and `SecretEncryptionService` rather than hand-rolling SQL + encryption. `InstanceCommandProviderModule` now imports the modules those services live in. Prerequisite: `twenty-sdk@2.1.0` and `twenty-exa@0.1.0` must be published on npm before this migration runs — otherwise the CDN fetch returns no manifest and the command logs a warning and exits clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
🚀 Preview Environment Ready! Your preview environment is available at: http://bore.pub:63089 This environment will automatically shut down after 5 hours. |
Adding imports to `InstanceCommandProviderModule` shifted GraphQL code-first emission order — pure reorder, no semantic schema change (267 insertions, 267 deletions, all reshuffling). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@claude could you please review this PR ? |
|
Claude finished @prastoin's task in 3m 52s —— View job Claude PR Review
SummaryTightly scoped, well-considered bridge migration. The data flow is clean: env var → catalog upsert → server variable seed → Findings1. Silent skip when the manifest doesn't declare
|
Summary
One-shot bridge for instances that set
EXA_API_KEYas an env var under the old nativeweb-searchdriver (removed in #19980). Prevents a gap between deploy and admin setup where Exa search is broken.Stacked on top of #19980 — depends on the
isPreInstalledcolumn + thetwenty-exanpm package.What it does
A slow instance command that runs at deploy time:
EXA_API_KEYis unset → no-op on fresh instances.twenty-examanifest from the app registry CDN and upserts theApplicationRegistrationvia the existing catalog-sync path.EXA_API_KEYonto the registration's server variable, encrypted viaSecretEncryptionService. Never overwrites a value already edited through the admin UI.isPreInstalled=true. New workspaces auto-install from that point; existing workspaces are backfilled with theinstall-pre-installed-appsCLI command.Idempotent — each step no-ops when its target already exists. Safe to rerun.
Prerequisites
Before merging / deploying:
twenty-sdk@2.1.0(new./billingsubpath, introduced in feat(app): migrate Exa web search to a pre-installed app #19980).twenty-exa@0.1.0to npm.Otherwise the migration's CDN fetch returns no manifest and the command logs a warning and exits clean (no crash).
Post-deploy
EXA_API_KEYfrom infra env vars — it now lives encrypted on the registration.npx nx run twenty-server:command install-pre-installed-appsto backfill existing workspaces (new workspaces pick it up automatically).Why a separate PR
Keeps #19980 focused on the architectural migration. This one is a deployment-lifecycle helper — reviewable in isolation and droppable without losing the core work if we change our mind on the bridging strategy.
Test plan
Registered 1 fast instance, 1 slow instance ... for 2.0.0)🤖 Generated with Claude Code