Vitals

Contactless pulse, breathing, and HRV from a camera — a cross-platform app on the SmartSpectra SDK, built for the code-til-dawn hackathon.

This repo is the solution. The Go vitals-sidecar that wraps the native SmartSpectra SDK is vendored in services/sidecar (self-contained — builds without the reference SDK checkout).

🎤 Pitch deck: presentation/index.html — open it in a browser, arrow keys to navigate.

Stack: Cloudflare Workers + Drizzle ORM on Turso (libSQL), a Go (cgo) vitals-sidecar, a dependency-free web client, and pnpm workspaces.

Live (deployed):

• Web: https://vitals-web.pages.dev
• Worker API: https://vitals-api.darthbitcoin.workers.dev (Turso-backed, verified)
• Sidecar: https://vitals-sidecar-nst.fly.dev (Fly app vitals-sidecar-nst)

🔒 All three tiers require a per-user sign-in. The web app shows a lock screen — enter a username + password; the Worker verifies it (PBKDF2 hashes in VITALS_USERS) and mints a short-lived HMAC-signed session (VITALS_SESSION_SECRET, 8h). Every call then carries Authorization: Bearer <session>, which the Worker and the sidecar both verify (constant-time). Anonymous requests get 401 (sidecar engine + Presage key, Worker API, and the sidecar's own dashboard are all gated). No password or secret is baked into the build. CORS is locked to the web origin.

Teardown when done: fly apps destroy vitals-sidecar-nst, wrangler delete (Worker), wrangler pages project delete vitals-web. The sidecar carries your Presage key, so don't share its URL widely without adding auth (see NEXT.md P2).

🏆 For judges

What it is. Contactless pulse, breathing & HRV from any webcam — a real web + edge product built on SmartSpectra's native vitals engine.

The hard part we solved. SmartSpectra ships native SDKs for iOS, Android, and C++ only — no Go, no web build, and the engine can't run in a browser or a serverless isolate. We wrote the Go (cgo) binding the SDK didn't have, wrapped it in a Linux vitals-sidecar, and put a Cloudflare Worker + Turso edge API in front, so a browser can measure vitals it otherwise couldn't touch — and persist them at the edge.

Evaluate it in ~2 minutes

1. See it live (visual): open https://vitals-web.pages.dev, sign in (ask us for a login), allow the camera, press Start, sit still ~15s → live pulse / breathing / HRV. Raw frames are processed and discarded; only the derived metrics are stored.
2. See the backend (scriptable): VITALS_USER=… VITALS_PASS=… ./scripts/demo.sh — logs in, checks health, runs a measurement through browser→Worker→sidecar→real engine→Turso, and prints your saved history. (Pass CLIP=<a clip of a face> for real numbers.)
3. Read the design: docs/ARCHITECTURE.md — the native-binary constraint and the trust boundaries.

What's live: web (Cloudflare Pages), edge API (Workers + Drizzle/Turso), and the native-engine sidecar (Fly.io) — all three deployed and gated behind per-user sign-in. Full breakdown in Status.

✨ Inspiration

Every heartbeat pushes blood into the skin of your face and shifts its colour by an amount far too small to see — but not too small for a camera. That faint signal carries your pulse, your breathing, and your heart-rate variability; the technique is remote photoplethysmography (rPPG). Today, getting those numbers means strapping hardware to your body or going to a clinic. We wanted to know: what if a $0 webcam was enough — hands-free, contactless, anywhere, in seconds? SmartSpectra ships a production-grade vitals engine that does exactly this, so we built a real product around it that runs on the web and on the edge, not just on a phone.

🩺 What it does

Point a camera at a face and Vitals reads, in real time:

	Metric	Unit
❤️	Pulse (with a confidence score)	bpm
🌬️	Breathing rate	breaths/min
📈	HRV (RMSSD) — a stress / recovery signal	ms
🙂	Facial expression	—

• Live, then saved. Watch the numbers stabilise as a stream, then stop to save a snapshot to history.
• One product, many cameras. The web client ships today; iOS, Android and macOS run the same SDK on-device and write the same measurements shape (the source column distinguishes them).
• Private by design. Raw video is processed and discarded — only the derived metrics are stored. On-device platforms never send frames anywhere.

Live measurement	Secured & deployed	Mobile

🚀 How to use it

Try it live: open https://vitals-web.pages.dev, sign in with your username + password (ask the team — credentials are never baked into the build), allow the camera, and press Start measurement. Sit still, well-lit, facing the camera; the first stable pulse takes ~10–20s. Press Stop to save the snapshot. To run the whole stack on your own machine, see Quick start (local) below (auth is off by default locally).

Why it's shaped this way

The SmartSpectra engine is a native binary with no web/WASM build and it can't run in a Cloudflare Workers isolate. So the web path measures on a Linux server (the Go vitals-sidecar) and the browser/edge just talk to it. Mobile/desktop measure on-device. Full picture: docs/ARCHITECTURE.md.

browser ──clip──► Cloudflare Worker ──► vitals-sidecar (Linux+SDK) ──► metrics ──► Turso
apps/web          workers/api            services/sidecar                         db/

🛠 How it was built

Layer	Tech
Web client	Vanilla JS + getUserMedia (no build) → Cloudflare Pages
Edge API	Cloudflare Workers + Drizzle ORM (type-safe DB)
Database	Turso (libSQL / SQLite)
Native engine	Go + cgo sidecar over SmartSpectra v3.1.0 (TFLite on CPU) → Fly.io
Monorepo	pnpm workspaces

The bug that reshaped the demo. The first plan was a simple clip upload. It failed deep in the engine: non_monotonic_timestamp 0<=0 — the SDK's file reader wasn't deriving frame timestamps from the video container. Rather than fight the file path against the clock, we pivoted to a live path: the browser JPEG-encodes webcam frames and streams them to the sidecar, which feeds the engine via custom input with monotonic timestamps (WithCustomInput). That drove 40 frames through the real engine with the real API key — face detection at ~54 fps, no timestamp or auth errors. Real pulse/breathing/HRV just need a real face in front of the camera.

We then circled back and fixed the upload path itself: POST /measure no longer touches the broken native reader — it decodes the clip with ffmpeg and feeds the same monotonic custom-input stream, so plain file upload works against the real engine too. The fixed sidecar is deployed (the real-engine container compiles and boots clean); end-to-end is runnable via scripts/demo.sh.

Layout

Path	What
apps/web	Web client — webcam capture → upload → vitals card (no build step)
workers/api	Cloudflare Worker — proxy to sidecar + persist to Turso via Drizzle ORM. POST /api/measure (clip), POST /api/measurements (save a snapshot), GET /api/measurements, GET /api/health
db	Turso (libSQL); Drizzle schema lives in workers/api/src/db, generated migrations in workers/api/drizzle
services/sidecar	Vendored Go (cgo) vitals-sidecar — wraps the native SmartSpectra engine. Includes the SDK .deb; builds standalone. Entrypoint: cmd/vitals-sidecar
docker-compose.yml	Runs the native sidecar locally on :8080
docs/ARCHITECTURE.md	The design + trust boundaries

The sidecar source lives in this repo: services/sidecar/cmd/vitals-sidecar.

Quick start (local)

# 0) one-time: install deps + native SDK + Turso schema
pnpm install
docker build -t vitals-sidecar:local services/sidecar       # compiles the native engine
turso db shell vitals-hackathon < db/schema.sql              # if not already applied
cp .env.example .env                                         # fill SMARTSPECTRA_API_KEY + Turso creds

# 1) sidecar (needs your Presage API key at runtime)
SMARTSPECTRA_API_KEY=… docker compose up sidecar             # :8080  /healthz -> engine_built:true

# 2) worker (set Turso secrets first — see db/README.md)
pnpm dev:worker                                              # wrangler dev -> :8787

# 3) web
pnpm web                                                     # http://localhost:5173

Open the web app, set API base to http://localhost:8787, and record a clip. Without a valid SMARTSPECTRA_API_KEY the sidecar returns an auth error (the pipeline still runs).

Status

Slice	State
Backend end-to-end	✅ verified — web→Worker→sidecar→Turso with a real DB write (mock mode); row confirmed in the live DB
Go vitals-sidecar	✅ deployed & live on Fly (vitals-sidecar-nst.fly.dev) — /healthz → engine_built:true, mock:false, v3.1.0; live webcam dashboard at /
Cloudflare Worker + Drizzle	✅ deployed & live at vitals-api.darthbitcoin.workers.dev — typed insert+select against live Turso; /api/health + /api/measurements respond
Turso DB	✅ live (vitals-hackathon, group nicos-tools)
Web client	✅ deployed to Cloudflare Pages (vitals-web.pages.dev) — sign-in gate
Auth (all tiers)	✅ enforced — per-user sign-in mints an HMAC-signed session that gates the web app, Worker /api/*, and every sidecar route (incl. its dashboard). Verified: 401 without session, 200 with. CORS locked to the web origin
SmartSpectra API key	✅ accepted by the engine (no auth error at Start())
Live real-time path	✅ validated — sidecar webcam dashboard at /; browser JPEG frames → custom-input (monotonic timestamps). Drove 40 frames through the real engine + key: no timestamp/auth errors, ran face detection at ~54fps, returned no_face_found on synthetic input. Real pulse/breathing/HRV just need a real face.
Real-engine file upload	✅ fixed & live — /measure now decodes uploads with ffmpeg → custom-input (synthesized monotonic timestamps), bypassing the broken native reader. Deployed to Fly (real-engine amd64 build compiles & boots healthy); decode path unit-tested. Run an authed end-to-end measure with scripts/demo.sh.

Deploy

See deploy/README.md for the full sidecar (Fly.io) + Worker + Pages steps. In short: deploy the sidecar container built from services/sidecar, set its SMARTSPECTRA_API_KEY secret, wrangler deploy the Worker with Turso secrets, then wrangler pages deploy apps/web and point the web's API base at the Worker.

Secrets live in .env (gitignored) and Fly/Wrangler secrets — never in git. Get a SmartSpectra key at https://physiology.presagetech.com/auth/register.

Auth (per-user sign-in). The Worker and sidecar share one signing secret, VITALS_SESSION_SECRET (fly secrets set on the sidecar, wrangler secret put on the Worker — generate with openssl rand -hex 32). Accounts live in the Worker secret VITALS_USERS, a JSON map { "<username>": { salt, hash, iterations } } where hash is PBKDF2-SHA256 of the password (never plaintext). A user signs in at the lock screen → POST /api/login returns an 8h HMAC-signed session, kept in localStorage and sent as Authorization: Bearer. Nothing is baked into the build. (VITALS_AUTH_TOKEN is a legacy secret from the earlier shared-token scheme — no longer read by any code; safe to remove.)