feat(telemetry): local product-event capture — activation funnel, Tier-1 (ent#184)

docs/PRODUCT_EVENTS.md

@@ -0,0 +1,64 @@
+# Local Product Events — Activation Funnel (Tier-1)
+
+Trinity records a small set of **local, anonymous product events** so you — the
+operator — can see how your own first-run users move through setup and reach
+their first value. This is **Tier-1** of a two-tier telemetry model.
+
+> **This data never leaves your instance.** There is no network egress from this
+> layer. Events are written to your local database and read back only by you, on
+> your own admin surface. Sharing anything externally is a separate, explicitly
+> opt-in feature (Tier-2, not enabled by default and not part of this layer).
+
+## What is recorded
+
+**Onboarding-wizard step transitions** (emitted by the first-run wizard):
+
+| Event | Meaning |
+|-------|---------|
+| `setup_started` | The first-run wizard was opened |
+| `setup_step_create` | An intent was picked; advanced to the create form |
+| `setup_step_credential` | The first agent was created; reached the credential step |
+| `setup_completed` | The wizard was finished (opened chat / went to credentials) |
+| `setup_dismissed` | The wizard was closed before creating an agent |
+
+**First-value events** — `first_agent_created`, `first_chat`,
+`first_schedule_created`, `first_channel_connected`. These are **not** captured
+by a separate beacon; they are **derived on read** from data Trinity already
+records (the audit log and agent-activity timeline), so they cost no extra write
+path and survive restarts by construction.
+
+Each event carries:
+
+- a stable, random **installation id** (the same anonymous per-install id used by
+  the operator-intake correlation key — not tied to any user account),
+- a **UTC timestamp**, and
+- optionally a tiny, non-sensitive context blob (e.g. which starter intent was
+  picked). No message contents, credentials, emails, or PII are recorded.
+
+The emit endpoint accepts only the fixed allow-list of event types above; it
+cannot be used to store arbitrary data.
+
+## What is NOT recorded
+
+- No chat/message contents, no agent outputs.
+- No credentials, tokens, API keys, or emails.
+- No IP addresses or user identities beyond the anonymous install id.
+
+## Turning it off
+
+Capture is on by default and is intentionally lightweight (a handful of rows per
+install). Because it never phones home, there is no privacy reason to disable it.
+If you want zero local capture, you can drop the `product_events` rows at any
+time — nothing else depends on them:
+
+```sql
+DELETE FROM product_events;
+```
+
+## Viewing the funnel
+
+The operator-facing **Activation funnel** view (Settings → Activation) shows
+step-by-step activation counts, drop-off between steps, and the first-value
+tiles, with an honest empty state before any data exists. The funnel view is an
+entitlement-gated enterprise surface; the **capture** described above runs in
+every edition.

docs/memory/requirements/lifecycle-observability.md

@@ -323,6 +323,59 @@ endpoint — reports flow agent → MCP → backend.
 
 ---
 
+## 45. Local Product-Event Capture — Activation Funnel, Tier-1 (ent#184)
+
+**Description**: A **local-only** product-event capture layer — the **Tier-1**
+half of the two-tier telemetry model (Tier-2 = opt-in anonymized fleet sharing,
+#758 / trinity-enterprise#12, which builds on this). Tier-1 records
+activation/usage events **on the operator's own instance, default-ON, with zero
+network egress**, so the operator can see where their own first-run users drop
+off. It is *not* a sovereignty concern — nothing leaves the box — and is distinct
+from the identifiable opt-in operator intake (§43.1): this is anonymous,
+instance-local instrumentation keyed by the same `installation_id`.
+
+**Open-core split** (product decision, gating confirmed ent#184): the **capture**
+is OSS-core (the edition-agnostic instrumentation primitive, default-on); the
+operator-facing **activation-funnel view** is an entitlement-gated enterprise
+surface (`telemetry` feature-id). The generic seam is documented here; the funnel
+module's design lives in the private submodule.
+
+- **FR-1 — Event set v1 (OSS capture)**: the genuinely-new client beacons are the
+  onboarding-wizard step transitions — `setup_started`, `setup_step_intro`,
+  `setup_step_create`, `setup_step_credential`, `setup_completed`,
+  `setup_dismissed` — emitted by `components/OnboardingWizard.vue` through
+  `stores/productTelemetry.js` → `POST /api/product-events`. **First-value
+  events** (`first_agent_created`, `first_chat`, `first_schedule_created`,
+  `first_channel_connected`) are **derived on read** from the rows Trinity
+  already writes (`audit_log`, `agent_activities`, `schedule_executions`), never
+  re-emitted — so they survive restart by construction and add no write path.
+- **FR-2 — Storage (OSS)**: a local SQLite/Postgres table `product_events`
+  (`installation_id`, `event_type`, `event_context` optional small JSON,
+  `created_at`; dual-track migration + `db/tables.py` MetaData). The emit
+  endpoint accepts only a **fixed allow-list** of `event_type` values (unknown →
+  422) so the table can't be spammed with arbitrary strings. Rows carry the
+  stable `installation_id` (§43.1) and a UTC timestamp so Tier-2's opt-in
+  **retroactive backfill at consent** can serialize history — the mechanism that
+  rescues early-funnel data despite consent arriving late.
+- **FR-3 — Zero egress**: the capture layer NEVER phones home; the emit endpoint
+  writes one local row and returns. All sharing/consent lives in Tier-2 (#12).
+  Verifiable and documented as local-only in user docs.
+- **FR-4 — Operator funnel view (enterprise-gated)**: an operator-facing
+  activation/funnel panel on an existing admin surface (Settings, admin-only)
+  shows step-by-step activation counts + drop-off with an honest empty state when
+  there's no data yet. It reads a gated enterprise endpoint
+  (`requires_entitlement("telemetry")`) that aggregates `product_events` +
+  derives the first-value events from the OSS tables above. The **panel Vue**
+  ships in the OSS bundle but is hidden unless `telemetry` is in
+  `enterprise_features` (the standard feature-flag gating). Explicitly **NOT** a
+  new standalone analytics dashboard in v1.
+
+**Deferred**: auto-retention sweep for `product_events` (volume is negligible —
+a handful of rows per install); per-user (vs per-install) funnel cohorts;
+Tier-2 opt-in sharing + backfill serialization (#12).
+
+---
+
 ## Ephemeral "Ghost" Agents (trinity-enterprise#69)
 
 **Description**: A disposable-agent lifecycle — an agent is created with a hard

src/backend/database.py

@@ -113,6 +113,7 @@
 from db.sessions import SessionOperations
 from db.activities import ActivityOperations
 from db.reports import ReportOperations
+from db.product_events import ProductEventOperations
 from db.reminders import RemindersOperations
 from db.connector import ConnectorOperations
 from db.permissions import PermissionOperations
@@ -432,6 +433,7 @@ def __init__(self):
         self._session_ops = SessionOperations()
         self._activity_ops = ActivityOperations()
         self._report_ops = ReportOperations()
+        self._product_event_ops = ProductEventOperations()
         self._reminder_ops = RemindersOperations()
         self._connector_ops = ConnectorOperations()
         self._permission_ops = PermissionOperations(self._user_ops, self._agent_ops)
@@ -1455,6 +1457,24 @@ def delete_report(self, agent_name: str, report_id: str):
     def prune_agent_reports(self, retention_days: int = 90, chunk_size: int = 1000):
         return self._report_ops.prune_agent_reports(retention_days, chunk_size)
 
+    # =========================================================================
+    # Local Product-Event Capture Methods (ent#184 — delegated to db/product_events.py)
+    # =========================================================================
+
+    def record_product_event(self, installation_id, event_type, event_context=None):
+        return self._product_event_ops.record_product_event(
+            installation_id, event_type, event_context
+        )
+
+    def count_product_events_by_type(self, since=None):
+        return self._product_event_ops.count_product_events_by_type(since)
+
+    def list_product_events(self, event_type=None, since=None, limit=1000, offset=0):
+        return self._product_event_ops.list_product_events(event_type, since, limit, offset)
+
+    def prune_product_events(self, retention_days, chunk_size=1000):
+        return self._product_event_ops.prune_product_events(retention_days, chunk_size)
+
     # =========================================================================
     # Agent Self-Reminder Methods (#1296 — delegated to db/reminders.py)
     # =========================================================================

src/backend/db/migrations.py

@@ -3083,6 +3083,41 @@ def _migrate_schedule_executions_redelivery_count(cursor, conn):
     conn.commit()
 
 
+def _migrate_product_events_table(cursor, conn):
+    """Create product_events table (ent#184).
+
+    Local product-event capture — activation funnel, Tier-1. Local-only,
+    default-on, zero egress. Records onboarding-wizard step transitions; the
+    first-value events are derived on read from audit_log/agent_activities.
+    Schema is also in db/schema.py for fresh installs; this handles existing
+    installs. Idempotent. Mirrored by the Alembic revision 0029_product_events
+    for PostgreSQL.
+    """
+    cursor.execute("PRAGMA table_info(product_events)")
+    if cursor.fetchall():
+        return  # already created (fresh-install path via init_schema)
+
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS product_events (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            installation_id TEXT NOT NULL,
+            event_type TEXT NOT NULL,
+            event_context TEXT,
+            created_at TEXT NOT NULL
+        )
+    """)
+    cursor.execute(
+        "CREATE INDEX IF NOT EXISTS idx_product_events_type_created "
+        "ON product_events(event_type, created_at)"
+    )
+    cursor.execute(
+        "CREATE INDEX IF NOT EXISTS idx_product_events_created "
+        "ON product_events(created_at)"
+    )
+    conn.commit()
+    print("Created product_events table (ent#184)")
+
+
 MIGRATIONS = [
     ("agent_sharing", _migrate_agent_sharing_table),
     ("schedule_executions_observability", _migrate_schedule_executions_observability),
@@ -3183,4 +3218,5 @@ def _migrate_schedule_executions_redelivery_count(cursor, conn):
     ("operator_queue_request_id", _migrate_operator_queue_request_id),
     ("users_github_pat", _migrate_users_github_pat),
     ("agent_reminders_table", _migrate_agent_reminders_table),
+    ("product_events_table", _migrate_product_events_table),
 ]

src/backend/db/product_events.py

@@ -0,0 +1,141 @@
+"""
+Local product-event capture database operations (ent#184).
+
+Tier-1 of the two-tier telemetry model: activation/usage events recorded **on
+the operator's own instance, default-ON, with zero network egress**. Only the
+onboarding-wizard step transitions are emitted here (the genuinely-new client
+beacons); first-value events (first_agent_created, first_chat, ...) are derived
+on read from ``audit_log``/``agent_activities`` and never re-emitted.
+
+SQLAlchemy Core over the ``product_events`` table in ``db/tables.py`` so it runs
+unchanged on SQLite and PostgreSQL. This layer holds the WRITE path + minimal
+read helpers; the operator-facing funnel aggregation is an entitlement-gated
+enterprise surface that reads these rows (open-core split, ent#184).
+"""
+
+import json
+from typing import Dict, List, Optional
+
+from sqlalchemy import select, insert, delete, func
+
+from .engine import get_engine
+from .tables import product_events
+from utils.helpers import utc_now_iso, iso_cutoff
+
+
+class ProductEventOperations:
+    """Local product-event capture operations (ent#184)."""
+
+    def record_product_event(
+        self,
+        installation_id: str,
+        event_type: str,
+        event_context: Optional[Dict] = None,
+    ) -> Dict:
+        """Insert one local product event. Zero egress — one local row.
+
+        ``event_context`` is an optional small dict serialized to JSON. The
+        caller (router) is responsible for allow-listing ``event_type``; this
+        layer just persists.
+        """
+        now = utc_now_iso()
+        ctx = json.dumps(event_context) if event_context else None
+        stmt = insert(product_events).values(
+            installation_id=installation_id,
+            event_type=event_type,
+            event_context=ctx,
+            created_at=now,
+        )
+        with get_engine().begin() as conn:
+            result = conn.execute(stmt)
+        new_id = result.inserted_primary_key[0] if result.inserted_primary_key else None
+        return {
+            "id": new_id,
+            "installation_id": installation_id,
+            "event_type": event_type,
+            "event_context": event_context,
+            "created_at": now,
+        }
+
+    def count_product_events_by_type(self, since: Optional[str] = None) -> Dict[str, int]:
+        """Counts grouped by ``event_type`` (optionally since an ISO cutoff).
+
+        The primitive the enterprise activation-funnel view aggregates over the
+        wizard-step slice. Returns ``{event_type: count}``.
+        """
+        stmt = select(product_events.c.event_type, func.count().label("n"))
+        if since:
+            stmt = stmt.where(product_events.c.created_at >= since)
+        stmt = stmt.group_by(product_events.c.event_type)
+        with get_engine().connect() as conn:
+            return {r["event_type"]: r["n"] for r in conn.execute(stmt).mappings()}
+
+    def list_product_events(
+        self,
+        event_type: Optional[str] = None,
+        since: Optional[str] = None,
+        limit: int = 1000,
+        offset: int = 0,
+    ) -> List[Dict]:
+        """Raw rows (oldest first) for Tier-2 backfill serialization + audit.
+
+        Ordered by ``created_at`` ASC so a later opt-in (#12) can serialize
+        history in chronological order.
+        """
+        stmt = select(product_events)
+        conditions = []
+        if event_type:
+            conditions.append(product_events.c.event_type == event_type)
+        if since:
+            conditions.append(product_events.c.created_at >= since)
+        if conditions:
+            for c in conditions:
+                stmt = stmt.where(c)
+        stmt = (
+            stmt.order_by(product_events.c.created_at.asc())
+            .limit(limit)
+            .offset(offset)
+        )
+        with get_engine().connect() as conn:
+            rows = conn.execute(stmt).mappings().all()
+        return [
+            {
+                "id": r["id"],
+                "installation_id": r["installation_id"],
+                "event_type": r["event_type"],
+                "event_context": json.loads(r["event_context"]) if r["event_context"] else None,
+                "created_at": r["created_at"],
+            }
+            for r in rows
+        ]
+
+    def prune_product_events(self, retention_days: int, chunk_size: int = 1000) -> int:
+        """Delete product events older than ``retention_days``. ``0`` disables.
+
+        Provided for completeness; not wired into an automatic sweep in v1 (the
+        table is negligible — a handful of rows per install). Chunked so a large
+        table never holds the write lock for the full purge.
+        """
+        if retention_days <= 0 or chunk_size <= 0:
+            return 0
+        cutoff = iso_cutoff(hours=retention_days * 24)
+        total = 0
+        while True:
+            with get_engine().begin() as conn:
+                ids = [
+                    row["id"]
+                    for row in conn.execute(
+                        select(product_events.c.id)
+                        .where(product_events.c.created_at < cutoff)
+                        .limit(chunk_size)
+                    ).mappings()
+                ]
+                if not ids:
+                    break
+                result = conn.execute(
+                    delete(product_events).where(product_events.c.id.in_(ids))
+                )
+                total += result.rowcount
+            if len(ids) < chunk_size:
+                break
+        return total

src/backend/db/schema.py

@@ -502,6 +502,21 @@
         )
     """,
 
+    # -------------------------------------------------------------------------
+    # Local product-event capture — activation funnel, Tier-1 (ent#184)
+    # Local-only, default-on, zero egress. Wizard step transitions are emitted;
+    # first-value events are derived on read from audit_log/agent_activities.
+    # -------------------------------------------------------------------------
+    "product_events": """
+        CREATE TABLE IF NOT EXISTS product_events (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            installation_id TEXT NOT NULL,
+            event_type TEXT NOT NULL,
+            event_context TEXT,
+            created_at TEXT NOT NULL
+        )
+    """,
+
     # -------------------------------------------------------------------------
     # Notifications (NOTIF-001)
     # -------------------------------------------------------------------------
@@ -1380,6 +1395,11 @@
     # Serves the retention sweep's `WHERE created_at < cutoff` scan (#918).
     "CREATE INDEX IF NOT EXISTS idx_agent_reports_created ON agent_reports(created_at)",
 
+    # Product-event capture (ent#184): funnel aggregation groups by event_type,
+    # backfill/query orders by created_at.
+    "CREATE INDEX IF NOT EXISTS idx_product_events_type_created ON product_events(event_type, created_at)",
+    "CREATE INDEX IF NOT EXISTS idx_product_events_created ON product_events(created_at)",
+
     # Permission indexes
     "CREATE INDEX IF NOT EXISTS idx_permissions_source ON agent_permissions(source_agent)",
     "CREATE INDEX IF NOT EXISTS idx_permissions_target ON agent_permissions(target_agent)",