feat(pii): export PII/audit events as a Prometheus counter#10641
Conversation
The PII EventStore ring buffer is capacity-bound and meant for
recent-audit browsing via /api/pii/events; operators also want a
monotonic, scrape-friendly signal on /metrics — how many
detections/masks/blocks per hour, per origin, and whether the filter
stopped firing after a deploy (silent-failure class).
EventStore.Record is the single choke point every producer already goes
through (request middleware, response scrubbing, MITM proxy
connects/intercepts), so one lazily-initialised counter there covers all
paths without touching any producer:
localai_pii_events_total{kind, origin, action, direction}
Same lazy otel.Meter pattern as core/services/routing/billing, so the
counter lands on the Prometheus-backed global MeterProvider installed by
the monitoring service. No behaviour change; label cardinality is
bounded (enum-like fields only, no pattern IDs or user IDs).
Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
|
Live verification as promised: deployed this patch on our LocalAI instance (v4.5.6 + this commit, PII middleware active on cloud-proxy models). First request containing PII (name + email, masked before egress) produced: Counter lands on /metrics via the existing Prometheus-backed MeterProvider, series appears on first event, labels populate as designed ( |
|
Note on the red "PII NER tier E2E (live GGUF, CPU)" check: the job hit the 30-minute go-test timeout ( 🤖 Generated with Claude Code |
Operators need a scrape-friendly signal for agent-turn health (completing,
erroring, cancelled, duration) — log-derived counters proved brittle (ANSI/
timezone parsing, restart gaps). Adds localai_agent_runs_total{agent,outcome}
and localai_agent_run_seconds histogram, recorded at the Chat() response
handoff (single choke point of the local execution path). Lazy meter init,
same pattern as the PII events counter (mudler#10641).
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Operators need a scrape-friendly signal for agent-turn health (completing,
erroring, cancelled, duration) — log-derived counters proved brittle (ANSI/
timezone parsing, restart gaps). Adds localai_agent_runs_total{agent,outcome}
and localai_agent_run_seconds histogram, recorded at the Chat() response
handoff (single choke point of the local execution path). Lazy meter init,
same pattern as the PII events counter (#10641).
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
3 participants
What
Adds a single Prometheus counter for the PII middleware / MITM audit pipeline:
Why
The PII
EventStorering buffer is capacity-bound and meant for recent-audit browsing via/api/pii/events. For operations you also want a monotonic, scrape-friendly signal on/metrics: how many detections/masks/blocks per hour, per origin — and most importantly whether the filter stopped firing after a deploy (silent-failure class; alertable with a simplerate()).How
EventStore.Recordis the single choke point every producer already goes through (request middleware, response scrubbing, MITM proxy connects/intercepts), so one lazily-initialised counter there covers all paths without touching any producer. Same lazyotel.Meterpattern ascore/services/routing/billing, so the counter lands on the Prometheus-backed global MeterProvider installed by the monitoring service.No behaviour change. Label cardinality is bounded — enum-like fields only (no pattern IDs, no user IDs).
Verification
go build ./core/services/routing/pii/+go test ./core/services/routing/pii/...green. Deployed on a LocalAI instance with the PII middleware + MITM proxy active; counter series appear on /metrics after the first detection/connect event.