Add optional in-VM OTLP telemetry export sink#303
Conversation
Pure mapping from telemetry envelopes to OTLP log records: event type to EventName, structured JSON body, severity, and kernel.* attributes. Excludes screenshot/monitor categories from export.
Mirrors S2StorageWriter: reads the event stream via an independent ring reader and exports records over OTLP/HTTP through the OTel log SDK. Configurable endpoint, path, headers, and resource attributes; drops excluded categories and oversized envelopes.
Adds OTLP_RELAY_* config and starts the OTLP storage writer at boot when an endpoint is set, alongside the existing S2 sink, with matching graceful shutdown. JWT is redacted in logs.
Receives OTLP/HTTP on :4318 and prints decoded records to stdout for local validation of the telemetry export path.
EventName is dropped by some backends (Datadog, Loki), and a structured body is only flattened by some (Honeycomb), so mirror the event type into kernel.event.type and lift high-value network/console fields into queryable attributes using semantic-convention names. Full payload stays in the body.
Envelopes are already capped at ~1MB at publish, so a 4MB per-record check could never fire. Removes the dead guard, its drop counter, and the unused storage logger.
Substrate-independent harness: a measurement OTLP receiver (record count, end-to-end lag p50/p99, bytes, slow-upstream delay) and a relay-shape load driver (light/heavy profiles at a target rate) feeding the shared converter. Reused unchanged by the server-side substrates; only the driver differs.
The Temporal and claim-goroutine substrates share the relay's convert/batch/POST hot path but add an S2 read hop. The knob injects that hop so end-to-end lag can be compared against the relay baseline.
Adds a durable-pusher mode (unbounded log + resumable cursor) and a delivery outage knob, plus unique-record counting in the receiver, so the relay's best-effort loss can be measured head-to-head against at-least-once recovery.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub. |
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
Wraps the exporter to log delivery failures (the SDK otherwise reports them only through an uninstalled global logger, so loss was silent). Removes the unused MaxBatchRecords knob, omits empty resource attributes, logs the two missing OTLP config fields, decodes the exported payload in the sink test to prove category exclusion, and notes the producer-schema tie in the converter.
The otlpbench rig (including the alternative-substrate durable-pusher model and fault injection) is delivery-substrate exploration, not part of the in-VM exporter. Moved to its own branch to keep this PR reviewable in isolation.
1 participant
Summary
OTLP_RELAY_ENDPOINT— no behavior change when unset.EventName(also mirrored to akernel.event.typeattribute for backends that dropEventName), structured JSON body, derived severity, andkernel.*attributes. High-value network/console fields are promoted to semantic-convention attributes (http.response.status_code,url.full,http.request.method). Screenshot/monitor categories are excluded from export.S2StorageWriter: an independent ring reader feeds records through the OTel log SDK (otlploghttp+ batch processor). Runs independently of S2 (both can be enabled). Export failures are surfaced via a logging exporter wrapper (the SDK otherwise reports them only through an uninstalled global logger).Testing
otelcol-contrib(config underserver/dev/otelcol/), including a real headless-Chromium CDP run confirming live events convert and export correctly.go build ./...,go vet, and the events test suite pass.Notes
hypeship/otlp-substrate-bench, to keep this PR reviewable in isolation.