[vercel_otel] Add alert rule and SLO templates for server errors and Core Web Vitals - #20699
Conversation
…Core Web Vitals Add 10 alert rule templates (errors, traffic, saturation, latency, and audit governance) and 5 SLO templates (server-side error rate and Core Web Vitals p75) with dashboard links, so operators have ready-to-enable detection and reliability tracking for Vercel OTel data on top of the existing dashboards.
Elastic Docs Style Checker (Vale)Summary: 1 suggestion found 💡 Suggestions (1): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
Point the 0.3.0 changelog entry at elastic#20699 now that the PR has been opened, replacing the placeholder link.
| | **[Vercel OTel] Web Analytics** | Traffic and engagement overview covering page views, geography, devices, top pages, and custom events. | | ||
| | **[Vercel OTel] Speed Insights** | Web Vitals performance overview tracking LCP, INP, CLS, and TTFB at the p75 benchmark over time and per page. | | ||
|
|
||
| ## Alert rules |
There was a problem hiding this comment.
These should be generated dynamically from template
## Alerting Rule Templates
{{alertRuleTemplates}}
|
|
||
| All rules use ES|QL queries and are filtered to `deployment.environment.name: production`. Thresholds are conservative defaults — adjust them to match your traffic volume and SLA. | ||
|
|
||
| ## SLO templates |
There was a problem hiding this comment.
These should be generated dynamically from template
## SLO Templates
{{sloTemplates}}
| "params": { | ||
| "searchType": "esqlQuery", | ||
| "esqlQuery": { | ||
| "esql": "FROM metrics-vercel.speedinsights.v1.otel-default\n| WHERE deployment.environment.name == \"production\"\n AND metrics.TTFB IS NOT NULL\n| STATS\n ttfb_p75 = PERCENTILE(metrics.TTFB, 75),\n samples = COUNT(*)\n// Minimum sample size, since Speed Insights data is sampled\n| WHERE samples >= 20\n// \"Poor\" threshold per the Web Vitals spec table (milliseconds); lower toward 800-1800 for a stricter,\n// earlier-warning threshold aligned with the \"needs improvement\" boundary\n| WHERE ttfb_p75 > 1800.0" |
There was a problem hiding this comment.
We should use TS query since its TSDB. In dashboards we already use TS queries - https://github.com/elastic/integrations/blob/main/packages/vercel_otel/kibana/dashboard/vercel_otel-speed-insights.json
| "params": { | ||
| "searchType": "esqlQuery", | ||
| "esqlQuery": { | ||
| "esql": "FROM metrics-vercel.speedinsights.v1.otel-default\n| WHERE deployment.environment.name == \"production\"\n// Evaluate LCP, INP, and CLS independently per route, then union the results (FORK branches must share a schema)\n| FORK\n (WHERE metrics.LCP IS NOT NULL\n | STATS p75 = PERCENTILE(metrics.LCP, 75), samples = COUNT(*) BY route = COALESCE(http.route, url.path)\n // \"Poor\" threshold per the Web Vitals spec table (milliseconds); adjust to your own SLA if stricter\n | EVAL metric = \"LCP\", poor_threshold = 4000.0)\n (WHERE metrics.INP IS NOT NULL\n | STATS p75 = PERCENTILE(metrics.INP, 75), samples = COUNT(*) BY route = COALESCE(http.route, url.path)\n | EVAL metric = \"INP\", poor_threshold = 500.0)\n (WHERE metrics.CLS IS NOT NULL\n | STATS p75 = PERCENTILE(metrics.CLS, 75), samples = COUNT(*) BY route = COALESCE(http.route, url.path)\n // CLS is unitless; \"poor\" threshold per the spec table\n | EVAL metric = \"CLS\", poor_threshold = 0.25)\n// Minimum sample size per route/metric, since Speed Insights data is sampled\n| WHERE samples >= 5\n| WHERE p75 > poor_threshold\n// Combine route + metric into a single grouping key since the rule can only group alerts by one field\n| EVAL group_key = CONCAT(route, \" - \", metric)\n| SORT p75 DESC\n| LIMIT 20" |
There was a problem hiding this comment.
We should use TS query since its TSDB. In dashboards we already use TS queries - https://github.com/elastic/integrations/blob/main/packages/vercel_otel/kibana/dashboard/vercel_otel-speed-insights.json
| "params": { | ||
| "searchType": "esqlQuery", | ||
| "esqlQuery": { | ||
| "esql": "FROM logs-vercel.logs.otel-default\n| WHERE vercel.log.source == \"firewall\"\n// vercel.proxy is a flattened field; sub-fields aren't queryable via dotted notation, use FIELD_EXTRACT instead\n| EVAL waf_action = FIELD_EXTRACT(vercel.proxy, \"waf_action\")\n// \"firewall\" source includes log-only entries too; isolate actual blocks\n| WHERE waf_action == \"deny\"\n| STATS denies = COUNT(*)\n// Alert threshold: tune based on your application's normal WAF deny baseline.\n// Consider grouping by client.address instead of a flat total if you want per-attacker alerting.\n| WHERE denies > 100" |
There was a problem hiding this comment.
Missing WHERE deployment.environment.name == \"production\"\n which is present in other alerts
| "params": { | ||
| "searchType": "esqlQuery", | ||
| "esqlQuery": { | ||
| "esql": "FROM logs-vercel.auditlog.v1.otel-default\n// Fall back through identifying fields since non-user actors (app/system/integration) lack user.* fields\n| EVAL actor = COALESCE(user.email, user.name, user.id, vercel.actor.type)\n// Sensitive action categories: env var access, drain/webhook config (log exfiltration risk), token issuance,\n// team/domain configuration, and project ownership transfers. Extend this list as needed.\n| WHERE event_name LIKE \"*env-variable*\"\n OR event_name LIKE \"*drain*\"\n OR event_name LIKE \"*webhook*\"\n OR event_name LIKE \"*token*\"\n OR event_name LIKE \"*team-*\"\n OR event_name LIKE \"*domain*\"\n OR event_name LIKE \"*project-transfer*\"\n| STATS sensitive_events = COUNT(*) BY actor\n// Alert threshold: tune based on your team's normal operational volume of sensitive actions\n| WHERE sensitive_events > 5\n| SORT sensitive_events DESC\n| LIMIT 20" |
There was a problem hiding this comment.
Missing WHERE deployment.environment.name == \"production\"\n which is present in other alerts
There was a problem hiding this comment.
Fixed for vercel_otel-firewall-deny-spike.json (logs-vercel.logs.otel-default), which does have this field.
For this one specifically I held off: logs-vercel.auditlog.v1.otel-default doesn't carry deployment.environment.name — I checked _field_caps against a live cluster and it's not part of that data stream's mapping (audit events are team/account-level governance actions — env var reads, token/webhook/domain changes, etc. — not tied to a specific deployment or environment). Adding the filter throws Unknown column [deployment.environment.name] (verification_exception) rather than just returning no rows, so it would break the rule outright. The pre-existing Audit Logs dashboard doesn't reference this field either, for the same reason.
Happy to add it if I'm missing something about how this field gets populated for audit events — let me know.
…docs
- Generate Alerting Rule Templates / SLO Templates sections in docs/README.md
dynamically via {{alertRuleTemplates}}/{{sloTemplates}} instead of hand-
maintained tables, matching the convention used elsewhere.
- Use TS instead of FROM for Speed Insights queries (TSDB data stream),
matching the existing dashboard; adjust COUNT(*) to COUNT(<field>) since
the TS command doesn't support COUNT(*).
- Add the missing deployment.environment.name == "production" filter to the
firewall deny spike alert, consistent with its sibling alerts on the same
data stream.
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
|
2 participants
Proposed commit message
Add 10 alert rule templates and 5 SLO templates to the
vercel_otelcontent package.WHAT: The alert rule templates cover HTTP 5xx error rate, lambda/edge crash rate, elevated ERROR/WARNING log rate, regional error rate skew, firewall deny spikes, elevated TTFB p75, poor Core Web Vitals by route, request rate drop, page view rate drop, and sensitive audit activity spikes — all as
.es-queryrules using ES|QL. The SLO templates cover server-side error rate (occurrence budgeting) and LCP/INP/CLS/TTFB p75 (timeslice budgeting, rolling 30 days). Each template links to the relevant existing dashboard viaartifacts.dashboards.WHY: The package previously shipped dashboards only, so operators had no ready-to-enable detection (alerts) or reliability tracking (SLOs) for Vercel OTel data and had to hand-author ES|QL queries and thresholds from scratch. These templates encode SRE-informed golden-signal thresholds (errors, traffic, saturation, latency) so operators can enable them directly from Fleet. Closes elastic/observability-dev#5912.
Checklist
changelog.ymlfile.Author's Checklist
artifacts.dashboards(vercel_otel-logs,vercel_otel-audit-logs,vercel_otel-web-analytics) match this package's actual dashboard saved-object IDs.data_stream.datasetfilter values (vercel.logs.otel,vercel.speedinsights.v1.otel) match the dataset names declared inmanifest.yml'sdiscovery.datasets.How to test this PR locally
Executed by agent (this session):
elastic-package lint— passed cleanly.elastic-package build— succeeded, producedbuild/packages/vercel_otel-0.3.0.zip.vercel_oteldata streams on a local Elastic stack; created live instances of all 10 alert rule templates via the Kibana Alerting API and all 5 SLO templates via the SLO API. Confirmed all 10 rules transitioned toactivewith correctly-targeted alert instances (and correctly recovered as the data aged out of each rule's window), and all 5 SLOs computed real (non-NO_DATA)sliValues reflecting the seeded conditions.Reviewer instructions:
elastic-package stack up, thenelastic-package install --zip build/packages/vercel_otel-0.3.0.zip.Related issues
Screenshots