[cloud_security_posture] GCP lifecycle fields + metering_state transform for CSPM metering

[seanrathier]

Summary

Adds the package-side support for stateful serverless CSPM metering: the ingest pipeline indexes GCP compute lifecycle fields, and a new metering_state transform maintains the per-incarnation scan state that the metering task bills from.

Warning

Draft — do not merge. The transform does not install in serverless; see Known limitation below. The ingest-pipeline commit is independently sound.

Related:

• https://github.com/elastic/security-team/issues/17662
• Kibana side (billing query + state index + tests): [Cloud Security] Stateful CSPM metering via per-incarnation state index kibana#285268

---

What's here

1. Ingest pipeline (962667c) — lifts GCP compute lifecycle data out of the unindexed resource.raw payload into indexed resource.lifecycle.* fields (status, incarnation, created_at, last_started_at, last_stopped_at, last_run_ms). Without this the metering transform has nothing to group or sort on.

2. metering_state transform — a pivot transform writing one document per (resource.id, incarnation, sub_type, account, posture_type) into logs-cloud_security_posture.metering_state-default.

Grouping on resource.lifecycle.incarnation is the point: a spot VM that is destroyed and re-created under the same name starts a fresh bucket instead of inheriting the previous instance's first_seen, so span_ms never bridges two physically different machines.

Two design notes worth review attention:

• max + bucket_script, not top_metrics. top_metrics emits the string "null" for a metric absent from a bucket, which the date/long destination mappings reject — dropping the whole document. That silently excluded every non-GCP resource and every never-stopped instance. max is equivalent here because these timestamps are monotonic within one incarnation, and the incarnation is part of the group key.
• last_run_ms is signed on purpose. stop - start; absent when the instance never stopped (default gap_policy: skip), negative when it was restarted after its last stop. The billing query uses that sign to tell a restarted-and-running instance from a stopped one, now that lifecycle status is no longer carried in the state document.

---

Known limitation (blocks merge)

The transform fails to install in serverless:

Cannot create transform [logs-cloud_security_posture.metering_state-default-0.1.0]
because user elastic/kibana lacks the required permissions
[logs-cloud_security_posture.metering_state-default:[delete, index]]

action [indices:admin/create] is unauthorized for service account [elastic/kibana]
  on indices [logs-cloud_security_posture.metering_state-default]

_meta.run_as_kibana_system: false is set, and Fleet does apply the installing user's secondary auth to putTransform (with defer_validation: true). It does not apply it to the destination-index create/delete, which are still attempted as elastic/kibana. That account has read on logs-*.* but no create_index/index/delete on this pattern.

The failure aborts the whole transform batch, so the pre-existing misconfiguration transform fails to install too, and POST /api/fleet/package_policies returns 400.

Needs one of: a Fleet change to pass secondary auth to the destination-index operations, or a service-account grant. Input from the Fleet team welcome — this is the part of the diagnosis I'm least certain of.

---

Test plan

elastic-package lint          # passes
elastic-package build         # produces cloud_security_posture-3.6.0.zip

Pipeline tests for the lifecycle fields are in _dev/test/pipeline/test-gcp-lifecycle.json.

Reproducing the install failure end to end:

1. elastic-package build in this package.
2. Run docker.elastic.co/kibana-ci/package-registry-distribution:lite with package_paths covering both /packages/package-storage and a mount of integrations/build/packages.
3. Start the serverless security FTR with --xpack.fleet.registryUrl pointed at it and CLOUD_SECURITY_PLUGIN_VERSION set to 3.6.0.
4. Observe the 400 from package-policy creation and the security_exception in the Kibana log.

🤖 Generated with Claude Code

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: f329efa

cc @seanrathier