Skip to content

Add support for OTEL metrics - #7459

Merged
pingsutw merged 5 commits into
flyteorg:mainfrom
Sovietaced:otel-metrics
Jun 8, 2026
Merged

pingsutw merged 5 commits into
flyteorg:mainfrom
Sovietaced:otel-metrics

Conversation

@Sovietaced

@Sovietaced Sovietaced commented May 29, 2026

Copy link
Copy Markdown
Member

Fixes #7447

Why are the changes needed?

This pull request generally adds support for emitting OTEL metrics. Additionally, it uses otelconnect to instrument all of the connect RPC clients/servers with OTEL metrics. Without configuration all trace/metric related calls will be a noop so this won't change the default behavior.

What changes were proposed in this pull request?

  • Generate meter providers alongside existing trace providers
  • Instrument RPC clients/servers with trace and meter providers

How was this patch tested?

We have a variation of this running in our fork in production. It still uses plain gRPC instead of connect RPC but we have used otelconnect in other capacities.

Labels

Please add one or more of the following labels to categorize your PR:

  • added: For new features.
  • changed: For changes in existing functionality.
  • deprecated: For soon-to-be-removed features.
  • removed: For features being removed.
  • fixed: For any bug fixed.
  • security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Stack

If you do use git town to manage PR Stacks, the stack relevant to this PR
will show below. Otherwise, you can ignore this section.

Docs link

Signed-off-by: Jason Parraga <sovietaced@gmail.com>
Signed-off-by: Jason Parraga <sovietaced@gmail.com>
Signed-off-by: Jason Parraga <sovietaced@gmail.com>
@Sovietaced Sovietaced added the added Merged changes that add new functionality label May 29, 2026
@Sovietaced
Sovietaced marked this pull request as ready for review May 29, 2026 23:49
)

const (
AdminClientTracer = "admin-client"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Old unused constants

Signed-off-by: Jason Parraga <sovietaced@gmail.com>
Signed-off-by: Jason Parraga <sovietaced@gmail.com>
@pingsutw
pingsutw merged commit 70242b9 into flyteorg:main Jun 8, 2026
21 checks passed
pingsutw added a commit that referenced this pull request Jun 10, 2026
Build on the meter provider from #7459 to instrument the TaskAction controller
with two metrics the framework doesn't provide:

- taskaction.active (Int64ObservableGauge): number of TaskAction CRDs by plugin
  phase, observed asynchronously by listing from the controller cache.
- taskaction.crd.size_bytes (Int64Histogram): serialized size of a TaskAction
  CRD, recorded per reconcile.

Both are emitted via otelutils.GetMeterProvider("executor"), so they flow
through the same OTLP pipeline as the RPC metrics. Registration is non-fatal
(degrades to no custom metrics) and observeCRDSize is nil-safe.

Reconcile rate/latency, workqueue depth, and k8s API r/w latency are already
exposed by the controller-runtime metrics server; event-proxy send latency is
already captured by the otelconnect-wrapped events client.

Signed-off-by: Kevin Su <pingsutw@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

added Merged changes that add new functionality flyte2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[flyte2] Add gRPC/Connect RPC metrics interceptor to the runs service

2 participants