[LiteLLM][Spend Tracking] Add spend_tracking data stream - #20206
[LiteLLM][Spend Tracking] Add spend_tracking data stream#20206muskan-agarwal26 wants to merge 8 commits into
Conversation
Elastic Docs Style Checker (Vale)Summary: 3 warnings, 5 suggestions found
|
| File | Line | Rule | Message |
|---|---|---|---|
| packages/lite_llm/manifest.yml | 135 | Elastic.DontUse | Don't use 'Please'. |
| packages/lite_llm/manifest.yml | 178 | Elastic.Latinisms | Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'. |
| packages/lite_llm/manifest.yml | 194 | Elastic.DontUse | Don't use 'Please'. |
💡 Suggestions (5): Optional style improvements. Apply when helpful.
| File | Line | Rule | Message |
|---|---|---|---|
| packages/lite_llm/changelog.yml | 1 | Elastic.Versions | Use 'later versions' instead of 'newer versions' when referring to versions. |
| packages/lite_llm/data_stream/spend_tracking/fields/fields.yml | 15 | Elastic.WordChoice | Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI. |
| packages/lite_llm/data_stream/spend_tracking/fields/fields.yml | 24 | Elastic.WordChoice | Consider using 'can, might' instead of 'may', unless the term is in the UI. |
| packages/lite_llm/data_stream/spend_tracking/fields/fields.yml | 147 | Elastic.WordChoice | Consider using 'can, might' instead of 'may', unless the term is in the UI. |
| packages/lite_llm/data_stream/spend_tracking/fields/fields.yml | 153 | Elastic.WordChoice | Consider using 'can, might' instead of 'may', unless the term is in the UI. |
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
🚀 Benchmarks reportTo see the full report comment with |
andrewkroh
added
Crest
…ntegrations into datastream-spend_tracking
ShourieG
added
the
Team:Security-Service Integrations
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
![vera-review-bot[bot]](https://avatars.github.com/in/3297022?s=60&v=4){kind=small}
| try { | ||
| def startMs = ZonedDateTime.parse(startTime).toInstant().toEpochMilli(); | ||
| def endMs = ZonedDateTime.parse(endTime).toInstant().toEpochMilli(); | ||
| ctx.event.duration = endMs - startMs; |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/lite_llm/data_stream/spend_tracking/elasticsearch/ingest_pipeline/default.yml:105
event.duration is computed in milliseconds, but ECS defines it as nanoseconds; multiply the millisecond delta by 1,000,000 (or omit and derive from event.start/event.end).
Details
The calculate_event_duration script parses start_time/end_time to epoch milliseconds and stores the difference (endMs - startMs) directly into ctx.event.duration. ECS specifies event.duration as the duration of the event in nanoseconds. Storing a millisecond value makes every event.duration value a factor of 1,000,000 too small and inconsistent with event.start/event.end. The pipeline tests pass only because start_time == end_time in all fixtures (duration 0), so the unit error is not exercised by the tests.
Recommendation:
Convert the millisecond delta to nanoseconds before assigning to event.duration:
if (startTime != null && endTime != null) {
try {
def startMs = ZonedDateTime.parse(startTime).toInstant().toEpochMilli();
def endMs = ZonedDateTime.parse(endTime).toInstant().toEpochMilli();
ctx.event.duration = (endMs - startMs) * 1000000L;
} catch(Exception e) {
ctx.event.duration = 0;
}
}
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| ### API usage | ||
|
|
||
| This integration uses the following API: |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/lite_llm/_dev/build/docs/README.md:176
The API usage section has two stacked intro sentences ('This integration uses the following API:' immediately followed by 'This integration dataset uses the following API:'); merge them into one.
Details
The new Spend Tracking API bullet was inserted with its own heading sentence, but the original 'This integration dataset uses the following API:' line was left in place directly below it. The rendered README now shows two consecutive introductory sentences before the API list, which reads as a copy/paste artifact.
Recommendation:
Keep a single intro sentence covering both APIs:
### API usage
This integration uses the following APIs:
- `Spend Tracking`: Collects spend tracking records via the **LiteLLM Spend Tracking API** (endpoint: `/spend/logs/v2`) or via **AWS S3/SQS** for organizations that export spend tracking data from LiteLLM to an S3 bucket.
- `Audit`: Collects audit logs via the **LiteLLM Audit API** (endpoint: `/audit`) or via **AWS S3/SQS** for organizations that export logs from LiteLLM to an S3 bucket.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| streams: | ||
| - input: aws-s3 | ||
| title: 'LiteLLM spend tracking logs via AWS S3' | ||
| description: 'Collect LiteLLM spend tracking logs (StandardAuditLogPayload) from an S3 bucket.' |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/lite_llm/data_stream/spend_tracking/manifest.yml:6
Both spend_tracking stream descriptions call the payload 'StandardAuditLogPayload', which is the audit-log object name; update to the spend/usage payload for accuracy.
Details
The aws-s3 and cel stream descriptions were copied from the audit data stream and still reference '(StandardAuditLogPayload)'. That object name describes LiteLLM audit-log records, not spend/usage records collected from /spend/logs/v2, so the label is inaccurate for this data stream.
Recommendation:
Drop or correct the payload name so it reflects spend tracking, e.g.:
- input: aws-s3
title: 'LiteLLM spend tracking logs via AWS S3'
description: 'Collect LiteLLM spend tracking logs from an S3 bucket.'🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| - **Direct API polling** via CEL input, which periodically queries the LiteLLM Audit API using API key authentication with cursor-based pagination. | ||
| - **Cloud storage** via AWS S3/SQS, for organizations that export audit logs from LiteLLM to an AWS S3 bucket. | ||
| - **Direct API polling** via CEL input, which periodically queries the LiteLLM API using API key authentication with cursor-based pagination. |
There was a problem hiding this comment.
| - **Direct API polling** via CEL input, which periodically queries the LiteLLM API using API key authentication with cursor-based pagination. | |
| - **Direct API polling** via the CEL input, which periodically queries the LiteLLM API using API key authentication with cursor-based pagination. |
| max([ | ||
| state.cursor.last_timestamp, | ||
| body.data[0].startTime | ||
| ]) |
There was a problem hiding this comment.
| max([ | |
| state.cursor.last_timestamp, | |
| body.data[0].startTime | |
| ]) | |
| max( | |
| state.cursor.last_timestamp, | |
| body.data[0].startTime | |
| ) |
| }).do_request().as(resp, resp.StatusCode == 200 ? | ||
| resp.Body.decode_json().as(body, { | ||
| "events": ( | ||
| has(body.data) ? |
There was a problem hiding this comment.
| has(body.data) ? | |
| has(body.data) ? |
| state.url.trim_right("/") + "/spend/logs/v2?" + | ||
| "page=" + string(state.?page.orValue(1)) + | ||
| "&page_size=" + string(state.batch_size) + | ||
| "&start_date=" + string(state.start_time).substring(0, 10) + "%20" + string(state.start_time).substring(11, 19) + | ||
| "&end_date=" + string(state.end_time).substring(0, 10) + "%20" + string(state.end_time).substring(11, 19) + | ||
| "&sort_order=desc" |
There was a problem hiding this comment.
Is there a reason to not use timestamp formatting and format_query here?
| - version: '0.1.0' | ||
| changes: | ||
| - description: Initial release of the LiteLLM integration with the audit logs data stream. | ||
| - description: Add support of spend_tracking datastream. |
There was a problem hiding this comment.
| - description: Add support of spend_tracking datastream. | |
| - description: Add spend_tracking data stream. |
| "layerType": "data" | ||
| } | ||
| }, | ||
| "title": "Requests by IP Address", |
| "layerType": "data" | ||
| } | ||
| }, | ||
| "title": "Failure by Error Class", |
| "id": "", | ||
| "params": { | ||
| "fontSize": 12, | ||
| "markdown": "#### Overview\n\nThis dashboard provides visibility into LiteLLM spend tracking activity collected from the LiteLLM proxy server. It highlights LiteLLM request trends, spending patterns, and resource utilization across the environment, helping administrators monitor API usage, track costs, and troubleshoot failures.\n\nVisualizations include LiteLLM operations over time, operation type distribution by model and provider, and outcome distribution showing successful versus failed requests. Interactive filters on status and model allow scoping the view to relevant records for focused troubleshooting.\n\nTotal Requests and Spend metrics provide overall API usage and cost overview. Requests by Models breaks down events by model type (Gemini, Claude, GPT-4, etc.). Top Requests by IP Address surfaces the top source IPs generating traffic and their activity volume. Request Volume over Time tracks operation activity per minute, helping identify usage spikes or periods of inactivity. Distribution of Request Types shows the split by model, provider, and call type. Requests by Status shows the split of successful versus failed operations. Failure Breakdown lists the most common failure reasons with their error class and request count.\n\nToken Usage Analysis provides granular visibility into input tokens, output tokens, and total token consumption per request, enabling cost optimization and performance monitoring across all API calls.\n\n**[Integration Page](/app/integrations/detail/lite_llm/overview)**\n", |
There was a problem hiding this comment.
Where is "Distribution of Request Types shows the split by model, provider, and call type"?
| "gen_ai": { | ||
| "request": { | ||
| "model": "openai/gpt-4o" | ||
| } |
There was a problem hiding this comment.
@muskan-agarwal26, gen_ai.* fields are explicitly defined in in other package. Example: claude_cowork, claude_code which sites the fields are not yet in the external ECS schema. Can you please confirm this?
cc: @efd6
| { | ||
| "@timestamp": "2026-07-18T10:12:56.825Z", | ||
| "client": { | ||
| "ip": "10.212.128.28" |
There was a problem hiding this comment.
Can you add geo handling for all ip addresses?
| - set: | ||
| field: user.email | ||
| tag: set_user_email_from_spend_tracking_metadata_user_api_key_user_id | ||
| copy_from: lite_llm.spend_tracking.metadata.user_api_key_user_id | ||
| ignore_empty_value: true |
There was a problem hiding this comment.
Should this be user.name instead of user.email?
| "lite_llm": { | ||
| "spend_tracking": { | ||
| "api_key": "8de15ff28eedf53bd4cf2e65b8e980b9231acc3f894648a9fa3c2dbf0ab1a6d9", | ||
| "completion_tokens": 0, |
There was a problem hiding this comment.
Can completion_tokens, prompt_tokens be used to derive gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens respectively?
There was a problem hiding this comment.
Do we have any test fixture without error?
| "completion_tokens": 0, | ||
| "metadata": { | ||
| "error_information": { | ||
| "error_code": "401", |
There was a problem hiding this comment.
| "metadata": { | ||
| "error_information": { | ||
| "error_code": "401", | ||
| "error_message": "Incorrect API key provided: sk-REPLA*E_ME" |
There was a problem hiding this comment.
Are these API Keys (inside errors) in clear text and not hashed or redacted?
If they are not already hashed or redacted, we should redact/delete them before indexing (maybe through a toggle).
andrewkroh
removed
the
New Integration
| tag: remove_custom_duplicate_fields_from_metadata_guardrail_information | ||
| if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields') | ||
| ignore_missing: true | ||
| - remove: |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/lite_llm/data_stream/spend_tracking/elasticsearch/ingest_pipeline/default.yml:2316
The pipeline never removes event.original, so the full raw record is stored on every document even though preserve_original_event defaults to false; add the standard guarded remove before remove_null_values.
Details
The pipeline renames message to event.original (line 21) and parses it into lite_llm.spend_tracking (line 35), but there is no processor anywhere in the file that removes event.original when the user has not opted in. The only references to preserve_original_event in this pipeline are the two append processors that ADD the tag on error (lines 2381 and 2400) - nothing consumes it.
The data stream manifest advertises the opposite behaviour: the preserve_original_event variable (spend_tracking/manifest.yml lines 130-138 and 199-207) is documented as "Preserves a raw copy of the original event, added to the field event.original" and defaults to false. As written, the field is retained unconditionally, so the setting has no effect.
This is not cosmetic for this data stream. _dev/test/pipeline/test-spend-tracking.log-expected.json shows the retained value: records 4 and 5 carry an event.original of roughly 6 KB each because LiteLLM embeds the entire metadata.model_map_information.model_map_value pricing catalogue in every spend record. For a busy LLM gateway that roughly doubles-to-triples _source size per document while duplicating data already parsed into lite_llm.spend_tracking.*.
Note the sibling audit data stream has the same gap, but it is outside this PR's changed set.
Recommendation:
Add the standard guarded remove immediately before the remove_null_values script (currently line 2345), then regenerate test-spend-tracking.log-expected.json and sample_event.json:
- remove:
field: event.original
tag: remove_event_original
if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event'))
ignore_missing: true
- script:
tag: remove_null_values
lang: painless
description: Remove null or empty values from the document.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
We aren't following the practice of removing event.original, so skipping this
| ), | ||
| }, | ||
| }, | ||
| "want_more": false, |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/lite_llm/data_stream/spend_tracking/agent/stream/cel.yml.hbs:93
The CEL error branch keeps the in-flight page and the already-advanced cursor, so any non-200 response mid-pagination makes the next polling cycle start at page 2+ and permanently skip the newest records; reset page to 1 and only advance the cursor on the last page.
Details
Two pieces of state survive the error branch and cause silent data loss.
-
pageis never reset. The success branch sets"page"to the next page number (line 74-77) and the error branch (lines 80-94) returns onlyeventsandwant_more: false. Becausestate.with()merges,pagekeeps its in-flight value. Concretely: cycle 1 fetches page 1 of 3 successfully (state.page becomes 2), the request for page 2 returns 500, sowant_morebecomes false andpagestays 2. On the next intervalwant_moreis false, so lines 25-32 compute a NEWstart_time/end_timewindow - but the very first request of that new window is issued withpage=2. Page 1 of that window, which undersort_order=descholds the newest records, is never fetched and never will be. -
The cursor advances before pagination completes. Lines 58-72 set
cursor.last_timestampfrombody.data[0].startTimeon EVERY page, including page 1. Withsort_order=desc, page 1's first element is the newest record in the window, so after page 1 succeeds the cursor already points past everything on pages 2..N. If any later page fails, those older records fall outside the next window'sstart_dateand are lost even afterpageis corrected.
Both paths are reachable with any transient 429/5xx from /spend/logs/v2, which the README itself warns about under "Pagination or rate limiting (CEL)".
Recommendation:
Reset the pagination counter on the error branch, and only publish the cursor once the final page has been consumed (the window end is the correct high-water mark):
"cursor": {
?"last_timestamp": int(state.?page.orValue(1)) < int(body.total_pages) ?
state.?cursor.last_timestamp
:
optional.of(state.end_time)
},
and in the non-200 branch:
{
"events": {
"error": {
"code": string(resp.StatusCode),
"id": resp.Status,
"message": "GET:" + state.url.trim_right("/") + "/spend/logs/v2: " + (
size(resp.Body) != 0 ?
string(resp.Body)
:
resp.Status + ' (' + string(resp.StatusCode) + ')'
),
},
},
"want_more": false,
"page": 1,
}
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| field: event.kind | ||
| tag: set_pipeline_error_to_event_kind | ||
| value: pipeline_error | ||
| if: ctx.error?.message != null |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/lite_llm/data_stream/spend_tracking/elasticsearch/ingest_pipeline/default.yml:2392
In the top-level on_failure, set event.kind: pipeline_error is guarded by if: ctx.error?.message != null but runs before the append that creates error.message, so it never fires; drop the guard and put the append first.
Details
The pipeline-level on_failure block (lines 2387-2405) runs in order: set event.kind: pipeline_error guarded by if: ctx.error?.message != null, then append error.message, then append tags: preserve_original_event.
The handler only executes when a processor failed without its own on_failure. In this pipeline that is the json processor at line 35, which has no on_failure at all - a malformed event.original routes straight here. At that moment ctx.error.message has not been written yet (the append that writes it is the next processor), so the guard evaluates false and event.kind is left as event from line 114. The documents that most need to be findable as event.kind: pipeline_error are exactly the ones that do not get tagged.
The third processor's identical guard does work, because it runs after the append - which confirms the ordering is the problem rather than the condition itself.
Recommendation:
Put the error.message append first and set event.kind unconditionally, matching the standard three-step handler:
on_failure:
- append:
tag: append_pipeline_failure_message
field: error.message
value: >-
Processor '{{{ _ingest.on_failure_processor_type }}}'
{{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
{{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}'
- set:
field: event.kind
tag: set_pipeline_error_to_event_kind
value: pipeline_error
- append:
field: tags
tag: append_preserve_original_event
value: preserve_original_event
allow_duplicates: false🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| }, | ||
| { | ||
| "embeddableConfig": { | ||
| "savedObjectId": "9ccf2a3d-765c-42b2-8b15-06fa3d1b1fd2" |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/lite_llm/kibana/dashboard/lite_llm-3e8b7d21-6c4a-4f19-b0d5-2a7e9c1f4b83.json:1312
The saved-search panel carries a stale embeddableConfig.savedObjectId that omits the lite_llm- prefix and duplicates the panelRefName reference; remove it so the panel resolves through references only.
Details
This panel declares the linked saved search twice, in two different forms. embeddableConfig.savedObjectId is 9ccf2a3d-765c-42b2-8b15-06fa3d1b1fd2, while panelRefName (line 1322) points at the references entry whose id is lite_llm-9ccf2a3d-765c-42b2-8b15-06fa3d1b1fd2 (line 1415). The packaged saved object is the prefixed one - kibana/search/lite_llm-9ccf2a3d-765c-42b2-8b15-06fa3d1b1fd2.json sets "id": "lite_llm-9ccf2a3d-765c-42b2-8b15-06fa3d1b1fd2". The unprefixed id in embeddableConfig does not exist after install.
By-reference panels are supposed to carry the id only in references, so that Fleet's reference rewriting stays authoritative; an inline savedObjectId bypasses that and is a leftover from exporting before the package prefix was applied. The sibling audit dashboard (lite_llm-7f3c2a1e-...json) has no savedObjectId anywhere, so this is specific to the new dashboard rather than a package convention.
Recommendation:
Drop the inline id and let the reference do the work:
{
"embeddableConfig": {},
"gridData": {
"h": 13,
"i": "5ebff807-1301-47c6-838d-62d9473640a2",
"w": 48,
"x": 0,
"y": 44
},
"panelIndex": "5ebff807-1301-47c6-838d-62d9473640a2",
"panelRefName": "panel_5ebff807-1301-47c6-838d-62d9473640a2",
"type": "search"
}🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ] | ||
| ], | ||
| "timeRestore": false, | ||
| "title": "Token Usage Analysis" |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/lite_llm/kibana/search/lite_llm-9ccf2a3d-765c-42b2-8b15-06fa3d1b1fd2.json:47
The saved search is titled just "Token Usage Analysis" with an empty description; prefix it with the integration namespace and describe it so it is identifiable in Kibana's global saved-object list.
Details
Packaged Kibana assets are namespaced so users can tell which integration installed them - the dashboard in this same PR follows that and is titled [Logs LiteLLM] Spend Tracking (dashboard JSON line 1327). This saved search is titled Token Usage Analysis with no prefix, and description (line 8) is an empty string, so in Discover's saved-search picker it appears as an anonymous entry with no indication that it belongs to LiteLLM.
Recommendation:
Namespace the title and fill in the description:
"description": "Token usage and spend breakdown for LiteLLM spend tracking records.",
"timeRestore": false,
"title": "[Logs LiteLLM] Token Usage Analysis"🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
|
||
| {{fields "spend_tracking"}} | ||
|
|
||
| ### Example event |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/lite_llm/_dev/build/docs/README.md:147
The new Spend Tracking block adds a second ### Example event heading identical to the Audit one, producing duplicate anchors; fold it into a single H4 under the Spend Tracking section.
Details
This PR adds ### Example event at line 147 while the same H3 already exists at line 161 for the Audit data stream. Two headings with identical text generate colliding anchors in the rendered docs, so in-page links and the sidebar table of contents cannot distinguish them, and a reader scanning the Reference section sees "Example event" twice with no context.
The surrounding structure already scopes content by data stream (### Spend Tracking at line 139, #### Spend Tracking fields at line 143), so the extra H3 adds a level without adding information.
Recommendation:
Replace the ### Example event / #### Spend Tracking pair with one scoped heading (and apply the same shape to the Audit block at lines 161-163):
#### Spend Tracking fields
{{fields "spend_tracking"}}
#### Spend Tracking example event
{{event "spend_tracking"}}🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| 6. Click **Validate Settings**, then **Save Settings**. | ||
|
|
||
| > **Note:** Audit logs are exported to S3 in JSON format at regular intervals. | ||
| > **Note:** logs are exported to S3 in JSON format at regular intervals. |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/lite_llm/_dev/build/docs/README.md:66
The S3 export note now starts mid-sentence with a lowercase "logs" after the word "Audit" was stripped; restore a complete sentence.
Details
The generalisation pass that removed audit-specific wording deleted the subject of this sentence, leaving "> Note: logs are exported to S3 in JSON format at regular intervals." - a sentence beginning with a lowercase word. The same pass also hollowed out the API-key instructions at lines 37 and 41, which now read "permission to access the endpoint" and "permission to read logs from the endpoint" without naming any endpoint, so a reader configuring a LiteLLM virtual key has no idea which scope to grant even though the data-stream table at lines 26-27 names /spend/logs/v2 and /audit.
Recommendation:
Restore the subject in the note, and name the endpoints in the prerequisites:
> **Note:** Spend tracking and audit logs are exported to S3 in JSON format at regular intervals.and at lines 37-41:
To collect data via the API, you need an **API key** with permission to access the `/spend/logs/v2` and `/audit` endpoints.
1. Sign in to your LiteLLM deployment admin panel.
2. Navigate to **API Keys** or **Credentials**.
3. Create or select an API key with permission to read from the `/spend/logs/v2` and `/audit` endpoints.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| if: ctx.lite_llm?.spend_tracking?.request_tags instanceof List | ||
| processor: | ||
| append: | ||
| field: tags |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: low path: packages/lite_llm/data_stream/spend_tracking/elasticsearch/ingest_pipeline/default.yml:2137
request_tags values are appended into ECS tags and the vendor field is then deleted, putting arbitrary vendor strings into the same list the pipeline uses for control flags; keep them under lite_llm.spend_tracking.request_tags instead.
Details
The foreach_request_tags processor (lines 2131-2140) copies every entry of lite_llm.spend_tracking.request_tags into ECS tags, and the source field is then deleted by remove_fields_mapped_to_ecs (line 2333), so tags becomes the only copy.
The committed fixture shows what lands there - test-spend-tracking.log-expected.json lines 342-347 record tags entries such as full and truncated User-Agent strings. ECS defines tags as a list of user-defined keywords for filtering, and this pipeline itself treats it as a control channel: line 2314 branches on ctx.tags.contains('preserve_duplicate_custom_fields') and lines 2381-2386 append preserve_original_event to it. Mixing free-form vendor strings into that list makes the tags facet unusable for filtering and blurs the boundary between agent/pipeline flags and vendor data.
Recommendation:
Delete the foreach_request_tags processor and stop deleting the vendor field, so the values stay addressable under the package namespace:
- remove:
field:
- lite_llm.spend_tracking.agent_id
- lite_llm.spend_tracking.api_base
# ... unchanged entries ...
- lite_llm.spend_tracking.request_id
# drop this line so request_tags is preserved:
# - lite_llm.spend_tracking.request_tags
- lite_llm.spend_tracking.requester_ip_address
tag: remove_fields_mapped_to_ecs
ignore_missing: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| }, | ||
| { | ||
| "id": "logs-*", | ||
| "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/lite_llm/kibana/dashboard/lite_llm-3e8b7d21-6c4a-4f19-b0d5-2a7e9c1f4b83.json:1434
The dashboard's references array lists the same entry twice (kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index) - drop the trailing duplicate.
Details
The reference named kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index (id logs-*, type index-pattern) appears both as the first entry of references (lines 1332-1336) and again as the last entry (lines 1432-1436). Saved-object reference names are keys and are expected to be unique; the duplicate is dead weight that will be re-emitted on every subsequent export and re-review of this file.
Recommendation:
Remove the trailing duplicate so the array ends after the controlGroup_ctrl-call-type:optionsListDataView entry:
{
"id": "logs-*",
"name": "controlGroup_ctrl-call-type:optionsListDataView",
"type": "index-pattern"
}
],
"type": "dashboard",
"typeMigrationVersion": "10.3.0"
}🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - lite_llm.spend_tracking.metadata.error_information.error_code | ||
| tag: remove_fields_mapped_to_ecs | ||
| ignore_missing: true | ||
| - script: |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/lite_llm/data_stream/spend_tracking/elasticsearch/ingest_pipeline/default.yml:2333
The pipeline never removes event.original, so the preserve_original_event toggle has no effect and every document permanently stores a full copy of the raw JSON. Add the standard conditional remove before the null-cleanup script.
Details
message is renamed to event.original at line 21-27, but nothing ever removes event.original again. The standard pattern is to keep it only when the user opted in via the preserve_original_event variable (declared in the data stream manifest, default false) which surfaces as the preserve_original_event tag on the event.
Evidence this is not being handled elsewhere: _dev/test/pipeline/test-spend-tracking.log-expected.json has event.original in all five expected documents even though no pipeline test config sets the tag (there is no test-common-config.yml for this data stream). So event.original is retained unconditionally.
Impact is material for this data stream specifically: spend records carry the metadata.model_map_information pricing catalogue, so a single raw record is several KB. Storing it verbatim on every document roughly doubles index size while silently ignoring the user's configured setting.
Note the sibling audit data stream has the same gap, but it is outside this PR's changed files.
Recommendation:
Insert the conditional remove before the remove_null_values script.
| - script: | |
| - remove: | |
| field: event.original | |
| tag: remove_event_original | |
| if: ctx.tags == null || !(ctx.tags.contains('preserve_original_event')) | |
| ignore_missing: true | |
| - script: |
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
We don't follow this practice.
| } | ||
| }, | ||
| "scale": "ratio", | ||
| "sourceField": "lite_llm.spend_tracking.response_cost" |
There was a problem hiding this comment.
Severity: 🟠 High confidence: high path: packages/lite_llm/kibana/dashboard/lite_llm-3e8b7d21-6c4a-4f19-b0d5-2a7e9c1f4b83.json:347
The dashboard's headline "Total Spend (USD)" metric sums lite_llm.spend_tracking.response_cost, a field nothing ever populates, so the panel always renders empty. It should sum lite_llm.spend_tracking.spend.
Details
response_cost appears nowhere in this PR except a single no-op convert processor at default.yml:2181-2194. It is absent from every fixture: the five records in _dev/test/pipeline/test-spend-tracking.log, the four records in _dev/deploy/tf/files/test-spend-tracking.log, the CEL mock in _dev/deploy/docker/files/config.yml, and sample_event.json.
The cost value the API and the S3 export actually emit is spend (declared as double at fields/fields.yml:589-591). It is present in all five expected documents, e.g. 0.0016485 in test-spend-tracking.log-expected.json:336. Because response_cost is mapped but never written, the metric renders as empty/zero rather than erroring — a silent failure of the dashboard's primary KPI.
Secondary issue in the same panel: "decimals": 0 at line 342. Observed per-request spend is fractions of a cent, so even after pointing at the right field the metric would display 0. The sibling audit dashboard's decimals: 0 is fine because it formats a count, not currency.
Recommendation:
Point the metric at lite_llm.spend_tracking.spend and give the currency sum enough precision to be readable:
"params": {
"emptyAsNull": false,
"format": {
"id": "number",
"params": {
"decimals": 4
}
}
},
"scale": "ratio",
"sourceField": "lite_llm.spend_tracking.spend"🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| : | ||
| [] | ||
| ), | ||
| "cursor": { |
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/lite_llm/data_stream/spend_tracking/agent/stream/cel.yml.hbs:58
The CEL program advances cursor.last_timestamp on page 1 of a descending-sorted paginated fetch, so any error on a later page permanently skips the remaining pages. Only promote the cursor once pagination completes.
Details
The request uses sort_order=desc, so body.data[0].startTime on page 1 is the newest record in the entire window. The cursor is set from it on every iteration, including the first, and CEL persists state after each evaluation.
Concrete failure path, entirely within the code as written:
- Iteration 1 fetches page 1, sets
cursor.last_timestampto the newest record'sstartTime, and setswant_more: true. - Iteration 2 fetches page 2 and gets a non-200. The error branch at lines 79-95 sets
want_more: falseandpage: 1but does not touchcursor, sostate.with()preserves the value committed in step 1. - The next scheduled run sees
want_more == false, sostart_timeis recomputed ascursor.last_timestamp— the newest record already ingested.
Everything on pages 2..N of that window is older than the committed cursor and is never requested again. The records are silently lost, with no error surfaced beyond the one-off HTTP failure. The same happens on an agent restart between pages.
The sibling audit data stream has the same construction, but it is outside this PR's changed files.
Recommendation:
Stage the newest timestamp in a scratch state key while paginating and promote it to cursor.last_timestamp only on the final page:
"pending_timestamp": (has(body.data) && size(body.data) > 0) ?
max(
state.?pending_timestamp.orValue(state.?cursor.last_timestamp.orValue(state.start_time)),
timestamp(body.data[0].startTime).format("2006-01-02 15:04:05")
)
:
state.?pending_timestamp.orValue(state.?cursor.last_timestamp.orValue(state.start_time)),
"cursor": {
?"last_timestamp": int(state.?page.orValue(1)) < int(body.total_pages) ?
state.?cursor.last_timestamp
:
optional.of(state.?pending_timestamp.orValue(state.start_time))
},
"want_more": int(state.?page.orValue(1)) < int(body.total_pages),
The error branch should likewise leave both cursor and pending_timestamp untouched so the window is retried in full.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
Not required.
| - name: completion_start_time | ||
| type: date | ||
| description: Timestamp when completion generation or the first streamed token began. | ||
| - name: messages |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/lite_llm/data_stream/spend_tracking/fields/fields.yml:43
messages, response, and metadata.proxy_server_request are mapped as keyword even though their own descriptions say LiteLLM emits object or list forms; an object value makes Elasticsearch reject the whole document. Map them as flattened.
Details
Three fields are declared keyword while being documented as variable-shape:
lite_llm.spend_tracking.messages(line 43-45): "Serialized request messages sent to the model; LiteLLM may also emit list or object forms."lite_llm.spend_tracking.response(line 577-579): "Model response retained in the spend log; LiteLLM may also emit list or object forms."lite_llm.spend_tracking.metadata.proxy_server_request(line 478-480): "Serialized proxy request information retained with the spend log."
No processor in elasticsearch/ingest_pipeline/default.yml stringifies or removes any of them. When LiteLLM populates them as JSON objects or arrays of objects, Elasticsearch fails to parse the value into a keyword field and rejects the entire document — a silent per-document data loss, not a partial one.
The fixtures do not exercise this: the pipeline test records carry proxy_server_request: null and no messages/response at all, and the remove_null_values script strips nulls and empty maps, so the empty-default case happens to pass. The same file already uses flattened for other variable-shape blobs (spend_logs_metadata at line 484-486, vector_store_request_metadata at line 562-564), so the mapping strategy is inconsistent here rather than deliberate.
Recommendation:
Use flattened for the three variable-shape fields, matching how spend_logs_metadata is already handled:
- name: messages
type: flattened
description: Request messages sent to the model; LiteLLM emits list or object forms. - name: response
type: flattened
description: Model response retained in the spend log; LiteLLM emits list or object forms. - name: proxy_server_request
type: flattened
description: 'Inferred: Proxy request information retained with the spend log.'If keeping them searchable as plain strings is preferred instead, add a json_processor/script step in the pipeline that serialises them before indexing.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
🟢 Reviewed the latest commits aecf3f8 — nothing new beyond already posted comments. Review summaryIssues found across earlier commits ac25869 — 3 high, 1 medium
Issues found across earlier commits 375ef86 — 1 low
Issues found across earlier commits 5b48707 — 2 high, 2 medium, 4 low
Issues found across earlier commits 75e8e05 — 1 high, 2 low
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
|
/test |
|
/test |
|
✅ All changelog entries have the correct PR link. |
💔 Build Failed
Failed CI StepsHistory
|
| tags = { | ||
| environment = var.ENVIRONMENT | ||
| repo = var.REPO | ||
| branch = var.BRANCH | ||
| build = var.BUILD_ID | ||
| created_date = var.CREATED_DATE | ||
| } | ||
| } |
There was a problem hiding this comment.
Add the following labels in order to run the CI successfully:
| tags = { | |
| environment = var.ENVIRONMENT | |
| repo = var.REPO | |
| branch = var.BRANCH | |
| build = var.BUILD_ID | |
| created_date = var.CREATED_DATE | |
| } | |
| } | |
| tags = { | |
| environment = var.ENVIRONMENT | |
| repo = var.REPO | |
| branch = var.BRANCH | |
| build = var.BUILD_ID | |
| created_date = var.CREATED_DATE | |
| division = "engineering" | |
| org = "obs" | |
| team = "security-service-integrations" # owner.github in manifest.yml | |
| project = "integrations-lite_llm-package" # name in manifest.yml | |
| } | |
| } |
FYI @teresaromero
In any case, these labels should be aligned with the ones that are being in review in these other Pull Requests for other packages:
- [ssi] Add Terraform resource labels to SSI packages (batch 1) #20704
- [ssi] Add Terraform resource labels to SSI packages (batch 3) #20706
- [cloudflare_logpush] Add Terraform resource labels #20705
So depending on the outcome of those, these labels would need to be updated too @muskan-agarwal26
If possible , I would wait until those PRs are merged to be sure we follow the same patterns.
There was a problem hiding this comment.
Keeping this PR draft until those PRs are merged.
CC: @efd6
7 participants
Proposed commit message
Checklist
changelog.ymlfile.How to test this PR locally
To test the LiteLLM package:
Related issues
Screenshots