Skip to content

SCHED-1943: Fix OpenTelemetry Collector JSON parsing errors for slurm_scripts#2690

Merged
Uburro merged 1 commit into
mainfrom
SCHED-1943/0
Jul 3, 2026
Merged

SCHED-1943: Fix OpenTelemetry Collector JSON parsing errors for slurm_scripts#2690
Uburro merged 1 commit into
mainfrom
SCHED-1943/0

Conversation

@Uburro

@Uburro Uburro commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

The jail-logs otel collector spams error-level logs with huge stacktraces: invalid character '\u0000' looking for beginning of value. Check output files on the shared jail FS are truncated and rewritten in place by check_runner.py, so the collector (tailing from another node) can read partially flushed lines with zero-filled tails, and the json_parser fails on them. Logs are still delivered — it's pure noise. Reproduces on some clusters (m42-h200, ami-labs) depending on storage/timing.

Solution

Hardened the parse_json_body operator in the jail-logs collector config:

  • extended the if guard to skip lines containing NUL bytes;
  • added on_error: send_quiet so any other parse failure sends the entry unparsed and logs at debug instead of error.

The parser is kept because it feeds slurm_run_status / slurm_run_resolution log labels (added in #2125).

Testing

  • make helmtest
  • helm template — verified the rendered operator config is valid and contains the new condition and on_error.
  • on the live cluster

Release Notes

Fix: jail-logs otel collector no longer emits error-level noise when reading partially flushed health check output files containing NUL bytes.

@Uburro Uburro added fix helm Functional changes in Helm charts labels Jul 3, 2026
@Uburro
Uburro marked this pull request as ready for review July 3, 2026 07:26
@Uburro
Uburro merged commit 95a9536 into main Jul 3, 2026
22 of 24 checks passed
@Uburro
Uburro deleted the SCHED-1943/0 branch July 3, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix helm Functional changes in Helm charts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants