Skip to content

Structured logs for each read request #967

Description

@TomAugspurger

Kvikio currently emits some logs related to reads. I'm working on optimizing cudf-polars IO, and these logs had a couple of shortcomings:

  1. The logs aren't machine readable. The lines are formatted like FileHandle::pread(buf=..., size=..., ...). It didn't have the path being read or the start/end times.
  2. Logical reads are logged, rather than physical reads (I think). kvikio automatically breaks large multi-MB reads into smaller chunks. Both are interesting, but I was especially interested in the physical reads.

Have we considered emitting structured logs (in JSON format, say, but this might be configurable)? https://github.com/rapidsai/kvikio/compare/main...TomAugspurger:kvikio:tom/read-log-traces?expand=1 has a POC for that. The lines include

  • source (path / URL)
  • start, end (int counters)
  • offset, size
  • threadId
  • bytesRead (the bytes actually read, which might differ from size)
  • backend (local or remote)
  • status
  • isDeviceBuffer (maybe this should be a "destination" enum?)
  • requestId: correlate with the logical request that induced this physical read (a logical read might be split into multiple physical reads)

It also includes a (private) tool for converting the structured logs to chrome://tracing format, which can be loaded into tools like https://ui.perfetto.dev/. https://gist.github.com/TomAugspurger/135c38917a3a4cbe0cfc6eb80b503bff is an example file, shown here:

Image

If this is helpful, I can clean things up and submit a pR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions