Skip to content

Add Features/ClassLabel schema views - #65

Merged
CarloLucibello merged 2 commits into
mainfrom
cl/features-classlabel-view
Jul 3, 2026
Merged

Add Features/ClassLabel schema views#65
CarloLucibello merged 2 commits into
mainfrom
cl/features-classlabel-view

Conversation

@CarloLucibello

@CarloLucibello CarloLucibello commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

What

Gives a dataset's schema a Julian home. ds.features now returns a Features view — an AbstractDict{String, Any} — instead of a raw Py mapping. Indexing a column yields a wrapped leaf:

julia> ds = ds.class_encode_column("label");

julia> cl = ds.features["label"]
ClassLabel(names=['neg', 'pos'])

julia> cl.names            # attribute/method access forwards to Python
2-element Vector{String}:
 "neg"
 "pos"

julia> cl.int2str(1)       # 0-based class id -> name
"pos"

ClassLabel/Value leaves are Py-backed and forward attribute/method access to Python (cl.names, cl.num_classes, cl.int2str, cl.str2int, v.dtype). Other feature types (nested Features, Sequence, Image, …) stay raw Py.

Highlights

  • Access-site handling. A :features branch in Dataset's getproperty — the generic py2jl batch hot path is untouched (Features subclasses dict, so detecting it there would tax every batch observation).
  • Tab-completion safe. Features caches its column names at construction, so keys/length/iteration never call Python (mirrors the DatasetDict fix).
  • Round-trips into Python. Build a schema from Julia (ClassLabel(names=[…]), Value("int64"), Features(Dict(…))) and hand it back via jl2py as a features= argument.
  • Label decoding in one call (public, unexported): class_names(ds, col), int2str(ds, col, i), str2int(ds, col, s). Class ids are 0-based data and pass through with no offset; the 0→1 bridge only appears when indexing the 1-based Julia names vector (names[ids .+ 1]), as documented.

API surface

Features and ClassLabel are exported; Value, features, class_names, int2str, str2int are public but unexported. The Pythonic idioms (ds.features, method chaining, datasets.ClassLabel(…) via the exported datasets handle) are the primary interface.

Tests & docs

  • test/features.jl — 47 assertions, all local (no network), wired into runtests.jl. Full suite passes.
  • API reference entries, a "Schema: features and labels" guide section, CHANGELOG + AGENTS.md notes. All doctests and the full docs build pass.

🤖 Generated with Claude Code

CarloLucibello and others added 2 commits July 3, 2026 10:00
Give a dataset's schema a Julian home. `ds.features` now returns a `Features`
view (an `AbstractDict{String, Any}`) instead of a raw `Py` mapping: indexing a
column yields a wrapped `ClassLabel`/`Value` leaf (other feature types stay raw
`Py`), each forwarding attribute/method access to Python (`cl.names`,
`cl.num_classes`, `cl.int2str(i)`, `cl.str2int(s)`, `v.dtype`).

The views are `Py`-backed and can be built from Julia (`ClassLabel(names=[…])`,
`Value("int64")`, `Features(Dict(…))`) and handed back to Python via `jl2py`
(e.g. a `features=` schema argument). Public-but-unexported Julian conveniences
decode labels in one call from a dataset + column: `class_names(ds, col)`,
`int2str(ds, col, i)`, `str2int(ds, col, s)`. Class ids are 0-based *data* and
pass through with no index offset; the 0→1 bridge only appears when indexing the
1-based Julia `names` vector (`names[ids .+ 1]`), as documented.

Handled at the access site (a `:features` branch in `Dataset`'s `getproperty`),
so the `py2jl` batch hot path is untouched — `Features` subclasses `dict` and
would otherwise tax every batch observation. `Features` caches its column names
at construction so `keys`/`length`/iteration never call Python (safe from the
REPL's async completion, mirroring `DatasetDict`).

`Features`/`ClassLabel` are exported; `Value` and the decode helpers are public
but unexported — the Pythonic idioms (`ds.features`, method chaining,
`datasets.ClassLabel(…)`) are the primary interface.

Tests: test/features.jl (47 assertions, all local/CI-safe). Docs: API entries,
a "Schema: features and labels" guide section, CHANGELOG + AGENTS.md notes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-review of the schema views found two gaps against the `AbstractDict`
contract the `Features` docstring advertises:

- `get(f, key, default)` threw `MethodError` — Julia has no generic
  `Base.get(::AbstractDict, …)` fallback. Add it (Python-free via cached `haskey`).
- Indexing a missing column surfaced Python's `KeyError` as a `PyException`;
  guard `getindex` with `haskey` so it raises a Julia `KeyError` instead.

Tests extended to cover `get` (present/absent) and the `KeyError` path (51 pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CarloLucibello CarloLucibello changed the title Add Features/ClassLabel schema views (review item 1) Add Features/ClassLabel schema views Jul 3, 2026
@CarloLucibello
CarloLucibello merged commit a0a3215 into main Jul 3, 2026
9 of 10 checks passed
CarloLucibello added a commit that referenced this pull request Jul 6, 2026
0.4.1 gets the Features/ClassLabel schema views (#65) and MLCore ≥ 1.1
getobs(::Py) change (#66); 0.4.0 keeps the breaking-release content; the
num_workers DataLoader entry stays under Unreleased. Compare links updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CarloLucibello added a commit that referenced this pull request Jul 6, 2026
* Add process-parallel DataLoader (num_workers) support for Dataset

`MLUtils.DataLoader(ds::Dataset; num_workers=N)` (MLUtils >= 0.4.10) spreads `getobs`
over N worker processes, each with its own CPython interpreter and GIL, so reads scale
where thread parallelism cannot (a shared GIL serializes them).

The loader serializes its data container from a background feeder task that does not hold
the GIL; a `Dataset` serializes by calling `pickle`/PythonCall, which segfaults off the
main task. Fix it with a `DistributedDataset` wrapper that precomputes the pickle bytes on
the calling (GIL-holding) task and ships those bytes — pure Julia, safe from any thread —
reconstructing a live `Dataset` on the worker. The `DataLoader(::Dataset)` method installs
the wrapper automatically when `num_workers > 0`; serial/`parallel` loaders are untouched.
`Dataset` itself is unchanged. An `@info` notes when an in-memory dataset is first
materialized to a temporary Arrow file so it can pickle by reference.

Tests: a DistributedDataset round-trip plus an end-to-end num_workers DataLoader over an
in-memory dataset; MLUtils is added as a test dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: dedicated Data loaders section in the guide

Split the combined MLUtils section into a focused integration section and a standalone
"Data loaders" section covering feeding a Dataset to DataLoader, on-the-fly vs. materialized
loading, and process-parallel `num_workers` loading (with DistributedDataset). Document
DistributedDataset in the API reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Rely on MLUtils for num_workers serialization; drop DistributedDataset

The GIL-safe, feeder-thread-proof serialization needed for
`DataLoader(ds; num_workers=N)` now lives in MLUtils (>= 0.4.11): it
serializes the data container on the main task and loads this package on
the workers. That makes the interim `DistributedDataset` wrapper and the
`DataLoader(::Dataset)` hook redundant here, and additionally makes
`mapobs`/`ObsView`-wrapped datasets work under `num_workers`.

- Remove src/dataloader.jl (DistributedDataset + the DataLoader hook)
- Drop the export and the now-unused `numobs` import
- Bump MLUtils compat to 0.4.11 (main + test)
- Update guide, API docs, changelog, and tests accordingly

The `Dataset` Serialization methods (pickle-by-reference) are unchanged --
they are exactly what MLUtils' main-task serialization invokes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Fix docs @ref, trim Data Loaders guide, num_workers MNIST example

- Fix the `[Data Loaders](@ref)` cross-reference (header case) that failed
  the Documentation CI job.
- Trim the guide's Data Loaders section to defer loader mechanics to the
  MLUtils guide; keep only the `Dataset`-specific notes.
- Move the Flux MNIST example into examples/flux_mnist/ with its own
  Project.toml, and switch it to process-parallel loading via `num_workers`
  (package-free transform; one-hot/loss on the main process).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Pin JULIA_PYTHONCALL_EXE so num_workers workers skip CondaPkg re-resolve

DataLoader(...; num_workers=N) spawns Distributed workers that each `using
PythonCall`. Without help, every worker re-resolves the same CondaPkg env in
lockstep: they serialize on CondaPkg's file lock (noisy "Waiting for lock to be
freed") and redo work the parent already did.

Exporting the already-resolved interpreter in `__init__` lets the workers (which
inherit ENV) use it directly and skip CondaPkg entirely. Mirrors the CI-only
hack inside PythonCall; guarded on `CTX.which === :CondaPkg` and with `get!` so a
user-set interpreter is left untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* examples: flux_mnist data-loading benchmark, rename, doc links

- Rename examples/flux_mnist/main.jl -> flux_mnist.jl and add a warm-up epoch, so
  the reported per-config timings exclude Julia's JIT compilation.
- Add PyTorch counterparts (a 1:1 port and an idiomatic HF version) and a README
  with the Julia vs PyTorch timing tables and takeaways (Apple M1 Pro, CPU, 4
  epochs). Headline: materializing into memory is the big win; num_workers/process
  parallelism does not pay off for this toy MLP.
- Point the example links in docs/src/guide.md at the new path.
- gitignore __pycache__; add DataStructures compat bound.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* CHANGELOG: split Unreleased into 0.4.0, 0.4.1, and Unreleased

0.4.1 gets the Features/ClassLabel schema views (#65) and MLCore ≥ 1.1
getobs(::Py) change (#66); 0.4.0 keeps the breaking-release content; the
num_workers DataLoader entry stays under Unreleased. Compare links updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: expand Data Loaders guide with runnable examples

Split the Data Loaders section into Iterating batches / Materializing /
num_workers, with two verified jldoctests (batch structure and the
mapobs -> (input, target) pattern) plus julia-repl Hub and num_workers
examples.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant