Add Features/ClassLabel schema views - #65
Merged
Merged
Conversation
Give a dataset's schema a Julian home. `ds.features` now returns a `Features`
view (an `AbstractDict{String, Any}`) instead of a raw `Py` mapping: indexing a
column yields a wrapped `ClassLabel`/`Value` leaf (other feature types stay raw
`Py`), each forwarding attribute/method access to Python (`cl.names`,
`cl.num_classes`, `cl.int2str(i)`, `cl.str2int(s)`, `v.dtype`).
The views are `Py`-backed and can be built from Julia (`ClassLabel(names=[…])`,
`Value("int64")`, `Features(Dict(…))`) and handed back to Python via `jl2py`
(e.g. a `features=` schema argument). Public-but-unexported Julian conveniences
decode labels in one call from a dataset + column: `class_names(ds, col)`,
`int2str(ds, col, i)`, `str2int(ds, col, s)`. Class ids are 0-based *data* and
pass through with no index offset; the 0→1 bridge only appears when indexing the
1-based Julia `names` vector (`names[ids .+ 1]`), as documented.
Handled at the access site (a `:features` branch in `Dataset`'s `getproperty`),
so the `py2jl` batch hot path is untouched — `Features` subclasses `dict` and
would otherwise tax every batch observation. `Features` caches its column names
at construction so `keys`/`length`/iteration never call Python (safe from the
REPL's async completion, mirroring `DatasetDict`).
`Features`/`ClassLabel` are exported; `Value` and the decode helpers are public
but unexported — the Pythonic idioms (`ds.features`, method chaining,
`datasets.ClassLabel(…)`) are the primary interface.
Tests: test/features.jl (47 assertions, all local/CI-safe). Docs: API entries,
a "Schema: features and labels" guide section, CHANGELOG + AGENTS.md notes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-review of the schema views found two gaps against the `AbstractDict` contract the `Features` docstring advertises: - `get(f, key, default)` threw `MethodError` — Julia has no generic `Base.get(::AbstractDict, …)` fallback. Add it (Python-free via cached `haskey`). - Indexing a missing column surfaced Python's `KeyError` as a `PyException`; guard `getindex` with `haskey` so it raises a Julia `KeyError` instead. Tests extended to cover `get` (present/absent) and the `KeyError` path (51 pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Features/ClassLabel schema views (review item 1)Features/ClassLabel schema views
CarloLucibello
added a commit
that referenced
this pull request
Jul 6, 2026
* Add process-parallel DataLoader (num_workers) support for Dataset `MLUtils.DataLoader(ds::Dataset; num_workers=N)` (MLUtils >= 0.4.10) spreads `getobs` over N worker processes, each with its own CPython interpreter and GIL, so reads scale where thread parallelism cannot (a shared GIL serializes them). The loader serializes its data container from a background feeder task that does not hold the GIL; a `Dataset` serializes by calling `pickle`/PythonCall, which segfaults off the main task. Fix it with a `DistributedDataset` wrapper that precomputes the pickle bytes on the calling (GIL-holding) task and ships those bytes — pure Julia, safe from any thread — reconstructing a live `Dataset` on the worker. The `DataLoader(::Dataset)` method installs the wrapper automatically when `num_workers > 0`; serial/`parallel` loaders are untouched. `Dataset` itself is unchanged. An `@info` notes when an in-memory dataset is first materialized to a temporary Arrow file so it can pickle by reference. Tests: a DistributedDataset round-trip plus an end-to-end num_workers DataLoader over an in-memory dataset; MLUtils is added as a test dependency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: dedicated Data loaders section in the guide Split the combined MLUtils section into a focused integration section and a standalone "Data loaders" section covering feeding a Dataset to DataLoader, on-the-fly vs. materialized loading, and process-parallel `num_workers` loading (with DistributedDataset). Document DistributedDataset in the API reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Rely on MLUtils for num_workers serialization; drop DistributedDataset The GIL-safe, feeder-thread-proof serialization needed for `DataLoader(ds; num_workers=N)` now lives in MLUtils (>= 0.4.11): it serializes the data container on the main task and loads this package on the workers. That makes the interim `DistributedDataset` wrapper and the `DataLoader(::Dataset)` hook redundant here, and additionally makes `mapobs`/`ObsView`-wrapped datasets work under `num_workers`. - Remove src/dataloader.jl (DistributedDataset + the DataLoader hook) - Drop the export and the now-unused `numobs` import - Bump MLUtils compat to 0.4.11 (main + test) - Update guide, API docs, changelog, and tests accordingly The `Dataset` Serialization methods (pickle-by-reference) are unchanged -- they are exactly what MLUtils' main-task serialization invokes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix docs @ref, trim Data Loaders guide, num_workers MNIST example - Fix the `[Data Loaders](@ref)` cross-reference (header case) that failed the Documentation CI job. - Trim the guide's Data Loaders section to defer loader mechanics to the MLUtils guide; keep only the `Dataset`-specific notes. - Move the Flux MNIST example into examples/flux_mnist/ with its own Project.toml, and switch it to process-parallel loading via `num_workers` (package-free transform; one-hot/loss on the main process). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Pin JULIA_PYTHONCALL_EXE so num_workers workers skip CondaPkg re-resolve DataLoader(...; num_workers=N) spawns Distributed workers that each `using PythonCall`. Without help, every worker re-resolves the same CondaPkg env in lockstep: they serialize on CondaPkg's file lock (noisy "Waiting for lock to be freed") and redo work the parent already did. Exporting the already-resolved interpreter in `__init__` lets the workers (which inherit ENV) use it directly and skip CondaPkg entirely. Mirrors the CI-only hack inside PythonCall; guarded on `CTX.which === :CondaPkg` and with `get!` so a user-set interpreter is left untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * examples: flux_mnist data-loading benchmark, rename, doc links - Rename examples/flux_mnist/main.jl -> flux_mnist.jl and add a warm-up epoch, so the reported per-config timings exclude Julia's JIT compilation. - Add PyTorch counterparts (a 1:1 port and an idiomatic HF version) and a README with the Julia vs PyTorch timing tables and takeaways (Apple M1 Pro, CPU, 4 epochs). Headline: materializing into memory is the big win; num_workers/process parallelism does not pay off for this toy MLP. - Point the example links in docs/src/guide.md at the new path. - gitignore __pycache__; add DataStructures compat bound. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * CHANGELOG: split Unreleased into 0.4.0, 0.4.1, and Unreleased 0.4.1 gets the Features/ClassLabel schema views (#65) and MLCore ≥ 1.1 getobs(::Py) change (#66); 0.4.0 keeps the breaking-release content; the num_workers DataLoader entry stays under Unreleased. Compare links updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: expand Data Loaders guide with runnable examples Split the Data Loaders section into Iterating batches / Materializing / num_workers, with two verified jldoctests (batch structure and the mapobs -> (input, target) pattern) plus julia-repl Hub and num_workers examples. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Gives a dataset's schema a Julian home.
ds.featuresnow returns aFeaturesview — anAbstractDict{String, Any}— instead of a rawPymapping. Indexing a column yields a wrapped leaf:ClassLabel/Valueleaves arePy-backed and forward attribute/method access to Python (cl.names,cl.num_classes,cl.int2str,cl.str2int,v.dtype). Other feature types (nestedFeatures,Sequence,Image, …) stay rawPy.Highlights
:featuresbranch inDataset'sgetproperty— the genericpy2jlbatch hot path is untouched (Featuressubclassesdict, so detecting it there would tax every batch observation).Featurescaches its column names at construction, sokeys/length/iteration never call Python (mirrors theDatasetDictfix).ClassLabel(names=[…]),Value("int64"),Features(Dict(…))) and hand it back viajl2pyas afeatures=argument.class_names(ds, col),int2str(ds, col, i),str2int(ds, col, s). Class ids are 0-based data and pass through with no offset; the 0→1 bridge only appears when indexing the 1-based Julianamesvector (names[ids .+ 1]), as documented.API surface
FeaturesandClassLabelare exported;Value,features,class_names,int2str,str2intare public but unexported. The Pythonic idioms (ds.features, method chaining,datasets.ClassLabel(…)via the exporteddatasetshandle) are the primary interface.Tests & docs
test/features.jl— 47 assertions, all local (no network), wired intoruntests.jl. Full suite passes.🤖 Generated with Claude Code