Add Features/ClassLabel schema views

AGENTS.md

@@ -43,6 +43,14 @@ that lazily converts observations to Julia types).
   PIL images into Julia types; `jl2py` is the write-path dual. The `"julia"`
   format is numpy-backed, so numeric array columns decode to real N-D Julia arrays
   and image columns decode to raw numeric arrays (not `Colorant` colorviews).
+- `src/features.jl` — `Py`-backed views over a dataset's schema: `Features` (an
+  `AbstractDict` returned by `ds.features`), and the `ClassLabel`/`Value` leaves it
+  wraps, each forwarding attribute/method access to Python (`cl.names`,
+  `cl.int2str`, `v.dtype`). Handled at the access site (a `:features` branch in
+  `Dataset`'s `getproperty`), never in the `py2jl` batch hot path. Also the Julian
+  label-decoding helpers `class_names`/`int2str`/`str2int` (`(ds, col, …)`), and
+  `jl2py` overloads so a Julia-built schema round-trips into a `features=` argument.
+  Everything here is public but unexported; the Pythonic idioms are primary.
 - `src/serialization.jl` — `Serialization.serialize`/`deserialize` for `Dataset`,
   so it can cross a process boundary (process-parallel data loaders). Never
   serializes the wrapped `Py` directly; instead uses `datasets`' own pickle

CHANGELOG.md

@@ -37,6 +37,18 @@ Julia values instead of raw Python objects. See **Breaking** below before upgrad
   `BoundsError`, instead of `AssertionError` — update any code catching `AssertionError`.
 
 ### Added
+- Julia views over a dataset's schema. `ds.features` now returns a `Features` view (an
+  `AbstractDict{String, Any}`) instead of a raw `Py` mapping: indexing a column yields a wrapped
+  `ClassLabel`/`Value` leaf (other feature types stay raw `Py`), each forwarding attribute and
+  method access to Python (`cl.names`, `cl.num_classes`, `cl.int2str(i)`, `cl.str2int(s)`,
+  `v.dtype`). The views can be built from Julia (`ClassLabel(names=[…])`, `Value("int64")`,
+  `Features(Dict(…))`) and handed back to Python via `jl2py` (e.g. a `features=` schema
+  argument). Public but unexported Julian conveniences look up a column's `ClassLabel` in one
+  call: `class_names(ds, col)`, `int2str(ds, col, i)`, `str2int(ds, col, s)` (label ids are
+  0-based data, passed through with no index offset), plus `features(ds)` as the function form
+  of `ds.features`. `Features`/`ClassLabel`/`Value` and these helpers are all public but not
+  exported — the Pythonic idioms (`ds.features`, method chaining, `datasets.ClassLabel(…)`) are
+  the primary interface.
 - `Serialization` support for `Dataset`: `Serialization.serialize`/`deserialize` now work,
   so a `Dataset` can cross a process boundary — the prerequisite for process-parallel data
   loaders (e.g. a `MLUtils.DataLoader(ds; num_workers=N)` that spreads `getobs` over worker

docs/make.jl

@@ -1,8 +1,13 @@
 using HuggingFaceDatasets
 using Documenter
 
+# Bring the public-but-unexported schema conveniences into doctest scope, and disable
+# `datasets`' tqdm progress bars so doctests that trigger them (e.g. `class_encode_column`)
+# don't emit a progress line into the captured output.
 DocMeta.setdocmeta!(HuggingFaceDatasets, :DocTestSetup,
-    :(using HuggingFaceDatasets, PythonCall); recursive=true)
+    :(using HuggingFaceDatasets, PythonCall;
+      using HuggingFaceDatasets: features, class_names, int2str, str2int, Value;
+      HuggingFaceDatasets.datasets.disable_progress_bars()); recursive=true)
 
 makedocs(;
     modules=[HuggingFaceDatasets],

docs/src/api.md

@@ -18,6 +18,18 @@ IterableDatasetDict
 Column
 ```
 
+## Schema (features)
+
+```@docs
+features
+Features
+ClassLabel
+Value
+class_names
+int2str
+str2int
+```
+
 ## Loading
 
 ```@docs

docs/src/guide.md

@@ -2,6 +2,8 @@
 CurrentModule = HuggingFaceDatasets
 DocTestSetup = quote
     using HuggingFaceDatasets, PythonCall
+    using HuggingFaceDatasets: features, class_names, int2str, str2int, Value
+    HuggingFaceDatasets.datasets.disable_progress_bars()
 end
 ```
 
@@ -128,6 +130,88 @@ Keyword arguments are forwarded as Python keyword arguments, so calls like
     [`datasets` documentation](https://huggingface.co/docs/datasets) for the exact
     meaning of each method's arguments.
 
+## Inspecting the schema: features and labels
+
+Every dataset carries a **schema** describing each column's type. `ds.features` returns it as
+a [`Features`](@ref) view — an `AbstractDict` from column name to feature type — so you can
+inspect dtypes and, crucially, decode integer class labels. Indexing a column yields its
+feature: a [`Value`](@ref) for a scalar column (carrying an Arrow `dtype`), a
+[`ClassLabel`](@ref) for an encoded label, and so on.
+
+```jldoctest guide
+julia> ds = Dataset((; text=["good", "bad", "good"], label=["pos", "neg", "pos"]));
+
+julia> ds.features
+{'text': Value('string'), 'label': Value('string')}
+
+julia> ds.features["text"]
+Value('string')
+```
+
+The most useful leaf is [`ClassLabel`](@ref), which maps integer class ids to names. Turn a
+string column into one with the forwarded `class_encode_column`, then read the mapping straight
+off the feature with Pythonic method chaining (`.names`, `.int2str`, `.str2int`):
+
+```jldoctest guide
+julia> ds = Dataset((; label=["pos", "neg", "pos"]));
+
+julia> ds = ds.class_encode_column("label");   # string column -> ClassLabel (names sorted)
+
+julia> cl = ds.features["label"]
+ClassLabel(names=['neg', 'pos'])
+
+julia> cl.names
+2-element Vector{String}:
+ "neg"
+ "pos"
+
+julia> cl.int2str(0)            # 0-based class id -> name (no index offset)
+"neg"
+
+julia> cl.str2int("pos")
+1
+
+julia> ds["label"]              # the stored ids are 0-based data, not 1-based indices
+3-element HuggingFaceDatasets.Column{Int64}:
+ 1
+ 0
+ 1
+```
+
+!!! note "Class ids are 0-based data"
+    A `ClassLabel` column stores **0-based class ids** (`ds["label"]` above is `[1, 0, 1]`),
+    not 1-based Julia indices. `int2str`/`str2int` pass ids through to Python unchanged; only
+    the wrapper's `getindex`/iteration interface is 1-based. Decoding a whole column is
+    therefore `cl.names[ds["label"] .+ 1]`, where the `+1` bridges a 0-based id to a 1-based
+    Julia position.
+
+For the common "from a dataset and column name" case there are also public (unexported) Julian
+shortcuts — [`class_names`](@ref), [`int2str`](@ref), and [`str2int`](@ref) — that look up the
+column's `ClassLabel` for you (and error clearly if it isn't one):
+
+```jldoctest guide
+julia> ds = Dataset((; label=["pos", "neg", "pos"]));
+
+julia> ds = ds.class_encode_column("label");
+
+julia> class_names(ds, "label")
+2-element Vector{String}:
+ "neg"
+ "pos"
+
+julia> int2str(ds, "label", [0, 1, 1])   # decode a batch of ids in one call
+3-element Vector{String}:
+ "neg"
+ "pos"
+ "pos"
+```
+
+Reach these as `HuggingFaceDatasets.class_names` etc., or bring them into scope with
+`using HuggingFaceDatasets: class_names, int2str, str2int`. To **construct** a schema from
+Julia — e.g. to pass as a `features=` argument — build the wrapper types (also public but
+unexported: `HuggingFaceDatasets.ClassLabel(names=["neg", "pos"])`,
+`HuggingFaceDatasets.Value("int64")`) and hand them back to Python with [`jl2py`](@ref).
+
 ## The `"julia"` format and transforms
 
 Datasets are returned in the `"julia"` format by default, so indexing yields native Julia

src/HuggingFaceDatasets.jl

@@ -45,6 +45,10 @@ export py2jl,
     jl2numpy,
     numpy2jl
 
+include("features.jl")
+export Features, ClassLabel
+@compat public Value, features, class_names, int2str, str2int
+
 include("load_dataset.jl")
 export load_dataset
 
@@ -53,12 +57,8 @@ export concatenate_datasets,
     interleave_datasets,
     load_from_disk
 
-# Recipe-based `Serialization` for `Dataset` (ships an on-disk path, never a `Py`), so a
-# `Dataset` can be sent to `Distributed` worker processes — the basis for process-parallel
-# data loaders. Included after `toplevel.jl` as it uses `load_from_disk`.
 include("serialization.jl")
 
-# `public` is a Julia 1.11+ keyword; `@compat` makes it a no-op on the supported 1.10.
 @compat public from_csv, from_json, from_parquet
 
 function __init__()

src/dataset.jl

@@ -224,6 +224,9 @@ function Base.getproperty(ds::Dataset, s::Symbol)
     if s in fieldnames(Dataset)
         return getfield(ds, s)
     end
+    # Return the schema as a Julia `Features` view (see `features.jl`) instead of the raw
+    # Python mapping, so `ds.features["label"]` yields a wrapped `ClassLabel`/`Value` leaf.
+    s === :features && return features(ds)
     # Route the format and `map`/`filter` methods to this package's own versions (see
     # `_method_override`); every other name forwards to the wrapped Python object.
     override = _method_override(ds, s)