feat: implement scale-offset and data type casting via codecs

[d-v-b]

this PR defines the scale-offset transformation as a pair of codecs: a ScaleOffset codec, which implements the Zarr V3 scale-offset codec , and the CastValueRustV1 codec from cast-value.py, which implements the Zarr V3 cast_value codec. This means float data can be decoded as float, but stored as ints, in a cf- and xarray-independent manner.

Right now it's exposed via a boolean parameter to convert_s2_optimized, and command-line flag --experimental-scale-offset-codec.

caveat: downstream consumers will need the ScaleOffset and CastValue codecs available in order to decode the data. We should work on ways to make this straightforward.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 89.65517% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/eopf_geozarr/conversion/geozarr.py	80.00%	2 Missing ⚠️
src/eopf_geozarr/conversion/utils.py	83.33%	2 Missing ⚠️
src/eopf_geozarr/cli.py	0.00%	1 Missing ⚠️
src/eopf_geozarr/s2_optimization/s2_multiscale.py	96.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[d-v-b]

I did not make a stand-alone library for this codec, because it is so simple.

[emmanuelmathot]

ok but will it live in zarr-python or will users have to import eopf-geozarr to use the scale_offset?

[d-v-b]

in the current state users have to import eopf-geozarr, but I don't like that outcome. I will see if we can get this into zarr-python

[emmanuelmathot]

great. Couple of comments only.

[emmanuelmathot]

The experimental_scale_offset_codec flag only activates when keep_scale_offset=False. if keep_scale_offset=True, the cdec branch is silently skipped. This may be surprising.

[d-v-b]

@emmanuelmathot we could remove the "keep_scale_offset" parameter, either here in a separate PR.

[emmanuelmathot]

fill_value is hardcoded to float("nan") in the codec path, which may not be appropriate for integer-stored arrays.

[lhoupert]

Hi @d-v-b and @emmanuelmathot . I deployed this branch pinned to c16bdb6 on our staging pipeline and the conversion completes with exit code 0 but the output is float32 with no scale_offset codec.
Reproducing locally confirmed the issue.

Claude traced it to the following:

Root cause: fill_value=float("nan") + CastValueRustV1(data_type="uint16") are incompatible

In create_measurements_encoding() (s2_multiscale.py ~line 398), the codec path sets:

var_encoding["filters"] = (so_codec, cv_codec)   # CastValueRustV1(data_type="uint16")
var_encoding["fill_value"] = float("nan")         # ← incompatible

When zarr initialises the codec pipeline, CastValueRustV1.resolve_metadata() tries to cast the array's fill_value (NaN) to the target dtype (uint16). This raises:

ValueError: Cannot cast NaN to integer type without scalar_map

This was already flagged in @emmanuelmathot's review comment.

Why the conversion exits 0

The exception is raised inside a dask-distributed compute path (write_job.persist() / distributed.progress()). The exception appears to be caught and silently swallowed somewhere in the dask/distributed error handling in stream_write_dataset(), causing the fallback write (without codecs) to succeed. The caller never sees an error.

This is maybe a second bug, I think a hard failure in the codec pipeline should not silently produce wrong output?

Suggested fix by Claude

Two options:

1. Set fill_value=0 in the codec path — the original nodata sentinel for packed S2 uint16 data is 0, so this is semantically correct:

   var_encoding["fill_value"] = 0  # uint16 nodata sentinel; NaN is only valid for float arrays

2. Use scalar_map on CastValueRustV1 to explicitly map NaN → 0, which is more self-documenting:

   cv_codec = CastValueRustV1(
       data_type=np.dtype(packed_dtype).name,
       rounding="nearest-even",
       scalar_map={float("nan"): 0},
   )
   var_encoding["fill_value"] = float("nan")  # kept for float-side semantics

Option 2 seems to aligns better with the codec's design intent.

Separately, could stream_write_dataset() let codec pipeline errors propagate rather than falling back silently?

[d-v-b]

The caller never sees an error.

unpleasant! it definitely makes sense to make stream_write_dataset noisily error!

Set fill_value=0 in the codec path — the original nodata sentinel for packed S2 uint16 data is 0, so this is semantically correct:

IMO we want the (decoded, float) fill value to be NaN. I think the fix here is to declare a scalar map that sends NaN -> 0, and ensures that the minimum real data value is 1

[d-v-b]

the changes in 4be784b should address the issues @lhoupert found. I will add some inline comments to explain some changes in context.

[d-v-b]

flagging this section as the location where we pick the int value that NaN gets mapped to. I first check and see if there's an existing _FillValue field in metadata, and use that if it's available. Otherwise, I use the lowest representable value in the output data type. If we want to make this configurable, we can expose it as an option.

[d-v-b]

@lhoupert have you had a chance to test the latest changes in the staging env? If you send me the command I could also test locally

[lhoupert]

Yes! I was going to message. I ran it locally and created a notebook to validate the file generated. Will deploy that in staging as soon https://github.com/EOPF-Explorer/data-pipeline/blob/feat/181-scale-offset-codec-staging/operator-tools/codec/validate_conversion.ipynb

[emmanuelmathot]

Could we have the samples on S3 to test all the tools with it? (TiTiler, OL, GDAL?)

[d-v-b]

I updated to use the latest zarr-python release, which includes scale-offset and cast-value! so we can remove some code from this PR :)

[d-v-b]

a few off-target changes incoming, since xarray versions are causing some skew in fill values. I'm going to fix this by setting the fill value explicitly for all variables, and not exposing us to xarray defaults.

[d-v-b]

Branch summary (from Claude)

This is a recap of everything that landed on feat/scale-offset-codecs. The PR ended up touching three concerns; the codec work is the headline, but two follow-on fixes (xarray version drift and pydantic-zarr 0.10) are tangled in.

1. CF scale-offset, optionally pushed into the zarr codec pipeline

New CLI flag for convert-s2-optimized:

--experimental-scale-offset-codec

When on (and --keep-scale-offset is off), the converter replaces CF scale_factor / add_offset attributes with a [ScaleOffset, CastValue] filter pipeline at the zarr level. Data stays as packed integers on disk; the codecs decode transparently on read. Without the flag, behaviour is unchanged: scale-offset attributes are stripped and arrays are written as decoded floats.

The codecs themselves come from zarr-python >= 3.2.0, pulled in via the zarr[cast-value-rs] extra. Locally we only carry scale_offset_from_cf — the small mapping from CF parameters to ScaleOffset constructor arguments (offset = add_offset, scale = 1 / scale_factor).

NaN handling: CastValue refuses to cast NaN to an integer dtype without an explicit map, so the codec branch builds a scalar_map that round-trips NaN ↔ a packed-dtype sentinel (the source's _FillValue if present, else iinfo.min). This is wired up in create_measurements_encoding.

2. Encoding-loss fixes around astype

Two latent bugs surfaced during the codec work: xr.DataArray.astype clears .encoding, and the multiscale path was relying on encoding surviving through astype calls.

• _coarsen_variable now captures .encoding before the cast and restores it after.
• The float64 → float32 loop in create_multiscale_from_datatree does the same.

Without these, downsampled levels saw an empty encoding dict, so the experimental codec pipeline was silently dropped on r120m / r360m / r720m (and CF metadata in general didn't propagate to coarsened levels).

3. Explicit fill_value, no longer xarray's call

Snapshot tests started failing with 0.0 → "NaN" diffs on conditions/geometry arrays nobody touched. Bisecting against uvx-pinned versions identified xarray as the source: in xarray 2025.7.1 and earlier, a float variable whose source had _FillValue=NaN was written with zarr-level fill_value=0.0 (silently substituting). xarray 2025.9+ correctly honours the source _FillValue and writes "NaN".

The fix removes the version-dependence by setting fill_value ourselves — see the new utils.explicit_fill_value helper and its three call sites:

• _create_encoding (legacy convert path)
• _create_geozarr_encoding (sharded convert path)
• create_original_encoding (S2 conditions/geometry groups)

For non-finite floats it emits the JSON-canonical string form ("NaN" / "Infinity" / "-Infinity") that zarr-python serialises.

4. Test split

Replaced one slow, single-parametrization snapshot test with two complementary ones:

• test_create_multiscale_from_datatree — snapshot test with the canonical (keep_scale_offset=False, experimental_scale_offset_codec=False) parametrization, comparing against the regenerated structure fixtures.
• test_create_multiscale_from_datatree_behavior — fast in-memory parametrized test over all four (keep × codec) combos. Builds a tiny CF-encoded DataTree and asserts (a) on-disk dtype + codec presence per combo on every pyramid level (original and downsampled), (b) decoded-value invariance: regardless of which storage path the converter chose, xarray reads back the same float values (modulo scale_factor quantisation).

Snapshot fixtures regenerated to capture the corrected explicit-fill_value output. CLI e2e fixtures regenerated similarly.

5. Misc

• TestCheckValidCoordinates constructors now pass attributes={} to GroupSpec(...) — pydantic-zarr 0.10 made it required.
• A standalone demo at xarray_example.py is in the working tree (not committed yet) — PEP 723 single-file script that round-trips CF-encoded data through the codec pipeline, intended to share with xarray devs as a worked example. Worth committing or moving to examples/ separately.

Open follow-ups

• The CLI flag is named --experimental- and is documented as such; before flipping the default we'd want to validate the codec behaviour on a wider range of S2 products and confirm downstream readers (titiler-xarray) handle the codec metadata.
• The fixture comparison still flags "37 paths differ" without naming the differing fields. A small refactor to a JSONPointer-keyed flat representation would make these diffs much more diagnostic on the next drift; deferred to a separate PR.

Full local test run: 257 passed, 6 skipped (titiler / docs), 0 failed.

🤖 Generated with Claude Code