Collocation and SPARE-ICE updates - #431
Open
olemke wants to merge 23 commits into
Open
Conversation
Improvements and fixes made by Lorena during her Masters' thesis. Collocator (typhon/collocations/collocator.py): - Extend `collocate_dataset` to handle multiple secondary files per primary: collect all secondaries matched to a primary, concatenate them along the time dimension, renumber the time index, and join their file paths with "::" in the `__file` attribute. The first secondary's file attributes are taken as representative for the combined group. - Return early from `collocate_dataset` when no matches are found, and propagate `skip_file_errors` through `FileSet.match`. - Sort the primary and secondary datasets by time at the start of `collocate` so unsorted input files cannot break xarray's stack/sel. - Switch the common-period computation to `pd.to_datetime` so NaT values introduced by concatenating secondaries no longer break the min/max calculation. - Cast the `interval` DataArray to `timedelta64[ns]` and fix the "tempoerally" typo in a comment. Collocations common (typhon/collocations/common.py): - Use `np.int64` explicitly for the row counters in `_rows_for_secondaries` so results stay consistent across platforms. FileSet (typhon/files/fileset.py): - Treat the `find` search interval as closed (no longer subtract a microsecond from `end`) so end-time files are included consistently with `collocate`'s common-period computation. - Add `skip_file_errors` to `match` and search the secondary fileset over an `max_interval`-extended window while keeping the primary search on the original [start, end]. - Append the original file name (plus a temp suffix) to the decompress target path so decompressed HDF4/CloudSat files keep a recognisable name in the temp directory. CloudSat handler (typhon/files/handlers/cloudsat.py): - Rewrite `get_info` to work for both R04 and R05 granules: parse year and DOY from the filename and combine `UTC_start` with the last `Profile_time` value instead of relying on the `start_time`/`end_time` global attributes. - Promote `scnline` to a coordinate (numbered from 1) after reading. - Update the 2C-ICE documentation link to the current CloudSat DPC page. NetCDF4 handler (typhon/files/handlers/common.py): - Add `read_from_list` helper that reads a list of FileInfo objects and concatenates the resulting datasets along a caller-supplied common dimension. AVHRR GAC handler (typhon/files/handlers/tovs.py): - Pass `file_info` into `_interpolate_packed_pixels` so the "too many NaNs" error message can name the affected file, and let the granule be read as None when the interpolation raises. RetrievalProduct (typhon/retrieval/common.py): - Exclude attributes starting with `_repr_` (e.g. `_repr_html_`) from being deep-copied into the product dict. SPAREICE (typhon/retrieval/spareice/common.py): - Mask negative elevations via a `.loc` assignment to avoid chained-assignment warnings. - Convert the input collocations to an xarray Dataset before pulling lat/lon/time/scnpos, and make the retrieval `index` 1-based to match `scnline`. - Add `no_files_error` to `retrieve_from_collocations` and pass it through to `Collocations.map`. Fix `Timer` instantiation (`Timer().start()` rather than the classmethod form) and simplify the elapsed-time log line. - Rework the SPARE-ICE report plots: enable the top/right axes spines for a framed look, save figures as PDF (instead of PNG) with `bbox_inches='tight'`, drop the per-experiment subdirectory, draw a zero line and set xlim on the bias/mfe plots, and stop hard-coding a y-range for the bias plot. Rewrite `_report_ice_cloud` to build the confusion matrix with seaborn `heatmap` (labels=[1,0], row-normalised with NaN-safe denominators), keeping the previous implementation as `_report_ice_cloud_old`.
Collocation end time is now inclusive instead of exclusive.
The secondary dataset sort guard was checking `isinstance(primary, tuple)` in both branches due to a copy-paste bug, so the secondary dataset was never sorted when the primary was not a tuple. Use `isinstance(secondary, tuple)` for the secondary check. Also drop a stray `print(collocated)` debug statement left inside the `collocate` docstring example.
Clean up several leftover print() debug statements added during development: - collocations/common.py: drop "default read_mode is collapse" print issued whenever Collocations.read() is called with the default read_mode. - files/fileset.py: drop print(file_info) on every read in _call_map_function. - retrieval/spareice/common.py: drop print of ice_cloud feature_importances_ in _report_ice_cloud.
FileSet's decompress target names previously relied on the private `tempfile._get_candidate_names()` generator (an underscore-prefixed stdlib API that may change across Python versions). Use the public `secrets.token_hex(8)` instead, which is stable and produces a random filename suffix of the same shape. Applied at both decompress call sites in fileset.py (_get_info_via_handler and FileSet.read).
The interpolation fallback in AVHRR_GAC_HDF.read caught every exception (including KeyboardInterrupt and unrelated bugs) and silently returned None, masking real failures. Narrow it to `except ValueError` (the only exception raised by _interpolate_packed_pixels) and log the skipped file via a module-level logger.warning.
olemke
force-pushed
the
apply-lorena-collocation-updates
branch
from
August 11, 2026 10:16
b3fa04f to
c5ffad5
Compare
_tree_from_dict referenced the deprecated `n_features_` coefficient key, which was renamed to `n_features_in_` in scikit-learn 1.0. Update the deserialization to match the new attribute name so RetrievalProduct trees reconstruct correctly with modern sklearn.
Replace typhon.utils.to_array with direct numpy handling when rebuilding decision trees from stored coefficients. n_features_in_ and n_outputs_ are passed through unchanged, while n_classes_ is coerced with np.atleast_1d/np.asarray to satisfy sklearn's expectation of an integer array.
Trees trained with scikit-learn <1.3 lack the missing_go_to_left field in their node ndarray dtype. Newer sklearn versions reject this in Tree.__setstate__ with an incompatible-dtype error, making trained SPAREICE products unloadable. Rebuild the nodes array with the expected dtype (missing_go_to_left defaulting to 0, matching pre-1.3 behaviour) before calling __setstate__.
Replace `dataset.dims.keys()` with `dataset.sizes.keys()` in common._xarray_rename_fields and AAPP_HDF._test_coords to avoid the FutureWarning about the upcoming change in `Dataset.dims` return type.
Remove the MultiIndex coordinates before assigning a plain integer coordinate, since a MultiIndex cannot be stored to a file.
netCDF4 returns unmasked variables as MaskedArrays. Passing these directly to xarray promotes int types to float (NaN fill), which turned int64 coordinates (e.g. scnline, scnpos, channel) of SPARE-ICE collocation files into float64 on newer xarray/numpy stacks. Unwrap fully unmasked arrays so the original dtype is kept. Add a regression test verifying int64 coordinate dtypes survive a write/read roundtrip.
Subgroups only inherit dimensions when the parent group declares them first, so older readers that look at locally-defined dimensions (group.dimensions) fail to map variables in the subgroups. Sort groups by nesting depth so deeper subgroups are written first and declare their dimensions locally. Add a regression test.
The helper that samples the elevation map computed cell indices from the wrong origin and flipped the latitude direction. Use lat + 90 (instead of 90 - lat) for south→north ordering at the (-90, -180) origin.
fsspec's LocalFileSystem reports symlinks to directories as type other rather than directory, so the trailing-slash glob pattern in _get_matching_dirs skipped them entirely. Drop the trailing slash and filter with isdir(), which follows symlinks. Add a regression test for MHS files read through a symlinked placeholder directory.
Broken MHS AAPP files that lack the scnline dimension crashed the reader. Return None and log a warning instead.
Also compute the mean of the squared data in each grid cell and return it alongside the mean and the number of points.
Only rename variables, coordinates and dimensions whose names still contain the group prefix, and use rename_dims for dimensions.
The bundled standard.json was trained with scikit-learn <1.0 and failed to deserialize with modern versions. Remap the pre-1.0 module paths (sklearn.preprocessing.data, ...) in _import_class, drop constructor params removed from the estimator signatures, and fall back to the old n_features_ key when n_features_in_ is missing or None. Also parse structured dtypes stored in their dictionary representation, which numpy >=2 emits for aligned dtypes such as decision tree nodes.
The inplace Series.replace silently does nothing under pandas 3.0 Copy-on-Write, leaving log10(0) = -inf in the iwp column. Clearsky pixels (iwp=0) were then never NaN, so the training dataset preparation found an empty clearsky subset and crashed. Assign the replaced Series back to the column, which works across all pandas versions.
olemke
force-pushed
the
apply-lorena-collocation-updates
branch
from
August 14, 2026 12:48
f52e2d1 to
0a259dc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR consolidates a series of updates to the collocation machinery and its satellite data pipeline, many of which were contributed by Lorena Kowalczyk as part of her Master's thesis on SPARE-ICE. The collocator now combines all matching secondary files into a single dataset per primary file (instead of processing only the first match), tolerates
NaTtime values, and sorts inputs by time before collocating. The PR also bundles fixes to theFileSet/NetCDF4 layer that these workflows depend on: symlinked directories are now discovered and read correctly, integer dtypes survive NetCDF4 round-trips, subgroups are written in an order that older readers can parse, and decompression uses collision-free temporary file names. SPARE-ICE retrieval benefits from corrected elevation-map indexing, compatibility with models saved by older scikit-learn versions, and updated validation plots. Along the way, the collocation and SPARE-ICE code was modernized to be compatible with Python 3.14.Changes
typhon/collocations/collocator.py: Combine all overlapping secondary files viaxr.concatbefore collocating, mark the combined files in the__filevariable, return early when no file matches are found, and sort input datasets by time to avoid xarray stacking problemstyphon/collocations/collocator.py: Replacepd.Timestamp(values.min().item())withpd.to_datetime(...).min()so datasets containingNaTare handled, cast interval output totimedelta64[ns], and drop the MultiIndex explicitly before storing collocation outputtyphon/files/fileset.py: Switchfind()from a semi-open to a closed interval (endinclusive) to matchcollocate(), allowmatch()to skip file-not-found errors viaskip_file_errors, and glob directories without a trailing slash plus anisdir()filter so symlinked directories are found; use unique (secrets.token_hex) decompression targets to avoid name clashes between processes and replace the privatetempfile._get_candidate_nameswithsecretstyphon/files/handlers/common.py: Write NetCDF4 subgroups before their parent groups so dimensions are declared locally, and unwrap fully-unmaskedMaskedArrays before handing them to xarray to preserve integer dtypes (e.g.scnline,channelcoordinates); skip identity renames when writing groups so variables already named without a group prefix are left untouched; add aread_from_list()helper for concatenating multiple files; useDataset.sizesinstead of the deprecatedDataset.dimstyphon/files/handlers/cloudsat.py: Derive file start/end times from the filename (year/day-of-year) and theUTC_start/Profile_timefields, makingget_infowork for both R04 and R05 data, and exposescnlineas a 1-based coordinatetyphon/files/handlers/tovs.py: Degrade gracefully when AVHRR packed-pixel interpolation fails (log a warning and returnNoneinstead of aborting), report the affected file in the error message, and skip broken MHS files that lack thescnlinedimensiontyphon/retrieval/common.py: Restore SPARE-ICE models saved with old scikit-learn versions — remap un-prefixed sklearn module names (e.g.sklearn.tree.tree→sklearn.tree._classes), fall back ton_features_whenn_features_in_isNone, backfill themissing_go_to_leftnode field, decode structured dtypes in their dictionary representation (numpy ≥ 2), and filter constructor params against the class signature; exclude_repr_*attributes when serializing modelstyphon/retrieval/spareice/common.py: Correct elevation-grid cell indexing (offset to-90/-180origin), zero theelevationcolumn via.loc, replace-infIWP values without in-place mutation, convert collocations withto_xarray()before retrieval, and rework the validation plots (PDF output, confusion-matrix heatmap with feature-importance panel, framing/spine and axis-limit fixes)typhon/geographical.py: Guard against division by zero ingridded_meanso empty cells return 0 instead ofnan, and additionally return the mean of the squared values (useful for variance estimates)typhon/tests/...: Add tests for symlinked directory discovery/reading, locally-declared subgroup dimensions, and int-coordinate dtype round-trips; update test time ranges for the closed-intervalfind()semanticsBreaking Changes
FileSet.find()now treatsendas inclusive (closed interval) instead of semi-open. Code that relied on the previous exclusive-end behavior will see one additional file at the interval boundary.gridded_mean()now returns three values (mean, mean of squares, count) instead of two.