Summary
Currently, all providers share a common cache directory at pooch.os_cache("climate_ref"). This can make it difficult to manage and debug provider-specific data. Additionally, data registries download to inconsistently named directories (e.g., pmp-climatology registry downloads to PMP_obs4MIPsClims/).
Current Behavior
- All providers use the same cache root:
~/.cache/climate_ref/
- Data registries download to directories determined by their internal structure, not their registry name
- Example:
pmp-climatology registry → {cache_dir}/PMP_obs4MIPsClims/
Desired Behavior
-
Per-provider cache directories: Each provider should have its own cache subdirectory, e.g.:
~/.cache/climate_ref/pmp/
~/.cache/climate_ref/esmvaltool/
~/.cache/climate_ref/ilamb/
-
Consistent registry naming: Data registries should download to folders matching their registry name:
pmp-climatology registry → {cache_dir}/pmp-climatology/
- This makes it easier to correlate registry names with filesystem locations
Benefits
- Easier to debug provider-specific issues
- Clearer disk usage attribution per provider
- Simpler cleanup of individual provider data
- More predictable directory structure
Files to Update
packages/climate-ref-core/src/climate_ref_core/dataset_registry.py - registry download paths
packages/climate-ref-pmp/src/climate_ref_pmp/__init__.py - get_data_path() and ingest_data()
- Similar changes needed for other providers (esmvaltool, ilamb)
Notes
The ingest_data() method currently hardcodes PMP_obs4MIPsClims as the climatology subdirectory name. This should use the registry name (pmp-climatology) instead.
Summary
Currently, all providers share a common cache directory at
pooch.os_cache("climate_ref"). This can make it difficult to manage and debug provider-specific data. Additionally, data registries download to inconsistently named directories (e.g.,pmp-climatologyregistry downloads toPMP_obs4MIPsClims/).Current Behavior
~/.cache/climate_ref/pmp-climatologyregistry →{cache_dir}/PMP_obs4MIPsClims/Desired Behavior
Per-provider cache directories: Each provider should have its own cache subdirectory, e.g.:
~/.cache/climate_ref/pmp/~/.cache/climate_ref/esmvaltool/~/.cache/climate_ref/ilamb/Consistent registry naming: Data registries should download to folders matching their registry name:
pmp-climatologyregistry →{cache_dir}/pmp-climatology/Benefits
Files to Update
packages/climate-ref-core/src/climate_ref_core/dataset_registry.py- registry download pathspackages/climate-ref-pmp/src/climate_ref_pmp/__init__.py-get_data_path()andingest_data()Notes
The
ingest_data()method currently hardcodesPMP_obs4MIPsClimsas the climatology subdirectory name. This should use the registry name (pmp-climatology) instead.