Skip to content

fix(config): remove nested none values before toml serialization#4249

Open
mattfaltyn wants to merge 2 commits into
dlt-hub:develfrom
mattfaltyn:fix/4248-strip-nested-toml-none
Open

fix(config): remove nested none values before toml serialization#4249
mattfaltyn wants to merge 2 commits into
dlt-hub:develfrom
mattfaltyn:fix/4248-strip-nested-toml-none

Conversation

@mattfaltyn

@mattfaltyn mattfaltyn commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Description

  • recursively remove None values from nested mappings before TOML serialization
  • sanitize mappings contained in lists through the shared provider path
  • normalize tuple inputs to TOML arrays so file-backed _config_doc and _config_toml stay synchronized
  • add regression coverage for string serialization and file write/reload behavior

Related Issues

Additional Context

BaseDocProvider._set_value previously removed None only from the outer dictionary. Nested None values reached tomlkit, causing StringTomlProvider.dumps() to raise _ConvertError. SettingsTomlProvider caught that conversion error but still updated its plain dictionary, so the value appeared in memory while being omitted from persisted TOML.

The fix sanitizes values in the shared provider path before either representation is updated. Tuple inputs are normalized to lists because tomlkit represents TOML arrays as lists; this also avoids reconstructing tuple subclasses with incompatible constructors.

Validation

  • make lint — passed, including mypy, Ruff, Flake8, Bandit, lockfile, dependency, and API compatibility checks
  • python -m pytest tests/common/configuration/test_toml_provider.py -q — 20 passed
  • python -m pytest tests/common/configuration -q — 247 passed, 11 skipped
  • parallel common phase — 7,056 passed, 57 skipped; the only failure was the live load_github example exhausting GitHub's 60-request unauthenticated API quota while fetching issue comments
  • serial/forked common phase — 209 passed, 1 skipped before the macOS pytest-forked harness stalled; the isolated five-test tail then passed
  • GitHub Actions — awaiting upstream maintainer approval for the fork workflow

@mattfaltyn
mattfaltyn marked this pull request as ready for review July 17, 2026 09:15
@mattfaltyn
mattfaltyn marked this pull request as draft July 17, 2026 09:41
@mattfaltyn
mattfaltyn marked this pull request as ready for review July 17, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nested None values break TOML provider serialization

1 participant