Skip to content

fix(release-controller): stop resubmitting election proposals for already-elected versions - #2023

Merged
pietrodimarco-dfinity merged 2 commits into
mainfrom
pmarco/release-controller-fix-duplicate-submission
May 26, 2026
Merged

fix(release-controller): stop resubmitting election proposals for already-elected versions#2023
pietrodimarco-dfinity merged 2 commits into
mainfrom
pmarco/release-controller-fix-duplicate-submission

Conversation

@pietrodimarco-dfinity

Copy link
Copy Markdown
Contributor

Why

The release-controller fired three different NNS election proposals for commit c7552446... (141776 -> failed, 141782 -> executed, then duplicates 141879 and 141880, both rejected by governance) because two latent bugs combined badly with a stale ignored_proposals entry that nobody removed once 141782 was executed:

  1. get_election_proposals_by_version in public_dashboard.py and dre_cli.py had a nested loop and a blind d[version] = proposal assignment that kept whichever proposal was iterated last per version. The dashboard API returns proposals newest-first, so the oldest (the rejected 141776) won. After _filtered_proposals_retriever then stripped 141776, the version looked unproposed on every cycle, and each pod restart submitted yet another one.
  2. The proposal-submission phase already fetched the elected (blessed) versions to compute unelect_versions, but it never used that set to short-circuit the submission, so the governance canister was left to refuse every duplicate on its own.

What

  • Replace the buggy per-version aggregator with dre_cli.proposals_by_version, a free helper that:
    • keeps the highest-id proposal per version (no reliance on iteration order), and
    • applies ignored_proposals filtering at the raw-list level before aggregation, so a stale ignored id can't hide a perfectly good non-ignored proposal targeting the same version -- the wrapper falls back to the next-highest-id automatically.
  • Refactor Reconciler._filtered_proposals_retriever to take a raw-list retriever (get_past_election_proposals) and call the new helper, so the filter happens at the right layer.
  • Add a release_commit in blessed short-circuit to the proposal submission phase that warns loudly and skips submission when the target version is already elected -- safety net for any future operator mistake of the same shape.
  • Drop the now-stale 141776 entry from release-index.yaml and update the README's example to a synthetic id, plus document the new fallback-to-next-highest-id behaviour in the caveats.

Tests

New tests in tests/test_reconciler_integration.py:

  • test_dashboard_get_election_proposals_by_version_keeps_highest_id_for_duplicates -- guards the dashboard aggregator.
  • test_dre_cli_get_election_proposals_by_version_keeps_highest_id_for_duplicates -- guards the dre_cli aggregator.
  • test_reconciler_filtered_proposals_retriever_falls_back_to_next_highest_id -- guards the filter-before-aggregation ordering (this is the user's exact scenario).
  • test_reconciler_skips_submission_when_version_already_in_blessed_set -- guards the safety net.

Updated tests:

  • test_reconciler_filtered_proposals_retriever_drops_ignored_ids / ..._is_noop_without_ids -- migrated to the new raw-list retriever signature.
  • test_reconciler_reconciles_without_error_already_submitted_proposals -- expected forum-post id updated from 138814 to 138817 to reflect the now-correct highest-id-wins behaviour.

Local runs:

bazel test //release-controller:integration_tests //release-controller:medium_tests
# both PASSED
bazel build //release-controller:public_dashboard //release-controller:dre_cli \
            //release-controller:release-controller //release-controller:dryrun \
            //release-controller:integration_tests //release-controller:medium_tests
# mypy clean

…eady-elected versions

The release-controller resubmitted three different NNS election proposals
for commit c7552446... (141776 -> failed, 141782 -> executed, then
duplicates 141879 and 141880) because two latent bugs combined badly with
a stale `ignored_proposals` entry left in `release-index.yaml` after the
replacement proposal was submitted:

1. `get_election_proposals_by_version` in `public_dashboard.py` and
   `dre_cli.py` had a nested loop and a blind `d[version] = proposal`
   that kept *whichever proposal was iterated last* per version.  Given
   that the dashboard API returns proposals newest-first, that meant the
   *oldest* (here the rejected 141776) won.  After
   `_filtered_proposals_retriever` then stripped 141776, the version
   looked unproposed every cycle, so each pod restart submitted a new
   one.

2. The proposal-submission phase already fetched the elected (`blessed`)
   versions to compute `unelect_versions`, but it never used that set to
   short-circuit the submission, so the governance canister was left to
   refuse every duplicate on its own.

Both fixes:

- Replace the buggy per-version aggregation with `proposals_by_version`,
  a free helper that keeps the highest-id proposal per version and
  applies `ignored_proposals` filtering at the raw-list level *before*
  aggregation, so a stale ignored id can't accidentally hide an
  otherwise-valid proposal that targets the same version.
- Add a `release_commit in blessed` short-circuit to the
  `proposal submission` phase that warns loudly and skips submission
  when the target version is already elected -- a safety net for any
  future operator mistake of the same shape.

Also drop the now-stale `141776` entry from `release-index.yaml` and
update the README so the example uses a synthetic id and the caveats
describe the new fallback-to-next-highest-id behaviour.
@pietrodimarco-dfinity
pietrodimarco-dfinity requested a review from a team as a code owner May 26, 2026 11:02
@pietrodimarco-dfinity
pietrodimarco-dfinity enabled auto-merge (squash) May 26, 2026 11:08
@pietrodimarco-dfinity

Copy link
Copy Markdown
Contributor Author

Reopening to retrigger CI after self-hosted runner outage cleared.

auto-merge was automatically disabled May 26, 2026 11:42

Pull request was closed

@pietrodimarco-dfinity
pietrodimarco-dfinity merged commit ddc9697 into main May 26, 2026
8 of 10 checks passed
@pietrodimarco-dfinity
pietrodimarco-dfinity deleted the pmarco/release-controller-fix-duplicate-submission branch May 26, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants