feat(3jane): monitor Accountable Proof of Solvency - #329
Draft
spalen0 wants to merge 4 commits into
Draft
Conversation
Adds a client for Accountable's public Proof of Solvency feed and wires 3Jane's dashboard (DFID 100000026) into the existing hourly monitor. The collateral ratio is recomputed from total_reserves / total_supply at full precision rather than read from the API's `collateralization` field, which is rounded to six decimals -- near the alert boundary a true ratio of 0.9999996 would present as 1.0 and pass a `< 1.00` test. The reported field is kept as a consistency cross-check. `net` and `collateralization` are defined against liabilities, which equal total_supply only when total_supply.fx == 1, so the client asserts that rather than assuming it. Staleness budgets are keyed by source type: the reserves side leans on manually uploaded document reports that routinely run past their declared cadence, so those get a 7-day grace while on-chain sources get 2 hours. A single global threshold would fire on day one. Alerts fire on band transitions (OK/HIGH/CRITICAL) rather than on every worsening tick, since the live margin sits a few basis points above 100%. CRITICAL additionally requires two consecutive sub-100% runs; a single reading is reported as HIGH so it stays visible without escalating on what is more likely a stale document-report refresh. Alerts use protocol key `3jane-accountable` with channel `3jane`: they reach the normal Telegram channel at full severity, but the key is absent from DISPATCHABLE_PROTOCOLS so a CRITICAL cannot trigger the emergency cap-zeroing webhook while the feed's noise profile is unproven. Refs #327 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Restores the alert bands specified in #327 and hardens the client against failure modes that could silently disable the solvency check. Bands (reverts 0c69cad): - HIGH threshold back to 1.05; 1.0001 left a 1bp warning window the ratio can skip entirely between hourly runs, going OK -> CRITICAL with no lead. - CRITICAL again requires two consecutive, newer sub-100% reports, per the issue's acceptance criteria. A frozen report cannot confirm itself and an unavailable run resets partial confirmation. Client: - A missing or malformed required data source is now STALE, not UNAVAILABLE. Freshness can no longer be established, but the collateral ratio is unaffected — an upstream source rename must not blind the sub-100% check. - A source lastUpdated in the future is treated as unusable rather than clamped to age 0, which made a broken clock look permanently fresh. - Decode the JSON body outside the RequestException handler; requests raises JSONDecodeError, which is itself a RequestException, so decode failures were reported as network failures. - Reject a non-HTTPS dashboard URL. The URL is env-overridable and v1 authenticates the feed with TLS alone. - Drop the ambiguous bare "M" cadence unit; reading it as months would grant a source a 30x freshness budget. - Drop the fx-absent fallback in _validate_usd_supply: it restated the same identity _validate_consistency already enforces, on the same tolerance. Monitor: - Widen set_cache_value to accept str; the band writes were an arg-type error under mypy. - An unrecognised cached band now falls back to OK instead of raising out of the check on every subsequent run. Also format negative values with K/M/B suffixes, so the shortfall in a CRITICAL alert reads "-2.50M" rather than "-2500000.00". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #327.
Adds monitoring for Accountable's Proof of Solvency feed and wires 3Jane's dashboard (
DFID=100000026) into the existing hourly3janemonitor.Data access
No scraping and no Weavechain client needed.
GET https://accountable.3jane.xyz/dashboardserves the full report as public JSON — no auth, no headers. The SPA only falls back to the encrypted Weavechain plugin path if that endpoint fails, so the plain JSON path is the integration.Three things that shaped the implementation
The reported ratio is rounded, so it isn't used for alerting.
collateralizationcomes back rounded to six decimals. Near the boundary that's a missed-insolvency path: a true0.9999996presents as1.0and passes a< 1.00test. The monitor recomputes fromtotal_reserves / total_supplyat full precision and keeps the reported field as a consistency cross-check (tolerance ≥1e-6, since the server's own rounding sets the floor).netandcollateralizationare defined against liabilities, which equaltotal_supplyonly whentotal_supply.fx == 1. The client asserts that when the field is present. The live response omits it, in which case the net cross-check enforces the same invariant — the server computesnetagainst liabilities, sonet ≈ reserves − supplyholds only when liabilities are raw supply. A non-USD-pegged feed fails loudly either way instead of silently comparing against the wrong denominator.Freshness is per source type. A fresh aggregate timestamp doesn't prove the inputs are fresh, and that matters here —
reserves_splitis essentially all "Morpho Credit", the bulk of which is off-chain loan receivables priced by manually uploaded document reports. Live, two of the four sources are already past their declared cadence:So budgets are keyed by source
type—Document Reportgets a 7-day grace on top of its cadence, everything else gets 2 hours. A single global threshold would have fired on day one. Sources with an unrecognised cadence are skipped rather than flagged, so a schema addition on Accountable's side can't spuriously page us.A required source that goes missing or malformed degrades the feed to STALE, not UNAVAILABLE: freshness can no longer be established, but the ratio is unaffected, so the report is still returned and the sub-100% check still runs. An upstream source rename cannot silently disable the CRITICAL solvency alert.
Band transitions, not drop-based dedupe. The live margin sits a few basis points above 100%, so the existing
should_alert_value_drop()helper would alert on essentially every run. Alerts fire onOK → HIGH → CRITICALtransitions; recovering re-arms the bands above without alerting. CRITICAL additionally requires two consecutive, newer sub-100% reports — a single reading is reported as HIGH so it stays visible without escalating on what is more likely a stale document-report refresh. Re-polling a frozen report cannot confirm it, and an unavailable run resets partial confirmation.No emergency dispatch in v1
Alerts use protocol key
3jane-accountablewith channel3jane. They reach the normal 3Jane Telegram channel at full severity, but the key is deliberately absent fromutils.dispatch.DISPATCHABLE_PROTOCOLS, so a CRITICAL here cannot fire the emergency cap-zeroing webhook.Alertalready separates dispatch keying from Telegram routing, so this needed no changes to shared alert infrastructure. Intentional for v1 — the margin is basis points and the feed's noise profile is unproven. There's a test asserting this stays true.Heads-up: this alerts on first run
Live collateralization is ~100.03%, which is below the 105% HIGH threshold. Deploying this will fire one HIGH alert on the first run, then go quiet (band dedupe). That's a real signal about a genuinely thin margin, not a false positive — but worth knowing before merge. The 105% threshold is the value flagged as TBD in #327 and is easy to adjust.
Changes
utils/accountable.py— DFID-keyed client: fetch, strict validation, consistency cross-checks, per-source-type freshness, typedOK/STALE/UNAVAILABLEoutcomes. Never returns a silentNone.protocols/3jane/main.py— Accountable check in its own failure boundary, run before the onchain reads so it still executes when RPC is degraded.protocols/3jane/README.md,monitoring.yaml— docs and metadata.tests/test_accountable.py,tests/test_3jane.py— tests against a recorded live fixture.utils/formatting.py—format_with_suffixnow suffixes negatives, so a CRITICAL shortfall reads-2.50Mrather than-2500000.00.Verification
752 passed, 6 skipped— full suiteruff check .andruff format --check .cleanmypy utils/accountable.pyclean (the one reported error is pre-existing inutils/http_client.py);mypy protocols/3jane/main.pyclean apart from a pre-existingno-any-returnOK, classifies the live 100.029% as HIGH, routes to3jane-accountable/3jane, and the second run is silentFollow-ups (not in scope)
Questions for Accountable, none blocking: whether
/dashboardis a supported endpoint with versioning guarantees, whether the response can carry a feed identifier so a misrouted response can be rejected, intended grace periods for the document-report inputs, and the supported verification procedure for the HTTP signatures / TEE attestations. Signature verification is deferred — v1 relies on TLS plus the consistency checks.Also deferred per #327: verifiability-drop and total-reserves-change alerts.
utils.http_clientdoes not retry 429, which the issue lists as desirable; changing it affects every protocol, so it is left alone here.🤖 Generated with Claude Code