Skip to content

feat(3jane): monitor Accountable Proof of Solvency - #329

Draft
spalen0 wants to merge 4 commits into
mainfrom
accountable
Draft

feat(3jane): monitor Accountable Proof of Solvency#329
spalen0 wants to merge 4 commits into
mainfrom
accountable

Conversation

@spalen0

@spalen0 spalen0 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Closes #327.

Adds monitoring for Accountable's Proof of Solvency feed and wires 3Jane's dashboard (DFID=100000026) into the existing hourly 3jane monitor.

Data access

No scraping and no Weavechain client needed. GET https://accountable.3jane.xyz/dashboard serves the full report as public JSON — no auth, no headers. The SPA only falls back to the encrypted Weavechain plugin path if that endpoint fails, so the plain JSON path is the integration.

Three things that shaped the implementation

The reported ratio is rounded, so it isn't used for alerting. collateralization comes back rounded to six decimals. Near the boundary that's a missed-insolvency path: a true 0.9999996 presents as 1.0 and passes a < 1.00 test. The monitor recomputes from total_reserves / total_supply at full precision and keeps the reported field as a consistency cross-check (tolerance ≥1e-6, since the server's own rounding sets the floor).

net and collateralization are defined against liabilities, which equal total_supply only when total_supply.fx == 1. The client asserts that when the field is present. The live response omits it, in which case the net cross-check enforces the same invariant — the server computes net against liabilities, so net ≈ reserves − supply holds only when liabilities are raw supply. A non-USD-pegged feed fails loudly either way instead of silently comparing against the wrong denominator.

Freshness is per source type. A fresh aggregate timestamp doesn't prove the inputs are fresh, and that matters here — reserves_split is essentially all "Morpho Credit", the bulk of which is off-chain loan receivables priced by manually uploaded document reports. Live, two of the four sources are already past their declared cadence:

Source Type Frequency Observed age
Slope — Forward Flows Document Report DAILY ~57h
LendSwift — Warehouse Senior Note Document Report WEEKLY ~177h
USD3 Minted Liabilities ERC4626 15 MIN <1h
USD3 On-Chain Reserves ERC4626 15 MIN <1h

So budgets are keyed by source typeDocument Report gets a 7-day grace on top of its cadence, everything else gets 2 hours. A single global threshold would have fired on day one. Sources with an unrecognised cadence are skipped rather than flagged, so a schema addition on Accountable's side can't spuriously page us.

A required source that goes missing or malformed degrades the feed to STALE, not UNAVAILABLE: freshness can no longer be established, but the ratio is unaffected, so the report is still returned and the sub-100% check still runs. An upstream source rename cannot silently disable the CRITICAL solvency alert.

Band transitions, not drop-based dedupe. The live margin sits a few basis points above 100%, so the existing should_alert_value_drop() helper would alert on essentially every run. Alerts fire on OK → HIGH → CRITICAL transitions; recovering re-arms the bands above without alerting. CRITICAL additionally requires two consecutive, newer sub-100% reports — a single reading is reported as HIGH so it stays visible without escalating on what is more likely a stale document-report refresh. Re-polling a frozen report cannot confirm it, and an unavailable run resets partial confirmation.

No emergency dispatch in v1

Alerts use protocol key 3jane-accountable with channel 3jane. They reach the normal 3Jane Telegram channel at full severity, but the key is deliberately absent from utils.dispatch.DISPATCHABLE_PROTOCOLS, so a CRITICAL here cannot fire the emergency cap-zeroing webhook.

Alert already separates dispatch keying from Telegram routing, so this needed no changes to shared alert infrastructure. Intentional for v1 — the margin is basis points and the feed's noise profile is unproven. There's a test asserting this stays true.

Heads-up: this alerts on first run

Live collateralization is ~100.03%, which is below the 105% HIGH threshold. Deploying this will fire one HIGH alert on the first run, then go quiet (band dedupe). That's a real signal about a genuinely thin margin, not a false positive — but worth knowing before merge. The 105% threshold is the value flagged as TBD in #327 and is easy to adjust.

Changes

  • utils/accountable.py — DFID-keyed client: fetch, strict validation, consistency cross-checks, per-source-type freshness, typed OK/STALE/UNAVAILABLE outcomes. Never returns a silent None.
  • protocols/3jane/main.py — Accountable check in its own failure boundary, run before the onchain reads so it still executes when RPC is degraded.
  • protocols/3jane/README.md, monitoring.yaml — docs and metadata.
  • tests/test_accountable.py, tests/test_3jane.py — tests against a recorded live fixture.
  • utils/formatting.pyformat_with_suffix now suffixes negatives, so a CRITICAL shortfall reads -2.50M rather than -2500000.00.

Verification

  • 752 passed, 6 skipped — full suite
  • ruff check . and ruff format --check . clean
  • mypy utils/accountable.py clean (the one reported error is pre-existing in utils/http_client.py); mypy protocols/3jane/main.py clean apart from a pre-existing no-any-return
  • Ran end to end against the live feed: parses OK, classifies the live 100.029% as HIGH, routes to 3jane-accountable/3jane, and the second run is silent

Follow-ups (not in scope)

Questions for Accountable, none blocking: whether /dashboard is a supported endpoint with versioning guarantees, whether the response can carry a feed identifier so a misrouted response can be rejected, intended grace periods for the document-report inputs, and the supported verification procedure for the HTTP signatures / TEE attestations. Signature verification is deferred — v1 relies on TLS plus the consistency checks.

Also deferred per #327: verifiability-drop and total-reserves-change alerts. utils.http_client does not retry 429, which the issue lists as desirable; changing it affects every protocol, so it is left alone here.

🤖 Generated with Claude Code

codex and others added 4 commits July 31, 2026 09:51
Adds a client for Accountable's public Proof of Solvency feed and wires
3Jane's dashboard (DFID 100000026) into the existing hourly monitor.

The collateral ratio is recomputed from total_reserves / total_supply at
full precision rather than read from the API's `collateralization` field,
which is rounded to six decimals -- near the alert boundary a true ratio
of 0.9999996 would present as 1.0 and pass a `< 1.00` test. The reported
field is kept as a consistency cross-check. `net` and `collateralization`
are defined against liabilities, which equal total_supply only when
total_supply.fx == 1, so the client asserts that rather than assuming it.

Staleness budgets are keyed by source type: the reserves side leans on
manually uploaded document reports that routinely run past their declared
cadence, so those get a 7-day grace while on-chain sources get 2 hours. A
single global threshold would fire on day one.

Alerts fire on band transitions (OK/HIGH/CRITICAL) rather than on every
worsening tick, since the live margin sits a few basis points above 100%.
CRITICAL additionally requires two consecutive sub-100% runs; a single
reading is reported as HIGH so it stays visible without escalating on what
is more likely a stale document-report refresh.

Alerts use protocol key `3jane-accountable` with channel `3jane`: they
reach the normal Telegram channel at full severity, but the key is absent
from DISPATCHABLE_PROTOCOLS so a CRITICAL cannot trigger the emergency
cap-zeroing webhook while the feed's noise profile is unproven.

Refs #327

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Restores the alert bands specified in #327 and hardens the client against
failure modes that could silently disable the solvency check.

Bands (reverts 0c69cad):
- HIGH threshold back to 1.05; 1.0001 left a 1bp warning window the ratio
  can skip entirely between hourly runs, going OK -> CRITICAL with no lead.
- CRITICAL again requires two consecutive, newer sub-100% reports, per the
  issue's acceptance criteria. A frozen report cannot confirm itself and an
  unavailable run resets partial confirmation.

Client:
- A missing or malformed required data source is now STALE, not UNAVAILABLE.
  Freshness can no longer be established, but the collateral ratio is
  unaffected — an upstream source rename must not blind the sub-100% check.
- A source lastUpdated in the future is treated as unusable rather than
  clamped to age 0, which made a broken clock look permanently fresh.
- Decode the JSON body outside the RequestException handler; requests raises
  JSONDecodeError, which is itself a RequestException, so decode failures
  were reported as network failures.
- Reject a non-HTTPS dashboard URL. The URL is env-overridable and v1
  authenticates the feed with TLS alone.
- Drop the ambiguous bare "M" cadence unit; reading it as months would grant
  a source a 30x freshness budget.
- Drop the fx-absent fallback in _validate_usd_supply: it restated the same
  identity _validate_consistency already enforces, on the same tolerance.

Monitor:
- Widen set_cache_value to accept str; the band writes were an arg-type
  error under mypy.
- An unrecognised cached band now falls back to OK instead of raising out of
  the check on every subsequent run.

Also format negative values with K/M/B suffixes, so the shortfall in a
CRITICAL alert reads "-2.50M" rather than "-2500000.00".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Accountable Proof of Solvency monitoring (3Jane first): alert CRITICAL below 100% collateral ratio

1 participant