Performance benchmark suite + O(1) wildcard resolution + faster layout crawl - #3954
Open
T4rk1n wants to merge 3 commits into
Open
Performance benchmark suite + O(1) wildcard resolution + faster layout crawl#3954T4rk1n wants to merge 3 commits into
T4rk1n wants to merge 3 commits into
Conversation
A standalone benchmark harness for the renderer's hot paths, kept out of the pytest suite on purpose: timing is noisy, so it reports rather than flaking the test matrix. - benchmarks/scenarios.py: 10 scenarios across the platform - initial/deep hydration, Patch append (top-level + nested), scalar Patch update, full-list replacement (contrast), callback fan-out, ALL-wildcard resolution, and a deep callback chain. Each is a real Dash app + a browser-side interaction with warn/fail thresholds. - benchmarks/run.py: runs each scenario in its own bench_app subprocess against the production bundle, driven by headless Chrome. Timings use in-page performance.now() (no selenium-poll latency), aggregated as median/p90/max plus a growth ratio (late vs early per-op time) that flags O(total) creep. Also gates against a committed baseline and CPU-profiles a scenario (--profile) into a .cpuprofile + a hottest-functions table. - benchmarks/baseline.json: reference numbers on ubuntu-latest-class hardware. - .github/workflows/benchmarks.yml: PR job that builds the production renderer, runs the harness vs the baseline, hard-fails only on an order-of-magnitude regression, warns (without failing) on smaller drift, and upserts a sticky PR comment with the results table. - .ai/PERFORMANCE.md: how to run, how to profile, and the findings - including that ALL/MATCH wildcard resolution is O(n^2) (linear deep-equals getPath over the objs table; a hash index would make it O(1)), that post-fix Patch append has no single hotspot left, and that layouts deeper than ~250 fail to serialize. Removes the earlier pytest timing guards (tests/integration/renderer/ test_patch_append_perf.py); that coverage now lives in the harness + CI job.
Profiling the wildcard benchmark (one input change resolving an ALL callback
over 400 components) showed ~45% of the time in ramda `_equals`/`_functionName`:
getPath for a pattern-matching (dict) id did a linear `find(propEq(values,
'values'), keyPaths)` over every component sharing that id's key set, and it is
called once per resolved component - so an ALL dispatch over N components was
O(N^2) deep-equality comparisons.
The paths table now carries `objIndex`: {[keyStr]: {[valuesKey]: path}} where
valuesKey is JSON.stringify(values), so getPath is an O(1) map lookup. The
ordered `objs` array is untouched (resolveDeps / getAllPMCIds still walk it in
order for MATCH/ALLSMALLER); objIndex is only for exact lookups. It is
maintained inline in computePaths - copy-on-write per keyStr, so re-resolving
one chunk doesn't rebuild the index for unrelated components - and extended
incrementally in appendPaths. A table without an index (initial empty state,
hand-built test fixture) makes getPath fall back to the linear scan, so the two
can never disagree.
Result on the benchmark: wildcard_all_resolve ~580ms -> ~140ms (~4x, isolated),
with the _equals/_functionName frames gone from the profile. Renderer unit
suite 55/55; integration green across test_wildcards, multiple_callbacks,
layout_paths_with_callbacks, basic_callback, patch, children_reorder (69).
Baseline regenerated (full-suite numbers run hotter than isolated, but the
wildcard drop from 632ms to 217ms p90 is clearly captured); loosened the
intentionally-slow full_children_replace reference threshold to match.
Profiling every benchmark showed ~10-15% in ramda's curry machinery
(f1/f2/f3, _isPlaceholder, curried path/pathOr). It came from crawlLayout -
run on every component on every path recompute and callback gather - calling
curried path(['props','children'], obj)/pathOr(...) per node, plus the
path(['props','id'], child) in each crawl callback (paths.js, dependencies.js).
Replace those with direct property access (obj.props && obj.props.children,
etc.) and native array concat on the hot common path, leaving the rare declared
childrenProps ([]/{}) branch untouched. The crawled nodes are always plain
component objects, so this is equivalent to the curried path, just without the
dispatch and placeholder checks.
Same-machine A/B: patch_append_nested ~16% faster, initial render and wildcard
resolution a few percent, no behavior change. Renderer unit 55/55; integration
green across wildcards, basic/multiple callbacks, layout paths, patch, reorder.
Also: the benchmark profiler now keys anonymous frames by source location
(they were collapsing into one opaque bucket), and the baseline is refreshed.
|
Dash performance benchmarks❌ perf regression
growth = late-third / early-third per-op time; ~1 is flat, a large value means the per-op cost scales with accumulated state. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3948 (base branch
perf/patch-append-rehydration), so the diff here is only the benchmark work and the two optimizations it surfaced. Review/merge #3948 first.What's here
1. A platform performance benchmark harness (
benchmarks/)Standalone, not part of the pytest suite — timing is noisy, so it reports rather than flaking the test matrix. 10 scenarios across the platform (initial/deep hydration, Patch append top-level + nested, scalar Patch update, full-list replacement, callback fan-out,
ALL-wildcard resolution, deep callback chain). Each runs in its own subprocess against the production bundle, driven by headless Chrome, timed with in-pageperformance.now(), aggregated as median/p90/max plus agrowthratio that flags O(total) creep.--profile <scenario>captures a Chrome CPU profile + a hottest-functions table.2. CI job (
.github/workflows/benchmarks.yml)Runs on PRs touching
dash/,benchmarks/, or components. Builds the production renderer, runs vs the committedbaseline.json, hard-fails only on an order-of-magnitude regression, warns without failing on smaller drift, and upserts a sticky PR comment with the results table.3. Wildcard resolution O(n²) → O(1) — found by the harness
getPathfor a pattern-matching (dict) id did a linear deep-equality scan over every component sharing the id's key set, called once per resolved component — so anALLdispatch over N components was O(N²). Added anobjIndexhash map (valuesKey → path) maintained inline incomputePaths(copy-on-write) andappendPaths; the orderedobjsarray stays for pattern iteration. ~580ms → ~140ms (≈4x) onALLover 400 components.4. Ramda currying overhead in the per-node layout crawl — also found by the harness
crawlLayout(run on every component on every path recompute + callback gather) used curriedpath/pathOrper node. Replaced with direct property access on the hot common path. Same-machine A/B: patch_append ~16%, initial render + wildcard a few %. No behavior change..ai/PERFORMANCE.mddocuments the methodology, how to profile, and the findings (including two that are not fixed here:callback_chainis network-bound, and layouts deeper than ~250 fail to serialize).Testing
test_wildcards,test_basic_callback,test_multiple_callbacks,test_layout_paths_with_callbacks,test_patch,test_children_reorder.Note
baseline.jsonwas generated locally; absolute thresholds work anywhere, but the baseline-ratio comparison is best regenerated onubuntu-latest(adopt the first CI run'sresults.json).