Enable PGO for Linux x86-64 Ruff releases by charliermarsh · Pull Request #27570 · astral-sh/ruff

charliermarsh · 2026-08-07T13:10:22Z

Summary

This PR enables PGO for Ruff releases, starting with Linux x86-64.

Design

The release pipeline is modified as follows:

We build an instrumented, stripped release binary.
We run check and format on a corpus of projects from our ecosystem reports -- specifically, eight of the pinned ecosystem projects that we use in the ty CI. (In total, it's 502 Python and stub files.)
We merge the profiles via llvm-profdata.
We feed the result back into the existing maturin build.

Results

We evaluate performance on a held-out corpus: Prefect, Django, Pandas, scikit-learn, SciPy, and SymPy.

Results are as follows:

Held-out project	`ruff check`	`ruff format`
Django	10.9% faster	6.6% faster
pandas	16.3% faster	9.7% faster
scikit-learn	10.8% faster	7.9% faster
SciPy	14.7% faster	5.6% faster
SymPy	17.1% faster	11.4% faster
Geometric mean	14.0% faster	8.3% faster

Beyond runtime:

Binary size decreased by 6.2% (27.96 MB to 26.23 MB).
Release pipeline gets about 2x longer (non-PGO release build took 7m22s; PGO pipeline took 15m14s (8m35s instrumented training plus 6m39s optimized wheel).

Stack

Additional platforms are covered in subsequent PRs in the stacked; platforms that are lower-priority at at-all difficult to run on natively are omitted. In the end, I'm targeting Linux x86-64, Linux ARM, macOS ARM, and Windows x86-64.

I also attempted BOLT in #27588, but I've decided against pursuing that for now; see the results in that PR which speak for themselves.

ty and uv will follow the same approach; see the stacks here:

See: #7055.

astral-sh-bot · 2026-08-07T13:17:00Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

ntBre

Thanks, this is awesome! I guess we can close #17369 now too.

MichaReiser

Nice

MichaReiser · 2026-08-10T13:59:18Z

+CORPUS_PROJECTS = (
+    EcosystemProject(
+        name="pytest",
+        repository="pytest-dev/pytest",
+        revision="28e86a6c2ae0173831e4925a4af89b02a2936d09",
+        source_directories=("src/_pytest",),
+    ),
+    EcosystemProject(
+        name="httpx",
+        repository="encode/httpx",
+        revision="b5addb64f0161ff6bfe94c124ef76f6a1fba5254",


Can you add a comment here or in the PR summary explaining how you selected corpus project (It's okay saying that they were mostly chosen at random, I just want to avoid us wondering what the selection criteria was coming back to this i 8 months)

## Summary This PR enables PGO for Ruff's macOS ARM64 releases, following the approach outlined in #27570. For macOS, `ruff check` gets 8.3% faster on the holdout set, `ruff format` gets 5.4% faster, and the release binary gets 1.1% smaller.

## Summary This PR enables PGO for Ruff's Windows x86-64 releases, following the approach outlined in #27570.

## Summary This PR enables PGO for Ruff's Linx ARM64 releases, following the approach outlined in #27570. (To enable PGO, we also move to a native ARM64 runner.) As a result, `ruff check` gets 11.2% faster on the holdout set, `ruff format` gets 8.6% faster, and the release binary gets 3.6% smaller.

## Summary This PR enables PGO for ty releases, starting with Linux x86-64. The approach follows that outlined in astral-sh/ruff#27570. ### Design The release pipeline is modified as follows: - We build an instrumented release binary using the Rust toolchain pinned by the Ruff submodule. - We run `ty check` on the same ten pinned ecosystem projects used by Ruff. In total, the corpus contains 618 Python and stub files. - We also exercise `ty server` against a temporary project, covering diagnostics, hover, go-to-definition, completion, and incremental cross-file edits. - We merge the profiles via `llvm-profdata`. - We derive LLVM's hot-code threshold from the profile's 95th percentile, matching its size-optimization threshold and avoiding unnecessary expansion of moderately hot functions. (Without this, performance was marginally better, but wheel size _increased_ by 5-10%!) - We feed the result back into the existing `maturin` build. ### Results Here's the initial, untuned PGO performance on seven held-out projects (and language-server performance on three held-out projects): | Held-out project | `ty check` | Incremental edits | | --- | ---: | ---: | | Black | 7.1% faster | 17.1% faster | | isort | 13.3% faster | 18.1% faster | | Jinja | 5.5% faster | 13.7% faster | | Django | 25.5% faster | — | | pandas | 15.9% faster | — | | scikit-learn | 19.6% faster | — | | SymPy | 21.5% faster | — | | Geometric mean | **15.8% faster** | **16.3% faster** | As an aside, profiling the language server explicitly (i.e., including the language server commands in the PGO training) improved incremental-edit latency by another 7.6% over CLI-only PGO; without that training, incremental edit performance was _still_ up (but not quite as much). Beyond raw performance, with the tuned PGO settings, we see the following results: - **Binary size decreased by 7.98%** (28.10 MB to 25.86 MB compared with non-PGO). - **Wheel size decreased by 1.32%** (12.71 MB to 12.54 MB compared with non-PGO), and by 14.11% compared with untuned PGO. - **Release archive size decreased by 1.53%** (12.35 MB to 12.16 MB compared with non-PGO), and by 14.25% compared with untuned PGO. - **Incremental language-server edits are 3.30% faster than untuned PGO**. - **Linux x86-64 release build finishes in 11m28s on eight-core Depot**, compared with 12m59s on four-core Depot and 15m45s on the previous GitHub runner. The full-stack bottleneck shifts to Windows at 12m24s; other Linux targets retain their four-core runners. ### Stack Additional platforms are covered in subsequent PRs: - macOS ARM64: #4216 - Windows x86-64: #4217 - Linux ARM64: #4218 Ruff and uv follow the same approach; see astral-sh/ruff#27570 and astral-sh/uv#21001.

## Summary This PR enables PGO for Ruff releases, starting with Linux x86-64. ### Design The release pipeline is modified as follows: - We build an instrumented, stripped release binary. - We run `check` and `format` on a corpus of projects from our ecosystem reports -- specifically, eight of the pinned ecosystem projects that we use in the ty CI. (In total, it's 502 Python and stub files.) - We merge the profiles via `llvm-profdata`. - We feed the result back into the existing `maturin` build. ### Results We evaluate performance on a held-out corpus: Prefect, Django, Pandas, scikit-learn, SciPy, and SymPy. Results are as follows: | Held-out project | `ruff check` | `ruff format` | | --- | ---: | ---: | | Django | 10.9% faster | 6.6% faster | | pandas | 16.3% faster | 9.7% faster | | scikit-learn | 10.8% faster | 7.9% faster | | SciPy | 14.7% faster | 5.6% faster | | SymPy | 17.1% faster | 11.4% faster | | Geometric mean | **14.0% faster** | **8.3% faster** | Beyond runtime: - **Binary size decreased by 6.2%** (27.96 MB to 26.23 MB). - **Release pipeline gets about 2x longer** (non-PGO release build took 7m22s; PGO pipeline took 15m14s (8m35s instrumented training plus 6m39s optimized wheel). ### Stack Additional platforms are covered in subsequent PRs in the stacked; platforms that are lower-priority at at-all difficult to run on natively are omitted. In the end, I'm targeting Linux x86-64, Linux ARM, macOS ARM, and Windows x86-64. I also attempted BOLT in astral-sh#27588, but I've decided against pursuing that for now; see the results in that PR which speak for themselves. ty and uv will follow the same approach; see the stacks here: - astral-sh/ty#4213 - astral-sh/uv#21001 See: astral-sh#7055.

## Summary This PR enables PGO for Ruff's macOS ARM64 releases, following the approach outlined in astral-sh#27570. For macOS, `ruff check` gets 8.3% faster on the holdout set, `ruff format` gets 5.4% faster, and the release binary gets 1.1% smaller.

## Summary This PR enables PGO for Ruff's Windows x86-64 releases, following the approach outlined in astral-sh#27570.

## Summary This PR enables PGO for Ruff's Linx ARM64 releases, following the approach outlined in astral-sh#27570. (To enable PGO, we also move to a native ARM64 runner.) As a result, `ruff check` gets 11.2% faster on the holdout set, `ruff format` gets 8.6% faster, and the release binary gets 3.6% smaller.

## Summary This PR enables PGO for uv releases, starting with Linux x86-64. ### Design The release pipeline is modified as follows: - We build instrumented `uv` and `uvx` release binaries inside the existing manylinux container. - We train directly on eleven shared `test/ecosystem` fixtures: cibuildwheel, Cookiecutter, Flask, HTTPX, LLM, the OpenAI Python SDK, Poetry, pytest-cov, Sentry, Zulip, and the 38-project pyx workspace. - Every project exercises cold and warm `uv pip compile` resolution, cold and warm `uv lock` resolution, `uv export`, `uv pip` installation, and `uv sync` installation on every release platform. - Per-project `exclude-dependencies` settings omit packages without compatible release-platform wheels while preserving the rest of each real dependency graph. - Raw profiles are merged online by workload family, then merged with `llvm-profdata` and fed back into the existing `maturin` build. - JupyterLab, Saleor, Semantic Kernel, Transformers, and Warehouse remain separate, held-out evaluation projects. ### Results We benchmarked the builds on Linux x86-64 using the expanded corpus for training and five separate held-out ecosystem projects for evaluation. | Held-out project | Dependency resolution | Wheel installation | Project locking | Lockfile export | | ---------------- | --------------------: | -----------------: | ---------------: | --------------: | | JupyterLab | 11.2% faster | 0.0% | 10.1% faster | 10.6% faster | | Semantic Kernel | 9.3% faster | 7.1% faster | 12.5% faster | 8.7% faster | | Transformers | 11.7% faster | 4.1% faster | 9.5% faster | 11.0% faster | | Saleor | — | — | 9.7% faster | 10.0% faster | | Warehouse | — | — | 9.7% faster | 7.6% faster | | Geometric mean | **10.7% faster** | **3.7% faster** | **10.3% faster** | **9.6% faster** | Across all sixteen held-out workloads, wall time decreased by **8.9%** (95% CI: 8.4–9.8%) and CPU time decreased by **8.4%** (95% CI: 7.9–9.0%). The matched Linux executable was **13.3% smaller** (58.91 MB to 51.05 MB), or **12.3% smaller** after gzip compression (22.68 MB to 19.90 MB). (Saleor and Warehouse include source-only dependencies, so they are benchmarked on locking and export rather than wheel-only installation.) ### Stack Additional platforms are covered in subsequent PRs in the stack: macOS ARM64, Windows x86-64, and Linux ARM64. Ruff and ty follow the same approach; see the stacks here: - astral-sh/ruff#27570 - astral-sh/ty#4213

Wire the PGO training script into the CICD release pipeline for native Linux targets (x86_64-unknown-linux-gnu and aarch64-unknown-linux-gnu). Cross-compiled targets are skipped as PGO training requires a native binary. Pipeline addition (before the existing 'Build coreutils' step): 1. Install llvm-tools (rustup component add llvm-tools) 2. Train PGO profiles via scripts/build_coreutils_pgo.py --train-only (instrumented build + representative workloads + llvm-profdata merge) 3. Inject -Cprofile-use=... into RUSTFLAGS so the existing build step produces a PGO-optimised binary automatically Benchmarks on aarch64 (CPU-bound utilities): sort +24%, join +31%, cut +22%, wc -w +21%, uniq +20%, nl +20% Follows the same approach as astral-sh/ruff#27570.

chirizxc reviewed Aug 7, 2026

View reviewed changes

Comment thread .github/workflows/build-binaries.yml

charliermarsh mentioned this pull request Aug 7, 2026

Add BOLT optimization to PGO Ruff releases #27571

Closed

charliermarsh added 4 commits August 7, 2026 16:29

Prototype profile-guided optimization for Ruff releases

683dc07

Train Ruff PGO on pinned ecosystem projects

f96ec0f

Retry PGO corpus Git operations

a26e45c

Harden Ruff PGO training script

a752a11

charliermarsh force-pushed the charlie/ruff-pgo-prototype branch from ff08211 to a752a11 Compare August 7, 2026 20:31

charliermarsh mentioned this pull request Aug 7, 2026

Enable PGO for macOS ARM64 Ruff releases #27572

Merged

charliermarsh changed the title ~~Prototype profile-guided optimization for Ruff releases~~ Enable PGO for Linux x86-64 Ruff releases Aug 7, 2026

Share the ty PGO training corpus

912fdb3

charliermarsh marked this pull request as ready for review August 10, 2026 13:41

charliermarsh added performance Potential performance improvement release Related to the release process labels Aug 10, 2026

This was referenced Aug 10, 2026

Enable PGO for Windows x86-64 Ruff releases #27573

Merged

Enable PGO for Linux ARM64 Ruff releases #27574

Merged

ntBre approved these changes Aug 10, 2026

View reviewed changes

MichaReiser approved these changes Aug 10, 2026

View reviewed changes

charliermarsh added 3 commits August 10, 2026 10:11

Diversify the PGO training corpus

6cf1f9d

Explain PGO corpus selection

3c046b8

Clarify PGO corpus selection

cb0c3f9

charliermarsh merged commit 60d2207 into main Aug 10, 2026
65 checks passed

charliermarsh deleted the charlie/ruff-pgo-prototype branch August 10, 2026 15:56

charliermarsh added a commit that referenced this pull request Aug 10, 2026

Enable PGO for Windows x86-64 Ruff releases (#27573)

f32cff5

## Summary This PR enables PGO for Ruff's Windows x86-64 releases, following the approach outlined in #27570.

This was referenced Aug 10, 2026

Enable PGO for Linux x86-64 ty releases astral-sh/ty#4213

Merged

Enable PGO for Linux x86-64 uv releases astral-sh/uv#21001

Merged

BrewTestBot mentioned this pull request Aug 13, 2026

ruff 0.16.3 Homebrew/homebrew-core#298673

Merged

George-Ogden pushed a commit to George-Ogden/ruff that referenced this pull request Aug 16, 2026

Enable PGO for Windows x86-64 Ruff releases (astral-sh#27573)

e3009e0

## Summary This PR enables PGO for Ruff's Windows x86-64 releases, following the approach outlined in astral-sh#27570.

scottames-github-bot Bot mentioned this pull request Aug 21, 2026

chore(deps): update ⬆️ mise-packages scottames/dots#1025

Merged

1 task

KarelZe mentioned this pull request Aug 25, 2026

perf: add support for pgo optimized builds🔧 KarelZe/lev#157

Closed

sylvestre mentioned this pull request Aug 29, 2026

perf: PGO release builds - up to 31% faster on Linux x86_64/aarch64 uutils/coreutils#14235

Merged

Navigation Menu

Uh oh!

Enable PGO for Linux x86-64 Ruff releases - #27570

Enable PGO for Linux x86-64 Ruff releases#27570
charliermarsh merged 8 commits into
mainfrom
charlie/ruff-pgo-prototype

charliermarsh commented Aug 7, 2026 •

edited

Loading

Uh oh!

astral-sh-bot Bot commented Aug 7, 2026

Uh oh!

Uh oh!

ntBre left a comment

Uh oh!

MichaReiser left a comment

Uh oh!

MichaReiser Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

charliermarsh commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design

Results

Stack

Uh oh!

astral-sh-bot Bot commented Aug 7, 2026

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

Uh oh!

ntBre left a comment

Choose a reason for hiding this comment

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

MichaReiser Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

charliermarsh commented Aug 7, 2026 •

edited

Loading

`ruff-ecosystem` results