devops(test-results-db): CLI to compact CI test results into DuckDB

.claude/skills/playwright-test-results/SKILL.md

@@ -0,0 +1,135 @@
+---
+name: playwright-test-results
+description: Query Playwright CI test results from the aggregated DuckDB database. Answers questions about flaky tests, failure rates, slow tests, and per-run/SHA/PR results without hunting through GitHub artifacts.
+user_invocable: true
+---
+
+# Playwright Test Results (DuckDB)
+
+A single DuckDB file holds recent Playwright CI test results, so you can answer
+questions about failures, flakiness, and slow tests with plain SQL. It is
+refreshed every few hours.
+
+## Get the database
+
+Download the latest snapshot:
+
+```bash
+npm ci                       # first time only, from the repo root
+GITHUB_TOKEN=$(gh auth token) node utils/test-results-db/cli.ts download
+```
+
+The snapshot may be missing the newest runs. To top it up locally, run `update`:
+
+```bash
+GITHUB_TOKEN=$(gh auth token) node utils/test-results-db/cli.ts update --lookback-days 3
+```
+
+Query it with the `duckdb` CLI (or any DuckDB client):
+
+```bash
+duckdb utils/test-results-db/test-results.duckdb "SELECT count(*) FROM test_results"
+```
+
+## Schema
+
+Single table `test_results`, one row per test result (**one row per retry**).
+The columns are inferred from the parquet the reporter emits
+(`tests/config/parquetReporter.ts`), plus two trailing columns this CLI adds:
+
+| Column | Meaning |
+| --- | --- |
+| `run_id`, `run_attempt` | GitHub Actions run identity |
+| `run_started_at` | when the run started |
+| `workflow_name` | e.g. `tests 1` / `tests 2` / `tests others` / `MCP` |
+| `event` | `push` / `pull_request` |
+| `head_sha`, `head_branch`, `pr_number` | what was tested |
+| `bot_name` | e.g. `chromium-ubuntu-22.04-node20`, `webkit-macos-15-large` — the CI bot. **OS and arch are encoded here**; there is no separate os column. |
+| `project_name` | CI project = browser + suite, e.g. `chromium-page`, `webkit-library`, `playwright-test` |
+| `test_title` | title path within the file, joined by ` › ` (`describe › test`) |
+| `file`, `line`, `column_number` | source location (file is relative to repo root) |
+| `expected_status` | `passed` / `skipped` / ... |
+| `status` | actual result: `passed` / `failed` / `timedOut` / `skipped` / `interrupted` |
+| `retry` | 0 = first attempt |
+| `result_started_at` | when this attempt started |
+| `duration_ms` | result duration |
+| `error_message` | all errors joined, ANSI-stripped (NULL when none) |
+| `tags` | **list** of strings, e.g. `['@slow', '@flaky']` (use list functions / `list_contains`) |
+| `annotations` | list of `{type, description}` structs, e.g. `[{'type': 'skip', 'description': 'flaky on CI'}]` (empty list when none) |
+| `artifact_id` | the GitHub artifact this row came from (dedupe key) |
+| `ingested_at` | debug only — when this row was imported |
+
+Notes:
+- **A test is identified by `(project_name, file, test_title)`** — group on that
+  tuple. (Playwright's `test_id` hash is deliberately not stored; those three
+  columns are its pre-image.)
+- **Flakiness is derived**, not stored. The signal that matters most is
+  **cross-run**: a test whose *final* verdict (after retries) flips between
+  runs — green in some, red in others. A separate **within-run** flake is a
+  test a retry rescued inside a single run (`failed`→`passed`).
+- **Real failures vs intentional ones:** filter `expected_status = 'passed'`.
+  Tests marked `test.fail()` record `status='failed'` *with*
+  `expected_status='failed'` and would otherwise dominate any "most failing" list.
+- The db is size-capped by **run count**: the oldest whole runs are evicted over
+  time, so it holds a recent window, not full history.
+
+## Example queries
+
+Group tests by `(project_name, file, test_title)` and (for failure/flakiness)
+scope to `expected_status = 'passed'` so intentional `test.fail()` tests don't
+skew the results.
+
+**Flaky across runs** — the test's final verdict flips between runs (this is
+what makes a red CI run ambiguous). `least(failed_runs, passed_runs)` ranks
+genuinely bimodal tests above both always-broken and one-off failures:
+
+```sql
+WITH per_run AS (
+  SELECT project_name, file, test_title, run_id, run_attempt,
+         arg_max(status, retry) AS final_status,
+         any_value(expected_status) AS expected
+  FROM test_results
+  GROUP BY project_name, file, test_title, run_id, run_attempt)
+SELECT project_name, test_title,
+       count(*) AS runs,
+       count(*) FILTER (WHERE final_status IN ('failed','timedOut')) AS failed_runs,
+       count(*) FILTER (WHERE final_status = 'passed') AS passed_runs,
+       round(100.0 * count(*) FILTER (WHERE final_status IN ('failed','timedOut'))
+             / count(*), 1) AS fail_pct
+FROM per_run
+WHERE expected = 'passed'
+GROUP BY project_name, test_title
+HAVING failed_runs > 0 AND passed_runs > 0 AND runs >= 10
+ORDER BY least(failed_runs, passed_runs) DESC, failed_runs DESC
+LIMIT 20;
+```
+
+**Filter by tag** (`tags` is a list, not a string):
+
+```sql
+SELECT project_name, test_title, count(*) AS runs
+FROM test_results
+WHERE list_contains(tags, '@slow')
+GROUP BY project_name, test_title
+ORDER BY runs DESC
+LIMIT 20;
+```
+
+## Fetching the full detail
+
+The db stores per-result summaries. For the full step tree / attachments / stdio,
+fetch the original blob report for that run, if the run uploaded one. A row
+identifies it by `run_id` + `bot_name`: the run's blob artifact is named
+`blob-report-<bot_name>`.
+
+```bash
+# List the run's blob artifacts and find the one for this bot_name:
+gh api /repos/microsoft/playwright/actions/runs/<run_id>/artifacts \
+  --jq '.artifacts[] | select(.name | startswith("blob-report")) | {id, name}'
+
+# Download it (name == "blob-report-<bot_name>"):
+gh api /repos/microsoft/playwright/actions/artifacts/<artifact_id>/zip > blob.zip
+```
+
+Blob and parquet artifacts have a 7-day retention, so this works only for recent
+runs; the db itself retains summaries longer (until run-count eviction).

.github/workflows/update_test_results_db.yml

@@ -0,0 +1,46 @@
+name: "Update test results DB"
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 */3 * * *"
+
+concurrency:
+  group: test-results-db
+  cancel-in-progress: false
+
+jobs:
+  update:
+    name: Update DuckDB
+    runs-on: ubuntu-22.04
+    timeout-minutes: 60
+    if: github.repository == 'microsoft/playwright'
+    permissions:
+      actions: read
+      contents: read
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-node@v6
+        with:
+          node-version: 26
+      - run: npm ci
+      - name: Download current database
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: node utils/test-results-db/cli.ts download
+      - name: Ingest new test results
+        id: ingest
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          NODE_OPTIONS: --max-old-space-size=8192
+        run: node utils/test-results-db/cli.ts update --lookback-days 7 --concurrency 32
+      - name: Truncate to run cap
+        if: steps.ingest.outputs.imported != '0'
+        run: node utils/test-results-db/cli.ts truncate --max-runs 2000
+      - name: Upload database
+        if: steps.ingest.outputs.imported != '0'
+        uses: actions/upload-artifact@v7
+        with:
+          name: test-results-db
+          path: utils/test-results-db/test-results.duckdb
+          retention-days: 7
+          overwrite: true

utils/test-results-db/.gitignore

@@ -0,0 +1,3 @@
+# The maintained database and DuckDB sidecar files (WAL, compaction temp).
+*.duckdb
+*.duckdb.*

utils/test-results-db/cli.ts

@@ -0,0 +1,121 @@
+/**
+ * Copyright (c) Microsoft Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import path from 'path';
+import { fileURLToPath } from 'url';
+import { parseArgs } from 'util';
+
+import { cmdDownload } from './download.ts';
+import { cmdTruncate } from './truncate.ts';
+import { cmdUpdate } from './update.ts';
+
+const USAGE = `Usage: node utils/test-results-db/cli.ts <command> [options]
+
+Compacts the per-run parquet CI artifacts into a single queryable DuckDB file.
+
+Commands:
+  download                 Fetch the latest maintained database artifact.
+                           Starts a fresh database if none exists yet.
+  update [options]         Ingest parquet artifacts that aren't in the database yet.
+    --lookback-days <n>    How many days back to scan (default 7).
+    --concurrency <n>      Parallel downloads per batch (default 16).
+    --stop-after-seen <n>  Stop after this many consecutive already-ingested
+                           artifacts (default 100). The list is newest-first, so
+                           this short-circuits the scan once caught up.
+  truncate --max-runs <n>  Keep only the newest <n> runs, delete the rest, compact.
+
+Environment:
+  GITHUB_TOKEN             Required for 'download' and 'update'.
+  TRDB_DB_PATH             Database file path (default utils/test-results-db/test-results.duckdb).
+`;
+
+function defaultDbPath(): string {
+  const here = path.dirname(fileURLToPath(import.meta.url));
+  return path.join(here, 'test-results.duckdb');
+}
+
+const UPDATE_OPTIONS = {
+  'lookback-days': { type: 'string' },
+  'concurrency': { type: 'string' },
+  'stop-after-seen': { type: 'string' },
+} as const;
+
+const TRUNCATE_OPTIONS = {
+  'max-runs': { type: 'string' },
+} as const;
+
+type Flags = Record<string, string | boolean | undefined>;
+
+function intFlag(flags: Flags, name: string, fallback: number): number {
+  const raw = flags[name];
+  if (raw === undefined)
+    return fallback;
+  const value = Number(raw);
+  if (!Number.isInteger(value) || value <= 0)
+    throw new Error(`--${name} must be a positive integer, got "${raw}"`);
+  return value;
+}
+
+function requireToken(): string {
+  const token = process.env.GITHUB_TOKEN;
+  if (!token)
+    throw new Error('GITHUB_TOKEN is required for this command.');
+  return token;
+}
+
+async function main(): Promise<void> {
+  const [command, ...rest] = process.argv.slice(2);
+  const dbPath = process.env.TRDB_DB_PATH || defaultDbPath();
+
+  switch (command) {
+    case 'download': {
+      await cmdDownload(dbPath, requireToken());
+      break;
+    }
+    case 'update': {
+      const { values } = parseArgs({ args: rest, options: UPDATE_OPTIONS, allowPositionals: false });
+      await cmdUpdate(dbPath, requireToken(), {
+        lookbackDays: intFlag(values, 'lookback-days', 7),
+        concurrency: intFlag(values, 'concurrency', 16),
+        stopAfterSeen: intFlag(values, 'stop-after-seen', 100),
+      });
+      break;
+    }
+    case 'truncate': {
+      const { values } = parseArgs({ args: rest, options: TRUNCATE_OPTIONS, allowPositionals: false });
+      if (values['max-runs'] === undefined)
+        throw new Error('truncate requires --max-runs <n>');
+      await cmdTruncate(dbPath, intFlag(values, 'max-runs', 0));
+      break;
+    }
+    case undefined:
+    case 'help':
+    case '--help':
+    case '-h': {
+      process.stdout.write(USAGE);
+      break;
+    }
+    default: {
+      process.stderr.write(`Unknown command: ${command}\n\n${USAGE}`);
+      process.exitCode = 1;
+    }
+  }
+}
+
+main().catch(error => {
+  console.error(error instanceof Error ? error.message : error);
+  process.exitCode = 1;
+});