feat(ci): adds to the GitHub QA Dashboard another dimension crossing flakies & failures

[daniel-herrero]

Done Definition Checks

Taiga URL

Taiga Ticket: 1623

Description

This PR adds a second dimension to the GitHub QA Dashboard from #772: cross-referencing the automated regression suite against which of those tests are actually reliable in CI.

What problem are you trying to solve?

The coverage-vs-bugs dashboard answers "do we have a test here?" but not "can we trust it?" — a flaky or chronically-failing test inflates the automated-coverage numbers without actually giving the confidence those numbers imply. There was also no easy way to see, at a glance, which regression tests are the noisiest/most prone to false failures, even though that data effectively already existed in the daily Playwright results.json reports — it just wasn't being aggregated or surfaced anywhere.

Solution

How did you solve the problem?

• flaky-tally.ts — dependency-free script that walks a Playwright results.json, extracts the Qase ID for every test that flaked or failed (same annotation/title parsing as triage.ts), and merges the day's occurrences into a rolling flaky-history.json aggregate. Only tests that actually flaked or failed get an entry, so the file stays proportional to the tests that are a problem, not the full Qase inventory; occurrences older than the (default 30-day) window are dropped on every merge.
• playwright_pre_daily.yml — new "Update flaky-test history" step runs flaky-tally.ts right after each daily run, pulling/pushing flaky-history.json to s3://kaleidos-qa-reports/ (public GET, no credentials needed to read it back).
• github_dashboard.ts — fetches that aggregate and joins it against the regression tests by Qase ID (buildReliabilityRows), writing a new flaky-tests.csv and a reliability block in the dashboard's data payload. Default sort is most-failed-first, ties broken by most-flaky.
• dashboard_template.html — new "Test reliability" section: click-to-sort columns, a KPI tile, and a "Last seen" link straight to that occurrence's run report (best-effort — raw run reports expire off S3 after ~10 days, so older links may 404).

How to test

• Check the code
• It complies with the test conventions
• Run the dashboard locally (QASE_TOKEN=... GITHUB_TOKEN=... npx tsx scripts/github-dashboard/github_dashboard.ts) and confirm the "Test reliability" section renders with real data
• Confirm sorting works on every column, and that "Last seen" links to a live (or gracefully-expired) run report
• Optionally run github-dashboard.yml manually against this branch with s3_prefix: github-dashboard-test to sanity-check the live-rendered dashboard without touching the real latest/ output

Screenshots 📸 (optional)

Anything Else? (optional)

[estefafdez]

LGTM!