Skip to content

feat: quarantine debt CI summary — surface suppressed tests in every run #3

@EmanueleMinotto

Description

@EmanueleMinotto

Opportunity

After every run, Skipper knows exactly which tests it suppressed, why, and until when. Today that information goes nowhere. The opportunity: surface a quarantine debt score in every CI run — making the cost of hiding tests visible and creating accountability pressure that no other tool in this space provides.

Proposed feature

Emit a structured quarantine report at the end of every test session, targeting two outputs:

  • GitHub Actions job summary (via GITHUB_STEP_SUMMARY)
  • A local skipper-report.json artifact for archiving and downstream tooling

The report includes: number of currently suppressed tests, tests expiring this week, tests re-enabled in this run, and a total quarantine-days of debt score (sum of disabled_until - today across all active rows).

Wire the emission into:

  • skipper-pytest — via pytest_sessionfinish
  • skipper-unittest — via a tearDownModule / runner hook on SkipperTestCase
  • skipper-playwright — via the existing session-end hook in SkipperSyncTest
import json, os
from pathlib import Path

def emit_summary(report: dict) -> None:
    md = build_markdown_summary(report)
    summary_file = os.environ.get("GITHUB_STEP_SUMMARY")
    if summary_file:
        Path(summary_file).open("a").write(md)
    else:
        print(md)
    Path("skipper-report.json").write_text(json.dumps(report, indent=2, default=str))

The helper lives in skipper-core and is invoked once per session by each integration.

Why this is the unique moat

The quarantine-days of debt number grows when tests are disabled and shrinks when they expire. A team with 200 quarantine-days of debt is hiding a lot of reality. That number in every PR comment is a forcing function that costs zero additional tooling — and no other tool in this space does it.

Acceptance criteria

  • After every session, a markdown summary is written to GITHUB_STEP_SUMMARY if the env var is set
  • If GITHUB_STEP_SUMMARY is not set, the summary is printed to stdout
  • skipper-report.json is written with the full suppression state of the run
  • Report includes: currently suppressed count, expiring this week, re-enabled this run, quarantine-days of debt
  • Quarantine-days of debt = sum of (disabled_until - today).days across all active disabled rows
  • Implemented once in skipper-core, invoked by skipper-pytest, skipper-unittest, skipper-playwright

Effort estimate

~80 lines in skipper-core plus ~5 lines per integration package, stdlib only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions