E2E: Add Jetpack AI sidebar coverage

[kat3samsin]

Fixes #

Proposed changes

• Add connected self-hosted E2E coverage for the Jetpack AI Sidebar in the post and page editors.
• Verify selected-block, editor-field, image alt-text, review, and non-Big-Sky fallback tool contracts against streamed browser evidence and persisted editor state.
• Sanitize Playwright attachments so public CI artifacts exclude request URLs, payloads, response bodies, and raw console messages.
• Keep the AI suites out of normal pull request, push, and repository-dispatch matrices.
• Add manual workflow choices for the post editor, page editor, or both, with independent concurrency and 60-minute job timeouts.
• Document local setup, individual test commands, provider-bundle overrides, scope, and manual CI execution.

Related product discussion/links

• No public issue.

Does this pull request change what data or activity we track or use?

No. This adds automated test coverage and sanitized diagnostic artifacts only.

Testing instructions

• From projects/plugins/jetpack/tests/e2e, run pnpm typecheck.
• From the monorepo root, run pnpm lint-file projects/plugins/jetpack/tests/e2e/helpers/jetpack-ai-sidebar-evidence.ts projects/plugins/jetpack/tests/e2e/specs/ai-sidebar/abilities.test.ts projects/plugins/jetpack/tests/e2e/specs/ai-sidebar/evidence.test.ts .github/files/e2e-tests/e2e-matrix.js.
• Run pnpm test:run specs/ai-sidebar/evidence.test.ts; the environment prerequisites and evidence collector test should pass.
• Run pnpm test:run specs/ai-sidebar/abilities.test.ts --grep "A site without Big Sky updates the selected block with the fallback tool"; the connected-site prerequisites and fallback scenario should pass.
• Validate manual matrices with MANUAL_SUITE set to standard-suites, jetpack-ai-sidebar, jetpack-ai-sidebar-post, and jetpack-ai-sidebar-page; the standard selection should exclude both AI jobs and the AI selections should return two, one post, and one page job respectively.
• After merge, use Actions → E2E Tests → Run workflow and choose the desired Jetpack AI Sidebar suite. To exercise this branch before merge, use gh workflow run e2e-tests.yml --ref add/jetpack-ai-sidebar-e2e -f suite=jetpack-ai-sidebar-post -f wp_version=latest.

Current draft status

• TypeScript, ESLint, PHP syntax, workflow YAML parsing, matrix generation, changelog validation, the deterministic evidence test, and the live fallback scenario pass locally.
• All 19 non-review ability scenarios have passed locally across the post and page editors.
• The review ability cases currently receive over-limit responses or time out before the ability call because the shared test account has no available AI quota. A manual AI workflow run will report these failures normally.
• The post and page subsets have observed runtimes of roughly 20–25 minutes, below their 60-minute job limits.

[github-actions]

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

• To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack), and enable the add/jetpack-ai-sidebar-e2e branch.
• To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack add/jetpack-ai-sidebar-e2e

Interested in more tips and information?

• In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
• Read more about our development workflow here: PCYsg-eg0-p2
• Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
3. You can use GitHub's Reviewers functionality to request a review.
4. When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

---

Jetpack plugin:

No scheduled milestone found for this plugin.

If you have any questions about the release process, please ask in the #jetpack-releases channel on Slack.

[jp-launch-control]

Code Coverage Summary

This PR did not change code coverage!

That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷

Full summary · PHP report · JS report