GitHub regression testing

Start from a test plan. GitHub Actions runs that plan. It does not choose the tests. For a review of the pull request diff, use GitHub pull request testing.

Set it up

Create a test plan

Pick the tests, then copy the plan short ID (pln_…).

Test Run Action

Run the plan from a workflow, with blocking and preview URL overrides.

Mobile build

Upload the build and run the plan against it.

Schedule

Run the plan on a cron schedule from Manage Schedules.

The post-merge agent is separate. After a reviewed pull request merges, it can promote tests from that review into the suite. It does not start the test plan.

What works on GitHub

Capability	GitHub
Review on a preview URL, before merge	Not this guide. The Test Run Action does not post a pull request review.
Review after merge, with no preview URL	Not this guide. See GitHub pull request testing.
Run a regression test plan from CI	Yes. The Test Run Action runs the plan you name, including against a preview URL you pass in.
Run that plan against a mobile build	Yes. Upload the build and pass `applicationBuildShortId`. See Mobile regression testing.
Block the workflow until the plan finishes	Yes. Set blocking on the Test Run Action.
Schedule	On the test plan, with Manage Schedules. GitHub is not required for that.

What this does not do

The Test Run Action does not post a pull request review and does not pick tests from the diff.
Envoyer is a different trigger for the same kind of plan. It is not required when you have GitHub Actions. See Envoyer.
A schedule configured only in GitHub Actions is optional. The product schedule is Manage Schedules on the test plan.

Post-merge agent

When a pull request is merged, QA.tech can run an autonomous post-merge agent to keep your test suite healthy — before the internal review retrospective runs. Configure it under Settings → Integrations → GitHub App with the Run post-merge agent toggle, which reveals a Post-merge prompt you can edit. These settings live on the GitHub App integration (alongside Auto-run on PRs and other merge-review options). New GitHub integrations default the agent on with the default prompt prefilled. Existing integrations keep the agent off until you enable it (the default prompt is still prefilled in the form). You can also pass per-PR options when starting a change review via the Start change review chat API or the Change Review Action. See API and CI options below. The key idea: the prompt is the agent’s entire goal. The agent only changes tests when the prompt asks it to. If you point the prompt at something else (for example, “open a Linear issue summarizing the merge”), that is all it does — it will not touch your tests. The default prompt focuses on conservative regression maintenance. The agent runs in the same conversation as the PR review, so it already has the review history — the tests that ran, their results, and the diff — as context. It reuses your existing test tools; there is nothing new to configure beyond the prompt.

What it can do

Depending on your prompt, the agent can:

Promote review tests into the regression suite. Only tests this PR’s own reviews created — surfaced to the agent as “main candidates,” still drafts and not last seen failing — can be promoted. Promoting activates a test (and its dependencies) and labels it regression + auto-added automatically, so you can filter for them in the test list. Promoting nothing is a normal outcome.
Update tests whose behavior the merge changed so they match the merged state.
Convert to draft tests that the merge made stale or irrelevant — cautiously, and only with strong, diff-grounded evidence. If another active test depends on one being drafted, the agent must confirm and draft the whole chain together; it never archives tests automatically.
Add tests to a test plan (for example Smoke or Regression) when your prompt asks for it and defines the criteria.
Use connected integrations — for example open a tracker issue, post or read a Slack update (when Enable Slack Tools in Chat is on), or (when you have connected MCP servers) run one of their tools — when your prompt asks for something other than test maintenance.

Before promoting or creating anything, the agent checks for an equivalent existing test to avoid duplicates, and it only keeps tests that will run reliably in a regression suite (no hard-coded names, IDs, or other data tied to a single PR). Labels are applied automatically on promotion — you do not manage them in the prompt.

Example prompts

Use these as inspiration — copy, combine, or adapt them: Conservative regression maintenance (default):

Now that this pull request is merged, update tests the merge changed, promote the
main-flow candidate tests that passed during review, and convert clearly stale tests to
draft. Skip edge cases. Never keep tests with hard-coded PR-specific data.

Only cover normal end-user flows, not admin tooling:

Maintain tests for this merge, but only for flows a normal end user would hit. Do not
promote or create tests that exercise admin-only screens or internal tooling.

Add to the Smoke plan when criteria are met:

Keep the regression suite healthy for this merge. If a promoted test covers a critical,
high-traffic flow (login, checkout, signup) and is stable, also add it to the "Smoke"
test plan.

Be more aggressive about removing stale tests:

After this merge, be proactive about converting tests to draft when the PR removed or
replaced the feature they cover. Explain your reasoning for each one you draft.

Do something other than test maintenance:

When this PR merges, open a Linear issue in the QA team summarizing the user-facing
changes and any test gaps you noticed during review. Do not modify any tests.

Post a Slack summary after merge: Requires Slack tools in chat to be enabled for the project.

When this PR merges, post a short Slack summary of the user-facing changes to the
project's default Slack channel. Do not modify any tests.

The post-merge agent runs before the internal review retrospective, which always runs afterward regardless of whether the post-merge agent is enabled, skipped, or fails. Non-merged (closed) PRs skip the post-merge agent and go straight to the retrospective.

Post-merge via API or CI

When you trigger a merge review with POST /v1/chat/change-review (mode: "pr"), you can pass optional postMerge options. Those options are stored on the review conversation and override the GitHub App integration settings for that PR only when it merges.

curl -sSf -X POST "https://api.qa.tech/v1/chat/change-review" \
  -H "Authorization: Bearer $QATECH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "pr",
    "prUrl": "https://github.com/acme/web/pull/42",
    "vcsProviderId": "github",
    "applicationOverrides": [
      { "id": "app_...", "environmentId": "env_..." }
    ],
    "postMerge": {
      "enabled": true,
      "prompt": "After merge, promote stable main-flow review tests and draft clearly stale ones."
    }
  }'

Field	Required	Description
`postMerge.enabled`	Yes (when `postMerge` is sent)	`true` runs the post-merge agent after merge; `false` skips it even if the integration has it on
`postMerge.prompt`	No	Full goal for the agent. Omit or leave empty to use the default maintenance prompt

Omit postMerge entirely to use the GitHub App integration toggle and prompt. From CI, pass the same JSON body to the change-review API (or wrap it in your own workflow step). Promoted tests are labelled auto-added for provenance today; a durable, structured link back to the originating pull request, and per-test-plan opt-in controls for automatic additions, are planned follow-ups.

Test run implementation patterns

Run on pull requests

name: PR Tests
on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'smoke-tests'
          blocking: true

Test preview deployments

name: Test Preview
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      preview_url: ${{ steps.deploy.outputs.url }}
    steps:
      - name: Deploy to Vercel
        id: deploy
        run: |
          # Your deployment logic
          echo "url=https://preview-${{ github.event.pull_request.number }}.vercel.app" >> $GITHUB_OUTPUT

  test:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'regression-suite'
          blocking: true
          applications_config: |
            {
              "applications": {
                "frontend-app": {
                  "environment": {
                    "url": "${{ needs.deploy.outputs.preview_url }}",
                    "name": "PR-${{ github.event.pull_request.number }}",
                    "customHeaders": [
                      {
                        "domains": ["*.vercel.app"],
                        "headers": {
                          "x-vercel-protection-bypass": "${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }}",
                          "x-vercel-set-bypass-cookie": "true"
                        }
                      }
                    ]
                  }
                }
              }
            }

This pattern passes the preview URL into the Test Run Action. It does not create GitHub deployment records. If you also want the GitHub App to pick up the same preview for automatic PR reviews, add the steps in GitHub Deployments.

Test a mobile PR build

Native apps have no preview URL. Build the APK or simulator .app in the workflow, upload it, then pass applicationBuildShortId in applications_config instead of url. Full workflows (test plan and change review) are in PR Testing for Mobile Apps.

Persist environment custom headers

Pass customHeaders on any environment in applications_config to persist host-pattern header rules (auth and protection bypass). The Test Run Action and Change Review Action both accept this field. The preview deployment example shows it in a full workflow.

{
  "url": "https://preview.example.com",
  "name": "PR-123",
  "customHeaders": [
    {
      "domains": ["*.preview.example.com"],
      "headers": {
        "x-vercel-protection-bypass": "YOUR_SECRET",
        "x-vercel-set-bypass-cookie": "true"
      }
    }
  ]
}

customHeaders works with url, shortId, or applicationBuildShortId. Omit the field to leave stored headers unchanged. Pass [] to clear them. The Start Run API uses the same customHeaders shape on applications[].environment (array of application objects, not a map). See Environment custom headers.

Scheduled testing

name: Nightly Tests
on:
  schedule:
    - cron: '0 2 * * *' # Daily at 2 AM UTC

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'full-regression'
          blocking: false

Use action outputs

- uses: QAdottech/run-action@v4
  id: qatech
  with:
    project_short_id: 'proj_abc123'
    api_token: ${{ secrets.QATECH_API_TOKEN }}
    blocking: true

- name: Check Results
  if: steps.qatech.outputs.run_result == 'FAILED'
  run: echo "Tests failed! See ${{ steps.qatech.outputs.run_url }}"

Regression Testing

Agents

Choose your CI

GitHub regression testing

Set it up

Create a test plan

Test Run Action

Mobile build

Schedule

What works on GitHub

What this does not do

Post-merge agent

What it can do

Example prompts

Post-merge via API or CI

Test run implementation patterns

Run on pull requests

Test preview deployments

Test a mobile PR build

Persist environment custom headers

Scheduled testing

Use action outputs

​Set it up

Create a test plan

Test Run Action

Mobile build

Schedule

​What works on GitHub

​What this does not do

​Post-merge agent

​What it can do

​Example prompts

​Post-merge via API or CI

​Test run implementation patterns

​Run on pull requests

​Test preview deployments

​Test a mobile PR build

​Persist environment custom headers

​Scheduled testing

​Use action outputs

Set it up

What works on GitHub

What this does not do

Post-merge agent

What it can do

Example prompts

Post-merge via API or CI

Test run implementation patterns

Run on pull requests

Test preview deployments

Test a mobile PR build

Persist environment custom headers

Scheduled testing

Use action outputs