GitHub regression testing

Start from a test plan. GitHub Actions runs that plan. It does not choose the tests. For a review of the pull request diff, use GitHub pull request testing.

Set it up

Test Run Action

Add the secret and a workflow that runs the plan.

Mobile build

Upload the build and run the plan against it.

The post-merge agent is separate. After a reviewed pull request merges, it can promote tests from that review into the suite. It does not start the test plan.

Configure Secrets

Add a secret to your GitHub repository (Settings → Secrets and variables → Actions). Create the token in Organization Settings → API Keys. Store it as QATECH_API_TOKEN. Project-scoped keys are bound to one project. Org-scoped keys can see multiple projects and need projectShortId on API calls.

Get the project short ID

Copy proj_… from the project URL or Settings (for example proj_abc123). Pass it as project_short_id on the action. See Understanding Different IDs.

Get the test plan short ID

Open the plan under Test Plans and copy pln_… from the URL (for example pln_abc123). See Find the test plan short ID.

Create Workflow

Create .github/workflows/qatech.yml:

name: QA.tech Tests
on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'pln_abc123'
          blocking: true

What works on GitHub

Capability	GitHub
Review on a preview URL, before merge	Not this guide. The Test Run Action does not post a pull request review.
Review after merge, with no preview URL	Not this guide. See GitHub pull request testing.
Run a regression test plan from CI	Yes. The Test Run Action runs the plan you name, including against a preview URL you pass in.
Run that plan against a mobile build	Yes. Upload the build and pass `applicationBuildShortId`. See Mobile regression testing.
Block the workflow until the plan finishes	Yes. Set blocking on the Test Run Action.
Schedule	On the test plan, with Manage Schedules. See Manual and scheduled runs.

What this does not do

The Test Run Action does not post a pull request review and does not pick tests from the diff.
Envoyer is a different trigger for the same kind of plan. It is not required when you have GitHub Actions. See Envoyer.

Patterns

Run on pull requests

name: PR Tests
on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'smoke-tests'
          blocking: true

Test preview deployments

name: Test Preview
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      preview_url: ${{ steps.deploy.outputs.url }}
    steps:
      - name: Deploy to Vercel
        id: deploy
        run: |
          # Your deployment logic
          echo "url=https://preview-${{ github.event.pull_request.number }}.vercel.app" >> $GITHUB_OUTPUT

  test:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'regression-suite'
          blocking: true
          applications_config: |
            {
              "applications": {
                "frontend-app": {
                  "environment": {
                    "url": "${{ needs.deploy.outputs.preview_url }}",
                    "name": "PR-${{ github.event.pull_request.number }}",
                    "customHeaders": [
                      {
                        "domains": ["*.vercel.app"],
                        "headers": {
                          "x-vercel-protection-bypass": "${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }}",
                          "x-vercel-set-bypass-cookie": "true"
                        }
                      }
                    ]
                  }
                }
              }
            }

This pattern passes the preview URL into the Test Run Action. It does not create GitHub deployment records. If you also want the GitHub App to pick up the same preview for automatic PR reviews, add the steps in GitHub Deployments.

Test a mobile pull request build

Native apps have no preview URL. Build the APK or simulator .app in the workflow, upload it, then pass applicationBuildShortId in applications_config instead of url. How application builds work is on Mobile regression testing. The shared upload script is on API pull request testing.

name: QA.tech mobile PR tests
on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build APK
        run: ./gradlew assembleDebug

      - name: Upload build to QA.tech
        id: upload
        env:
          QATECH_API_TOKEN: ${{ secrets.QATECH_API_TOKEN }}
          APK_PATH: app/build/outputs/apk/debug/app-debug.apk
        run: |
          APP_ID="app_gXeBl2"
          FILE_NAME=$(basename "$APK_PATH")
          UPLOAD_RESPONSE=$(curl -sSf -X POST "https://api.qa.tech/v1/applications/$APP_ID/builds/upload-url" \
            -H "Authorization: Bearer $QATECH_API_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"fileName\": \"$FILE_NAME\"}")
          UPLOAD_URL=$(echo "$UPLOAD_RESPONSE" | jq -r '.uploadUrl')
          BUILD_TOKEN=$(echo "$UPLOAD_RESPONSE" | jq -r '.buildToken')
          curl -sSf -X PUT "$UPLOAD_URL" \
            --upload-file "$APK_PATH" \
            -H "Content-Type: application/octet-stream"
          BUILD_RESPONSE=$(curl -sSf -X POST "https://api.qa.tech/v1/applications/$APP_ID/builds" \
            -H "Authorization: Bearer $QATECH_API_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"platform\": \"android\", \"buildToken\": \"$BUILD_TOKEN\"}")
          echo "build_short_id=$(echo "$BUILD_RESPONSE" | jq -r '.applicationBuildShortId')" >> "$GITHUB_OUTPUT"

      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'pln_abc123'
          blocking: true
          applications_config: |
            {
              "applications": {
                "app_gXeBl2": {
                  "environment": {
                    "applicationBuildShortId": "${{ steps.upload.outputs.build_short_id }}"
                  }
                }
              }
            }

Replace app_gXeBl2, pln_abc123, proj_abc123, and the APK path. Set blocking: true if the workflow should fail when tests fail. Native Android and iOS examples assume android/ or an .app already exist. Expo and React Native managed apps gitignore those folders. In CI, generate them before upload:

- run: npm ci
- run: npx expo prebuild --platform android --non-interactive --no-install
- run: cd android && ./gradlew assembleDebug --no-daemon

The debug APK path is typically android/app/build/outputs/apk/debug/app-debug.apk. EAS preview APKs also work if you wait for the build and download the artifact. Still upload a simulator or emulator binary, not a store .ipa. Full build steps are in Preparing Your App Build. iOS builds need a macOS runner and a simulator .app. Zip it, use "platform": "ios", and point BUILD_FILE at the archive. Full xcodebuild flags are in Mobile App Testing.

Persist environment custom headers

Pass customHeaders on any environment in applications_config to persist host-pattern header rules (auth and protection bypass). The Test Run Action and Change Review Action both accept this field. The preview deployment example shows it in a full workflow.

{
  "url": "https://preview.example.com",
  "name": "PR-123",
  "customHeaders": [
    {
      "domains": ["*.preview.example.com"],
      "headers": {
        "x-vercel-protection-bypass": "YOUR_SECRET",
        "x-vercel-set-bypass-cookie": "true"
      }
    }
  ]
}

customHeaders works with url, shortId, or applicationBuildShortId. Omit the field to leave stored headers unchanged. Pass [] to clear them. The Start Run API uses the same customHeaders shape on applications[].environment (array of application objects, not a map). See Environment custom headers.

Persist environment basic auth

Pass basicAuth next to customHeaders when the preview sits behind HTTP basic auth. Only hosts that match domains receive the credentials.

{
  "url": "https://pr-123.preview.example.com",
  "name": "PR-123",
  "basicAuth": [
    {
      "domains": ["*.preview.example.com"],
      "username": "preview",
      "password": "${{ secrets.PREVIEW_BASIC_AUTH_PASSWORD }}"
    }
  ]
}

basicAuth follows the same rules as customHeaders: omit it to leave stored rules unchanged, pass [] to clear them, or pass a list to replace them. See Environment basic auth.

Use action outputs

- uses: QAdottech/run-action@v4
  id: qatech
  with:
    project_short_id: 'proj_abc123'
    api_token: ${{ secrets.QATECH_API_TOKEN }}
    blocking: true

- name: Check Results
  if: steps.qatech.outputs.run_result == 'FAILED'
  run: echo "Tests failed! See ${{ steps.qatech.outputs.run_url }}"

Test Run Action reference

Inputs

Input	Description	Required	Default
`project_short_id`	QA.tech project short ID (for example `proj_abc123`)	Yes	-
`api_token`	QA.tech API token	Yes	-
`test_plan_short_id`	Test plan short ID to run	No	All tests
`blocking`	Wait for test results before completing	No	`false`
`applications_config`	JSON with application environment and device preset overrides. Optional `customHeaders` and `basicAuth` on each environment persist auth/bypass headers and basic auth; see Environment custom headers.	No	-
`api_url`	Custom API URL	No	`https://api.qa.tech`

Outputs

Output	Description
`run_created`	Whether the test run was created successfully
`run_short_id`	The short ID of the run
`run_url`	The URL of the run
`run_status`	Final status (`COMPLETED`, `ERROR`, `CANCELLED`, or `TIMED_OUT`) - only when `blocking: true`
`run_result`	Test result (`PASSED`, `FAILED`, `SKIPPED`) - only when `blocking: true`

Regression Testing

Agents

Choose your CI

GitHub regression testing

Set it up

Test Run Action

Mobile build

What works on GitHub

What this does not do

Patterns

Run on pull requests

Test preview deployments

Test a mobile pull request build

Persist environment custom headers

Persist environment basic auth

Use action outputs

Test Run Action reference

Inputs

Outputs

​Set it up

Test Run Action

Mobile build

​What works on GitHub

​What this does not do

​Patterns

​Run on pull requests

​Test preview deployments

​Test a mobile pull request build

​Persist environment custom headers

​Persist environment basic auth

​Use action outputs

​Test Run Action reference

​Inputs

​Outputs

Set it up

What works on GitHub

What this does not do

Patterns

Run on pull requests

Test preview deployments

Test a mobile pull request build

Persist environment custom headers

Persist environment basic auth

Use action outputs

Test Run Action reference

Inputs

Outputs