GitHub App for PR Reviews

Get autonomous AI-powered test coverage and reviews on every pull request. The QA.tech GitHub App uses AI exploratory testing to analyze code changes, create missing tests, and post comprehensive reviews based on test results.

Integration modes:

AI exploratory (this page): Automatic AI-powered PR reviews - GitHub only
API-driven: Manual test plan triggers via GitHub Actions or GitLab CI

See CI/CD Integration Overview to understand both modes and choose the right approach for your needs.If you use GitLab merge requests, see GitLab Merge Request Reviews.

What It Does

Feature	Description
✅ Intelligent Test Selection	AI semantically matches PR changes to relevant tests (typically 5-15 tests selected)
✅ Gap-Only Test Generation	Creates 1-3 tests only when coverage gaps exist; most PRs create zero new tests
✅ Persistent Test Suite	Auto-generated tests become permanent regression tests for future PRs
✅ Preview Environment Testing	Tests against PR preview deployments
✅ Approval/Rejection	Posts reviews with pass/fail verdicts
✅ Manual Trigger	Comment `@qa.tech` on a PR to trigger, re-run, or steer a review on demand
❌ Backend-Only Testing	Requires UI access - can’t test microservices without frontend

How PR Reviews Work

PR opened/updated
  ↓
1. Classify Changes
   → User-facing? Continue
   → Docs/infra only? Skip testing, post info comment
  ↓
2. Assess Coverage
   → Find relevant existing tests
   → Identify gaps in coverage
  ↓
3. Create Tests (if needed)
   → Generate tests for untested functionality
   → Configure dependencies (e.g., login tests)
  ↓
4. Run Tests
   → Execute against PR preview environment
   → Wait for completion
  ↓
5. Post Review
   → ✅ Approve if all tests pass
   → ❌ Decline if tests fail
   → ℹ️ Informational if untestable

The agent posts reviews with:

Verdict: Pass/fail/unable to verify
What was tested: Description of coverage
Results summary: Patterns and themes
Test details: Table with individual test results

Reviews focus on test results only - no code quality opinions, implementation suggestions, or references to other bot comments.

What the GitHub App Writes

The GitHub App needs write permissions to communicate test results back to your PR. Here is what it posts and when:

Operation	When	Description
PR comment (in progress)	When the agent starts testing	Comment with link to the QA.tech conversation and details (branch, commit SHA, event type). Lets you track progress.
PR comment (quota limit)	When the organization is out of credits	Single comment with upgrade link. No tests run.
PR review	After tests complete	Native GitHub review with verdict (approve/request changes/comment), summary, test results table, and evaluation details. Replaces any pending review from the bot.
Reaction (eyes emoji)	When processing starts	Added to the triggering comment or PR to acknowledge it has been seen.
Commit status	During and after test run	GitHub check named QA.tech / PR Review on the PR commit. See PR Review check on GitHub below for when it appears and what each state means.

If you see permission errors when installing the app, ensure the repository grants write access for pull requests and statuses. The app only writes when tests run or when the organization is out of credits.

PR Review check on GitHub

QA.tech registers a GitHub check run named QA.tech / PR Review on PR commits. You can require it in branch protection as a merge gate. Whether a check appears on PR open or sync depends on your integration settings and how the review was triggered.

Situation	Check created on PR open/sync?	What you see
Auto-run on PRs enabled (default)	Yes	`in progress` while the agent reviews, then a final verdict (approve, request changes, or comment)
Auto-run on PRs disabled	No	No check until you explicitly trigger a review (see below)
Draft PR and Include draft PRs disabled	Yes (skipped)	Skipped check titled Draft PR - auto-review does not run on drafts until the PR is marked ready for review, or until you comment `@qa.tech`
Comment `@qa.tech` on the PR	Yes (when the review starts)	`in progress`, then final verdict
Change Review Action or Start change review chat API	Yes (when triggered)	`in progress`, then final verdict. Runs even when Auto-run on PRs is off

Auto-run on PRs controls only the automatic PR-open trigger. It does not block explicit triggers: @qa.tech comments, the Change Review Action, and the change-review API always start a review (and create or update the check) when invoked.When Auto-run on PRs is off, QA.tech does not post a skipped check that says auto-review is disabled. The PR simply has no QA.tech check until you trigger a review manually or from CI.

Integration settings

Configure these under Settings → Integrations → GitHub App for the repository:

Setting	Effect on the PR Review check
Auto-run on PRs	When on, creates the check on PR open/sync (subject to draft rules below). When off, no check on PR open - use `@qa.tech`, the Change Review Action, or the API.
Include draft PRs	When off (default), draft PRs get a skipped Draft PR check instead of an in-progress review. When on, draft PRs are reviewed the same as non-draft PRs (if auto-run is enabled).
Register a commit check	On by default. When off, QA.tech never posts the QA.tech / PR Review check for any trigger (auto-run, `@qa.tech`, the Change Review Action, or the API). Reviews still post the sticky PR comment and appear in the dashboard, but there is no merge gate to require.

Post-merge agent

When a pull request is merged, QA.tech can run an autonomous post-merge agent to keep your test suite healthy — before the internal review retrospective runs. Configure it under Settings → Integrations → GitHub App with the Run post-merge agent toggle, which reveals a Post-merge prompt you can edit. These settings live on the GitHub App integration (alongside Auto-run on PRs and other merge-review options). New GitHub integrations default the agent on with the default prompt prefilled. Existing integrations keep the agent off until you enable it (the default prompt is still prefilled in the form). You can also pass per-PR options when starting a change review via the Start change review chat API or the Change Review Action. See API and CI options below. The key idea: the prompt is the agent’s entire goal. The agent only changes tests when the prompt asks it to. If you point the prompt at something else (for example, “open a Linear issue summarizing the merge”), that is all it does — it will not touch your tests. The default prompt focuses on conservative regression maintenance. The agent runs in the same conversation as the PR review, so it already has the review history — the tests that ran, their results, and the diff — as context. It reuses your existing test tools; there is nothing new to configure beyond the prompt.

What it can do

Depending on your prompt, the agent can:

Promote review tests into the regression suite. Only tests this PR’s own reviews created — surfaced to the agent as “main candidates,” still drafts and not last seen failing — can be promoted. Promoting activates a test (and its dependencies) and labels it regression + auto-added automatically, so you can filter for them in the test list. Promoting nothing is a normal outcome.
Update tests whose behavior the merge changed so they match the merged state.
Convert to draft tests that the merge made stale or irrelevant — cautiously, and only with strong, diff-grounded evidence. If another active test depends on one being drafted, the agent must confirm and draft the whole chain together; it never archives tests automatically.
Add tests to a test plan (for example Smoke or Regression) when your prompt asks for it and defines the criteria.
Use connected integrations — for example open a tracker issue, post or read a Slack update (when Enable Slack Tools in Chat is on), or (when you have connected MCP servers) run one of their tools — when your prompt asks for something other than test maintenance.

Before promoting or creating anything, the agent checks for an equivalent existing test to avoid duplicates, and it only keeps tests that will run reliably in a regression suite (no hard-coded names, IDs, or other data tied to a single PR). Labels are applied automatically on promotion — you do not manage them in the prompt.

Example prompts

Use these as inspiration — copy, combine, or adapt them: Conservative regression maintenance (default):

Now that this pull request is merged, update tests the merge changed, promote the
main-flow candidate tests that passed during review, and convert clearly stale tests to
draft. Skip edge cases. Never keep tests with hard-coded PR-specific data.

Only cover normal end-user flows, not admin tooling:

Maintain tests for this merge, but only for flows a normal end user would hit. Do not
promote or create tests that exercise admin-only screens or internal tooling.

Add to the Smoke plan when criteria are met:

Keep the regression suite healthy for this merge. If a promoted test covers a critical,
high-traffic flow (login, checkout, signup) and is stable, also add it to the "Smoke"
test plan.

Be more aggressive about removing stale tests:

After this merge, be proactive about converting tests to draft when the PR removed or
replaced the feature they cover. Explain your reasoning for each one you draft.

Do something other than test maintenance:

When this PR merges, open a Linear issue in the QA team summarizing the user-facing
changes and any test gaps you noticed during review. Do not modify any tests.

Post a Slack summary after merge: Requires Slack tools in chat to be enabled for the project.

When this PR merges, post a short Slack summary of the user-facing changes to the
project's default Slack channel. Do not modify any tests.

The post-merge agent runs before the internal review retrospective, which always runs afterward regardless of whether the post-merge agent is enabled, skipped, or fails. Non-merged (closed) PRs skip the post-merge agent and go straight to the retrospective.

Post-merge via API or CI

When you trigger a merge review with POST /v1/chat/change-review (mode: "pr"), you can pass optional postMerge options. Those options are stored on the review conversation and override the GitHub App integration settings for that PR only when it merges.

curl -sSf -X POST "https://api.qa.tech/v1/chat/change-review" \
  -H "Authorization: Bearer $QATECH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "pr",
    "prUrl": "https://github.com/acme/web/pull/42",
    "vcsProviderId": "github",
    "applicationOverrides": [
      { "id": "app_...", "environmentId": "env_..." }
    ],
    "postMerge": {
      "enabled": true,
      "prompt": "After merge, promote stable main-flow review tests and draft clearly stale ones."
    }
  }'

Field	Required	Description
`postMerge.enabled`	Yes (when `postMerge` is sent)	`true` runs the post-merge agent after merge; `false` skips it even if the integration has it on
`postMerge.prompt`	No	Full goal for the agent. Omit or leave empty to use the default maintenance prompt

Omit postMerge entirely to use the GitHub App integration toggle and prompt. From CI, pass the same JSON body to the change-review API (or wrap it in your own workflow step). Promoted tests are labelled auto-added for provenance today; a durable, structured link back to the originating pull request, and per-test-plan opt-in controls for automatic additions, are planned follow-ups.

Test Selection and Creation

How Tests Are Selected

The agent uses AI to semantically match PR changes to test goals - not keyword matching. Example: PR changes checkout payment flow

Your project: 645 total tests

Agent analyzes:
├─ "Complete checkout with credit card" → RELEVANT ✓
├─ "Complete checkout with PayPal" → RELEVANT ✓
├─ "Verify payment confirmation email" → RELEVANT ✓
├─ "User profile photo upload" → NOT RELEVANT ✗
└─ "Search products by category" → NOT RELEVANT ✗

Selected: 12 tests covering payment & checkout flows

Selection mechanics:

Runs ALL tests it determines are relevant (no arbitrary limits)
Only considers tests with status='enabled'
More intelligent than running all tests every time

When Tests Are Created

Tests are created only when coverage gaps exist, such as when you’re developing a new feature not yet covered by your testing suite.

Writing acceptance criteria? Include test requirements in your Linear/Jira ticket or PR description. The agent reads this context and uses it to create more accurate tests for your new feature. See Recommended Workflow for New Features for details.

PR Type	Existing Tests Selected	New Tests Created	Total
Bug fix in login	3-7	0 (already covered)	3-7
Small feature	5-10	1-2 (fill gaps)	6-12
Major feature	15-25	3-5 (new functionality)	18-30
Refactor	10-20	0 (no new behavior)	10-20
Docs/infra only	0	0 (untestable via UI)	0

Example: PR adds Apple Pay to checkout

Agent assesses existing coverage:
├─ "Complete checkout with credit card" ✓ Exists
├─ "Complete checkout with PayPal" ✓ Exists
└─ Apple Pay integration ✗ Gap identified

Decision: Create 1 new test
→ "Complete checkout with Apple Pay"

Created tests:

Stay in your suite labeled ‘ephemeral’
Available for future PRs
Manage in Settings → Test Cases (filter by ‘ephemeral’ to find them)

Example lifecycle:

PR #42: Add Apple Pay
├─ Agent creates "Complete checkout with Apple Pay"
├─ Test runs on PR #42 ✅
└─ Test persists with 'ephemeral' label

PR #58: Refactor checkout UI
├─ Agent finds existing test
├─ Runs it (no new test created) ✅
└─ Your suite now protects against Apple Pay regressions

The agent prevents duplicates by reading all existing tests first and using semantic deduplication. If one slips through, simply delete it in the UI. Self-limiting: As your test suite grows, fewer tests are created automatically - better coverage means fewer gaps. Most PRs (bug fixes, refactors) create zero new tests.

Mobile and Responsive Testing: When a PR mentions mobile, tablet, or responsive design changes, the AI agent may automatically test with appropriate device presets. The agent detects keywords like “mobile”, “tablet”, “responsive”, or “Safari mobile” and can override device presets when creating or running tests. See Device Presets for configuring device settings.

How to Set It Up

Install GitHub App

Go to Settings → Organization → Connections and add the GitHub App connection. Follow the OAuth flow to grant access to your repositories.

Configure Repository

Navigate to Settings → Integrations and select the repository you want to enable PR reviews for. PR reviews are enabled automatically once you select a repository. Optional: Add review context to guide the agent (e.g., “Focus on security vulnerabilities” or “Validate accessibility standards”). This appears in the integration settings.

Configure Preview Deployments (if needed)

If you use Vercel, Netlify, Render, Railway, or Fly.io, your preview deployments are detected automatically. Skip to step 4.

QA.tech creates preview environment records in your project when it detects your preview deployments. Your CI/CD platform handles the actual deployment. QA.tech tests against the preview URLs. See Preview Environments.

For GitHub Actions or other custom CI/CD: QA.tech finds preview URLs through GitHub’s Deployments API. After your workflow deploys, it must create a deployment on the PR head commit, set the latest status to success, and put the preview URL in environment_url. Follow GitHub Deployments for the Actions recipe, required fields, and permissions. Any CI that can call the GitHub REST API can create the same records.

Map Environments (optional)

If you have multiple Applications (for example frontend and admin), map GitHub deployment environments to the matching QA.tech applications under Settings → Integrations → GitHub App → Mapped GitHub Environments. Use a stable name per app (such as Preview - frontend). See Map environments to applications.

Without GitHub deployments, QA.tech cannot test against PR-specific URLs and uses your default environment.

Create a Pull Request

Once configured, the agent automatically runs on PRs:

Detects code changes when PRs are opened or updated
Determines which tests are relevant
Creates new tests for untested functionality
Runs all relevant tests against the PR preview
Posts a review with approval or decline based on results

Triggering a PR Review

Automatic trigger (simple)

With Run automatically on PRs enabled, QA.tech starts a review when a PR is opened or updated and its preview deployment is ready. This is the simplest option once a repository is configured.

Automatic triggering works best when you have a single application deployed via GitHub Deployments (or a provider that creates them, such as Netlify or Vercel) and you don’t need extra control over when reviews run. If you have multiple applications per PR, don’t use deployment-based previews, or need to gate reviews on labels, paths, or a specific deploy job, use the GitHub Action trigger instead.

GitHub Action trigger (recommended)

For the most control, trigger reviews from your CI with the Change Review Action. It runs the same review agent and posts the same native PR review as the automatic trigger, but lets you decide exactly when it fires and pass an applications_config per PR - so you can route each application to its own preview URL, gate reviews on labels or paths, or order the review after a specific deploy job. Turn off Run automatically on PRs in Settings → Integrations → GitHub App when driving reviews from the action, so each PR isn’t reviewed twice. See GitHub Actions for setup details.

Manual trigger with an `@qa.tech` comment

You can trigger a review at any time by commenting @qa.tech on the pull request. This works on the main PR conversation and on inline code review comments, even when Run automatically on PRs is turned off or a previous review was abandoned. The mention is case-insensitive (@QA.tech, @qatech, and similar variants all match). When QA.tech picks up the comment, it reacts with an eyes emoji to acknowledge it.

Comment	What happens
`@qa.tech`	Re-runs the review. QA.tech waits for a ready preview deployment on the PR’s latest commit, then starts automatically.
`@qa.tech test the checkout flow`	Runs a focused review. Any instructions after the mention are passed to the agent to steer what it tests.
`@qa.tech https://preview.example.com`	Runs immediately against the URL in the comment, which is treated as the preview deployment to test (skips waiting for a deployment).

Without a URL in the comment, QA.tech waits for a ready preview deployment on the PR’s latest commit. It posts a short “waiting for a preview deployment” comment and starts the review automatically once one is ready. Include a preview URL in the comment to start immediately against that environment.

Re-run from the GitHub checks UI

Every review posts a status check on the PR commit. You can re-trigger a review by using GitHub’s Re-run button on the QA.tech check, the same way you re-run any other CI check.

Common Questions

Will auto-generated tests pollute my test suite? No. The agent only creates tests for coverage gaps - most PRs (bug fixes, refactors) create zero new tests. Even major features typically add 3-5 focused tests. The system is self-limiting: better coverage → fewer gaps → less generation. Can I control which tests run? The GitHub App selects tests autonomously, but you can steer a run by commenting @qa.tech <instructions> on the PR (e.g. @qa.tech test the checkout flow) - see Triggering a PR Review. For full manual test selection, use GitHub Actions instead. You can use both: automatic PR reviews + manual deep testing on-demand. How do I trigger or re-run a review manually? Comment @qa.tech on the pull request. Add instructions to focus the review (@qa.tech test the checkout flow) or a preview URL to test a specific environment (@qa.tech https://preview.example.com). You can also use GitHub’s Re-run button on the QA.tech check. See Triggering a PR Review. My project has multiple applications per PR and the App’s environment mapping isn’t a good fit. What can I do? Install the GitHub App as usual, then turn off Auto-run on PRs in Settings → Integrations → GitHub App and drive reviews from CI with the Change Review Action. The action runs the same review agent (and posts the same native PR review) but lets you pass an applications_config per PR, so you can route each application short ID to its own preview URL without relying on the App’s static environment mapping. With auto-run off, no QA.tech / PR Review check appears on PR open; the check is created when the action runs. Can I gate reviews on labels, paths, or a specific deploy job? Yes. Use the Change Review Action instead of the automatic trigger and put the gating logic in your workflow (if: conditions on labels or paths, needs: on a deploy job, etc.). Disable Auto-run on PRs if you want only the action to fire; the action posts the native PR review the same way the App does. PR reviews aren’t posting - what should I check?

Verify GitHub App is installed and has repository access
Check repository is configured in QA.tech Settings → Integrations
Ensure PR has user-facing changes (docs/infra-only PRs are skipped)
If you see 403 Forbidden, check that the app has write permissions for pull requests and statuses

We ran out of credits - what happens? QA.tech posts a single comment on the PR with an upgrade link. No tests run. No review is posted. How do I run PR reviews on a native iOS or Android app? Automatic GitHub App reviews look for a preview URL. A mobile PR is a new binary, not a website. Upload the PR build from CI and start the Change Review Action (or a test plan) with applicationBuildShortId. Turn off Auto-run on PRs for that repository so the App does not also review the default stored build. See PR Testing for Mobile Apps. Tests running against wrong URL?

Map GitHub environments to applications in Settings → Integrations → GitHub App → Mapped GitHub Environments
Verify your CI/CD creates GitHub deployment records with environment_url set to the preview URL. See GitHub Deployments.
Check environment names match between GitHub and QA.tech

Agent created irrelevant tests?

Add review context with specific guidelines in integration settings
Update existing tests to better cover the functionality
The agent learns from your existing test patterns

CI/CD Integration Overview - Learn about integration modes and capabilities
GitHub Actions - API-driven testing with manual control over test plans
GitHub Deployments - Publish preview URLs from GitHub Actions for automatic PR reviews
Test Plans - Organize tests for API-driven runs
PR Testing for Mobile Apps - CI upload and pinned builds for native PRs

Get Started

Core Concepts

Configuration

Best Practices

Test Features

Integrations

Applications

CLI

GitHub App for PR Reviews

What It Does

How PR Reviews Work

What the GitHub App Writes

PR Review check on GitHub

Integration settings

Post-merge agent

What it can do

Example prompts

Post-merge via API or CI

Test Selection and Creation

How Tests Are Selected

When Tests Are Created

How to Set It Up

Triggering a PR Review

Automatic trigger (simple)

GitHub Action trigger (recommended)

Manual trigger with an `@qa.tech` comment

Re-run from the GitHub checks UI

Common Questions

​What It Does

​How PR Reviews Work

​What the GitHub App Writes

​PR Review check on GitHub

​Integration settings

​Post-merge agent

​What it can do

​Example prompts

​Post-merge via API or CI

​Test Selection and Creation

​How Tests Are Selected

​When Tests Are Created

​How to Set It Up

​Triggering a PR Review

​Automatic trigger (simple)

​GitHub Action trigger (recommended)

​Manual trigger with an @qa.tech comment

​Re-run from the GitHub checks UI

​Common Questions

​Related Documentation

What It Does

How PR Reviews Work

What the GitHub App Writes

PR Review check on GitHub

Integration settings

Post-merge agent

What it can do

Example prompts

Post-merge via API or CI

Test Selection and Creation

How Tests Are Selected

When Tests Are Created

How to Set It Up

Triggering a PR Review

Automatic trigger (simple)

GitHub Action trigger (recommended)

Manual trigger with an `@qa.tech` comment

Re-run from the GitHub checks UI

Common Questions

Related Documentation