> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qa.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# Regression review agent

> How the agent investigates every test plan result, retries suspected flakes, and writes one verdict.

Every time a test plan runs, the results get a regression review. When the tests finish, a review agent looks at each test that did not pass, works out why, reruns the tests where a rerun can answer the question, and writes one review: what works, what is broken, and what could not be verified. You read the review instead of a list of red tests.

A red test is not a finding until the agent has looked at it. A browser agent is nondeterministic, so one failed attempt is weak evidence. The review separates real product bugs from flaky tests, outdated tests, and setup problems such as an expired login.

## What gets a review

A test plan, however it starts:

* The **Run Tests** button on a test plan
* A schedule
* CI, including the [GitHub Test Run Action](/regression-testing/github), [GitLab](/regression-testing/gitlab), [Bitrise](/regression-testing/bitrise) and [Envoyer](/regression-testing/envoyer)
* The [Start Run API](/api-reference/runs/start-test-run) and [`qatech run`](/cli/commands/run)

A single test started on its own, and tests a chat starts itself, do not get a separate review. A [change review](/pr-testing/overview) of a pull request has its own review.

## The flow

```mermaid theme={null}
graph TD
    A["Test plan starts<br/>(Run Tests, schedule, CI, API)"] --> B["Review chat attached"]
    B --> C["Tests execute"]
    C --> D["Tests finish, review agent wakes"]
    D --> E["Investigate every test that did not pass"]
    E --> F{"Would a rerun<br/>answer the question?"}
    F -->|Yes| G["Rerun those tests"]
    G --> D
    F -->|No| H{"Only a person<br/>can fix it?"}
    H -->|Yes| I["File a todo"]
    H -->|No| J["Classify the test"]
    I --> J
    J --> K["Record the review<br/>(verdict, failures, flaky, setup, tests to update)"]
    K --> L["Analysis tab, notifications and CI status"]
```

<Steps>
  <Step title="A review chat is attached">
    When the test plan starts, QA.tech opens a chat for it. If you started it
    with **Run Tests**, you land in that chat and it appears in your chat
    history. A test plan started from a schedule, CI or the API gets a
    background chat; open it from the test results.
  </Step>

  <Step title="The tests execute">
    Nothing changes about how the tests execute: same tests, same environment,
    same parallelism and dependencies. The agent waits until every test has
    finished.
  </Step>

  <Step title="The agent gathers the results">
    Passed tests count as verified. For every test that did not pass, the agent
    reads the test's assessment and its recent history. A test that passed most
    of the recent times and failed now is most likely flaky. A test that has
    failed consistently is more likely a real problem. Where the assessment is
    unclear, the agent opens the screenshots, steps and evidence, and compares
    them with the last passing result of the same test.
  </Step>

  <Step title="The agent investigates each failure">
    Failures that share one root cause are grouped. When login fails, every
    test behind it fails too: that is one finding, and the dependent tests are
    unverifiable, not broken. For each failure, the agent checks the evidence
    against concrete causes:

    * **Product bug**: the app misbehaves, such as an error page, wrong data, or a control that does nothing.
    * **Outdated test**: the app changed on purpose and the test still describes the old flow.
    * **Test data or state**: the data the test needs is missing, already exists, or another test in the same plan changed it.
    * **Timing**: the app was slow and the agent acted before the page was ready.
    * **Agent mistake**: the app was fine, but the agent clicked the wrong element or misjudged the result.
    * **Environment**: the environment was down or redeploying, and many unrelated tests failed at once.
    * **Credentials or third party**: login rejected, an expired token or license, an OTP or captcha, or an external service failing.

    All results from the test plan count as evidence. If login passed in other
    tests, the failure is not a credentials problem.
  </Step>

  <Step title="The agent reruns to test a hypothesis">
    Each rerun answers a question. A timing or agent problem is rerun as is,
    and an agent that could not operate the UI can be switched to a more
    capable agent for that attempt. A suspected data clash is rerun alone, so
    no other test touches its data. A test blocked by an upstream failure is
    rerun through that upstream test. A test with no recent passes and a clear
    app failure is not rerun: it is very likely a real bug.

    Each test is rerun at most twice per review, in at most three rounds. A
    test that fails the same way twice is not flaky. A rerun replaces that
    test's result in place, so the test plan results stay in one place.
  </Step>

  <Step title="Blockers go to a person">
    When only a person can fix the cause (missing or expired credentials, a
    misconfigured environment, missing seed data, a license to renew), the
    agent files a todo under **Todos**, one per need rather than one per test.
    A matching open todo gets a new occurrence instead of a duplicate. Product
    bugs, flaky tests and agent mistakes never become todos.
  </Step>

  <Step title="The agent records the review">
    The review is recorded once every failed test has a cause backed by
    evidence, or is marked unverifiable with what a person needs to do. The
    agent never edits, creates or archives tests during a review. It reports
    what should change.
  </Step>
</Steps>

## What the review contains

The review opens with a verdict and a one-line headline, such as "2 issues to fix" or "Blocked by an expired license", followed by a short summary of what the plan covered and what the reruns showed.

The agent decides the verdict from what the test plan is meant to prove, not by counting red tests.

| Verdict | Meaning |
| - | - |
| **Verified** | The flows the plan covers work. Flaky tests that passed on a rerun do not prevent this. |
| **Needs attention** | No confirmed breakage, but outdated or unverifiable tests leave part of the plan unproven. |
| **Broken** | At least one confirmed product bug in a flow the plan covers, with the failing step and a screenshot as proof. |

The sections under the verdict:

| Section | What it lists |
| - | - |
| **Issues** | One entry per root cause: what happens, expected vs actual, and links to the affected test results. |
| **Setup, not a code bug** | Environment, data or credential problems that blocked tests, with the ask for a person and the todo. |
| **Flaky, passed on retry** | Tests that failed first and passed on a rerun, with the likely cause and a suggested fix for the test. |
| **Tests to update** | Tests where the app changed on purpose, with what the test should say instead. |

Every finding links to the test results it is based on. Pass and fail counts always come from the real results, not from the review.

## Where you see it

* **Results**: open the test plan's results in **Results**. The **Analysis** tab opens by default once the tests finish. While the agent is still working, it shows what the agent is doing right now. The **Review chat** link in the header opens the conversation.
* **Review chat**: the agent's reasoning, every rerun it started and why, and the review as a card at the end with a **View run** link back to the results. Ask follow-up questions in the same chat.
* **Notifications**: Slack, Microsoft Teams and email notifications wait until the review is recorded, so you get one message after the reruns instead of a red result that a rerun clears.
* **CI and the API**: while the review is in progress, the [Get Run](/api-reference/runs/get-run) status stays `RUNNING` and the GitHub commit status stays pending. A test that passes on a rerun counts as passed, so a test plan that was red only because of flakes reports green. A **Broken** verdict always reports failed. The GitHub commit status shows the verdict and headline.

If a review does not finish within 4 hours of the last attempt, notifications and CI get the raw results instead, so a stuck review never blocks a pipeline.

## Start a review from chat

To start a test plan and review it from a chat, type `/regression` and name the plan, for example `/regression Smoke plan on staging`. The agent starts the test plan, replies once with the plan, environment and test count, and waits until the tests finish. It then follows the same steps and writes the review in that chat and on the **Analysis** tab.

Notifications and CI status for a test plan started this way do not wait for the review.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.