Skip to main content
Every time a test plan runs, the results get a regression review. When the tests finish, a review agent looks at each test that did not pass, works out why, reruns the tests where a rerun can answer the question, and writes one review: what works, what is broken, and what could not be verified. You read the review instead of a list of red tests. A red test is not a finding until the agent has looked at it. A browser agent is nondeterministic, so one failed attempt is weak evidence. The review separates real product bugs from flaky tests, outdated tests, and setup problems such as an expired login.

​
What gets a review

A test plan, however it starts: A single test started on its own, and tests a chat starts itself, do not get a separate review. A change review of a pull request has its own review.

​
The flow

1

A review chat is attached

When the test plan starts, QA.tech opens a chat for it. If you started it with Run Tests, you land in that chat and it appears in your chat history. A test plan started from a schedule, CI or the API gets a background chat; open it from the test results.
2

The tests execute

Nothing changes about how the tests execute: same tests, same environment, same parallelism and dependencies. The agent waits until every test has finished.
3

The agent gathers the results

Passed tests count as verified. For every test that did not pass, the agent reads the test’s assessment and its recent history. A test that passed most of the recent times and failed now is most likely flaky. A test that has failed consistently is more likely a real problem. Where the assessment is unclear, the agent opens the screenshots, steps and evidence, and compares them with the last passing result of the same test.
4

The agent investigates each failure

Failures that share one root cause are grouped. When login fails, every test behind it fails too: that is one finding, and the dependent tests are unverifiable, not broken. For each failure, the agent checks the evidence against concrete causes:
  • Product bug: the app misbehaves, such as an error page, wrong data, or a control that does nothing.
  • Outdated test: the app changed on purpose and the test still describes the old flow.
  • Test data or state: the data the test needs is missing, already exists, or another test in the same plan changed it.
  • Timing: the app was slow and the agent acted before the page was ready.
  • Agent mistake: the app was fine, but the agent clicked the wrong element or misjudged the result.
  • Environment: the environment was down or redeploying, and many unrelated tests failed at once.
  • Credentials or third party: login rejected, an expired token or license, an OTP or captcha, or an external service failing.
All results from the test plan count as evidence. If login passed in other tests, the failure is not a credentials problem.
5

The agent reruns to test a hypothesis

Each rerun answers a question. A timing or agent problem is rerun as is, and an agent that could not operate the UI can be switched to a more capable agent for that attempt. A suspected data clash is rerun alone, so no other test touches its data. A test blocked by an upstream failure is rerun through that upstream test. A test with no recent passes and a clear app failure is not rerun: it is very likely a real bug.Each test is rerun at most twice per review, in at most three rounds. A test that fails the same way twice is not flaky. A rerun replaces that test’s result in place, so the test plan results stay in one place.
6

Blockers go to a person

When only a person can fix the cause (missing or expired credentials, a misconfigured environment, missing seed data, a license to renew), the agent files a todo under Todos, one per need rather than one per test. A matching open todo gets a new occurrence instead of a duplicate. Product bugs, flaky tests and agent mistakes never become todos.
7

The agent records the review

The review is recorded once every failed test has a cause backed by evidence, or is marked unverifiable with what a person needs to do. The agent never edits, creates or archives tests during a review. It reports what should change.

​
What the review contains

The review opens with a verdict and a one-line headline, such as “2 issues to fix” or “Blocked by an expired license”, followed by a short summary of what the plan covered and what the reruns showed. The agent decides the verdict from what the test plan is meant to prove, not by counting red tests. The sections under the verdict: Every finding links to the test results it is based on. Pass and fail counts always come from the real results, not from the review.

​
Where you see it

  • Results: open the test plan’s results in Results. The Analysis tab opens by default once the tests finish. While the agent is still working, it shows what the agent is doing right now. The Review chat link in the header opens the conversation.
  • Review chat: the agent’s reasoning, every rerun it started and why, and the review as a card at the end with a View run link back to the results. Ask follow-up questions in the same chat.
  • Notifications: Slack, Microsoft Teams and email notifications wait until the review is recorded, so you get one message after the reruns instead of a red result that a rerun clears.
  • CI and the API: while the review is in progress, the Get Run status stays RUNNING and the GitHub commit status stays pending. A test that passes on a rerun counts as passed, so a test plan that was red only because of flakes reports green. A Broken verdict always reports failed. The GitHub commit status shows the verdict and headline.
If a review does not finish within 4 hours of the last attempt, notifications and CI get the raw results instead, so a stuck review never blocks a pipeline.

​
Start a review from chat

To start a test plan and review it from a chat, type /regression and name the plan, for example /regression Smoke plan on staging. The agent starts the test plan, replies once with the plan, environment and test count, and waits until the tests finish. It then follows the same steps and writes the review in that chat and on the Analysis tab. Notifications and CI status for a test plan started this way do not wait for the review.