Set it up
Pull request review
Azure Pipelines, the API, or an
@qa.tech comment starts the review.After merge
No preview environment. Review the merged pull request after the deploy
step.
What works on Azure DevOps
What this does not do
- Azure DevOps has no deployment event QA.tech can wait on. A review starts from Azure Pipelines, the API, or an
@qa.techcomment. - Prepare change review accepts GitHub pull request URLs. An Azure DevOps review starts when you send the preview URL.
- The
triagefields on the change review API apply to GitHub. On Azure DevOps, triage follows the repository toggle. - The post-merge agent updates the suite. It does not run the GitHub retro that reads comment reactions.
- This connection is for Azure DevOps Services (
https://dev.azure.com/{organization}orhttps://{organization}.visualstudio.com).
Connect Azure DevOps
Azure DevOps has no installable app. QA.tech uses an organization URL and a personal access token. Comments are posted as the user who created the token.Create the token on a dedicated Azure DevOps user, not a personal account. A
personal token stops working when that user is removed, and every comment
shows that user’s name.
1
Create a personal access token
In Azure DevOps, create a token with Code (Read & write). That scope lets QA.tech read the repository and post pull request comments, including screenshots.
2
Connect the organization
Open Settings → Integrations → Azure DevOps. Enter the organization URL, such as
https://dev.azure.com/your-organization, and the token. The connection is shared by every project in the organization.3
Add a repository
On the same page, add each repository QA.tech should review. Pick the applications those pull requests are tested against. With none selected, reviews use the project default application.
4
Choose how reviews start
Review Trigger defaults to Azure Pipelines. API is for a script you already run. Manually starts a review only from an
@qa.tech comment. A comment that mentions @qa.tech starts or re-runs a review for every trigger.5
Subscribe to service hooks
Saving the repository shows Service hook setup: a URL, the username
qatech, and a basic auth secret. In the Azure DevOps project, open Project settings → Service hooks, add a Web Hooks subscription for each event below, and paste those three values. Azure DevOps does not sign the payload, so that secret is what authenticates the delivery. Azure DevOps webhook documentation covers the form in their UI.Subscribe to:- Pull request created
- Pull request updated
- Pull request merge attempted
- Pull request commented on
@qa.tech mention. Pull request merge attempted is what tells QA.tech a pull request merged, which the post-merge agent needs.Start a review from Azure Pipelines
Put this step in the same job as the deploy step, after that step has setDEPLOY_URL to this pull request’s preview URL. Azure DevOps service hooks never include that URL.
DEPLOY_URL:
$(System.CollectionUri) already ends with a slash. The JSON body is
single-quoted so Azure Pipelines expands $(...) before the shell runs.
$QATECH_API_TOKEN is a shell variable and stays in double quotes. Secret
pipeline variables are not exposed to scripts on their own, which is why the
step maps the token through env.- Create
QATECH_API_TOKENas a secret pipeline variable. The value is a QA.tech API key from Organization Settings → API Keys. - Replace
proj_abc123andapp_gXeBl2with your project and application short IDs. An organization-scoped key needsprojectShortId. A project-scoped key may omit it. - Keep
curl -sSfso a rejected request fails the step. The step does not wait for the review to finish. - If the deploy step is in another job, pass
DEPLOY_URLacross jobs with an output variable. - The repository settings include Copy agent prompt when the trigger is Azure Pipelines. That prompt contains this step with your project’s short IDs filled in.
commitHash as $(Build.SourceVersion) so the review is tied to the commit the pipeline built.
Start a review from the API
Use this when something other than the pipeline step above already knows the preview URL. The repository still has to be connected, with service hooks, so QA.tech can read the diff and post the review.prUrl is the pull request URL: https://dev.azure.com/{organization}/{project}/_git/{repository}/pullrequest/{id}. The same path works on {organization}.visualstudio.com.
Optional postMerge (enabled, and an optional prompt of up to 4000 characters) overrides the repository’s post-merge settings for that pull request. See Post-merge agent.
Start a review from a comment
Comment on the pull request and mention@qa.tech. This needs the Pull request commented on service hook.
Add instructions (@qa.tech test the checkout flow) or a preview URL to test a specific environment. With no URL, the review uses the application’s configured environment.
QA.tech replies with a link to the conversation. If the organization cannot start a run, that reply says so and the review does not start.
What QA.tech posts
The finished review is one comment on the pull request. A later review of the same pull request updates that comment instead of adding another. Screenshots in the review are uploaded to the pull request, because Azure DevOps only renders images from its own host. The comment is authored by the user who created the personal access token.Code access
Code Access is on for a new repository. Chat and reviews can clone the repository with the connection’s token and search its code. Turn it off on the repository to stop cloning. With Code Access on, Logic Verification (beta) can model rule-heavy changes (dates, money, permissions, limits) from source and check them with a solver. That can add a few minutes to those reviews.PR triage
PR triage is on for a new repository. QA.tech skips pull requests that cannot be tested with end-to-end or API tests, and does not post a review comment for those. It needs the pull request service hooks, because triage reads the pull request ingested from them. The skip rules are the ones on PR triage. Azure DevOps does not post a status when a pull request is skipped. Comment@qa.tech to run the review anyway.
The API triage override is for GitHub pull requests. On Azure DevOps, use the repository’s PR Triage toggle and PR triage prompt.
Post-merge agent
Post-merge agent is off until you enable it on the repository. After a pull request merges, the agent keeps the test suite up to date. It only does what the prompt asks. It needs the Pull request merge attempted service hook, and a completed review of that pull request. Example prompts are on GitHub’s post-merge agent. Azure DevOps uses the same agent and does not run the GitHub retro afterward.postMerge on a change review request overrides enabled and prompt for that pull request. Omit it to use the repository settings.
After merge
When pull requests have no preview environment, run the review after the merge lands and the deploy finishes. The agent reads the merged diff and tests staging or production. The result is posted on the merged pull request. Before you add the step:- The repository is connected, with the pull request service hooks, so the agent can read the diff and post the result.
- Staging exists in Settings → Applications & Envs, and you have its environment short ID (for example
env_aB3xY9). - QA.tech can reach staging. If staging is private or behind a VPN, see Environment access.
Merged PR 123:. If your merge commits do not include that line, the step skips them.
environment.url instead of environment.shortId.
ubuntu-latest includes curl and jq. The token is a secret pipeline variable. The other env entries pass predefined variables into the script without putting them inside the JSON by macro expansion.