Foundation for TestingBefore using any testing approach below, ensure you’ve crawled your website to build the knowledge graph and configured authentication. These prerequisites enable the AI agent to test efficiently.
Choosing Your Testing Strategy
Different testing approaches fit different scenarios. Choose based on when you need results and what you’re validating:| Testing Approach | Best For | When to Use | Setup Guide |
|---|---|---|---|
| Manual test runs | Ad-hoc testing, debugging, development | Quick validation, iterating on test design | Running Tests |
| Scheduled test plans | Production monitoring, regression suites | Daily/nightly checks, SLA monitoring | Test Plans |
| PR-triggered testing | Pre-merge validation, preview environments | Every PR, feature branches | GitHub Actions |
| API-triggered testing | Custom workflows, deployment pipelines | Post-deploy validation, multi-environment | API Reference |
- Start with: Manual runs while building tests, then add PR testing
- Add next: Scheduled nightly runs for comprehensive coverage
- Advanced: API integration for custom deployment workflows
- Pro tip: Use test plans to organize tests for different scenarios (smoke vs. full regression)
Organizing Tests with Test Plans
Test plans help you run the right tests at the right time. Organize tests strategically to separate fast smoke tests from comprehensive regression suites. Common test plan patterns:- Smoke tests plan — 5-10 critical tests, runs on every PR (~5 min)
- Regression plan — Full suite, runs nightly (~30-60 min)
- Production monitoring plan — Key flows, runs every 4 hours
- Preview environment plan — Dynamic URLs, runs on deploy
Test plans enable parallel execution - 20 tests complete in ~3 minutes instead of 60 minutes sequential. See Test Plans.
- Separate fast tests (PR-triggered) from slow tests (scheduled)
- Target different environments (staging vs. production)
- Control what runs when based on your workflow
Quick Triage Workflow
1
Filter to failures
Use RESULT=FAILED + STATUS=COMPLETED to see only actionable failures. See Tests and Results for filter options.
2
Review by scenario
Grouped failures often share root cause - review scenario by scenario
3
Categorize each failure
Determine if it’s a product bug, test needs update, flaky test, or environment issue
4
Take action
Create issues for bugs, update tests for intentional changes, add waits for flaky tests
Getting Notified
Configure notifications based on your testing approach:- PR tests — GitHub comments post automatically (see GitHub Actions)
- Scheduled runs — Slack/email for failures (see Notifications)
- Production monitoring — Immediate alerts for critical failures
Exporting Issues
When you find bugs during review:- Export to Linear, Jira, or Trello: Configure integrations
- Include playback URL and context before exporting
- Tag with environment and test run for traceability