Tester Agent Manual
Everything you need to know about your resident QA engineer — what it tests, how to read test reports, how to set up your test suite, and how to direct it to investigate regressions.
1. What the Tester Agent does
The Tester Agent is your always-on QA engineer. It runs your test suite on a schedule, detects regressions the moment they appear, and automatically files detailed bug reports with reproduction steps. It covers end-to-end flows, API contracts, load limits, accessibility compliance, authentication edge cases, and schema changes.
Unlike a CI pipeline that only runs on push, the Tester Agent also runs proactively on a schedule — so a regression caused by a third-party dependency update or an infrastructure change will be caught even if no code was deployed.
2. Test types
| Test type | What it checks | Run frequency |
|---|---|---|
| E2E flows | Critical user journeys: signup, login, checkout, core feature use | Every 30 min (Pro), every 5 min (Studio) |
| API contract | HTTP response shapes match expected schema; status codes correct | On every API schema change + scheduled |
| Load test | Throughput, latency, error rate under simulated concurrent users | Weekly (Pro), daily (Studio), on demand |
| Accessibility | WCAG 2.1 AA compliance: contrast, ARIA, keyboard nav, screen reader flow | Weekly crawl on all public pages |
| Auth & security | Auth token expiry, CSRF protection, privilege escalation, rate limiting | Daily on all auth endpoints |
| Schema watcher | API response shape changes; database schema migrations; OpenAPI spec drift | On every deploy + continuous monitoring |
3. Setting up your test suite
Getting started takes one message. Tell the agent your app URL, the critical flows to test, and any credentials needed. The agent generates a test plan, confirms it with you, and starts running.
"My app is at staging.myapp.com. The critical flows are: user signup, login, creating a project, and inviting a team member. Login credentials for the test account: test@myapp.com / TestPass123. Also monitor the public /api/health and /api/v1/projects endpoints. Start with a full E2E run and set up daily scheduling."
What to give the agent upfront
| Info | Why it matters |
|---|---|
| Base URL (staging or prod) | Agent needs the target environment |
| Test credentials | Login + any API keys needed to exercise auth flows |
| Critical user flows (list them) | Agent writes E2E scripts for these first |
| Endpoints to monitor | Agent adds these to continuous API contract testing |
| Known slow pages | Agent prioritises load testing here |
| Accessibility standard | Default is WCAG 2.1 AA; tell agent if you need Section 508 or AAA |
4. E2E testing
The agent writes and runs E2E test scripts using a headless browser. Each script follows a real user journey — no mocking, no stubs. The test interacts with your actual UI, clicks real buttons, fills real forms, and verifies the outcome.
How the agent writes E2E tests
Give the agent the flow in plain English and it generates the test:
E2E test results
| Status | Meaning |
|---|---|
| PASS | All assertions passed, flow completed without errors |
| FAIL | At least one assertion failed — bug report created automatically |
| FLAKY | Test passed and failed on the same code — intermittent issue flagged |
| TIMEOUT | Page or element took too long — performance issue flagged |
| ERROR | Test itself errored (bad locator, auth failed) — agent self-heals or alerts you |
5. Load testing
The load test service simulates concurrent users hitting your application and measures throughput, response time percentiles, and error rates. Results are compared to the previous week's baseline so you know if performance is improving or degrading.
How to run a load test
Load test metrics explained
| Metric | What it means | Target |
|---|---|---|
| Throughput | Requests per second your server handled | Depends on your expected traffic |
| p50 latency | Median response time — half of requests were faster | Should match normal production latency |
| p95 latency | 95th percentile — slowest 5% of requests | ≤ 2× your normal p50 |
| p99 latency | 99th percentile — slowest 1% | No more than 5× your p50 |
| Error rate | Percentage of requests that returned 5xx or timed out | < 0.1% for critical endpoints |
| Saturation point | Number of concurrent users where error rate starts climbing | Know this before traffic spikes in production |
6. Accessibility testing
The agent crawls every public page of your site and checks for WCAG 2.1 AA compliance. Violations are categorised by impact: critical (blocks screen reader users), serious (significant barrier), moderate, and minor.
Common accessibility issues the agent catches
| Issue | WCAG criterion | Impact |
|---|---|---|
| Missing image alt text | 1.1.1 Non-text Content | Critical — screen readers read filename instead |
| Insufficient color contrast | 1.4.3 Contrast (Minimum) | Serious — text invisible for low-vision users |
| Missing form labels | 1.3.1 Info and Relationships | Critical — form fields unusable with screen reader |
| Non-keyboard-navigable modals | 2.1.1 Keyboard | Critical — keyboard-only users trapped |
| Missing ARIA landmarks | 1.3.6 Identify Purpose | Moderate — navigation harder for screen reader users |
| Focus not visible | 2.4.7 Focus Visible | Serious — keyboard users cannot see where they are |
| Auto-playing media | 1.4.2 Audio Control | Serious — disorienting for screen reader users |
How to fix an accessibility issue
Ask the agent for the fix:
The agent gives you the specific element, the current and required contrast ratio, and the exact hex value to use to pass the AA threshold.
7. Auth & security testing
The auth check service verifies that your authentication and authorisation flows behave correctly under adversarial conditions. It does not perform penetration testing — that is the Security Agent's domain — but it does verify that your auth layer is wired correctly.
| Check | What is verified |
|---|---|
| Token expiry | Expired JWT/session cookies are rejected with 401, not silently accepted |
| CSRF protection | State-changing endpoints reject requests without CSRF token |
| Role isolation | User A cannot access User B's resources using a valid User A token |
| Rate limiting | Login endpoint is rate-limited after N failed attempts |
| Logout completeness | After logout, the old session token no longer works |
| Password reset flow | Reset tokens are single-use and expire within the expected window |
| OAuth redirect safety | OAuth redirect_uri is validated against allowlist — no open redirect |
8. Schema change detection
The schema watcher monitors your API responses and compares them to the last known schema. When a field is added, removed, or changes type — even in a nested object — the agent detects it and alerts you.
This is critical for catching breaking changes before they reach consumers. If your API returns { "user_id": 123 } today and { "userId": "abc-123" } tomorrow (field renamed, type changed to string), every client that expected user_id as an integer will break.
How to add an endpoint to schema watching
9. Automatic bug reports
When the agent detects a failure, it immediately files a structured bug report in your session chat and optionally in your ticketing system (Linear, GitHub Issues, or Jira on Pro/Studio).
What a bug report contains
| Field | What it includes |
|---|---|
| Title | Short description: "E2E FAIL: Checkout flow — order confirmation not shown" |
| Severity | Critical / High / Medium / Low based on which flow failed |
| Environment | URL, browser, viewport, timestamp |
| Steps to reproduce | Exact steps the test took before the failure |
| Expected result | What the test was asserting |
| Actual result | What actually happened (error message, wrong text, missing element) |
| Screenshot | Captured at the moment of failure (Studio tier) |
| Logs | Browser console errors and network request failures during the test |
| AI root cause | LLM analysis: most likely cause based on the failure pattern |
10. Reading test reports
The agent sends a daily test summary at 8 AM in your timezone (configurable).
Daily report sections
| Section | What it tells you |
|---|---|
| Overall health | PASS / DEGRADED / FAILING — worst status across all tests |
| Run summary | Total tests, passed, failed, flaky, skipped |
| New failures | Tests that passed yesterday but failed today — regressions |
| Recovered | Tests that were failing and now pass again |
| Flaky tests | Tests with inconsistent results — intermittent bugs, not regressions |
| Performance trend | Whether p95 latency improved or degraded vs last week |
| Accessibility delta | New violations or fixed violations since last week |
| Recommended action | Top priority: what to investigate or fix today |
11. Communicating with your agent
Your Tester Agent session has a dedicated email (tester-a3f9b2c1@agents.boboyka.com) and an in-session chat panel. Both share the same history.
Things you can ask the agent
| Request type | Example |
|---|---|
| Run a specific test now | "Run the checkout E2E test right now and report back" |
| Investigate a failure | "Why is the login test failing since yesterday? Is this a code change or infra?" |
| Add a new test | "Write an E2E test for the new team invite flow we shipped today" |
| Start a load test | "Ramp to 500 concurrent users on /api/search and tell me the saturation point" |
| Accessibility audit | "Run a full accessibility audit on the new /dashboard page" |
| Auth check | "Check if our rate limiting on /login is working after the Nginx config change" |
| Schema check | "Compare the current /api/v1/orders response to what it was two weeks ago" |
12. Best practices
Test on staging, alert on production differences
Run the full test suite on your staging environment continuously. For production, run a lighter "smoke test" of the 5 most critical flows every 15 minutes. When staging and production diverge unexpectedly, the agent alerts you.
Define test ownership
Tell the agent which engineer owns each feature area: "Authentication tests are owned by alice@mycompany.com. When auth tests fail, add her to the CC on bug reports."
Keep test credentials fresh
Test accounts get stale — passwords expire, email services block them, or the account gets deleted. Tell the agent when test credentials change: "The test account password changed to NewPass456."
Treat flaky tests as real bugs
A test that fails 20% of the time is not a bad test — it's a real intermittent bug. Ask the agent to investigate flaky tests: "The 'invite team member' test is flaky — it fails about 1 in 5 runs. What is causing the inconsistency?"
13. Tier capabilities
| Feature | Starter | Pro | Studio |
|---|---|---|---|
| E2E test runs | Every 30 min | Every 15 min | Every 5 min |
| Max concurrent tests | 2 | 10 | Unlimited |
| Load testing | ❌ | ✅ (100 VUs) | ✅ (2,000 VUs) |
| Accessibility audit | ✅ (weekly) | ✅ (daily) | ✅ (on every deploy) |
| Auth checks | ✅ (daily) | ✅ (hourly) | ✅ (real-time) |
| Schema watcher | ❌ | ✅ | ✅ |
| Bug report to ticketing system | ❌ | ✅ (Linear, GitHub) | ✅ (all + Jira) |
| Screenshots on failure | ❌ | ❌ | ✅ |
| Daily email report | ✅ | ✅ | ✅ |
| Slack alerts | ❌ | ✅ | ✅ |
| Agent chat panel | ✅ | ✅ | ✅ |
14. Troubleshooting
Tests pass locally but fail in the agent
The agent runs tests in a headless browser in a cloud environment — results can differ from local if your app behaves differently without a GUI, depends on local environment variables, or has timing-sensitive UI interactions. Ask: "The checkout test fails but works locally. What is the error message and where does it fail?"
The agent reports WCAG violations my design team approved
Some WCAG violations are acceptable by design (e.g., a specific contrast ratio exception for decorative elements). Tell the agent to suppress specific rules: "Suppress WCAG 1.4.3 violations for elements with the CSS class .decorative — these are intentional."
Load test is hitting production accidentally
Test credentials stopped working
Send the agent fresh credentials via the secure chat panel (not email, to avoid credentials in email history): "Test credentials updated: email test-new@myapp.com, password: [paste in chat]."