01Start line· Pole position
Harry Bevan
Lead Quality AI Engineer
Nine years in quality engineering, from regression testing core banking systems to leading quality for AI products. I’d rather show the work than list it, so this site is built the way I build software: specified, automated and tested on every change. Take a lap: first the site’s own test results and AI evals, then how I design tests, and the AI workflow that ships every change.
- years in quality & test engineering
- 9+
- automated tests run before every deploy
- 100+
- of the start sequence’s criteria traced to tests
- 100%
- Playwright
- AI/LLM validation
- Test design
- CI/CD
- Risk-based testing
02Opening corners
A portfolio that tests itself
A quality engineer’s portfolio should prove the work, not just describe it. This site is specified, automated and tested on every change, and it publishes the evidence.
Build & test · This site
Only a green run can deploy
100%controls with a test id
- Problem
- Claims about quality are cheap. The proof is a site that can’t ship broken, and shows you why.
- Approach
- A stable data-testid on every control, covered by Playwright end-to-end tests on desktop and mobile Chromium, visual regression screenshots, axe accessibility scans, and CSP and privacy checks. GitHub Actions runs lint, type checks and the full suite on every change.
- Outcome
- Each deploy publishes its own test results, below. Replay that run, run live checks in your own browser, or break the page and watch the checks catch it.
- Playwright
- Visual regression
- axe
- GitHub Actions
Latest CI run
- Passed
- –
- Failed
- –
- Skipped
- –
- Run time
- –
“Break it” knocks a wheel off the car and plants real bugs in this page. Then run the live checks and watch them get caught.
- Press “Replay the CI run” to watch the suite that shipped this build, or “Run live checks” to test this page in your browser now.
Raw results (JSON)CI runs on GitHub (opens in a new tab)
Lighthouse, measured on the production build, 9 Nov 2026.
- Performance
- 96
- mobile
- Accessibility
- 100
- Lighthouse
- Best practices
- 100
- Lighthouse
- SEO
- 100
- Lighthouse
AI evals · scorecard
The pit wall assistant
A small AI assistant that answers questions about this portfolio, using only the text on this page. It was asked questions it should answer, questions it can’t answer, requests for personal data, prompt injections and off-topic requests. Its answers were recorded once and are scored by code, and every CI run re-scores them, so this scorecard always matches the suite. No model runs when you load this page.
–of eval questions passed
–average LLM judge score
Each answer was then independently marked out of 100 by a different model, against a fixed rubric: accuracy, following its rules, helpfulness and clarity. The judge had no part in writing the answers and saw only this page, the questions and the replies.
Every question and answer (0)
03The esses
From requirement to test cases
How an AI test-design agent works, with a tester reviewing every step, shown on something you’ve just used: this site’s start sequence. The agent turns a requirement into acceptance criteria, maps the states and risks, picks techniques and drafts prioritised test cases. A coverage matrix then proves every criterion is tested, and the gaps it finds become new tests.
01 · Requirement
The user story
As a visitor, I want to start the car myself, so the intro never plays without me and I stay in control of it.
Acceptance criteria
- AC1START only works once the safety guard is lifted, and lowering the guard disarms it.
- AC2Pressing START runs the sequence once: five red lights, all green, then the lap.
- AC3“Skip intro” is the first focusable element and goes straight to the content, with focus on the hero.
- AC4Everything works with a keyboard alone.
- AC5With reduced motion, the sequence uses fades only and ends in the same place.
- AC6No sound plays before the visitor’s first click, and muting is remembered.
- AC7A finished intro is remembered for the session, and “Replay start” runs it again.
02 · Analysis
States, risks and techniques
States
introarmedstartingtrack
- intro → armed (lift the guard)
- armed → intro (lower the guard)
- armed → starting → track (press START)
- any stage → track (Skip intro)
- track → intro (Replay start)
Risks
- An accidental or repeated start
- Keyboard focus lost or trapped
- Motion that makes visitors unwell
- Audio playing without consent
- Content hidden if something fails
Techniques
- State transition
- Negative testing
- Equivalence partitioning (input methods)
- Accessibility
- Cross-device
03 · Test cases
14 prioritised test cases
9 P1 · 5 P2 · each with its technique and the Playwright test that automates it
| ID | Test case | Covers | Technique | Priority | Automated by |
|---|---|---|---|---|---|
| TC-01 | START is locked until the guard is lifted | AC1 | Negative | P1 | ✓ guard then button reaches the track state |
| TC-02 | Lifting the guard arms START and announces it | AC1 | State transition | P1 | ✓ guard then button reaches the track state |
| TC-03 | Lowering the guard disarms START again | AC1 | State transition | P2 | ✓ the guard can be lowered again before starting |
| TC-04 | START runs five red lights, then green, then the lap | AC2 | State transition | P1 | ✓ with sound on (the default), the sequence plays audio without errors |
| TC-05 | Pressing START again mid-sequence doesn’t restart it | AC2 | Negative | P2 | ✓ pressing START again mid-sequence does not restart it |
| TC-06 | “Skip intro” is first in the tab order | AC3 | Accessibility | P1 | ✓ initial screen: Skip intro is the first focusable element |
| TC-07 | “Skip intro” lands on the hero, with focus | AC3 | State transition | P1 | ✓ Skip intro jumps straight to the track/hero state |
| TC-08 | Keyboard alone: Tab, Enter and Space complete the start | AC3, AC4 | Equivalence partitioning | P1 | ✓ keyboard-only flow works end to end |
| TC-09 | No serious accessibility issues at any stage | AC3, AC4 | Accessibility | P1 | ✓ start screen has no serious or critical violations✓ armed state has no serious or critical violations |
| TC-10 | Reduced motion: fades only, same end state, focus on the hero | AC5 | Accessibility | P1 | ✓ shows the content without the full animation✓ Skip intro moves focus to the hero, not back onto the skip link |
| TC-11 | No audio exists before the first click | AC6 | Negative | P1 | ✓ no audio is created and no AudioContext exists before any click |
| TC-12 | Muting is remembered after a reload | AC6 | State transition | P2 | ✓ sound is on by default, and muting is remembered |
| TC-13 | A finished intro is skipped on reload; “Replay start” runs it again | AC7 | State transition | P2 | ✓ completed intro is remembered for the session, and can be replayed |
| TC-14 | The whole start works on a phone-sized screen | AC1, AC2, AC3 | Cross-device | P2 | ✓ guard then button reaches the track state (mobile Chromium) |
04 · Coverage
Every criterion covered
7/7 criteria covered ·14/14 test cases automated
| Criterion | Covered by |
|---|---|
| AC1 START only works once the safety guard is lifted, and lowering the guard disarms it. | TC-01, TC-02, TC-03, TC-14 |
| AC2 Pressing START runs the sequence once: five red lights, all green, then the lap. | TC-04, TC-05, TC-14 |
| AC3 “Skip intro” is the first focusable element and goes straight to the content, with focus on the hero. | TC-06, TC-07, TC-08, TC-09, TC-14 |
| AC4 Everything works with a keyboard alone. | TC-08, TC-09 |
| AC5 With reduced motion, the sequence uses fades only and ends in the same place. | TC-10 |
| AC6 No sound plays before the visitor’s first click, and muting is remembered. | TC-11, TC-12 |
| AC7 A finished intro is remembered for the session, and “Replay start” runs it again. | TC-13 |
The matrix exposed one gap: nothing tested a second press of START mid-sequence. TC-05 was added, along with its Playwright test, which now runs on every deploy.
Demo: drafted by an AI agent from the requirement, then reviewed and edited by me, the way I’d run it on a real feature. The automated tests are this site’s own; their latest results are in the Telemetry panel above.
04Back straight
Plan to deploy, hands-free
How a change to this site ships. I describe it once; a Claude Code workflow plans it, builds it, tests it and has a second agent verify it, then stops at the one step that can’t be undone and waits for me. It’s how I’d want any team to use AI agents: fast between the gates, and never trusted to mark their own work.
AI workflow · Claude Code
One command, five gates, one human decision
/ship make the lap timer pause when the tab is hidden
Plan
Planner agent, read-only
Turns the request into acceptance criteria, risks and a named Playwright test for every criterion. It can read the code but can’t change it.
GateEvery criterion has a test
.claude/agents/planner.md (opens in a new tab)Build
Claude Code, inside guard rails
Writes the planned tests first, then the smallest change that passes them. One hook formats every edit; another blocks the shortcuts listed below.
GateOnly the planned files changed
.claude/hooks/guard.mjs (opens in a new tab)Test
ESLint, Playwright and axe
Lint, type checks, the build and the full suite on desktop and mobile: the same command CI runs. A failure means fixing the code, and after three failed attempts it stops and reports.
Gatenpm run verify passes
.claude/skills/ship/SKILL.md (opens in a new tab)Verify
Verifier agent, fresh context
A second agent that never saw the build reads the diff, reruns the checks itself and marks each criterion pass or fail. The agent that wrote the change never signs it off.
GateA pass on every criterion
.claude/agents/verifier.md (opens in a new tab)Deploy
Me, then GitHub Actions
The agent commits and stops, because pushing is blocked for it. I push; CI reruns every check and deploys only a green run; then a smoke check asks the live site which commit it is serving.
GateThe live site reports the new commit
scripts/smoke-live.mjs (opens in a new tab)
What the agent can’t do, whatever it’s asked
- Push or merge to main: deploying stays a human decision
- Switch a test off, or skip one without a written reason
- Rewrite the visual baselines to hide a difference
- Skip commit hooks or throw away uncommitted work
- Add an email address or phone number to anything that ships
- Add an inline style the Content Security Policy would block
Hands-free, not eyes-off: the agents do the work between the gates, the hooks hold the lines that mustn’t move, and the one irreversible step stays with a person. The skill, agents and hooks are in this site’s repo, and the guard rails have their own tests in the suite above.
05Chequered flag
Let’s talk quality
Lap complete. Talk to me about quality engineering, AI validation or test automation.
Chequered flag!
Lap time --:--.---
Lap complete
The lap from above
Every marker is a section of the portfolio, with the split time you reached it in. Pick one to go back to it.