Sample QA audit report
This is the exact structure of the report delivered at the end of an AutomationDataCamp QA diagnostic. The company, figures and findings are made up: they show what you receive, not a real client.
QA diagnostic €1,000 excl. VAT · delivered in 5 working days
1. Summary
- Client
- Facturo (fictional), publisher of an online invoicing product, 12 developers, 2 QA engineers.
- Scope
- Web application, public API, GitLab CI pipeline.
- Method
- Two interviews (technical and product leads), review of the repository and CI, analysis of 30 days of runs.
- Conclusion
- The existing suite gives false comfort: it mostly covers secondary screens, often fails for the wrong reasons and does not protect the journeys that generate revenue.
2. Baseline metrics
| Metric | Value | What it means |
|---|---|---|
| E2E tests | 214 | Only 9 cover the 6 critical journeys |
| Flaky tests | 11% of runs | Rerun until they pass, failures ignored |
| CI duration | 47 minutes | Full suite on every commit |
| Defects found in production | 14 last quarter | 5 of them in recurring billing |
3. Risk-to-test map
| Journey | Business risk | Current coverage |
|---|---|---|
| Create and send an invoice | High | Partial (UI only) |
| Recurring billing | High | None |
| Online payment | High | None (provider sandbox not configured) |
| Accounting export | Medium | Manual, before each release |
| User management | Medium | Good |
| Display settings | Low | Very high (62 tests) |
4. Prioritised findings
- P1 — The revenue journeys are not protected. Recurring billing and payment have no automated tests. Evidence: 5 of the 14 production defects last quarter.
- P1 — Flaky tests hide regressions. Selectors rely on generated CSS classes; 11% of runs fail for reasons unrelated to the code change.
- P2 — CI sets the release pace. 47 minutes per commit, with no test selection based on changed files.
- P2 — Test data is shared. Tests overwrite each other's data when they run in parallel.
- P3 — No requirement-to-test link. Traceability is rebuilt by hand before every client audit.
5. Target test strategy
- Test billing rules (calculations, taxes, due dates) through the API first: faster and more stable than the UI.
- Keep E2E for the 6 critical journeys only, with accessible selectors (roles and labels).
- Create each test's data on the fly and remove it afterwards.
- Run targeted tests on every commit and the full suite every night.
6. Three-month action plan
| Period | Actions | Estimated effort |
|---|---|---|
| Week 1 | Quarantine the 23 flaky tests, fix selectors on critical journeys | 3 days |
| Month 1 | API tests for billing and payment, isolated test data | 8 days |
| Month 2 | E2E for the 6 critical journeys, test selection in CI | 7 days |
| Month 3 | Requirement-to-test traceability, dashboard, handover to the team | 4 days |
Illustrative estimate. In a real report every line is costed, and the client decides what to do in-house or with us.
7. Proof of feasibility
The “create and send an invoice” journey was automated with Playwright in the client's repository, wired into CI, with its own test data. Run time: 38 seconds. This test is the template for the next ones.
8. Backlog excerpt
- QA-01 — Quarantine flaky tests and track them on a dedicated board.
- QA-02 — Replace CSS selectors on critical journeys with accessible selectors.
- QA-03 — Write API tests for tax and due-date calculations.
- QA-04 — Isolate test data (create and clean up per test).
- QA-05 — Add changed-file test selection in GitLab CI.
9. What the report does not cover
Security (no penetration test), performance under load and the mobile app were out of scope. They can be covered by a separate engagement.
The same report, on your product
Written report, 1-hour debrief, a backlog ready to prioritise and, if the environment allows it, one critical flow automated: delivered in 5 working days for €1,000 excl. VAT.