AI in Software Testing: Tools, Methods and Training (2026)

AI and software testing: what are we talking about?

The expression "AI in software testing" refers to two distinct activities. First, testing with AI, meaning using artificial intelligence models as the tester's assistants to design, write, maintain and analyse tests. Second, testing AI, meaning checking the quality of a system that embeds a model, for example a chatbot built on a large language model.

First meaning: AI as the tester's assistant

Here, the software under test is conventional (a web application, an API, a mobile application) and AI appears in the tester's tooling. It suggests test cases from a specification, generates a skeleton Playwright script, suggests a fix when a selector has changed, groups similar test failures or summarises a report. The tester remains the decision-maker: they validate, correct, complete and take responsibility for the coverage achieved.

Second meaning: AI as the object of the test

Here, the system under test itself contains an AI component, often a large language model (LLM). Its answers are not deterministic: the same question can produce two different wordings. Conventional methods ("expected result = actual result") are no longer enough. You need to build representative test sets, define quality metrics, involve domain experts and run attack tests (red teaming) to check that the system does not produce dangerous content.

Why both matter for a tester in 2026

Companies are deploying AI assistants in their products and equipping their teams with generative AI tools. A tester who knows how to use these tools critically and who understands how to evaluate a model is useful on both fronts. That is why the ADC course integrates generative AI applied to testing throughout its 399 hours, with a constant requirement for critical, traceable use.

Which AI tools can you use to test software?

AI tools for testing software and automating tests fall into four families: generating test cases and scripts, self-healing selectors, triaging and analysing failures, and code assistants built into the editor. Each saves time on a specific task; none replaces test strategy or human verification.

1. Generating test cases and scripts

From a user story, a mock-up or a web page, a language model proposes a list of scenarios (nominal case, edge cases, error cases) and then a first draft of code. General-purpose conversational assistants such as ChatGPT are used this way with a well-constructed prompt. On the specialised tooling side, Playwright's test generator records your actions in the browser and writes the corresponding code, described in the official documentation as a tool that "generates code for your interactions automatically" (source: Playwright, accessed 27 September 2026). It is not generative AI in the strict sense, but it is often the starting point that the tester then enriches with an assistant.

What to check: that the proposed cases match the real product (not an imagined feature), that the assertions check something meaningful, and that no important edge cases are missing.

2. Self-healing selectors

A large part of UI test maintenance comes from selectors that break when the interface changes. Commercial platforms such as Testim or Mabl offer mechanisms that identify an element by several attributes and try to find it again when one attribute changes. We make no claim here about their performance or pricing, which change over time; remember the principle: fewer tests broken for a purely cosmetic reason.

What to check: that an automatic "repair" has not pointed the test at the wrong element, which would produce a green test for the wrong reason. Note that Playwright's semantic locators (getByRole, getByLabel), described as "resilient" by the official documentation (source: Playwright, accessed 27 September 2026), already greatly reduce this problem without AI.

3. Visual testing and failure triage

Visual testing tools compare screenshots between two versions and distinguish a perceptible difference from simple rendering noise; Applitools is one of the well-known players in this category. Assisted triage groups hundreds of failures by probable cause (same error, same component, same environment), which speeds up the analysis of an overnight run.

What to check: the sensitivity setting (too tolerant and you miss regressions; too strict and you drown the team in false positives) and the relevance of the groupings.

4. Code assistants in the editor

GitHub Copilot and equivalent assistants complete code as you type, suggest whole functions and explain a snippet. For a tester, they speed up writing repetitive tests, creating test data and documentation. The Playwright documentation also mentions an MCP server that lets AI agents drive the browser (source: Playwright, accessed 27 September 2026).

What to check: every suggested line, as you would for a junior colleague's code. Assistants readily reproduce outdated patterns or fixed waits (waitForTimeout) that best practice advises against.

Summary table

Family	Task sped up	Publicly cited examples	Main risk
Test case and script generation	Initial design, first draft of code	ChatGPT, Playwright test generator	Invented cases, empty assertions
Self-healing selectors	UI test maintenance	Testim, Mabl	Repair pointing to the wrong element
Visual testing and triage	Results analysis	Applitools	False positives or missed regressions depending on the setting
Code assistants	Writing, test data, documentation	GitHub Copilot	Plausible but incorrect or outdated code

What are the risks of AI in software testing?

The main risks of AI in software testing are hallucinations (plausible but false content), loss of traceability (nobody knows why a test exists any more), leaks of sensitive data to external services, and regulatory non-compliance, notably with the European AI regulation. Each is managed through precise practices, not through trust.

Hallucinations

A language model produces probable text, not true text. Applied to testing, that means test cases for features that do not exist, selectors that match nothing on the page, or assertions that always pass. The ISTQB CT-GenAI syllabus explicitly lists hallucinations among the risks to manage (source: ISTQB, accessed 27 September 2026). The remedy: always check the AI's output against the real product, run the test and verify that it fails when it should fail.

Traceability

A test must be linkable to a requirement, a risk or a defect. When dozens of tests are generated in bulk, that link is lost and nobody knows what is really covered. That is why ADC insists on a "critical and traceable" use of AI: document the prompt used, the model version, what was kept and what was rejected, and link each generated test to an identified requirement.

Sensitive data

Sending an extract of a customer database, a production dataset or proprietary code to an external AI service can breach the GDPR or a confidentiality agreement. The CT-GenAI syllabus ranks confidentiality and security among the central risks (source: ISTQB, accessed 27 September 2026). Good practice: anonymise or synthesise test data, use AI instances approved by the company, and know the internal policy before opening an assistant.

Bias

A model trained on biased data reproduces those biases in its suggestions: test data that covers only one type of name, address format or language. The tester must consciously broaden the diversity of cases.

The European AI regulation (AI Act)

The European regulation on artificial intelligence takes a risk-based approach, with four levels: unacceptable risk (prohibited practices), high risk, transparency risk and minimal risk. For high-risk systems, the European Commission lists obligations that speak directly to the tester: "adequate risk assessment and mitigation systems", "high quality of the datasets", "logging of activity to ensure traceability of results", "detailed documentation", "appropriate human oversight measures" and "a high level of robustness, cybersecurity and accuracy". Transparency obligations require AI-generated content to be identifiable. According to the timeline published by the Commission, general application begins on 2 August 2026 and the rules for high-risk systems apply from 2 December 2027 (source: European Commission, accessed 27 September 2026).

For the tester: testing a high-risk AI system becomes a documented regulatory requirement, and the use of AI tools in the testing process must itself remain traceable.

How do you test AI and evaluate an LLM?

To test AI and evaluate an LLM, meaning a system based on a large language model, you rely on four pillars: a test set representative of real questions, suitable metrics (relevance, factual accuracy, safety, cost, latency), evaluation by domain experts, and attack tests (red teaming). Non-determinism means reasoning in terms of success rate over several runs rather than a single result.

Building a representative test set

In an article from June 2024, LeMagIT reports the approach of a consulting firm that builds "a list of typical questions, a representative corpus of about a hundred topics" to compare different models, and that compares "the relevance of the answers with domain experts" identified beforehand (source: LeMagIT, article of 20 June 2024, accessed 27 September 2026; the article is in French and the quotations are our translation). The transferable idea: start from users' real questions, cover frequent and sensitive topics, and version this corpus like any other test set.

Defining metrics

The same article stresses that generative AI has no classic quantitative performance metrics and recommends tracking, alongside answer quality, response times, costs and environmental impact, treating "tokens" as a reasonable unit for measuring a model's impact (source: LeMagIT, accessed 27 September 2026). In practice, an evaluation grid combines:

• Factual accuracy: is the answer true against a reference?
• Relevance: does it answer the question asked, without digressing?
• Safety: does it refuse dangerous or out-of-scope requests?
• Consistency: does the answer stay stable from one run to the next?
• Cost and latency: tokens consumed, response time.

RAG architecture and testability

The LeMagIT article recommends separating the documents from the model through a retrieval-augmented generation (RAG) layer, which processes the data separately before an LLM rephrases the result in human language (source: LeMagIT, accessed 27 September 2026). For the tester, this separation is a boon: you can test retrieval (are the right documents found?) independently of generation (is the answer faithful to the documents provided?).

Managing non-determinism

Two runs of the same prompt can give two different answers. Three techniques make testing possible nonetheless:

1. Semantic assertions: check for the presence of key information or the absence of forbidden content, rather than strict text equality.
2. Repeated runs: run each case several times and measure a success rate, with an acceptance threshold defined in advance.
3. Evaluator model: use a second model to grade the answer against a rubric (LLM-as-a-judge), keeping a sample reviewed by humans to calibrate the evaluator.

Red teaming

Red teaming consists of deliberately attacking the system: prompt injections ("ignore your instructions and…"), attempts to extract confidential data, requests for dangerous content phrased indirectly, and ambiguous questions designed to provoke hallucinations. These tests are scripted like conventional test cases, with an expected result (refusal, safe answer), and are replayed with each new version of the model or system prompt.

Which skills does a tester need in the age of AI?

In the age of AI, the tester keeps the fundamental skills (test design, risk analysis, automation, CI) and adds four more: knowing how to write an effective, reproducible prompt, knowing how to evaluate and critique an AI output, knowing how to test a non-deterministic system, and knowing the regulatory and ethical framework. None of these skills replaces the fundamentals; they are added to them.

The fundamentals remain the foundation

An AI assistant amplifies what you already know how to do. If you do not master test design, you will not see what is missing from the generated list; if you cannot read a Playwright test, you will not spot the empty assertion. That is why the ADC course starts with test design before introducing AI as an accelerator.

Prompting applied to testing

The ISTQB CT-GenAI syllabus aims to "develop practical skills in prompting large language models for software testing" (source: ISTQB, accessed 27 September 2026). In practice: give the context (product, technology, constraints), provide examples of the expected format, explicitly ask for edge cases and error cases, and iterate. A good test prompt is kept and versioned like a tool.

Equipped critical thinking

Reviewing an AI output does not mean skimming it: it means running it, checking it against the product, and comparing it with what an expert would have produced. The tester becomes the quality control of the AI itself, with extra vigilance: AI is always confident, even when it is wrong.

Testing non-determinism

Designing semantic assertions, defining statistical thresholds, maintaining an evaluation corpus, organising a red teaming campaign: skills that are new for most testers and in high demand among teams deploying AI assistants.

The regulatory and ethical framework

Knowing the broad lines of the European AI regulation, the GDPR and your company's internal policy lets you say "no" in time: to sending production data into an external assistant, or to putting a high-risk system into production without test documentation.

What is the ISTQB CT-GenAI certification?

The ISTQB CT-GenAI certification ("Certified Tester – Testing with Generative AI"), version 1.1, validates the ability to use generative AI in software testing in a controlled way. It requires you to already hold the ISTQB Foundation certification (CTFL), consists of 40 questions in 60 minutes and requires 30 points out of 46 (65%) to pass, according to the official ISTQB page.

Who is it for?

According to the ISTQB, the certification is aimed at "anyone involved in using generative AI for software testing", including testers, test analysts, test automation engineers, test managers, user-acceptance testers and software developers, as well as quality and project management professionals who need knowledge of testing with generative AI (source: ISTQB, accessed 27 September 2026).

Prerequisites

The official page is explicit: "Before taking the CT-GenAI exam, you must be certified ISTQB® Certified Tester Foundation Level (CTFL)" (source: ISTQB, accessed 27 September 2026). CTFL v4.0, for its part, has no prerequisite: 40 questions, 60 minutes (75 for non-native English speakers), 26 points out of 40 to pass (source: ISTQB, accessed 27 September 2026). The logical order is therefore CTFL, then CT-GenAI. See our ISTQB CTFL certification page.

Exam format

Item	ISTQB CTFL v4.0	ISTQB CT-GenAI v1.1
Prerequisite	None	CTFL
Number of questions	40	40
Duration	60 min (75 min for non-native English speakers)	60 min (75 min for non-native English speakers)
Pass mark	26 points out of 40	30 points out of 46 (65%)

Source for both columns: official ISTQB pages accessed on 27 September 2026.

What the syllabus covers

The objectives published by the ISTQB: "understand the fundamental concepts, capabilities, and limitations of generative AI", "develop practical skills in prompting large language models for software testing", "gain insight into the risks and mitigations of using generative AI for software testing" and "the applications of generative AI solutions for software testing", and contribute to the adoption strategy within the organisation. The areas covered: prompt engineering, risk management (hallucinations, bias, confidentiality, security), LLM-based solutions and adoption strategies (source: ISTQB, accessed 27 September 2026). See our ISTQB CT-GenAI certification page and the overview of certifications.

Where can you train in AI applied to software testing?

To train in AI applied to software testing, the ADC "QA Automation & AI Tester" course (399 hours over 12 weeks, 100% online, no technical prerequisite) does not isolate AI in an optional module: it integrates it throughout the programme, from test design to CI, with a requirement for critical, traceable use. It prepares for the ISTQB CTFL v4.0 and CT-GenAI v1.1 exams, included in the price.

How AI is taught at ADC

• In test design: use an assistant to generate cases, then critique them with the techniques of the ISTQB syllabus. The aim is to learn to spot what is missing and what is wrong.
• In Playwright + TypeScript automation: start from the test generator and a code assistant's suggestions, then refactor towards robust tests (semantic locators, web-first assertions, isolation).
• In API testing: produce datasets and assertions with AI, taking care that no real data passes through it.
• In GitHub Actions CI: analyse failures and logs with the help of an assistant, never concluding without a verified run.
• In traceability: document prompts, versions and choices, so that each generated test is linked to a requirement.

Programme structure

The 399 hours break down into 371 hours spread over 6 blocks (each validated by an ADC block certification), 14 hours of coaching, 7 hours of preparation for ISTQB CTFL v4.0 and 7 hours of preparation for ISTQB CT-GenAI v1.1, exams included. The programme concludes with the defence of a Capstone project, which awards the ADC Professional Certificate CP-QAI, and prepares for the TestMu AI certifications. ADC does not award an RNCP title.

Formats, dates and funding

Two formats: evening classes (2 hours per evening with a mentor, Monday to Friday) with personal funding at €1,500 in 1 to 4 instalments at no extra cost, or full-time (9:00–17:00) with corporate funding. State funding is announced as "coming soon". Next session from 15 October 2026 to 15 January 2027, 12 places, enrolment open until 22 October. Details on the QA Automation & AI Tester course and Financing pages.

Where to start

If you are a beginner, the free mini-course (FR) (7 Playwright lessons) shows concretely what an automated test is. If you are already a tester, our guide Becoming a software tester describes how the profession is evolving in 2026.

Frequently asked questions about AI and software testing

What is AI in software testing?

Two distinct things: using AI as the tester's assistant (generating test cases and code, repairing selectors, triaging failures) and testing systems that embed AI, such as an assistant based on a large language model, with methods adapted to non-determinism.

Will AI replace software testers?

No. AI speeds up certain tasks but does not take responsibility for what is verified, does not know the real product and can hallucinate. The tester becomes the quality control of AI outputs, and AI systems themselves create a new need for specialised testing.

Which AI tools exist for test automation?

Four families: generation of cases and scripts (assistants such as ChatGPT, the Playwright test generator), self-healing selectors (Testim, Mabl), visual testing and triage (Applitools) and code assistants (GitHub Copilot). Each family speeds up a specific task and requires human verification.

How do you test an AI or an LLM?

Build a representative corpus of real questions, define metrics (accuracy, relevance, safety, cost, latency), have domain experts evaluate the answers, run attack tests (red teaming) and manage non-determinism with semantic assertions and repeated runs against a success threshold.

What is LLM red teaming?

Red teaming consists of deliberately attacking an AI system: prompt injections, attempts to extract data, requests for dangerous content phrased indirectly. Each attack becomes a test case with an expected result (refusal or safe answer), replayed with each new version.

What is the ISTQB CT-GenAI certification?

The ISTQB "Certified Tester – Testing with Generative AI" certification (v1.1) validates the controlled use of generative AI in software testing. It requires CTFL as a prerequisite, consists of 40 questions in 60 minutes and requires 30 points out of 46 (65%). Source: istqb.org, accessed 27 September 2026.

What are the risks of using AI to write tests?

Hallucinations (invented test cases, empty assertions), loss of traceability between tests and requirements, leaks of sensitive data to external services, bias in generated test data, and non-compliance with the European AI regulation for high-risk systems.

How does ADC integrate AI into its tester course?

Generative AI applied to testing is present throughout the 399 hours (design, Playwright + TypeScript, API testing, GitHub Actions CI), with a requirement for critical, traceable use. The course includes 7 hours of preparation and the ISTQB CT-GenAI v1.1 exam, after CTFL v4.0.