CT-AI Sample Questions Answers

The ISTQB CT-AI syllabus explains inSection 4.4 – Experience-Based Testing for AI Systemsthat AI-based systems frequently suffer frominsufficient specifications, unpredictable model behavior, andtest oracle problems, especially when outputs depend on probabilistic or learned patterns. The syllabus explicitly states thatexploratory testingis especially valuable in such contexts because it allows testers to investigate the system interactively, observe unexpected behavior, and evaluate system responses that cannot be fully predicted beforehand. Thus, OptionCaccurately reflects the role and justification of exploratory testing for AI systems.

Option A describes data analysis rather than intuitive test design. Option B is incorrect because checklist-based testing does not dynamically adapt test cases; instead, it follows predetermined checklists. Option D incorrectly defines “tour-based testing”; tours refer to structured exploratory approaches, not biased datasets.

Therefore,Option Cis the syllabus-aligned correct statement.

The ISTQB CT-AI syllabus (Section4.3 – Test Design for AI-Based Systems) highlights pairwise testing as an effectivetest-case reduction techniquefor systems with many input parameters. Lane-keeping assist systems typically include environmental, sensor, and vehicle-dynamic parameters, making exhaustive testing infeasible. Pairwise testing significantly reduces the number of test cases while still capturingall 2-way interactions, which are responsible for a large proportion of software defects.

OptionBaligns with this syllabus description: pairwise testing reduces otherwise extremely large parameter combinations, making test effort manageable.

Option A overstates feasibility guarantees; the syllabus never claims pairwise testing always makes testing “typically feasible.” Option C is unsupported and incorrect because pairwise testing doesnotrestrict parameters to driver-controlled ones. Option D is incorrect because, although some defects arise from higher-order interactions, pairwise testing captures many relevant defects and is widely recognized as a pragmatic compromise.

Thus,Option Bis the correct statement.

For AI-enabled medical devices, the most required aspects for certification are safety, transparency, and side effects. Here’s why:

Safety (Aspect III): Critical for ensuring that the AI system does not cause harm to patients.

Transparency (Aspect IV): Important for understanding and verifying the decisions made by the AI system.

Side Effects (Aspect V): Necessary to identify and mitigate any unintended consequences of the AI system.

Why Not Other Options:

Autonomy and Maintainability (Aspects I and II): While important, they are secondary to the immediate concerns of safety, transparency, and managing side effects in life-critical processes.

The syllabus describes metamorphic testing as:

“Testing involves defining metamorphic relations and then applying those relations to check that the transformations result in expected outcomes, even when the expected output of the system is unknown or not well-defined.”

In this scenario, applying the metamorphic relation (speed differences) and checking the transformed outcome (arrival times) fits the definition of metamorphic testing.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.5, page 69 of 99)

Recognizing digits from a scan of handwritten numbers using machine learning is an example of classification. Here’s a breakdown:

Classification: This type of machine learning involves categorizing input data into predefined classes. In this scenario, the input data (handwritten digits) are classified into one of the 10 digit classes (0-9).

Why Not Other Options:

Reinforcement Learning: This involves learning by interacting with an environment to achieve a goal, which does not fit the problem of recognizing digits.

Regression: This is used for predicting continuous values, not discrete categories like digit recognition.

Clustering: This involves grouping similar data points together without predefined classes, which is not the case here.

The syllabus describes exploratory testing as especially useful when there are poor specifications or test oracle problems:

“Exploratory testing is especially useful when there are poor specifications or test oracle problems, which is often the case for AI-based systems.”

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.6, page 70 of 99)

The syllabus defines reward hacking as:

"Reward hacking can result from an AI-based system achieving a specified goal by using a 'clever' or 'easy' solution that perverts the spirit of the designer’s intent."

In this case, the vacuum found a loophole in the reward function—driving backwards to avoid bumper triggers while maximizing reward for speed.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.6, page 24 of 99)

The ISTQB CT-AI syllabus indicates inSection 5.2 – AI for Testingthat AI-generated test cases may not come withpredefined expected results. This is because test-case generation methods—such as evolutionary algorithms, reinforcement learning, or clustering-based sampling—produceinputs, but the tester must still determine the correct outputs. Therefore, OptionCis correct: expected results may not be available, especially when AI produces novel or previously unseen input combinations.

Option A is incorrect: AI-based test generation can improve coverage by exploring large input spaces. Option B is incorrect because the need for oracles does not depend on whether the test case is AI-generated. Option D is incorrect because the syllabus allows using a model as an oracle inback-to-back testing, provided limitations are understood.

Therefore,Option Cis the correct syllabus-aligned statement.

The syllabus highlights that testing for unnecessary human intervention is a key focus for autonomous AI-based systems:

"For autonomous AI-based systems, testers must ensure that the system does not prompt for unnecessary human intervention, as this contradicts the autonomy concept."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 8.2, page 59 of 99)

The syllabus explains that ML models can be used to analyze reported defects and suggest which developers are best suited to fix them based on historical data about defect assignment and resolution speed:

"Assignment: ML models can suggest which developers are best suited to fix particular defects, based on the defect content and previous developer assignments."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 11.2, page 78 of 99)

Defect prediction models aim to identify parts of the software that are likely to contain defects by analyzing historical data and code quality metrics. The primary goal is to use this predictive information to allocate testing and maintenance resources effectively. Let's break down why option D is the correct choice:

Understanding Classification Models:

Classification models are a type of supervised learning algorithm used to categorize or classify data into predefined classes or labels. In the context of defect prediction, the classification model would classify parts of the code as either "defective" or "non-defective" based on the input features.

Input Data - Code Quality Metrics:

The input data for these classification models typically includes various code quality metrics such as cyclomatic complexity, lines of code, number of methods, depth of inheritance, coupling between objects, etc. These metrics help the model learn patterns associated with defects.

Historical Data:

Historical versions of the code along with their defect records provide the labeled data needed for training the classification model. By analyzing this historical data, the model can learn which metrics are indicative of defects.

Why Option D is Correct:

Option D specifies using a classification model to predict the presence of defects by using code quality metrics as input data. This accurately describes the process of defect prediction using historical bug data and quality metrics.

Eliminating Other Options:

A. Identifying the relationship between developers and the modules developed by them: This does not directly involve predicting defects based on code quality metrics and historical data.

B. Search of similar code based on natural language processing: While useful for other purposes, this method does not describe defect prediction using classification models and code metrics.

C. Clustering of similar code modules to predict based on similarity: Clustering is an unsupervised learning technique and does not directly align with the supervised learning approach typically used in defect prediction models.

The syllabus states that while pairwise testing is effective at finding defects by reducing the number of test cases needed, the resulting test suite can still be extensive and require automation:

"Even the use of pairwise testing can result in extensive test suites… automation and virtual test environments often become necessary to allow the required tests to be run."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.2, Page 67 of 99)

The syllabus defines bias as:

"Bias is the systematic difference in treatment of certain objects, people or groups in comparison to others."

It also discusses:

"Sample bias can occur if the data used for training the model does not represent the operational environment, or if some relevant faulty conditions are excluded deliberately."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 7.6 and 8.3)

To calculate the accuracy, recall, and specificity from the confusion matrix provided, we use the following formulas:

Confusion Matrix:

Actually Rotten: 45 (True Positive), 8 (False Positive)

Actually Fresh: 5 (False Negative), 42 (True Negative)

Accuracy:

Accuracy is the proportion of true results (both true positives and true negatives) in the total population.

Formula: Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN​

Calculation: Accuracy=45+4245+42+8+5=87100=0.87\text{Accuracy} = \frac{45 + 42}{45 + 42 + 8 + 5} = \frac{87}{100} = 0.87Accuracy=45+42+8+545+42​=10087​=0.87

Recall (Sensitivity):

Recall is the proportion of true positive results in the total actual positives.

Formula: Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}Recall=TP+FNTP​

Calculation: Recall=4545+5=4550=0.9\text{Recall} = \frac{45}{45 + 5} = \frac{45}{50} = 0.9Recall=45+545​=5045​=0.9

Specificity:

Specificity is the proportion of true negative results in the total actual negatives.

Formula: Specificity=TNTN+FP\text{Specificity} = \frac{TN}{TN + FP}Specificity=TN+FPTN​

Calculation: Specificity=4242+8=4250=0.84\text{Specificity} = \frac{42}{42 + 8} = \frac{42}{50} = 0.84Specificity=42+842​=5042​=0.84

Therefore, the correct combinations of accuracy, recall, and specificity are 0.87, 0.9, and 0.84 respectively.

A. Black box attacks based on adversarial examples create an exact duplicate model of the original.

Black box attacks do not create an exact duplicate model. Instead, they exploit the model by querying it and using the outputs to craft adversarial examples without knowledge of the internal workings.

B. These attack examples cause a model to predict the correct class with slightly less accuracy even though they look like the original image.

Adversarial examples typically cause the model to predict the incorrect class rather than just reducing accuracy. These examples are designed to be visually indistinguishable from the original image but lead to incorrect classifications.

C. These attacks can't be prevented by retraining the model with these examples augmented to the training data.

This statement is incorrect because retraining the model with adversarial examples included in the training data can help the model learn to resist such attacks, a technique known as adversarial training.

D. These examples are model specific and are not likely to cause another model trained on the same task to fail.

Adversarial examples are often model-specific, meaning that they exploit the specific weaknesses of a particular model. While some adversarial examples might transfer between models, many are tailored to the specific model they were generated for and may not affect other models trained on the same task.

Therefore, the correct answer isDbecause adversarial examples are typically model-specific and may not cause another model trained on the same task to fail​​.

TheAI Effectis clearly defined in theISTQB Certified Tester AI Testing Syllabus v1.0under Section1.1 – Definition of AI and AI Effect. The document explains that society’s understanding of what qualifies as “AI” changes over time. Technologies once considered AI—such as expert systems from the 1970s and 1980s or early chess-playing systems—are no longer viewed as AI today. This phenomenon is explicitly labeled the“AI Effect,”described as“the changing perception of what constitutes AI.”The syllabus states that as AI capabilities become routine or widely implemented, they often stop being perceived as true artificial intelligence .

Options B, C, and D do not capture this definition. While AI learning from data (B) is a property of ML, it does not describe the shifting perception of AI. Option C describes a technological achievement, not a perceptual shift. Option D references a historical AI milestone (Deep Blue defeating Kasparov) that the syllabus specifically uses as an example of technology that is no longer considered AI due to the AI Effect. Therefore, onlyOption Aaccurately reflects the AI Effect as defined by the ISTQB syllabus.

According to the ISTQB CT-AI syllabus,metamorphic testingworks by applyingmetamorphic relations (MRs): predictable input transformations that should lead to predictable output changes. From the initial test data, clear relations emerge for defensive driving scoring. The score increases when:

v_mean decreases,

a_pos decreases,

a_neg becomes less negative,and decreases when the opposite occurs.

A valid metamorphic follow-up test must modify inputs in a direction consistent with at least one MR while keeping the expected output direction predictable.

Test Alowers v_mean compared to Test 1, with similar acceleration values. This directly satisfies the MR that lower speed → higher score.

Test Cincreases both acceleration and braking intensity compared to Test 2, making the reduced score range (30–70) consistent with more aggressive driving.

Test Dmodifies acceleration and braking magnitudes in ways consistent with Test 3’s defensive-driving scoring boundaries.

Test B, however, changes multiple variables in contradictory directions:

v_mean increases (worse)

a_pos increases (worse)

a_neg becomes less negative (better)

Because these changes conflict,the expected score trend becomes unpredictable, violating the premise of a metamorphic follow-up test.

ThusTest B cannot be considered a metamorphic follow-up, which makesOption Ccorrect.

The ISTQB CT-AI syllabus describes theML data preparation workflowin Section2.2 – Data Preparation. Data preparation consists ofdata gathering,cleaning,transformation, andsampling. The syllabus emphasizes that one significant challenge duringdata gatheringis combining data frommultiple heterogeneous sources, which often differ in structure, quality, and format. Ensuring the resulting dataset is accurate, complete, and representative can be complex, making this a critical challenge in the ML workflow. This aligns directly with OptionC.

Option A is incorrect because erroneous data correction is part ofcleaning, not transformation. Option B contradicts the syllabus: while automation can help,not all steps should be automateddue to the need for expert oversight, especially in detecting subtle data quality issues. Option D is incorrect because sampling continues to involve risk—particularly around representativeness—and the syllabus emphasizes caution, not complacency.

Thus, OptionCis the only statement that accurately reflects the syllabus.

The ISTQB CT-AI syllabus explains in Section1.6 – Machine Learning Approachesthatsupervised learningis appropriate when labeled data exists and the goal is to predict an output based on known historical examples. Predicting a customer’spurchase probabilityis aclassificationtask when the output corresponds to discrete categories such as“likely to purchase”vs.“not likely to purchase.”The syllabus gives similar examples in describing classification as the process of assigning instances to predefined classes based on learned patterns in labeled data. Because the retail company wants to determine whether a customer will make a purchase based on marketing actions, classification is the most appropriate choice .

Option A (regression) predicts continuous numeric values and is less suitable because the task centers oncategorical likelihood, not estimating exact monetary values. Option C (unsupervised learning) is used when labels are not available—here, the company has labeled purchase histories. Option D (reinforcement learning) requires an interactive environment with reward-driven behavior, which is not applicable to this scenario.

Thus,supervised learning (classification)is the most suitable approach according to the syllabus.

 Coverage for Neuron Activation Values:Sign change coverage is used to ensure that test cases cause each neuron to achieve both positive and negative activation values. This type of coverage ensures that the neurons are thoroughly tested under different activation states.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 6.2 Coverage Measures for Neural Networks, which details different types of coverage measures, including sign change coverage​​.

A. The difficulty of defining criteria for improvement before the model can be accepted.

Defining criteria for improvement is a challenge in the acceptance of AI models, but it is not directly related to the performance drop in real-world scenarios. It relates more to the evaluation and deployment phase rather than affecting the model's real-time performance post-deployment.

B. The fast pace of change did not allow sufficient time for testing.

This can significantly affect the model's performance. If the system is self-learning, it needs to adapt quickly, and insufficient testing time can lead to incomplete learning and poor performance.

C. The unknown nature and insufficient specification of the operating environment might have caused the poor performance.

This is highly likely to affect performance. Self-learning AI systems require detailed specifications of the operating environment to adapt and learn effectively. If the environment is insufficiently specified, the model may fail to perform accurately in real-world scenarios.

D. There was an algorithmic bias in the AI system.

Algorithmic bias can significantly impact the performance of AI systems. If the model has biases, it will not perform well across different scenarios and data distributions.

Given the context of the self-learning nature and the need for real-time adaptability, optionAis least likely to describe the fall in performance because it deals with acceptance criteria rather than real-time performance issues​​.

The ISTQB CT-AI syllabus (Section5.3 – AI Support for Defect Analysis) explains that AI can categorize defect reports using natural language processing or classification models. Categorization helps route defects efficiently and determine which areas of the system are affected. Thus, OptionCis correct: AI canidentify defect categories, supporting assignment and triage.

Option A is incorrect because ML cannot infer severity or consequence without domain-specific risk modeling. Option B inaccurately ties duplicate detection to criticality; the syllabus separatesduplicate detectionfromseverity analysis. Option D is incorrect because AI may assist in routing defects using categories, but the syllabus does not claim AI identifies specific developers.

Thus,Option Cis the syllabus-accurate statement.

The syllabus explicitly mentions challenges of handling personal data and ensuring privacy when testing AI-based systems:

"The management of personal data and sensitive data is often a concern during testing, as testing typically requires realistic data and it is difficult to fully anonymize data."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 7.3, page 52 of 99)

The question asks which combination of tests would be most appropriate to include in the strategy for optimal detection in a workflow system using multiple ML models.

Pairwise testing of combinations (I): This method is useful for testing interactions between different components in the workflow to ensure they work well together, identifying potential issues in the integration.

Testing each individual model for accuracy (II): Ensuring that each model in the workflow performs accurately on its own is crucial before integrating them into a combined workflow.

A/B testing of different sequences of models (III): This involves comparing different sequences to determine which configuration yields the best results. While useful, it might not be as fundamental as pairwise and individual accuracy testing in the initial stages.

A. Regression

Regression is used to predict a continuous value and is not suitable for grouping people based on similar tastes.

B. Association

Association is used to find relationships between variables in large datasets, often in the form of rules (e.g., market basket analysis). It does not directly group individuals but identifies patterns of co-occurrence.

C. Clustering

Clustering is an unsupervised learning method used to group similar data points based on their features. It is ideal for identifying groups of people with similar tastes without prior knowledge of the group labels. This technique will help the company segment its customer base effectively.

D. Classification

Classification is a supervised learning method used to categorize data points into predefined classes. It requires labeled data for training, which is not the case here as we want to identify groups without predefined labels.

Therefore, the correct answer isCbecause clustering is the most suitable method for grouping people with similar tastes for targeted product marketing​​.

 Testing Autonomy:Testing for human handover when it should not be relinquishing control is the least appropriate because it contradicts the very definition of autonomous systems. The other tests are relevant to ensuring smooth operation and transitions between human and AI control.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Testing Autonomous AI-Based Systems and Testing for Human-AI Interaction​​​​. ​

The question asks which characteristic is least likely to cause safety-related issues for an AI system. Let's evaluate each option:

Non-determinism (A): Non-deterministic systems can produce different outcomes even with the same inputs, which can lead to unpredictable behavior and potential safety issues.

Robustness (B): Robustness refers to the ability of the system to handle errors, anomalies, and unexpected inputs gracefully. A robust system is less likely to cause safety issues because it can maintain functionality under varied conditions.

High complexity (C): High complexity in AI systems can lead to difficulties in understanding, predicting, and managing the system's behavior, which can cause safety-related issues.

Self-learning (D): Self-learning systems adapt based on new data, which can lead to unexpected changes in behavior. If not properly monitored and controlled, this can result in safety issues.

According to the syllabus, pre-trained models often inherit biases and limitations from the data and processes used in their original training, which may not align with the new use case. Specifically, the syllabus states:

"When using a pre-trained model, the training data and process cannot be fully controlled or known by the user of the model. As a result, the model can inherit biases or inaccuracies that were part of its original development and training process."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 1.8.3)

When written requirements are given in text documents, the best way to generate test cases is by using Natural Language Processing (NLP). Here's why:

Natural Language Processing (NLP): NLP can analyze and understand human language. It can be used to process textual requirements to extract relevant information and generate test cases. This method is efficient in handling large volumes of textual data and identifying key elements necessary for testing.

Why Not Other Options:

Analyzing source code for generating test cases: This is more suitable for white-box testing where the code is available, but it doesn't apply to text-based requirements.

Machine learning on logs of execution: This approach is used for dynamic analysis based on system behavior during execution rather than static textual requirements.

GUI analysis by computer vision: This is used for testing graphical user interfaces and is not applicable to text-based requirements.

The ISTQB CT-AI syllabus explains ML performance metrics in Section3.2 – Evaluating ML Models. Forclustering, which is an unsupervised learning method, the syllabus lists metrics such asintra-cluster distance,inter-cluster distance, and coherence measures. Intra-cluster metrics evaluate how close data points are within a cluster, which directly corresponds to Option A.

Option B is incorrect becauseR-squaredis a regression metric measuring goodness-of-fit, not classification performance, and has no connection to ROC curves. Option C is wrong because thesilhouette coefficientis also a clustering metric, measuring cohesion vs. separation—not regression accuracy. Option D is incorrect because ROC curves evaluatebinary or multiclass classification, not clustering.

Thus, OptionAis the only accurate statement based on the syllabus.

The syllabus mentions that experience-based testing and EDA are effective for detecting biases:

“Experience-based testing can be used to verify that the training dataset is operationally relevant and identify potential sources of bias. EDA is also useful for exploring the data and understanding any relationships that might lead to bias in the model.”

(Reference: ISTQB CT-AI Syllabus v1.0, Section 8.3, page 58 of 99)

The syllabus states:

“Supervised learning… divides problems into two categories: classification and regression. Regression is used when the problem requires the ML model to predict a numeric output, for example predicting the age of a person based on their habits.”

(Reference: ISTQB CT-AI Syllabus v1.0, Section 3.1.1, Page 26 of 99)

The ISTQB CT-AI syllabus explains supervised learning under Section1.6 – Machine Learning Approaches. It definesclassificationas predictingcategorical labels, whereasregressionpredictscontinuous numerical values. OptionB—deciding whether an object is a bicycle or a motorcycle—fits the definition of classification precisely because the model chooses between discrete categories. The syllabus also uses similar examples to illustrate classification tasks, reinforcing that this is the correct interpretation .

Option A is incorrect because image recognition of a dog is aclassificationtask, not regression. Option C is incorrect because predicting a 10% price rise involves forecasting anumerical value, which is aregressionproblem. Option D is incorrect because classification can involveany number of classes, not only two. Multiclass classification is explicitly mentioned in the syllabus.

Therefore, OptionBis the only answer aligned with the syllabus’ definitions.