CT-GenAI Sample Questions Answers

Understanding the hierarchy of LLM types is vital for selecting the right tool for specific testing tasks.Foundation LLMsare trained on massive datasets to predict the next token; they excel at broad, creative "ideation" (Statement i) but often struggle with following specific instructions or constraints (making Statement iv incorrect).Instruction-tuned LLMshave undergone additional training (Fine-tuning) to follow explicit commands and templates. They are highly effective at structured tasks like converting requirements into Gherkin feature files (Statement ii).Reasoning LLMs(or those utilizing specialized prompting like Chain-of-Thought) are designed to handle complex, multi-stage logic. This makes them the superior choice for diagnostic tasks like root-cause analysis, where the model must synthesize information across logs and requirements to find a defect's origin (Statement iii). Statement v is incorrect because while instruction-tuned models are capable, complex "stepwise reasoning" usually requires specific prompting techniques or the inherent logic of specialized reasoning models. Therefore, the combination of i, ii, and iii represents the correct alignment of model capability to testing functionality.

The effectiveness of an LLM is heavily dependent on the specificity of itsOutput Format. While role definition (Option C) and technical instructions (Option D) are helpful, the most significant "value add" for a test lead is receiving information that is directlyactionable. By expanding the output format to include structuredrisk predictions, severity levels, and recommended actions(Option B), the tester is forcing the LLM to perform a deeper level of analysis. Instead of just "flagging trends," the model must now synthesize the data to determinewhya trend is a risk andwhatthe team should do about it. This aligns with the "Advanced Prompting" section of the CT-GenAI syllabus, which emphasizes using AI for decision support. A structured report that includes a "timeline for intervention" allows the human tester to quickly validate the AI's logic and make informed decisions, transforming the LLM from a simple data summarizer into a strategic predictive tool that actively supports the maintenance of release quality and schedule adherence.

A Retrieval-Augmented Generation (RAG) workflow is designed to "ground" an LLM's output in specific, verifiable data. The logical flow begins with an initial input or "focused query" (Step iv) that defines the tester's goal—in this case, generating cases for a new payments rule. The system then uses this query to perform a semantic search in avector database(Step i) to find the most relevant "chunks" of the new API specification. Following this, the system retrieves complementary data from therelational database(Step iii), such as historical test cases that might provide structural patterns or regression context. Finally, all the retrieved information—the new specs and the historical context—is bundled together and "fed" into the LLM as part of an augmented prompt (Step ii). This ensures the LLM doesn't hallucinate rules but instead synthesizes the new requirements with established organizational testing standards. Following the order in Option B ensures that the model is provided with the most relevant and logically organized context prior to generating the final testware.

Hallucinations—where an LLM generates factually incorrect or nonsensical information—occur primarily when the model lacks sufficient specific information and "fills in the gaps" using probabilistic patterns from its training data. The most effective mitigation strategy is "grounding," which involves providing the model with detailed, project-specific context. By including technical specifications, existing API schemas, business rules, and identified constraints within the prompt, the tester restricts the model’s operational space to the "project realities." This ensures the model does not have to guess or improvise details about the System Under Test (SUT). In contrast, randomizing prompts (Option B) or relying on generic examples (Option C) increases the likelihood of inconsistent and inaccurate outputs. Furthermore, using "longer" or higher temperature settings (Option D) actually encourages creativity and randomness, which is the opposite of the precision required for testing and significantly increases the risk of hallucinations. Therefore, rich contextual grounding is the technical foundation for reliable AI-assisted test analysis.

As Generative AI is integrated into the testing lifecycle, the role of the human tester undergoes a significant shift from "author" to "orchestrator and reviewer." In traditional testing, a significant portion of a tester's time is spent manually drafting test cases, scripts, and documentation. With GenAI, these artifacts can be generated in seconds. Consequently, the tester’s responsibility shifts towardreviewing, refining, and validatingthe AI-generated testware to ensure accuracy, relevance, and compliance with project goals. This "Human-in-the-Loop" (HITL) approach is critical because LLMs are prone to hallucinations and may lack the deep domain context of a human expert. Testers must apply their critical thinking to verify that the AI-generated scripts actually cover the necessary edge cases and do not contain logical errors. This evolution does not mean the end of human oversight (Option B) or a move exclusively to white-box testing (Option C). Instead, it elevates the tester to a higher-level analytical role, focusing on quality strategy and the final verification of AI outputs rather than the repetitive task of initial content creation.

A/B testing, also known as split testing, is a systematic empirical method used to compare two versions of a prompt (Version A and Version B) to determine which one performs better based on predefined evaluation metrics. In the realm of LLMs, where outputs can be stochastic (probabilistic), A/B testing is essential for mitigating inconsistency. When a team encounters vague or varying results for a user story, simply modifying the prompt iteratively (Option B) may improve the result but does not provide a statistical or objective basis for why one version is superior. Byrunning A/B tests, testers can evaluate prompts against specific KPIs such as accuracy, relevance, format adherence, or the absence of hallucinations. This process involves sending the same input data through both prompt versions multiple times and scoring the outputs. The version that consistently yields the "stronger wording" or more precise testware is then selected as the production standard. This data-driven approach is a cornerstone of prompt engineering in professional environments, ensuring that the most effective linguistic structures are utilized to maximize the model's performance and reliability.

This scenario describes a form ofRequest Manipulation, specifically a type of "Prompt Injection" or "Adversarial Prompting." In this attack vector, the user (or an external attacker) provides malicious or deceptive input—in this case, via an image in a multimodal LLM—to bypass the model's intended constraints or to steer its logic toward an unintended outcome. By crafting an image that tricks the LLM into seeing "acceptance criteria" that aren't actually there, the attacker manipulates the model’s request processing to generate false validation results. This is different fromData Poisoning(Option A), which involves corrupting the training data before the model is even built. It is also distinct fromData Exfiltration(Option B), which aims to steal data from the model. In a testing environment, request manipulation is a significant risk because it can lead to "Silent Failures," where the AI reports that tests have passed or requirements are met based on deceptive input, thereby compromising the integrity of the entire Quality Assurance process.

A successful Generative AI strategy for testing is heavily dependent on the quality of the data used for grounding (RAG) and prompting. The principle of "Garbage In, Garbage Out" is magnified with LLMs; therefore, a key strategic pillar is the prioritization of accurate, relevant, and high-quality input data. This involves establishing defined quality procedures to ensure that the requirements, codebases, and historical defect logs fed into the model are "clean" and representative of the current system state. Strategy must avoid the "unfiltered" approach (Option C), as including contradictory or obsolete data can lead to hallucinations or irrelevant test cases. While synthetic data (Option D) is a powerful tool for privacy, it cannot entirely replace the nuanced reality found in secured enterprise data. Furthermore, legacy data (Option A) often contains valuable insights for regression testing. Consequently, the strategy should focus on building a robust data pipeline that ensures only verified, contextually appropriate information is utilized, thereby increasing the reliability of AI-generated testware and ensuring it aligns with the organization's quality standards.

Few-shot promptingis the technique of providing a few examples (exemplars) within the prompt to demonstrate the desired task and output format to the LLM. In this scenario, providing 10 existing, high-quality test cases acts as a "pattern" for the model to follow. This is significantly more effective than "Zero-shot prompting" (Option D), where the model is given a task without examples and may deviate from the specific organizational format required (e.g., specific JSON structures or assertion styles). While "Prompt chaining" (Option A) is useful for breaking down complex tasks into sub-tasks, the primary need here is pattern recognition and replication, which is the core strength of Few-shot learning. "Meta prompting" (Option C) involves having the AI write the prompt itself, which is unnecessary when the team already has clear examples. By using Few-shot prompting, the tester "conditions" the model's latent space to prioritize the provided format, ensuring that all 500 generated test cases maintain consistency with the HTTP methods, headers, and assertion logic defined in the exemplars.