CCAR-F Sample Questions Answers

Option C is the only approach that guarantees the required terminal condition independently of the model’s behavior. When the agent loop stops, the orchestration layer evaluates the recorded outcome. If neither a successful resolution nor a confirmed human escalation occurred, deterministic application code invokes escalate_to_human using the information accumulated before termination.

The max_turns limit is a host-controlled safety boundary. Once that boundary is reached, the model may no longer have an opportunity to follow a prompt instruction or make another tool call. Therefore, the required fallback cannot depend entirely on Claude deciding to escalate before its remaining turns are exhausted.

Option A escalates based on an arbitrary 80% threshold and may prematurely terminate cases that could have been resolved successfully within the remaining budget. Option B provides additional turn capacity but does not guarantee that the second agent will complete its work or escalate before its own limit is reached. Option D remains probabilistic and cannot operate after an unexpected termination has already occurred.

The orchestration fallback should be idempotent, ensuring that repeated completion checks cannot create duplicate escalation records. It should also include verified customer information, relevant order data, completed actions, failure reasons, and the unresolved request in the handoff payload.

Official references/topics: Agent-loop termination, deterministic fallback orchestration, terminal-state validation, human handoff guarantees.

A concrete test case converts an imprecise correction into an executable success criterion. The input demonstrates the failing null-value condition, while the expected output defines exactly how the migration must handle it. Claude can modify the implementation, run the test, inspect the result, and continue iterating until the behavior passes.

Anthropic recommends giving Claude a verification mechanism such as a test suite, build result, linter, fixture comparison, or screenshot. Without such a mechanism, Claude can only judge that its output appears correct. A pass-or-fail test closes the feedback loop by allowing Claude to perform the work, evaluate the result, and refine it using objective evidence. ( https://code.claude.com/docs/en/best-practices )

Option A provides no additional specification and may generate a different but still incorrect implementation. Option B fixes the immediate defect manually but fails to use Claude’s iterative capabilities or establish a regression check. Option C supplies more description, but regenerating the entire script increases the risk of altering already-correct behavior and still lacks objective validation.

The best workflow is to add the failing case to the migration test suite, request the smallest necessary correction, run the relevant tests, and retain that test permanently to prevent the null-handling defect from recurring.

Official references/topics: Executable verification, test-driven iteration, regression protection, tight feedback loops.

A mandatory compliance threshold must be enforced outside probabilistic model reasoning. A PreToolUse hook can inspect every attempted process_refund call before execution, compare its amount with the $500 threshold, and deny the call when the limit is exceeded. The host application can then create the human-escalation case using the validated customer and order context.

Anthropic describes hooks as deterministic controls that ensure required actions occur instead of relying on the model to choose them. Agent SDK documentation confirms that a single denying PreToolUse hook blocks the tool call, including MCP tools matched through their qualified tool names. ( https://docs.anthropic.com/en/docs/claude-code/hooks-guide )

Options A and B improve expected behavior but cannot guarantee compliance. The stated production failure rate demonstrates that prompting alone is insufficient. Option C prevents the refund from completing, but the subsequent escalation still depends on Claude correctly interpreting the returned error and invoking the human tool. It therefore enforces the financial block but not the complete escalation requirement.

Option D enforces both sides of the policy at the orchestration boundary: deny the unauthorized action and route the case to an approved human process. The escalation action should be idempotent and auditable.

Official references/topics: PreToolUse hooks, deterministic policy enforcement, MCP tool interception, human-approval controls.

The failure is caused by an underspecified extraction policy rather than an inadequate JSON structure. “Extract all skills mentioned” does not define whether coordinated phrases should be split, whether inferred capabilities qualify, or what level of specificity constitutes one skill. Few-shot examples can demonstrate all three decisions using concrete inputs and expected outputs.

Anthropic identifies examples as one of the most effective ways to improve output accuracy and consistency because they communicate behavior more precisely than abstract instructions alone. Representative examples can show that “Python and SQL” becomes two entries, that an unstated skill must not be inferred from a job title, and that broad descriptions should not be decomposed into dozens of speculative micro-skills. ( https://docs.anthropic.com/en/docs/about-claude/use-case-guides/ticket-routing )

Option A adds provenance and confidence metadata but does not define the correct extraction granularity. Option C addresses explicitness and splitting, but the arbitrary 10–20 limit can truncate documents containing more than 20 legitimate skills or force padding when fewer than 10 appear. Option D is useful for canonicalization and deduplication, but post-processing cannot reliably remove plausible implied skills or determine how the original phrase should have been interpreted.

The strongest prompt should combine explicit rules with diverse few-shot examples covering normal cases, compound phrases, implied information, duplicates, and unusually long or short skill lists.

Official references/topics: Few-Shot Prompting, Output Consistency, Explicit Extraction Criteria, Representative Examples.

Limited review capacity should be concentrated on records with the highest probability of semantic error. Schema validation confirms that the response has the correct structure and data types; it does not establish that the extracted values are accurate. Ambiguous wording, contradictory passages, missing evidence, and model-reported uncertainty are direct indicators that an extraction requires human judgment.

Anthropic’s reliability guidance recommends permitting Claude to express uncertainty, grounding factual outputs in source material, and validating critical information because hallucination-reduction methods do not eliminate errors completely. These principles support routing uncertain or evidentially conflicted records to reviewers rather than treating syntactically valid output as automatically trustworthy. ( https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/reduce-hallucinations )

Option B may waste reviewer capacity on clear, well-supported values while overlooking ambiguous errors in other fields. Option C is reactive: downstream acceptance does not guarantee semantic correctness, and silent errors may never produce processing failures. Option D provides an unbiased estimate of overall quality and should be retained as a secondary audit mechanism, but random selection is not the most efficient way to intercept risky records when only 5% can be reviewed.

In production, model confidence should not be treated as a calibrated probability by itself. It should be combined with objective signals such as contradictory source spans, missing citations, OCR quality, validation warnings, document type, and business impact.

Official references/topics: Human-in-the-Loop Review, Uncertainty Handling, Source Grounding, Risk-Based Escalation.

The checklist is a reusable, task-specific procedure that should load only when a pull-request review is being performed. A /review command provides an explicit invocation mechanism, prevents repeated copy-and-paste, and keeps the checklist out of unrelated coding, debugging, and documentation sessions.

In current Claude Code terminology, custom commands have been consolidated into skills. A .claude/skills/review/SKILL.md file creates a /review command, while the legacy .claude/commands/review.md format remains supported. Anthropic recommends skills when users repeatedly supply the same checklist or multi-step procedure; unlike CLAUDE.md content, the skill body is loaded only when invoked or determined relevant. ( https://code.claude.com/docs/en/skills )

Option B introduces an isolated agent context when the requirement is primarily to inject a repeatable review procedure. A subagent could be useful for independent verification, but it is not necessary merely to avoid repasting the checklist. Option C bloats every session with instructions that apply only to code review. Option D controls editing permissions and planning behavior; it does not encode review criteria.

The review skill can also pre-authorize read-only tools and accept a pull-request identifier or branch as an argument.

Official references/topics: Claude Code Skills; Custom Slash Commands; On-Demand Instructions; Reusable Review Workflows.

The root cause is an incomplete tool contract. Claude sees that process_refund requires an order_id , but the tool description does not explain the parameter’s trusted source or the prerequisite lookup sequence. Option A makes the dependency explicit and prohibits fabricated values.

Anthropic states that tool descriptions should explain what a tool does, when it should be used, how it behaves, and any important limitations. Its prompting guidance also states that dependent tool calls must be executed sequentially and that Claude must never use placeholders or guess missing tool parameters. ( https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools )

Option B merely forces some tool use; it does not force the correct tool or prevent a fabricated identifier. Option C is an essential defense-in-depth control, but it detects the invalid ID after the model has already made the faulty call. It does not correct the selection logic that caused the fabrication. Option D works only when the customer actually provides an identifier and introduces an unnecessary preprocessing dependency.

The process_refund schema should describe order_id as a verified identifier returned by lookup_order , and the tool description should state that the tool is unavailable until the corresponding order has been retrieved and eligibility established.

Official references/topics: Tool descriptions, prerequisite tool calls, parameter provenance, sequential tool orchestration.

The search must first establish the function’s complete naming surface. Reading the core definition and every wrapper or re-export reveals aliases such as computeOrderTax , names introduced through intermediate modules, and potentially different import paths. Once those names are known, the agent can search the entire codebase for each exposed identifier and inspect the resulting call sites.

Claude Code’s Grep tool searches patterns inside file contents, whereas Read provides the surrounding implementation needed to determine whether a wrapper delegates to, renames, or conditionally invokes the original function. These tools are complementary: Read establishes semantic identity; Grep provides broad reference discovery. ( https://code.claude.com/docs/en/tools-reference?utm_source=chatgpt.com )

Option B misses every caller using a wrapper alias. Option A may find importing files, but import-level discovery is indirect and can miss re-exports, namespace imports, dependency injection, or calls made through a locally renamed symbol. It also requires unnecessary manual inspection of every consumer of the module. Option D relies on documentation, which may be incomplete or stale and cannot prove that all executable references have been found.

Before removal, the agent should also inspect tests, dynamic registrations, configuration-driven references, and generated code where relevant. The defining requirement is to map the alias chain first and then search all discovered public names.

Official references/topics: Read and Grep Tool Behavior; Symbol and Alias Discovery; Codebase Reference Tracing.

The implementation approach is unresolved and carries architectural consequences. WebSockets, Server-Sent Events, and polling differ in connection lifecycle, bidirectional communication, proxy compatibility, scaling requirements, deployment topology, reconnection behavior, and operational complexity. Claude should therefore inspect the existing application and infrastructure before modifying code.

Anthropic recommends separating exploration and planning from implementation when the approach is uncertain, the work affects multiple files, or the developer is unfamiliar with the relevant architecture. In plan mode, Claude can read files and analyze the system without making changes, produce a detailed proposal, and allow the developer or team to review the plan before execution. ( https://code.claude.com/docs/en/best-practices )

Option A commits to polling before determining whether its latency and load characteristics meet the requirement. Option B delegates a consequential architectural decision and immediate implementation to a single execution step without a review checkpoint. Option D prematurely selects the most infrastructure-sensitive option and assumes later refactoring will be inexpensive.

Option C creates the correct sequence: examine the frontend and backend architecture, document requirements, compare alternatives, identify infrastructure constraints, obtain approval, and only then implement the selected design with appropriate tests and observability.

Official references/topics: Plan mode, explore–plan–implement workflow, architectural trade-off analysis, review checkpoints.

Forking the existing analysis session creates independent continuations that inherit the accumulated conversation context. Each branch begins with the same understanding of the service module, request flow, source files, and error-handling patterns, but subsequent work on one testing strategy does not alter the other branch or the original session.

Anthropic’s Agent SDK documentation states that sessions can be resumed with their full context and forked to explore different approaches. In the SDK, enabling fork_session while resuming causes the continuation to receive a new session identifier rather than modifying the original session. ( https://docs.anthropic.com/en/docs/claude-code/sdk?utm_source=chatgpt.com )

Option B wastes time, tokens, and tool calls by requiring both new sessions to rebuild the same 23-file analysis. Option C mixes two experimental implementations into one conversation, increasing the risk that assumptions, edits, or conclusions from the first strategy influence the second. Option D preserves only a manually selected summary, which may omit details contained in the full session history.

The appropriate design is to create one fork for the end-to-end strategy and another fork for the snapshot strategy. The original analysis remains a stable parent, while each child session develops and evaluates its approach independently.

Official references/topics: Agent SDK Sessions, Session Forking, Context Preservation, Alternative-Approach Evaluation.

MCP prompts are exposed as user-invoked commands rather than autonomous tools or permanently loaded system instructions. Claude Code dynamically discovers prompts from connected MCP servers and displays them in the command list using the naming convention /mcp__servername__promptname .

Arguments are supplied as space-separated values after the command. When executed, the MCP server resolves the prompt and its returned content is injected into the active conversation. Anthropic’s official documentation provides examples such as /mcp__github__list_prs and /mcp__jira__create_issue "Bug in login flow" high . ( https://code.claude.com/docs/en/mcp )

Option A would consume context continuously and incorrectly treat optional workflow templates as mandatory system instructions. Option B confuses MCP prompts with MCP tools: tools are model-callable operations, while prompts are reusable prompt templates invoked as commands. Option C describes MCP resources, which can be referenced and attached but are a distinct MCP capability.

For the stated server, the team could invoke commands such as /mcp__devops__deploy_checklist or /mcp__devops__incident_response service-name . The exact server segment is derived from the configured server name, with normalization applied where necessary.

Official references/topics: MCP Prompts; Dynamic Prompt Discovery; MCP Slash-Command Naming; Prompt Arguments.

The investigation contains several bounded research questions that can be delegated independently: locating the complete test suite, tracing payment and refund flows, identifying conditional branches, and mapping external dependencies. Each subagent can read the relevant files in its own context and return a focused summary to the coordinating agent.

Anthropic recommends subagents for codebase exploration because extensive file reading rapidly consumes the main context window. Subagents isolate that volume and return only their conclusions, preserving the main conversation for synthesis and implementation. ( https://docs.anthropic.com/en/docs/claude-code/common-workflows ) Anthropic also describes parallel research as appropriate when separate investigation paths can proceed independently and the main agent can synthesize the results afterward. ( https://docs.anthropic.com/en/docs/claude-code/sub-agents )

Option B sacrifices the current conversational state and requires reconstruction after /clear . Option C may reduce token usage, but isolated text matches cannot reliably reveal full execution paths, indirect calls, or test coverage relationships. Option D converts the current analysis into a single lossy summary and risks omitting details needed later.

Option A directly addresses the demonstrated context degradation while retaining a high-level coordinating thread. The subagent prompts should be narrowly scoped and require concrete outputs such as file paths, uncovered branches, call-chain evidence, and existing tests associated with each flow.

Official references/topics: Subagent Context Isolation; Parallel Research; Context Preservation; Coordinated Codebase Analysis.

Option B preserves the formal failure indicator while making the returned content operationally useful. Anthropic’s tool-use guidance states that when a tool fails, the application should return the error through tool_result content and mark it with is_error: true . Claude can then use that information to retry with corrected input, seek clarification, or explain the limitation. ( https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/bash-tool )

The two example messages identify both the cause and the appropriate recovery path. “Order not found” indicates that repeating the same identifier will not help and proposes an alternative lookup method. “Database timeout” identifies a transient infrastructure condition for which a retry is reasonable. This eliminates blind repetition without removing Claude’s ability to adapt.

Option A can be useful for tightly bounded low-level retries, but it does not solve permanent errors or tell the agent what happened after retries fail. Option C incorrectly disguises a failed operation as successful tool content. Option D moves semantic knowledge into hardcoded orchestration logic, making the system more rigid and duplicating information already known by the tool.

The most reliable tool interface combines the error flag, a stable error type, a retryability indicator, a concise causal explanation, and a recommended recovery action.

Official references/topics: is_error handling, instructive tool results, recovery guidance, resilient MCP tool design.

The pipeline must preserve source evidence while making inconsistencies explicit. Option D records the amount stated on the invoice separately from the total derived from extracted line items. A mismatch then becomes a machine-detectable validation condition rather than an invisible extraction defect.

This approach is superior because it does not silently overwrite source data or ask another model to guess which value is correct. Anthropic’s evaluation guidance recommends automated, code-based grading whenever the criterion can be expressed deterministically. Arithmetic reconciliation is precisely such a criterion. ( https://docs.anthropic.com/en/docs/build-with-claude/develop-tests ) In production, the summation should preferably be calculated by application code using normalized decimal values, even though the option describes the model populating calculated_total . The essential design principle remains the same: preserve stated_total , compute an independent total, compare them, and route discrepancies for adjudication.

Option A might improve behavior but cannot resolve genuine OCR corruption and could encourage the model to modify extracted values merely to create mathematical consistency. Option B introduces a second probabilistic judgment without new evidence. Option C is unacceptable for accounting data because it fabricates adjusted amounts and destroys fidelity to the invoice.

The schema should therefore expose both values and attach a validation status or review reason when they differ.

Official references/topics: Deterministic Validation; Human-in-the-Loop Review; Structured Output Design; Source-Fidelity Controls.

Prompt chaining is appropriate because the workflow consists of predictable, fixed subtasks that apply to every pull request. The system can run a focused style-compliance analysis, then a security analysis, then a documentation review, and finally synthesize the three results into a consistent report.

Anthropic defines prompt chaining as decomposing a task into a sequence of steps in which each call handles a smaller component. It is recommended when a task can be cleanly divided into fixed subtasks, trading additional latency for better focus and accuracy. Intermediate checks can also be inserted between stages to confirm that each report section meets its requirements. ( https://www.anthropic.com/engineering/building-effective-agents )

Option A concentrates all considerations into one call, making it easier for one aspect to receive inadequate attention. Option B is unnecessary because orchestrator-workers is intended for complex work where the required subtasks cannot be predicted in advance. Here, the three review dimensions are already known and remain constant. Option D solves a different problem: routing is appropriate when inputs belong to distinct categories that require different downstream processes.

Although the three analyses could potentially be parallelized for lower latency, the option that accurately represents the stated repeatable decomposition is prompt chaining with final synthesis.

Official references/topics: Prompt Chaining, Fixed Subtasks, Intermediate Validation, Workflow Selection.