CCDAK Sample Questions Answers

According to the Apache Kafka producer documentation, serialization is configured independently for message keys and values. The key and value can use different serializers depending on their data types.

In this scenario, the message key is a string, so the producer must be configured with org.apache.kafka.common.serialization.StringSerializer for the key. The value contains JSON, so a suitable JSON serializer must be configured for the value, such as a custom JSON serializer, a library-based serializer, or a byte-array serializer after converting the JSON to bytes.

Option B correctly reflects this separation by specifying a string serializer for the key and a JSON serializer for the value. Kafka does not automatically infer serializers, and setting a serializer to null is invalid. Additionally, there is no built-in “JSON” serializer for keys, making Option D incorrect.

Therefore, the correct approach—fully aligned with Kafka’s producer configuration model—is to explicitly configure a StringSerializer for the key and a JSON-capable serializer for the value.

=========================

Comprehensive and Detailed Explanation From Exact Extract:

✅ C. Consumers can consume both committed and uncommitted transactions.By default, Kafka consumers only read committed messages if they are configured with isolation.level=read_committed. However, if configured as read_uncommitted, they can also consume uncommitted (potentially aborted) transactional messages.

From Kafka Documentation:

"The isolation.level setting controls whether the consumer will read only committed messages or all messages, including uncommitted messages from ongoing or aborted transactions."

✅ D. Information about producers and their transactions is stored in the _transaction_state topic.Kafka uses an internal topic named __transaction_state to maintain metadata about producer transactions. This topic is essential for tracking the transaction lifecycle, fencing, and recovery.

From Kafka Internals:

“Kafka stores the state of active and completed transactions in an internal topic called __transaction_state.”

These three metrics are critical for cluster health:

OfflinePartitionsCount: Indicates partitions without active leaders — a sign of broker failure.

ActiveControllerCount: There should be exactly one active controller. A count ≠ 1 signals controller failure.

UncleanLeaderElectionsPerSec: Tracks leader elections where out-of-sync replicas were selected — risky for data loss.

From Kafka Monitoring Documentation:

“Offline partitions and unclean leader elections should trigger alerts. Also, ensure a single active controller is running.”

A is about topics pending deletion — not critical.

E is a per-topic record metric, not broker-level.

To increase consumer throughput, one effective strategy is to increase the amount of data fetched in each poll by raising fetch.max.bytes or max.partition.fetch.bytes. This allows each poll to retrieve more records per request, improving processing efficiency.

From Kafka Consumer Config Docs:

“Increasing fetch size allows consumers to retrieve larger batches of messages, improving throughput and reducing request overhead.”

Removing consumers (A) may reduce parallelism.

Manual commit (C) adds complexity, not throughput.

Decreasing session timeout (D) risks unnecessary rebalances.

The Apache Kafka Streams documentation clearly distinguishes between the Kafka Streams DSL and the Processor API. While the DSL is designed for common stream processing operations such as filtering, mapping, joining, and aggregations, it does not support fully custom processing logic.

For use cases that require custom transformations, fine-grained control over record processing, access to headers, timestamps, state stores, and punctuation, Kafka Streams provides the Processor API. This API allows developers to implement custom Processor, Transformer, or ValueTransformer classes and explicitly define the processing topology.

Option A (TopologyTestDriver) is a testing utility, not a development API. Option C (DSL) is higher-level and not suitable for advanced custom logic. Option D does not exist in Kafka.

Therefore, the correct and officially supported approach for defining custom processors in Kafka Streams is to use the Processor API.

=========================

The Apache Kafka producer API is asynchronous by default, and proper error handling requires explicit mechanisms. The official documentation recommends using a callback function with producer.send() to determine whether a record was successfully acknowledged or failed due to an exception. The callback provides access to both RecordMetadata and any exception encountered during sending.

Additionally, certain errors—such as serialization errors or configuration issues—are thrown synchronously when calling producer.send(). These must be handled using a try/catch block that catches KafkaException or its subclasses.

Option A is invalid because ProducerRecord has no status() method. Option C is incorrect because producer.send() returns a Future, not a RecordMetadata object directly.

Thus, combining a callback for asynchronous errors and a try/catch block for synchronous errors is the correct and officially documented approach to ensure reliable producer error handling when using acks=all.

=========================

Kafka Connect can run in standalone mode or distributed mode. For simple tasks like reading logs from a file on a single host, standalone mode is recommended.

From Kafka Connect User Guide:

“Standalone mode is useful when running connectors on a single machine (e.g., for development or simple deployments like log collection from a local file).”

Distributed mode is preferred for scalability and fault tolerance but overkill for this use case.

Kafka Connect source connectors use the offset.storage.topic configuration parameter to define where to store offsets that track the connector's position in the source system (e.g., database, file, etc.).

From Kafka Connect Worker Configuration:

“offset.storage.topic specifies the topic in Kafka where the connector stores offsets for each source partition.”

This allows Kafka Connect to resume from the correct position on restart.

The other options (B–D) are not valid Kafka configs.

In the Kafka Streams API, operations are classified as stateless or stateful based on whether they require maintaining local state stores. According to the official Kafka Streams documentation, stateless operations process each record independently, without storing or accessing prior records.

Filter is a stateless operation because it evaluates each record individually and decides whether to pass it downstream. It does not require any state or historical context.

GroupBy is also considered stateless because it merely repartitions the stream by assigning a new key and forwarding records to downstream processors. While it triggers the creation of an internal repartition topic, the GroupBy operation itself does not maintain a state store.

In contrast, Reduce is a stateful operation because it aggregates records over time and requires maintaining intermediate results in a state store. Similarly, Join operations are stateful because Kafka Streams must buffer and store records from one or both input streams or tables to perform the join within a defined time window.

Thus, the correct stateless operations are Filter and GroupBy, as documented in the Kafka Streams developer guide.

The Apache Kafka documentation states that consumer offsets are stored per consumer group and must be explicitly reset if an application needs to reprocess data from an earlier position. The officially supported and safest way to reset offsets for an existing consumer group is by using the kafka-consumer-groups CLI tool with the --reset-offsets option and the --to-earliest flag.

Setting auto.offset.reset=earliest (Option D) only applies when no committed offset exists for a partition. Since this consumer is already running in production, offsets are already committed, making this option ineffective. Option C does not change offsets because Kafka identifies consumers by group ID, not application instance. Option B is extremely dangerous and incorrect, as setting retention.ms=0 deletes data for all consumers and impacts other applications.

Therefore, using the CLI to reset offsets for the specific consumer group ensures controlled, auditable, and targeted offset management without affecting other consumers or the topic itself. This approach is explicitly documented in Kafka’s consumer group administration tools.

=========================

The Apache Kafka documentation explains that Kafka’s parallelism model is based on topic partitions and consumer groups. Within a single consumer group, each partition can be assigned to at most one consumer at a time. As a result, the maximum level of parallelism for a consumer group is equal to the number of partitions in the topic.

In this scenario, the topic has five partitions, which means that up to five consumers in the same consumer group can actively consume data in parallel. If more than five consumers are added, the extra consumers will remain idle because there are no additional partitions to assign.

Option B is incorrect because Kafka does not enforce equal message consumption across partitions; message distribution depends on producers and partitioning strategy. Option C is false because partitions may be hosted on different brokers, and consumers fetch data from multiple brokers. Option D is incorrect because a single partition cannot be consumed by multiple consumers within the same group simultaneously.

Therefore, the only valid assumption is that a maximum of five consumers can actively consume messages in the same consumer group.

=========================

Kafka message headers are the right place to include metadata such as:

Payload format (e.g., schema version)

Timestamps (custom)

Trace or correlation IDs

From Kafka Producer API Docs:

“Headers are a map of key-value pairs that can be used to include metadata alongside the message, without modifying the message body.”

Keys determine partitioning and ordering

Values are the message payload

Brokers do not store arbitrary metadata fields

In Kafka Streams, the number of stream tasks = number of input partitions × num.stream.threads × number of instances, but only as many as the number of partitions can actively consume at once.

However, in this case, Kafka Streams assigns one task per partition, and because there are 5 partitions and 15 threads (3 instances × 5 threads), 15 tasks are created, and all 15 can be active depending on processing topology.

From Kafka Streams Developer Guide:

"Kafka Streams creates one task per input partition. If you increase the number of stream threads, it runs multiple tasks in parallel within a single instance."

So, 15 stream tasks will be created and 15 will be actively consuming.

RetriableException is a subclass of KafkaException, indicating transient issues that might succeed if retried. Examples include:

LeaderNotAvailableException: Happens when metadata is not yet propagated or leader election is in progress.

NotEnoughReplicasException: Happens when the number of in-sync replicas is insufficient. This can be transient if replicas come back online.

From the Apache Kafka Java Client documentation:

"These exceptions (like LeaderNotAvailableException, NotEnoughReplicasException) are transient and the client will retry them automatically depending on retry configuration."

RecordTooLargeException and AuthorizationException are non-retriable as they indicate client-side or permission errors.

When using Kafka Connect with sink connectors (like S3 Sink), each task is assigned one or more topic partitions. The total number of partitions (6) is evenly distributed across the available tasks (2). Thus, each task will handle 3 partitions.

From Kafka Connect Documentation:

“Kafka Connect divides the topic partitions among available tasks. For example, a topic with six partitions and two tasks results in each task handling three partitions.”

Kafka Connect sink connectors use the standard Kafka consumer mechanism to track offsets, which means offsets are stored in the __consumer_offsets internal topic.

From Kafka Connect Documentation:

“Sink connectors are regular Kafka consumers and store their offsets in the __consumer_offsets topic like any other consumer.”

The other options refer to source connectors or worker configuration.

offsets.storage.topic is relevant for source connectors.

Table → Product, Customers

Stream → Orders_Placed, Shipment_Of_Orders

Tables represent slowly changing, reference data — e.g., customer profiles, product info.

Streams represent real-time event data — e.g., orders placed, shipments processed.

From Kafka Streams Documentation:

“Use KTable for reference datasets, and KStream for event-based processing such as orders or logs.”

For real-time applications, the key metric to monitor is consumer lag — the difference between the latest offset on the broker and the last committed offset by the consumer.

From Kafka Monitoring Guide:

“Consumer lag is the most important metric for real-time applications. It tells you how far behind the consumer is from the latest data in Kafka.”

B relates to replication, not client responsiveness.

C tracks group membership stability, not data delay.

D tracks input throughput, not processing latency.

Unit Testing → MockProducer

Integration Testing → Testcontainers

Performance Testing → Trogdor

Mock Data Generation → Connect Datagen

MockProducer: Simulates a Kafka producer in unit tests (no real broker interaction).

Testcontainers: Spawns Kafka in Docker for real environment testing.

Trogdor: Kafka’s built-in performance load testing framework.

Connect Datagen: Creates sample source records for test and demo purposes.

From Kafka Developer Tools Guide:

“Kafka developers commonly use MockProducer for unit tests, Testcontainers for integration, and Trogdor for performance tests.”

According to the official Apache Kafka Streams documentation, session windows are specifically designed to model periods of activity separated by gaps of inactivity. A session window groups events that occur close together in time and closes when no new events arrive within a defined inactivity gap.

In this use case, tracking user shopping cart behavior requires detecting when a user becomes inactive (for example, stops adding or removing items). Session windows naturally represent this pattern by creating a new window when activity resumes after a period of silence. This makes them ideal for modeling user sessions, cart abandonment detection, and inactivity analysis.

Hopping and tumbling windows (Options B and C) are fixed-time windows and are not well-suited for detecting inactivity, as they segment data based on clock time rather than behavior. Sliding windows are primarily used for continuous joins and do not explicitly model inactivity gaps.

Therefore, session windows are the correct and officially recommended choice for identifying periods of user inactivity in Kafka Streams applications.

=========================

This scenario describes vertical scaling, as defined in the Apache Kafka Streams documentation and general distributed systems terminology. Vertical scaling refers to adding more resources (CPU, memory, or disk) to a single machine to increase processing capacity.

In this example, the Kafka Streams application is moved from a server with four CPU threads to one with ten CPU threads, increasing the available compute resources without increasing the number of application instances. Kafka Streams can take advantage of additional CPU threads through its num.stream.threads configuration, allowing more partitions to be processed in parallel within the same application instance.

Horizontal scaling (also known as scaling out) would involve adding more application instances across multiple servers, not upgrading a single server. Option D is therefore incorrect, and Option C is not a recognized scaling term.

Thus, this example clearly illustrates vertical scaling, where performance is improved by increasing the capacity of a single node.

According to the official Apache Kafka documentation, log compaction is the mechanism used to retain only the latest record for each key in a topic. By setting cleanup.policy=compact, Kafka ensures that older records with the same key are eventually removed, leaving only the most recent value for each key.

This behavior is exactly suited for use cases such as tracking the current state of an entity, including GPS tracker locations, user profiles, or configuration data. In this scenario, the key (trackerId) uniquely identifies a tracker, and the value represents its most recent latitude and longitude.

Option B (retention.ms=infinite) retains all historical data, which contradicts the requirement. Option C (min.cleanable.dirty.ratio) controls when compaction runs, not what data is retained. Option D (retention.ms=0) would immediately delete all data and is unsafe.

Therefore, setting cleanup.policy=compact is the correct and officially documented solution to always retain the latest location per tracker.

=========================

The official CLI utility for consuming messages from Kafka topics is kafka-console-consumer.sh. It connects to the broker, consumes messages, and prints them to standard output.

From Kafka CLI Tools Documentation:

"kafka-console-consumer.sh is used to read data from a Kafka topic and write it to standard output."

The other options are not valid Kafka CLI tools.

A: Kafka batches store compression metadata in their headers, indicating how the batch was compressed.

B: Compression is controlled via the producer config compression.type, which defaults to none. Producers explicitly set Gzip or other formats.

From Kafka Producer Docs:

“Compression is done by the producer, and the type is stored in the message batch metadata. The broker does not recompress.”

C is false: messages are serialized before compression.

D is false: messages without keys are distributed across multiple partitions.