Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Sample Questions Answers

Comprehensive and Detailed Explanation From Exact Extract:

The number of tasks is controlled by the number of partitions. By default,spark.sql.shuffle.partitionsis 200. If stages are showing very few tasks (less than total cores), you may not be leveraging full parallelism.

From the Spark tuning guide:

"To improve performance, especially for large clusters, increasespark.sql.shuffle.partitionsto create more tasks and parallelism."

Thus:

A is correct: increasing shuffle partitions increases parallelism

B is wrong: it further reduces parallelism

C is invalid: increasing dataset size doesn’t guarantee more partitions

D is irrelevant to task count per stage

Final Answer: A

Comprehensive and Detailed Explanation From Exact Extract:

Spark's best practice is to estimate partition count based on data volume and a reasonable partition size — typically 128 MB to 256 MB per partition.

With 1 TB of data: 1 TB / 128 MB ≈ ~8000 partitions

This ensures that tasks are distributed across available CPUs for parallelism and that each task processes an optimal volume of data.

Option A (equal to cores) may result in partitions that are too large.

Option B (fixed 200) is arbitrary and may underutilize the cluster.

Option C (nodes) gives too few partitions (10), limiting parallelism.

The correct way to overwrite an existing file using the DataFrameWriter is:

df.write.mode("overwrite").json("path/to/file")

Option D is also technically valid, but Option A is the most concise and idiomatic PySpark syntax.

Comprehensive and Detailed Explanation:

Exactly-once guarantees in Spark Structured Streaming require micro-batch mode (default), not continuous mode.

Continuous mode (.trigger(continuous=...)) only supports at-least-once semantics and lacks full fault-tolerance.

trigger(availableNow=True)is a batch-style trigger, not suited for low-latency streaming.

So:

Option A uses micro-batching with a tight trigger interval → minimal latency + exactly-once guarantee.

Final Answer: A

Comprehensive and Detailed Explanation From Exact Extract:

Pandas API on Spark (pyspark.pandas) allows interoperability with PySpark DataFrames. To convert apyspark.pandas.DataFrameto a standard PySpark DataFrame, you use.to_spark().

Example:

df = psdf.to_spark()

This is the officially supported method as per Databricks Documentation.

Incorrect options:

B, D: Invalid or nonexistent methods.

C: Converts to a local pandas DataFrame, not a PySpark DataFrame.

Comprehensive and Detailed Explanation From Exact Extract:

Adaptive Query Execution (AQE) dynamically selects join strategies based on actual data sizes at runtime. If the size of post-shuffle partitions is below the threshold set by:

spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold

then Spark prefers to use a shuffled hash join.

From the Spark documentation:

“AQE selects a shuffled hash join when the size of post-shuffle data is small enough to fit within the configured threshold, avoiding more expensive sort-merge joins.”

Therefore:

A is wrong — Cartesian joins are only used with no join condition.

B is correct — this is the optimized join for small partitioned shuffle data under AQE.

C and D are used under other scenarios but not for this case.

Final Answer: B

Comprehensive and Detailed Explanation From Exact Extract:

The correct approach to perform a parallelizedgroupByoperation across Spark worker nodes using Pandas API is viaapplyInPandas. This function enables grouped map operations using Pandas logic in a distributed Spark environment. It applies a user-defined function to each group of data represented as a Pandas DataFrame.

As per the Databricks documentation:

"applyInPandas()allows for vectorized operations on grouped data in Spark. It applies a user-defined function to each group of a DataFrame and outputs a new DataFrame. This is the recommended approach for using Pandas logic across grouped data with parallel execution."

Option A is correct and achieves this parallel execution.

Option B (mapInPandas) applies to the entire DataFrame, not grouped operations.

Option C uses built-in aggregation functions, which are efficient but not customizable with Pandas logic.

Option D creates a scalar Pandas UDF which does not perform a group-wise transformation.

Therefore, to run agroupBywith parallel Pandas logic on Spark workers, Option A usingapplyInPandasis the only correct answer.

Comprehensive and Detailed Explanation From Exact Extract:

df.cache()is shorthand fordf.persist(StorageLevel.MEMORY_AND_DISK)

df.persist()allows specifying any storage level such asMEMORY_ONLY,DISK_ONLY,MEMORY_AND_DISK_SER, etc.

By default,persist()usesMEMORY_AND_DISK, unless specified otherwise.

Comprehensive and Detailed Explanation From Exact Extract:

The Spark UI is the standard and most effective way to inspect executor logs, task time, input size, and shuffles.

From the Databricks documentation:

“You can monitor job execution via the Spark Web UI. It includes detailed logs and metrics, including task-level execution time, shuffle reads/writes, and executor memory usage.”

(Source: Databricks Spark Monitoring Guide)

Option A is incorrect: logs are not guaranteed to be in/tmp, especially in cloud environments.

B.—verbosehelps during job submission but doesn't give detailed executor logs.

D.spark-sqlis a CLI tool for running queries, not for inspecting logs.

Hence, the correct method is using the Spark UI → Stages tab → Executor logs.

Comprehensive and Detailed Explanation From Exact Extract:

Option B is the correct and idiomatic approach in PySpark to split a string column (like email) based on a delimiter such as "@".

The split(col("email"), "@") function returns an array with two elements: username and domain.

getItem(0) retrieves the first part (username).

getItem(1) retrieves the second part (domain).

withColumn() is used to create new columns from the extracted values.

Example from official Databricks Spark documentation on splitting columns:

from pyspark.sql.functions import split, col

df.withColumn("username", split(col("email"), "@").getItem(0)) \

withColumn("domain", split(col("email"), "@").getItem(1))

⚠️Why other options are incorrect:

A uses fixed substring indices (substr(0, 5)), which won’t correctly extract usernames and domains of varying lengths.

C uses substring_index, which is available but less idiomatic for splitting emails and is slightly less readable.

D removes "@" from the email entirely, losing the separation between username and domain, and ends up duplicating values in both fields.

Therefore, Option B is the most accurate and reliable solution according to Apache Spark 3.5 best practices.

Comprehensive and Detailed Explanation From Exact Extract:

In Spark, the default join type is an inner join, which returns only the rows with matching keys in both DataFrames. To retain all records from the left DataFrame (purch_df) and include matching records from the right DataFrame (cust_df), a left outer join should be used.

By specifying the join type as'left', the modified code ensures that all records frompurch_dfare preserved, and matching records fromcust_dfare included. Records inpurch_dfwithout a corresponding match incust_dfwill havenullvalues for the columns fromcust_df.

This approach is consistent with standard SQL join operations and is supported in PySpark's DataFrame API.

A Sparkjobis triggered by an action (e.g., count, show).

The job is broken intostages, typically one per shuffle boundary.

Eachstageis divided into multipletasks, which are distributed across worker nodes.

Comprehensive and Detailed Explanation From Exact Extract:

In the Spark architecture, the driver node is responsible for orchestrating the execution of a Spark application. It converts user-defined transformations and actions into a logical plan, optimizes it into a physical plan, and then splits the plan into tasks that are distributed to the executor nodes.

As per Databricks and Spark documentation:

“The driver node is responsible for maintaining information about the Spark application, responding to a user's program or input, and analyzing, distributing, and scheduling work across the executors.”

This means:

Option A is correct because the driver schedules and coordinates the job execution.

Option B is incorrect because the driver does more than just UI monitoring.

Option C is incorrect since data and computations are distributed across executor nodes.

Option D is incorrect; results are returned to the driver but not stored long-term by it.

Comprehensive and Detailed Explanation From Exact Extract:

The question requires creating a dictionary where keys areregionvalues and values are the correspondingregion_idintegers. Furthermore, it asks to retrieve only the smallest 3region_idvalues.

Key observations:

select('region', 'region_id')puts the column order as expected bydict()— where the first column becomes the key and the second the value.

sort('region_id')ensures sorting in ascending order so the smallest IDs are first.

take(3)retrieves exactly 3 rows.

Wrapping the result indict(...)correctly builds the required Python dictionary:{ 'AFRICA': 0, 'AMERICA': 1, 'ASIA': 2 }.

Incorrect options:

Option B flips the order toregion_idfirst, resulting in a dictionary with integer keys — not what's asked.

Option C uses.limit(3)without sorting, which leads to non-deterministic rows based on partition layout.

Option D sorts in descending order, giving the largest rather than smallestregion_ids.

Hence, Option A meets all the requirements precisely.

Comprehensive and Detailed Explanation From Exact Extract:

The message"GC overhead limit exceeded"typically indicates that the JVM is spending too much time in garbage collection with little memory recovery. This suggests that the driver or executor is under-provisioned in memory.

The most effective remedy is to increase the driver memory using:

--driver-memory 4g

This is confirmed in Spark's official troubleshooting documentation:

“If you see a lot ofGC overhead limit exceedederrors in the driver logs, it’s a sign that the driver is running out of memory.”

—Spark Tuning Guide

Why others are incorrect:

Amay help but does not directly address the driver memory shortage.

Bis not a valid action; GC cannot be disabled.

Dincreases memory usage, worsening the problem.

Comprehensive and Detailed Explanation:

Adaptive Query Execution (AQE) is a Spark 3.x feature that dynamically optimizes query plans at runtime. One of its core features is:

Dynamically switching join strategies (e.g., from sort-merge to broadcast) based on runtime statistics.

Other AQE capabilities include:

Coalescing shuffle partitions

Skew join handling

Option A is correct.

Option B refers to statistics collection, which is not AQE's primary function.

Option C is too broad and not AQE-specific.

Option D refers to Delta Lake optimizations, unrelated to AQE.

Final Answer: A

na.drop(how='any')drops any row that has at least one null value.

This is exactly what's needed when the goal is to retain only fully complete records.

Usage:CopyEdit

filtered_df = users_raw_df.na.drop(how='any')

Explanation of incorrect options:

A: thresh=0 is invalid — thresh must be ≥ 1.

B: how='all' drops only rows where all columns are null (too lenient).

D: spark.na.drop doesn't support mixing how and thresh in that way; it's incorrect syntax.

Comprehensive and Detailed Explanation From Exact Extract:

The correct way to load specific columns from an ORC file is to first load the file using.load()and then apply.select()on the resulting DataFrame. This is valid with.read.format("orc")or the shortcut.read.orc().

df = spark.read.format("orc").load("/file/test_data.orc").select("col1","col2")

Why others are incorrect:

Aperforms selection after filtering, but doesn’t match the intention to minimize memory at load.

Bincorrectly tries to use.select()before.load(), which is invalid.

Cuses a non-existent.selected()method.

Dcorrectly loads and then selects.

Comprehensive and Detailed Explanation From Exact Extract:

To flatten an array of structs into individual rows and access fields within each struct, you must:

Useexplode()to expand the array so each struct becomes its own row.

Access the struct fields via dot notation (e.g.,record_exploded.sensor_id).

Option C does exactly that:

First, explode therecordarray column into a new columnrecord_exploded.

Then, access fields of the struct using the dot syntax inselect.

This is standard practice in PySpark for nested data transformation.

Final Answer: C

Comprehensive and Detailed Explanation From Exact Extract:

Shuffling occurs in operations likegroupBy,reduceByKey, orjoin—which cause data to be moved across partitions. Therepartition()operation can also cause a shuffle, but in this context, it follows an aggregation.

InOption D, thegroupByfollowed byaggresults in a shuffle due to grouping across nodes.

Therepartition(10)is a partitioning transformation but does not involve a new shuffle since the data is already grouped.

This sequence — shuffle (groupBy) followed by non-shuffling (repartition) — is correct.

Option A does the opposite: thefilterdoes not cause a shuffle, butgroupBydoes — this makes it the wrong order.

To read each file as a single record, use:

spark.read.text(path, wholetext=True)

This ensures that Spark reads the entire file contents into one row.

Comprehensive and Detailed Explanation:

The original code has several issues:

It references a variable answer that is undefined.

The function is annotated to return a str, but the logic attempts numeric multiplication.

The UDF return type is declared as T.IntegerType() but the function performs a floating-point operation, which is incompatible.

Option B correctly:

Uses DoubleType to reflect the fact that the multiplication involves a float (3.14159).

Declares the input as float, which aligns with the multiplication.

Returns a float, which matches both the logic and the schema type annotation.

This structure aligns with how PySpark expects User Defined Functions (UDFs) to be declared:

"To define a UDF you must specify a Python function and provide the return type using the relevant Spark SQL type (e.g., DoubleType for float results)."

Example from official documentation:

from pyspark.sql.functions import udf

from pyspark.sql.types import DoubleType

@udf(returnType=DoubleType())

def multiply_by_pi(x: float) -> float:

return x * 3.14159

This makes Option B the syntactically and semantically correct choice.