SPARK-DEV practice questions 21–30

1. Question 21 · 219 in bank

   The code block shown below should return a DataFrame containing all columns from DataFrame storesDF except for column sqft and column customerSatisfaction. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: __1__.__2__(__3__)

   • A1. drop 2. storesDF 3. col(“sqft”), col(“customerSatisfaction”)
   • B1. storesDF 2. drop 3. sqft, customerSatisfaction
   • C1. storesDF 2. drop 3. “sqft”, “customerSatisfaction”
   • D1. storesDF 2. drop 3. col(sqft), col(customerSatisfaction)
   • E1. drop 2. storesDF 3. col(sqft), col(customerSatisfaction)

   Show answer

   Answer: C

   Next question

2. Question 22 · 219 in bank

   Which of the following describes the difference between DataFrame.repartition(n) and DataFrame.coalesce(n)?

   • ADataFrame.repartition(n) will split a DataFrame into n number of new partitions with data distributed evenly. DataFrame.coalesce(n) will more quickly combine the existing partitions of a DataFrame but might result in an uneven distribution of data across the new partitions.
   • BWhile the results are similar, DataFrame.repartition(n) will be more efficient than DataFrame.coalesce(n) because it can partition a Data Frame by the column.
   • CDataFrame.repartition(n) will split a Data Frame into any number of new partitions while minimizing shuffling. DataFrame.coalesce(n) will split a DataFrame onto any number of new partitions utilizing a full shuffle.
   • DWhile the results are similar, DataFrame.repartition(n) will be less efficient than DataFrame.coalesce(n) because it can partition a Data Frame by the column.
   • EDataFrame.repartition(n) will combine the existing partitions of a DataFrame but may result in an uneven distribution of data across the new partitions. DataFrame.coalesce(n) will more slowly split a Data Frame into n number of new partitions with data distributed evenly.

   Show answer

   Answer: A

   Explanation

   A The correct answer is A. DataFrame.repartition(n) will split a DataFrame into n number of new partitions with data distributed evenly. DataFrame.coalesce(n) will more quickly combine the existing partitions of a DataFrame but might result in an uneven distribution of data across the new partitions. The `repartition()` method can be used to increase or decrease the number of partitions in a DataFrame, while the `coalesce()` method is used to only decrease the number of partitions in an efficient way². The `repartition()` method does a full shuffle and creates new partitions with data that's distributed evenly. On the other hand, `coalesce()` avoids a full shuffle by allowing only the reduction of partitions.

   Source: Adapted from community discussion — thanab

   Next question

3. Question 23 · 219 in bank

   Which of the following cluster configurations is most likely to experience delays due to garbage collection of a large Dataframe?

   

   Note: each configuration has roughly the same compute power using 100GB of RAM and 200 cores.

   • AMore information is needed to determine an answer.
   • BScenario #5
   • CScenario #4
   • DScenario #1
   • EScenario #2

   Show answer

   Answer: D

   Next question

4. Question 24 · 219 in bank

   Which of the following statements about Spark’s stability is incorrect?

   • ASpark is designed to support the loss of any set of worker nodes.
   • BSpark will rerun any failed tasks due to failed worker nodes.
   • CSpark will recompute data cached on failed worker nodes.
   • DSpark will spill data to disk if it does not fit in memory.
   • ESpark will reassign the driver to a worker node if the driver’s node fails.

   Show answer

   Answer: E

   Explanation

   Option E is incorrect because the driver program in Spark is not reassigned to another worker node if the driver's node fails. The driver program is responsible for the coordination and control of the Spark application and runs on a separate machine, typically the client machine or cluster manager. If the driver's node fails, the Spark application as a whole may fail or need to be restarted, but the driver is not automatically reassigned to another worker node.

   Source: Adapted from community discussion — TmData

   Next question

5. Question 25 · 219 in bank

   Which of the following DataFrame operations is classified as a transformation?

   • ADataFrame.select()
   • BDataFrame.count()
   • CDataFrame.show()
   • DDataFrame.first()
   • EDataFrame.collect()

   Show answer

   Answer: A

   Next question

6. Question 26 · 219 in bank

   Which of the following operations will fail to trigger evaluation?

   • ADataFrame.collect()
   • BDataFrame.count()
   • CDataFrame.first()
   • DDataFrame.join()
   • EDataFrame.take()

   Show answer

   Answer: D

   Next question

7. Question 27 · 219 in bank

   The code block shown below should read a JSON at the file path filePath into a DataFrame with the specified schema schema. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: __1__.__2__.__3__(__4__).format("csv").__5__(__6__)

   • A1. spark 2. read() 3. schema 4. schema 5. json 6. filePath
   • B1. spark 2. read() 3. json 4. filePath 5. format 6. schema
   • C1. spark 2. read() 3. schema 4. schema 5. load 6. filePath
   • D1. spark 2. read 3. schema 4. schema 5. load 6. filePath
   • E1. spark 2. read 3. format 4. "json" 5. load 6. filePath

   Show answer

   Answer: D

   Next question

8. Question 28 · 219 in bank

   Which of the following code blocks returns a new DataFrame with a new column customerSatisfactionAbs that is the absolute value of column customerSatisfaction in DataFrame storesDF? Note that column customerSatisfactionAbs is not in the original DataFrame storesDF.

   • AstoresDF.withColumn(“customerSatisfactionAbs”, abs(col(“customerSatisfaction”)))
   • BstoresDF.withColumnRenamed(“customerSatisfactionAbs”, abs(col(“customerSatisfaction”)))
   • CstoresDF.withColumn(col(“customerSatisfactionAbs”, abs(col(“customerSatisfaction”)))
   • DstoresDF.withColumn(“customerSatisfactionAbs”, abs(col(customerSatisfaction)))
   • EstoresDF.withColumn(“customerSatisfactionAbs”, abs(“customerSatisfaction”))

   Show answer

   Answer: A

   Explanation

   it works for A and E: pyspark.sql.functions.abs(col: ColumnOrName) → pyspark.sql.column.Column[source] Computes the absolute value. New in version 1.3.0. Changed in version 3.4.0: Supports Spark Connect. Parameters colColumn or str target column to compute on.

   Source: Adapted from community discussion — cookiemonster42

   Next question

9. Question 29 · 219 in bank

   Which of the following statements about the Spark driver is true?

   • ASpark driver is horizontally scaled to increase overall processing throughput.
   • BSpark driver is the most coarse level of the Spark execution hierarchy.
   • CSpark driver is fault tolerant — if it fails, it will recover the entire Spark application.
   • DSpark driver is responsible for scheduling the execution of data by various worker nodes in cluster mode.
   • ESpark driver is only compatible with its included cluster manager.

   Show answer

   Answer: D

   Explanation

   I believe D is the correct one according to documentation from Databricks [1]: "The driver process runs your main() function, sits on a node in the cluster, and is responsible for three things: maintaining information about the Spark Application; responding to a user’s program or input; and analyzing, distributing, and scheduling work across the executors (defined momentarily)." Addittionaly: "The cluster manager controls physical machines and allocates resources to Spark Applications." Based on the above we could say that cluster manager is charge assign resources (CPU, Memory, etc) to the VMs used. Keep in mind that this is based on the definition from Databricks other definitions may include what was mentioned by cookiemonster42. [1] https://www.databricks.com/glossary/what-are-spark-applications

   Source: Adapted from community discussion — martcerv

   Next question

10. Question 30 · 219 in bank

    The code block shown below should write DataFrame storesDF to file path filePath as parquet and partition by values in column division. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: storesDF.__1__.__2__(__3__).__4__(__5__)

    • A1. write 2. partitionBy 3. “division” 4. path 5. filePath, node = parquet
    • B1. write 2. partitionBy 3. “division” 4. parquet 5. filePath
    • C1. write 2. partitionBy 3. col(“division”) 4. parquet 5. filePath
    • D1. write() 2. partitionBy 3. col(“division”) 4. parquet 5. filePath
    • E1. write 2. repartition 3. “division” 4. path 5. filePath, mode = “parquet”

    Show answer

    Answer: B

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 21. 219 questions available.