SPARK-DEV practice questions 131–140

1. Question 131 · 219 in bank

   The code block shown below contains an error. The code block intended to return a new DataFrame that is the result of an inner join between DataFrame storesDF and DataFrame employeesDF on column storeId. Identify the error. Code block: StoresDF.join(employeesDF, "inner", "storeID")

   • AThe key column storeID needs to be wrapped in the col() operation.
   • BThe key column storeID needs to be in a list like ["storeID"].
   • CThe key column storeID needs to be specified in an expression of both DataFrame columns like storesDF.storeId == employeesDF.storeId.
   • DThere is no DataFrame.join() operation – DataFrame.merge() should be used instead.
   • EThe column key is the second parameter to join() and the type of join in the third parameter to join() – the second and third arguments should be switched.

   Show answer

   Answer: E

   Explanation

   E storesDF.join(employeesDF, "storeID", "inner") The column key is the second parameter to join() and the type of join is in the third parameter to join() – the second and third arguments should be switched

   Source: Adapted from community discussion — Jtic

   Next question

2. Question 132 · 219 in bank

   Which of the following operations can perform an outer join on two DataFrames?

   • ADataFrame.crossJoin()
   • BStandalone join() function
   • CDataFrame.outerJoin()
   • DDataFrame.join()
   • EDataFrame.merge()

   Show answer

   Answer: D

   Next question

3. Question 133 · 219 in bank

   Which of the following pairs of arguments cannot be used in DataFrame.join() to perform an inner join on two DataFrames, named and aliased with "a" and "b" respectively, to specify two key columns?

   • Aon = [a.column1 == b.column1, a.column2 == b.column2]
   • Bon = [col("column1"), col("column2")]
   • Con = [col("a.column1") == col("b.column1"), col("a.column2") == col("b.column2")]
   • DAll of these options can be used to perform an inner join with two key columns.
   • Eon = ["column1", "column2"]

   Show answer

   Answer: B

   Explanation

   According to the following code, only response B returns an error. The key concept here is that dataframes must be "named" AND "aliased". from pyspark.sql.functions import col a = spark.createDataFrame([(1, 2), (3, 4)], ['column1', 'column2']) b = spark.createDataFrame([(1, 2), (5, 6)], ['column1', 'column2']) a = a.alias('a') b = b.alias('b') df = a.join(b, on = [a.column1 == b.column1, a.column2 == b.column2]) display(df) # df = a.join(b, on = [col("column1"), col("column2")]) df = a.join(b, on = [col("a.column1") == col("b.column1"), col("a.column2") == col("b.column2")]) display(df) df = a.join(b, on = ["column1", "column2"]) display(df)

   Source: Adapted from community discussion — juliom6

   Next question

4. Question 134 · 219 in bank

   The below code block contains a logical error resulting in inefficiency. The code block is intended to efficiently perform a broadcast join of DataFrame storesDF and the much larger DataFrame employeesDF using key column storeId. Identify the logical error. Code block: storesDF.join(broadcast(employeesDF), "storeId")

   • AThe larger DataFrame employeesDF is being broadcasted rather than the smaller DataFrame storesDF.
   • BThere is never a need to call the broadcast() operation in Apache Spark 3.
   • CThe entire line of code should be wrapped in broadcast() rather than just DataFrame employeesDF.
   • DThe broadcast() operation will only perform a broadcast join if the Spark property spark.sql.autoBroadcastJoinThreshold is manually set.
   • EOnly one of the DataFrames is being broadcasted rather than both of the DataFrames.

   Show answer

   Answer: A

   Explanation

   The answer is A. The logical error in the code block is that the larger DataFrame, employeesDF, is being broadcasted instead of the smaller DataFrame, storesDF. This defeats the purpose of a broadcast join, which is to optimize performance by broadcasting the smaller DataFrame to all the worker nodes, avoiding the need to shuffle data over the network. To perform a broadcast join efficiently, the smaller DataFrame should be broadcasted, which in this case is storesDF. The corrected code should be: broadcast(storesDF).join(employeesDF, "storeId")

   Source: Adapted from community discussion — 4be8126

   Next question

5. Question 135 · 219 in bank

   The code block shown below contains an error. The code block is intended to return a new DataFrame that is the result of a cross join between DataFrame storesDF and DataFrame employeesDF. Identify the error. Code block: storesDF.join(employeesDF, "cross")

   • AA cross join is not implemented by the DataFrame.join() operations – the standalone CrossJoin() operation should be used instead.
   • BThere is no direct cross join in Spark, but it can be implemented by performing an outer join on all columns of both DataFrames.
   • CA cross join is not implemented by the DataFrame.join()operation – the DataFrame.crossJoin()operation should be used instead.
   • DThere is no key column specified – the key column "storeId" should be the second argument.
   • EA cross join is not implemented by the DataFrame.join() operations – the standalone join() operation should be used instead.

   Show answer

   Answer: C

   Explanation

   C is correct. # https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.crossJoin.html a = spark.createDataFrame([(1, 2), (3, 4)], ['column1', 'column2']) b = spark.createDataFrame([(5, 6), (7, 8)], ['column3', 'column4']) df = a.crossJoin(b) display(df)

   Source: Adapted from community discussion — juliom6

   Next question

6. Question 136 · 219 in bank

   The code block shown below contains an error. The code block is intended to return a new DataFrame that is the result of a position-wise union between DataFrame storesDF and DataFrame acquiredStoresDF. Identify the error. Code block: storesDF.unionByName(acquiredStoresDF)

   • AThere is no DataFrame.unionByName() operation – the concat() operation should be used instead with both DataFrames as arguments.
   • BThere are no key columns specified – similar column names should be the second argument.
   • CThe DataFrame.unionByName() operation does not union DataFrames based on column position – it uses column name instead.
   • DThe unionByName() operation is a standalone operation rather than a method of DataFrame – it should have both DataFrames as arguments.
   • EThere are no column positions specified – the desired column positions should be the second argument.

   Show answer

   Answer: C

   Explanation

   The error in the code block is: C. The DataFrame.unionByName() operation does not union DataFrames based on column position – it uses column name instead. The unionByName() operation performs a position-wise union based on column names, not based on column positions. Therefore, the error in the code block is that the intended operation should be union(), which performs a position-wise union regardless of column names. The correct code block to perform a position-wise union between DataFrame storesDF and DataFrame acquiredStoresDF would be: storesDF.union(acquiredStoresDF)

   Source: Adapted from community discussion — 4be8126

   Next question

7. Question 137 · 219 in bank

   Which of the following code blocks writes DataFrame storesDF to file path filePath as JSON?

   • AstoresDF.write.option("json").path(filePath)
   • BstoresDF.write.json(filePath)
   • CstoresDF.write.path(filePath)
   • DstoresDF.write(filePath)
   • EstoresDF.write().json(filePath)

   Show answer

   Answer: B

   Next question

8. Question 138 · 219 in bank

   In what order should the below lines of code be run in order to write DataFrame storesDF to file path filePath as parquet and partition by values in column division? Lines of code: 1. .write() \ 2. .partitionBy("division") \ 3. .parquet(filePath) 4. .storesDF \ 5. .repartition("division") 6. .write \ 7. .path(filePath, "parquet")

   • A4, 1, 2, 3
   • B4, 1, 5, 7
   • C4, 6, 2, 3
   • D4, 1, 5, 3
   • E4, 6, 2, 7

   Show answer

   Answer: C

   Next question

9. Question 139 · 219 in bank

   The code block shown below contains an error. The code block intended to read a parquet at the file path filePath into a DataFrame. Identify the error. Code block: spark.read.load(filePath, source – "parquet")

   • AThere is no source parameter to the load() operation – the schema parameter should be used instead.
   • BThere is no load() operation – it should be parquet() instead.
   • CThe spark.read operation should be followed by parentheses to return a DataFrameReader object.
   • DThe filePath argument to the load() operation should be quoted.
   • EThere is no source parameter to the load() operation – it can be removed.

   Show answer

   Answer: E

   Explanation

   Answer should be E. Removing source and default is 'parquet' anyway. However, it is not ideal to use load, rather the respective method. https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrameReader.load.html?highlight=dataframereader%20load#pyspark.sql.DataFrameReader.load

   Source: Adapted from community discussion — Larrave

   Next question

10. Question 140 · 219 in bank

    Which of the following operations is most likely to result in a shuffle?

    • ADataFrame.join()
    • BDataFrame.filter()
    • CDataFrame.union()
    • DDataFrame.where()
    • EDataFrame.drop()

    Show answer

    Answer: A

    Explanation

    The most likely operation to result in a shuffle is: A. DataFrame.join() Explanation: A shuffle operation in Spark involves redistributing and reorganizing data across partitions. It typically occurs when data needs to be rearranged or merged based on a specific key or condition. DataFrame joins involve combining two DataFrames based on a common key column, and this operation often requires data to be shuffled to ensure that matching records are located on the same executor or partition. The shuffle process involves exchanging data between nodes or executors in the cluster, which can incur significant data movement and network communication overhead.

    Source: Adapted from community discussion — TmData

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 131. 219 questions available.