SPARK-DEV practice questions 91–100

1. Question 91 · 219 in bank

   Which of the following code blocks returns a DataFrame where column managerName from DataFrame storesDF is split at the space character into column managerFirstName and column managerLastName? A sample of DataFrame storesDF is displayed below:

   

   • A(storesDF.withColumn("managerFirstName", split(col("managerName"), " ")[0]) .withColumn("managerLastName", split(col("managerName"), " ")[1]))
   • B(storesDF.withColumn("managerFirstName", col("managerName"). split(" ")[1]) .withColumn("managerLastName", col("managerName").split(" ")[2]))
   • C(storesDF.withColumn("managerFirstName", split(col("managerName"), " ")[1]) .withColumn("managerLastName", split(col("managerName"), " ")[2]))
   • D(storesDF.withColumn("managerFirstName", col("managerName").split(" ")[0]) .withColumn("managerLastName", col("managerName").split(" ")[1]))
   • E(storesDF.withColumn("managerFirstName", split("managerName"), " ")[0]) .withColumn("managerLastName", split("managerName"), " ")[1]))

   Show answer

   Answer: A

   Explanation

   Right answer is A as the array returned by the split function is 0-based not 1-based. You can try yourself with the following code: df = spark.createDataFrame([('John Doe',)], ['Person',]) df2 = df \ .withColumn('first_name', split(col('Person'), ' ')[0]) \ .withColumn('last_name', split(col('Person'), ' ')[1]) df2.show()

   Source: Adapted from community discussion — max_manfred

   Next question

2. Question 92 · 219 in bank

   Which of the following cluster configurations will fail to ensure completion of a Spark application in light of a worker node failure?

   

   Note: each configuration has roughly the same compute power using 100GB of RAM and 200 cores.

   • AScenario #5
   • BScenario #4
   • CScenario #6
   • DScenario #1
   • EThey should all ensure completion because worker nodes are fault tolerant

   Show answer

   Answer: D

   Explanation

   scenario #1 uses a configuration with only 1 executor. If the worker node running that executor fails, the entire application will fail because there are no other executors available to continue processing the tasks. Spark needs redundancy across multiple worker nodes or executors to tolerate node failures. Since this configuration has only one executor, it lacks fault tolerance.

   Source: Adapted from community discussion — Oks_An

   Next question

3. Question 93 · 219 in bank

   Which of the following code blocks returns a new DataFrame where column managerName from DataFrame storesDF has had its missing values replaced with the value "No Manager"? A sample of DataFrame storesDF is below:

   

   • AstoresDF.na.fill("No Manager", "managerName")
   • BstoresDF.nafill("No Manager", col("managerName"))
   • CstoresDF.na.fill("No Manager", col("managerName"))
   • DstoresDF.fillna("No Manager", col("managerName"))
   • EstoresDF.nafill("No Manager", "managerName")

   Show answer

   Answer: A

   Next question

4. Question 94 · 219 in bank

   Which of the following statements about Spark DataFrames is incorrect?

   • ASpark DataFrames are the same as a data frame in Python or R.
   • BSpark DataFrames are built on top of RDDs.
   • CSpark DataFrames are immutable.
   • DSpark DataFrames are distributed.
   • ESpark DataFrames have common Structured APIs.

   Show answer

   Answer: A

   Explanation

   A. Spark DataFrames are the same as a data frame in Python or R.: This statement is incorrect because Spark DataFrames, while conceptually similar to data frames in Python (e.g., pandas) or R, are not the same. Spark DataFrames are distributed, immutable, and built on top of RDDs, designed to handle large-scale data processing across a cluster. In contrast, data frames in Python (pandas) or R are typically in-memory, single-node constructs.

   Source: Adapted from community discussion — Oks_An

   Next question

5. Question 95 · 219 in bank

   The code block shown below contains an error. The code block is intended to return a DataFrame containing all columns from DataFrame storesDF except for column sqft and column customerSatisfaction. Identify the error. Code block: storesDF.drop(sqft, customerSatisfaction)

   • AThe drop() operation only works if one column name is called at a time – there should be two calls in succession like storesDF.drop("sqft").drop("customerSatisfaction").
   • BThe drop() operation only works if column names are wrapped inside the col() function like storesDF.drop(col(sqft), col(customerSatisfaction)).
   • CThere is no drop() operation for storesDF.
   • DThe sqft and customerSatisfaction column names should be quoted like "sqft" and "customerSatisfaction".
   • EThe sqft and customerSatisfaction column names should be subset from the DataFrame storesDF like storesDF."sqft" and storesDF."customerSatisfaction".

   Show answer

   Answer: D

   Explanation

   The error in the code block is that the column names sqft and customerSatisfaction should be quoted, like "sqft" and "customerSatisfaction", since they are strings. The correct code block should be: storesDF.drop("sqft", "customerSatisfaction") Option D correctly identifies this error.

   Source: Adapted from community discussion — 4be8126

   Next question

6. Question 96 · 219 in bank

   Which of the following code blocks returns a new DataFrame with the mean of column sqft from DataFrame storesDF in column sqftMean?

   • AstoresDF.withColumn(mean(col("sqft")).alias("sqftMean"))
   • BstoresDF.agg(col("sqft").mean().alias("sqftMean"))
   • CstoresDF.agg(mean("sqft").alias("sqftMean"))
   • DstoresDF.agg(mean(col("sqft")).alias("sqftMean"))
   • EstoresDF.withColumn("sqftMean", mean(col("sqft")))

   Show answer

   Answer: D

   Next question

7. Question 97 · 219 in bank

   Which of the following code blocks fails to return a DataFrame sorted alphabetically based on column division?

   • AstoresDF.sort(asc("division"))
   • BstoresDF.orderBy(["division"], ascending = [1])
   • CstoresDF.orderBy(col("division").desc())
   • DstoresDF.orderBy("division")
   • EstoresDF.sort("division")

   Show answer

   Answer: D

   Next question

8. Question 98 · 219 in bank

   Which of the following describes the relationship between nodes and executors?

   • AExecutors and nodes are not related.
   • BAnode is a processing engine running on an executor.
   • CAn executor is a processing engine running on a node.
   • DThere are always the same number of executors and nodes.
   • EThere are always more nodes than executors.

   Show answer

   Answer: C

   Explanation

   In a Spark cluster, each node typically has multiple executors, which are responsible for executing tasks on that node. An executor is a separate process that runs on a node and is responsible for executing tasks assigned to it by the driver. Therefore, a node can have multiple executors running on it. The number of executors on a node depends on the resources available on that node and the configuration settings of the Spark application. So, option C is the correct answer.

   Source: Adapted from community discussion — TC007

   Next question

9. Question 99 · 219 in bank

   Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30?

   • AstoresDF.filter(col("sqft") <= 25000 | col("customerSatisfaction") >= 30)
   • BstoresDF.filter(col("sqft") <= 25000 or col("customerSatisfaction") >= 30)
   • CstoresDF.filter(sqft <= 25000 or customerSatisfaction >= 30)
   • DstoresDF.filter(col(sqft) <= 25000 | col(customerSatisfaction) >= 30)
   • EstoresDF.filter((col("sqft") <= 25000) | (col("customerSatisfaction") >= 30))

   Show answer

   Answer: E

   Explanation

   The correct code block to return a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30 is: storesDF.filter((col("sqft") = 30)) Option A uses a single pipe (|) instead of the correct syntax of two vertical bars (||) to represent "OR" logic, and also uses the wrong syntax for column referencing. Option B uses the correct or operator, but also uses the wrong syntax for column referencing. Option C uses the correct operator and syntax for column referencing, but does not use the col() function to reference column names. Option D uses the col() function, but also uses the wrong syntax for column referencing. Option E uses the correct syntax for both column referencing and logical operator, and correctly specifies the parentheses to ensure the proper order of operations. Therefore, the correct answer is E. storesDF.filter((col("sqft") = 30))

   Source: Adapted from community discussion — 4be8126

   Next question

10. Question 100 · 219 in bank

    Which of the following code blocks returns a new DataFrame from DataFrame storesDF where column storeId is of the type string?

    • AstoresDF.withColumn("storeId, cast(col("storeId"), StringType()))
    • BstoresDF.withColumn("storeId, col("storeId").cast(StringType()))
    • CstoresDF.withColumn("storeId, cast(storeId).as(StringType)
    • DstoresDF.withColumn("storeId, col(storeId).cast(StringType)
    • EstoresDF.withColumn("storeId, cast("storeId").as(StringType()))

    Show answer

    Answer: B

    Explanation

    The correct code block to return a new DataFrame from DataFrame storesDF where column storeId is of the type string is: storesDF.withColumn("storeId", col("storeId").cast(StringType())) Option A has an extra quotation mark after "storeId" and is missing a closing parenthesis for the cast() function. Option B correctly uses the cast() function to change the data type, but has a typo where "storeId" is repeated inside the string argument for the withColumn() function. Option C is missing the col() function to reference the storeId column, and also has a typo with the closing parentheses for the cast() function. Option D correctly references the storeId column using col(), but has a typo with the quotation marks and parentheses. Option E has a syntax error where the cast() function is inside the quotation marks, and is also missing the col() function to reference the storeId column. Therefore, the correct answer is B. storesDF.withColumn("storeId", col("storeId").cast(StringType()))

    Source: Adapted from community discussion — 4be8126

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 91. 219 questions available.