SPARK-DEV practice questions 101–110

1. Question 101 · 219 in bank

   Which of the following code blocks returns a new DataFrame with a new column employeesPerSqft that is the quotient of column numberOfEmployees and column sqft, both of which are from DataFrame storesDF? Note that column employeesPerSqft is not in the original DataFrame storesDF.

   • AstoresDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft"))
   • BstoresDF.withColumn("employeesPerSqft", "numberOfEmployees" / "sqft")
   • CstoresDF.select("employeesPerSqft", "numberOfEmployees" / "sqft")
   • DstoresDF.select("employeesPerSqft", col("numberOfEmployees") / col("sqft"))
   • EstoresDF.withColumn(col("employeesPerSqft"), col("numberOfEmployees") / col("sqft"))

   Show answer

   Answer: A

   Explanation

   Answer: A All other options do not work Code: from pyspark.sql import SparkSession from pyspark.sql.functions import col spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, 20, "A"), (1, 1, 50, "A"), (2, 2, 70, "A"), ] storesDF = spark.createDataFrame(data, ["storeID", "numberOfEmployees", "sqft", "division"]) storesDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft")) # A.

   Source: Adapted from community discussion — jds0

   Next question

2. Question 102 · 219 in bank

   The code block shown below should return a new DataFrame from DataFrame storesDF where column modality is the constant string "PHYSICAL", Assume DataFrame storesDF is the only defined language variable. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: storesDF. _1_(_2_,_3_(_4_))

   • A1. withColumn 2. "modality" 3. col 4. "PHYSICAL"
   • B1. withColumn 2. "modality" 3. lit 4. PHYSICAL
   • C1. withColumn 2. "modality" 3. lit 4. "PHYSICAL"
   • D1. withColumn 2. "modality" 3. SrtringType 4. "PHYSICAL"
   • E1. newColumn 2. modality 3. SrtringType 4. PHYSICAL

   Show answer

   Answer: C

   Explanation

   Option C is the correct answer. Here's why: The withColumn function is used to add a new column to the DataFrame based on an existing column or a constant value. The first blank (_1_) should be replaced with withColumn to indicate that we want to add a new column. The second blank (_2_) should be replaced with the name of the column we want to add. In this case, we want to add a column called modality. The third blank (_3_) should be replaced with a function that will create the values for the new column. In this case, we want to create a column that has the constant value "PHYSICAL". The lit function can be used to create a column with a literal value. Finally, the fourth blank (_4_) should be replaced with the actual value we want to use for the new column. Since we want to use the string "PHYSICAL", it should be wrapped in quotation marks to indicate that it is a string. Therefore, option C correctly fills in the blanks to give us the following code block: storesDF.withColumn("modality", lit("PHYSICAL"))

   Source: Adapted from community discussion — 4be8126

   Next question

3. Question 103 · 219 in bank

   Which of the following code blocks returns a DataFrame where column storeCategory from DataFrame storesDF is split at the underscore character into column storeValueCategory and column storeSizeCategory? A sample of DataFrame storesDF is displayed below:

   

   • A(storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[1]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[2]))
   • B(storesDF.withColumn("storeValueCategory", col("storeCategory").split("_")[0]) .withColumn("storeSizeCategory", col("storeCategory").split("_")[1]))
   • C(storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1]))
   • D(storesDF.withColumn("storeValueCategory", split("storeCategory", "_")[0]) .withColumn("storeSizeCategory", split("storeCategory", "_")[1]))
   • E(storesDF.withColumn("storeValueCategory", col("storeCategory").split("_")[1]) .withColumn("storeSizeCategory", col("storeCategory").split("_")[2]))

   Show answer

   Answer: C

   Explanation

   Option C returns a DataFrame where column storeCategory from DataFrame storesDF is split at the underscore character into column storeValueCategory and column storeSizeCategory. The correct code is: (storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1])) Explanation: split(col("storeCategory"), "_") splits the values in column storeCategory by the "_" character and returns an array of strings. [0] gets the first element of the resulting array and assigns it to the new column storeValueCategory. [1] gets the second element of the resulting array and assigns it to the new column storeSizeCategory. withColumn is used to create the new columns and returns a new DataFrame.

   Source: Adapted from community discussion — 4be8126

   Next question

4. Question 104 · 219 in bank

   Which of the following code blocks returns a new DataFrame where column productCategories only has one word per row, resulting in a DataFrame with many more rows than DataFrame storesDF? A sample of storesDF is displayed below:

   

   • AstoresDF.withColumn("productCategories", explode(col("productCategories")))
   • BstoresDF.withColumn("productCategories", split(col("productCategories")))
   • CstoresDF.withColumn("productCategories", col("productCategories").explode())
   • DstoresDF.withColumn("productCategories", col("productCategories").split())
   • EstoresDF.withColumn("productCategories", explode("productCategories"))

   Show answer

   Answer: A

   Explanation

   A is correct, use below code to test: from pyspark.sql import SparkSession # Initializing Spark session spark = SparkSession.builder.appName("test").getOrCreate() # 1. Creating DataFrame with an array column data_array = [ (1, ["electronics", "clothes", "toys"]), (2, ["groceries", "electronics"]), (3, ["books", "clothes"]), ] storesDF = spark.createDataFrame(data_array, ["ID", "productCategories"]) storesDF.show() df_array = storesDF.withColumn("productCategories", explode(col("productCategories"))) df_array.show()

   Source: Adapted from community discussion — newusername

   Next question

5. Question 105 · 219 in bank

   Which of the following code blocks returns a new DataFrame with column storeDescription where the pattern "Description: " has been removed from the beginning of column storeDescription in DataFrame storesDF? A sample of DataFrame storesDF is below:

   

   • AstoresDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: "))
   • BstoresDF.withColumn("storeDescription", col("storeDescription").regexp_replace("^Description: ", ""))
   • CstoresDF.withColumn("storeDescription", regexp_extract(col("storeDescription"), "^Description: ", ""))
   • DstoresDF.withColumn("storeDescription", regexp_replace("storeDescription", "^Description: ", ""))
   • EstoresDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: ", ""))

   Show answer

   Answer: E

   Explanation

   The correct answer is option E: storesDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: ", "")). This code block uses the withColumn() function to create a new column called storeDescription. It uses the regexp_replace() function to replace the pattern "^Description: " at the beginning of the string in the storeDescription column with an empty string. This effectively removes the pattern from the beginning of the string in each row of the column.

   Source: Adapted from community discussion — 4be8126

   Next question

6. Question 106 · 219 in bank

   Which of the following code blocks returns a new DataFrame where column division from DataFrame storesDF has been replaced and renamed to column state and column managerName from DataFrame storesDF has been replaced and renamed to column managerFullName?

   • A(storesDF.withColumnRenamed(["division", "state"], ["managerName", "managerFullName"])
   • B(storesDF.withColumn("state", col("division")) .withColumn("managerFullName", col("managerName")))
   • C(storesDF.withColumn("state", "division") .withColumn("managerFullName", "managerName"))
   • D(storesDF.withColumnRenamed("state", "division") .withColumnRenamed("managerFullName", "managerName"))
   • E(storesDF.withColumnRenamed("division", "state") .withColumnRenamed("managerName", "managerFullName"))

   Show answer

   Answer: E

   Explanation

   Option E is the correct answer. The withColumnRenamed function renames an existing column, whereas withColumn creates a new column. To replace the "division" column with a new column "state" and rename the "managerName" column to "managerFullName", we need to use withColumnRenamed. So option E is correct, where we first rename "division" to "state" and then rename "managerName" to "managerFullName".

   Source: Adapted from community discussion — 4be8126

   Next question

7. Question 107 · 219 in bank

   The code block shown contains an error. The code block is intended to return a new DataFrame where column sqft from DataFrame storesDF has had its missing values replaced with the value 30,000. Identify the error. A sample of DataFrame storesDF is displayed below:

   

   Code block: storesDF.na.fill(30000, col("sqft"))

   • AThe argument to the subset parameter of fill() should be a string column name or a list of string column names rather than a Column object.
   • BThe na.fill() operation does not work and should be replaced by the dropna() operation.
   • Che argument to the subset parameter of fill() should be a the numerical position of the column rather than a Column object.
   • DThe na.fill() operation does not work and should be replaced by the nafill() operation.
   • EThe na.fill() operation does not work and should be replaced by the fillna() operation.

   Show answer

   Answer: A

   Explanation

   The most correct answer seems to be A: Code below with Spark 3.5.1. ```python from pyspark.sql import SparkSession from pyspark.sql.functions import col from pyspark.errors import PySparkTypeError spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 43161), (1, 51200), (2, None), (3, 78367), (4, None), ] storesDF = spark.createDataFrame(data, ["storeID", "sqft"]) # storesDF.show() try: storesDF.na.fill(30000, col("sqft")) except PySparkTypeError as e: print(e) storesDF.na.fill(30000, "sqft").show() storesDF.na.fill(30000, ["sqft"]).show() storesDF.fillna(30000, ["sqft"]).show() storesDF.fillna(30000, "sqft").show() ```

   Source: Adapted from community discussion — jds0

   Next question

8. Question 108 · 219 in bank

   Which of the following will occur if there are more slots than there are tasks?

   • AThe Spark job will likely not run as efficiently as possible.
   • BThe Spark application will fail – there must be at least as many tasks as there are slots.
   • CSome executors will shut down and allocate all slots on larger executors first.
   • DMore tasks will be automatically generated to ensure all slots are being used.
   • EThe Spark job will use just one single slot to perform all tasks.

   Show answer

   Answer: A

   Explanation

   Slots are the basic unit of parallelism in Spark, and represent a unit of resource allocation on a single executor. If there are more slots than there are tasks, it means that some of the slots will be idle and not being used to execute any tasks, leading to inefficient resource utilization. In this scenario, the Spark job will likely not run as efficiently as possible. However, it is still possible for the Spark job to complete successfully. Therefore, option A is the correct answer.

   Source: Adapted from community discussion — TC007

   Next question

9. Question 109 · 219 in bank

   Which of the following operations fails to return a DataFrame with no duplicate rows?

   • ADataFrame.dropDuplicates()
   • BDataFrame.distinct()
   • CDataFrame.drop_duplicates()
   • DDataFrame.drop_duplicates(subset = None)
   • EDataFrame.drop_duplicates(subset = "all")

   Show answer

   Answer: E

   Explanation

   A. DataFrame.dropDuplicates(): This method returns a new DataFrame with distinct rows based on all columns. It should return a DataFrame with no duplicate rows. B. DataFrame.distinct(): This method returns a new DataFrame with distinct rows based on all columns. It should also return a DataFrame with no duplicate rows. C. DataFrame.drop_duplicates(): This is an alias for DataFrame.dropDuplicates(). It should also return a DataFrame with no duplicate rows. D. DataFrame.drop_duplicates(subset=None): This method returns a new DataFrame with distinct rows based on all columns. It should return a DataFrame with no duplicate rows. E. DataFrame.drop_duplicates(subset="all"): This method attempts to drop duplicates based on all columns but returns an error, because "all" is not a valid argument for the subset parameter. So this operation fails to return a DataFrame with no duplicate rows. Therefore, the correct answer is E.

   Source: Adapted from community discussion — 4be8126

   Next question

10. Question 110 · 219 in bank

    Which of the following code blocks will most quickly return an approximation for the number of distinct values in column division in DataFrame storesDF?

    • AstoresDF.agg(approx_count_distinct(col("division")).alias("divisionDistinct"))
    • BstoresDF.agg(approx_count_distinct(col("division"), 0.01).alias("divisionDistinct"))
    • CstoresDF.agg(approx_count_distinct(col("division"), 0.15).alias("divisionDistinct"))
    • DstoresDF.agg(approx_count_distinct(col("division"), 0.0).alias("divisionDistinct"))
    • EstoresDF.agg(approx_count_distinct(col("division"), 0.05).alias("divisionDistinct"))

    Show answer

    Answer: C

    Explanation

    I see you reply in a lot of question, barely correct. bro, you need to stop comment wrong information here. This question only ask for efficiency, no need to balance between accuracy and efficiency. Stop posting ChatGPT answer here

    Source: Adapted from community discussion — ZSun

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 101. 219 questions available.