SPARK-DEV practice questions 111–120

1. Question 111 · 219 in bank

   The code block shown below contains an error. The code block is intended to return a new DataFrame with the mean of column sqft from DataFrame storesDF in column sqftMean. Identify the error. Code block: storesDF.agg(mean("sqft").alias("sqftMean"))

   • AThe argument to the mean() operation should be a Column abject rather than a string column name.
   • BThe argument to the mean() operation should not be quoted.
   • CThe mean() operation is not a standalone function – it’s a method of the Column object.
   • DThe agg() operation is not appropriate here – the withColumn() operation should be used instead.
   • EThe only way to compute a mean of a column is with the mean() method from a DataFrame.

   Show answer

   Answer: A

   Explanation

   Correct answer is A: from pyspark.sql.functions import col, mean students =[ {'rollno':'001','name':'sravan','sqft':23, 'height':5.79,'weight':67,'address':'guntur'}, {'rollno':'002','name':'ojaswi','sqft':16, 'height':3.79,'weight':34,'address':'hyd'}] storesDF = spark.createDataFrame( students) storesDF.agg(mean(col('sqft')).alias('sqftMean')).show()

   Source: Adapted from community discussion — juliom6

   Next question

2. Question 112 · 219 in bank

   Which of the following operations can be used to return the number of rows in a DataFrame?

   • ADataFrame.numberOfRows()
   • BDataFrame.n()
   • CDataFrame.sum()
   • DDataFrame.count()
   • EDataFrame.countDistinct()

   Show answer

   Answer: D

   Explanation

   The operation that can be used to return the number of rows in a DataFrame is: D. DataFrame.count() The count() method in Spark DataFrame returns the number of rows in the DataFrame, and it is the standard way to determine the row count. Options A, B, C, and E are not valid methods for counting the number of rows in a DataFrame.

   Source: Adapted from community discussion — outwalker

   Next question

3. Question 113 · 219 in bank

   Which of the following operations returns a GroupedData object?

   • ADataFrame.GroupBy()
   • BDataFrame.cubed()
   • CDataFrame.group()
   • DDataFrame.groupBy()
   • EDataFrame.grouping_id()

   Show answer

   Answer: D

   Explanation

   D. DataFrame.groupBy() The groupBy() method is used to group the DataFrame based on one or more columns, and it returns a GroupedData object, which can then be used to perform various aggregation operations on the grouped data. Options A, B, C, and E do not return a GroupedData object.

   Source: Adapted from community discussion — outwalker

   Next question

4. Question 114 · 219 in bank

   Which of the following code blocks returns a collection of summary statistics for all columns in DataFrame storesDF?

   • AstoresDF.summary("mean")
   • BstoresDF.describe(all = True)
   • CstoresDF.describe("all")
   • DstoresDF.summary("all")
   • EstoresDF.describe()

   Show answer

   Answer: E

   Explanation

   E is the right option. See code below with Spark 3.5.1 # Summary statistics of a DataFrame from pyspark.sql import SparkSession from pyspark.sql.functions import col from pyspark.errors import PySparkTypeError spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 43161), (1, 51200), (2, None), (3, 78367), (4, None), ] storesDF = spark.createDataFrame(data, ["storeID", "sqft"]) try: storesDF.summary("mean").show() except Exception as e: print(e) try: storesDF.describe(all = True).show() except Exception as e: print(e) try: storesDF.describe("all").show() except Exception as e: print(e) try: storesDF.summary("all").show() except Exception as e: print(e) try: storesDF.describe().show() except Exception as e: print(e)

   Source: Adapted from community discussion — jds0

   Next question

5. Question 115 · 219 in bank

   Which of the following code blocks fails to return a DataFrame reverse sorted alphabetically based on column division?

   • AstoresDF.orderBy("division", ascending – False)
   • BstoresDF.orderBy(["division"], ascending = [0])
   • CstoresDF.orderBy(col("division").asc())
   • DstoresDF.sort("division", ascending – False)
   • EstoresDF.sort(desc("division"))

   Show answer

   Answer: C

   Next question

6. Question 116 · 219 in bank

   Which of the following code blocks returns all the rows from DataFrame storesDF?

   • AstoresDF.head()
   • BstoresDF.collect()
   • CstoresDF.count()
   • DstoresDF.take()
   • EstoresDF.show()

   Show answer

   Answer: B

   Explanation

   Answer: B Explanation: head() returns the first n rows of the DataFrame. By default, it returns the first 5 rows. collect() returns an array of Row objects that represent the entire DataFrame. count() returns the number of rows in the DataFrame. take(n) returns the first n rows of the DataFrame as an array of Row objects. show() prints the first 20 rows of the DataFrame in a tabular form. Only collect() returns all the rows from the DataFrame.

   Source: Adapted from community discussion — 4be8126

   Next question

7. Question 117 · 219 in bank

   Which of the following code blocks applies the function assessPerformance() to each row of DataFrame storesDF?

   • A[assessPerformance(row) for row in storesDF.take(3)]
   • B[assessPerformance() for row in storesDF]
   • CstoresDF.collect().apply(lambda: assessPerformance)
   • D[assessPerformance(row) for row in storesDF.collect()]
   • E[assessPerformance(row) for row in storesDF]

   Show answer

   Answer: D

   Explanation

   The correct answer is D. Explanation: Option A uses the take() method to extract three rows from the DataFrame, but it applies the assessPerformance() function to each row outside of the DataFrame context. Option B attempts to apply the assessPerformance() function to each row, but it doesn't reference the row object in any way. Option C tries to apply the assessPerformance() function to the entire DataFrame but does so using an incorrect syntax. Option D correctly applies the assessPerformance() function to each row of the DataFrame using a list comprehension over the result of the collect() method. Option E is similar to D, but it will iterate over rows individually instead of using the collect() method to retrieve all rows at once. While this is still a valid approach, it may be less efficient.

   Source: Adapted from community discussion — 4be8126

   Next question

8. Question 118 · 219 in bank

   Which of the following is the most granular level of the Spark execution hierarchy?

   • ATask
   • BExecutor
   • CNode
   • DJob
   • ESlot

   Show answer

   Answer: A

   Explanation

   The correct answer is A. Task. Explanation: In the Spark execution hierarchy, the most granular level is the task. A task represents a unit of work that is executed on a partitioned portion of the data in parallel. Tasks are created by the Spark driver program and assigned to individual executors for execution. Each task operates on a subset of the data and performs a specific operation defined by the Spark application, such as a transformation or an action.

   Source: Adapted from community discussion — TmData

   Next question

9. Question 119 · 219 in bank

   The code block shown below contains an error. The code block is intended to print the schema of DataFrame storesDF. Identify the error. Code block: storesDF.printSchema

   • AThere is no printSchema member of DataFrame – schema and the print() function should be used instead.
   • BThe entire line needs to be a string – it should be wrapped by str().
   • CThere is no printSchema member of DataFrame – the getSchema() operation should be used instead.
   • DThere is no printSchema member of DataFrame – the schema() operation should be used instead.
   • EThe printSchema member of DataFrame is an operation and needs to be followed by parentheses.

   Show answer

   Answer: E

   Explanation

   E. The printSchema member of DataFrame is an operation and needs to be followed by parentheses. The correct code block should be storesDF.printSchema() with parentheses to indicate that it's a method call.

   Source: Adapted from community discussion — 4be8126

   Next question

10. Question 120 · 219 in bank

    The code block shown below should create and register a SQL UDF named "ASSESS_PERFORMANCE" using the Python function assessPerformance() and apply it to column customerSatisfaction in table stores. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: spark._1_._2_(_3_, _4_) spark.sql("SELECT customerSatisfaction, _5_(customerSatisfaction) AS result FROM stores")

    • A1. udf 2. register 3. "ASSESS_PERFORMANCE" 4. assessPerformance 5. ASSESS_PERFORMANCE
    • B1. udf 2. register 3. assessPerformance 4. "ASSESS_PERFORMANCE" 5. "ASSESS_PERFORMANCE"
    • C1. udf 2. register 3."ASSESS_PERFORMANCE" 4. assessPerformance 5. "ASSESS_PERFORMANCE"
    • D1. register 2. udf 3. "ASSESS_PERFORMANCE" 4. assessPerformance 5. "ASSESS_PERFORMANCE"
    • E1. udf 2. register 3. ASSESS_PERFORMANCE 4. assessPerformance 5. ASSESS_PERFORMANCE

    Show answer

    Answer: A

    Explanation

    Answer: A See code example below with Spark 3.5.1: from pyspark.sql import SparkSession spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 43161, "A"), (1, 51200, "A"), (3, 78367, "B"), ] storesDF = spark.createDataFrame(data, ["storeID", "sqft", "division"]) def assess_performance(x): return "Large" if x > 50000 else "Small" spark.udf.register("ASSESS_PERFORMANCE", assess_performance, "STRING") storesDF.createOrReplaceTempView("stores") df = spark.sql("SELECT StoreID, ASSESS_PERFORMANCE(sqft) AS performance FROM stores") df.show()

    Source: Adapted from community discussion — jds0

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 111. 219 questions available.