SPARK-DEV practice questions 181–190

1. Question 181 · 219 in bank

   Which of the following code blocks prints the schema of DataFrame storesDF?

   • Aprint(storesDF)
   • BstoresDF.printSchema()
   • Cprint(storesDF.schema())
   • DstoresDF.schema
   • EstoresDF.schema()

   Show answer

   Answer: B

   Next question

2. Question 182 · 219 in bank

   The code block shown below should return a new 4-partition DataFrame from the 8-partition DataFrame storesDF without inducing a shuffle. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: _1_._2_(_3_)

   • A1. storesDF 2. coalesce 3. Nothing
   • B1. storesDF 2. coalesce 3. 4
   • C1. storesDF 2. coalesce 3. 4, "storeId"
   • D1. storesDF 2. coalesce 3. "storeId"

   Show answer

   Answer: B

   Next question

3. Question 183 · 219 in bank

   The code block shown below contains an error. The code block intended to return a new DataFrame that is the result of an outer join between DataFrame storesDF and DataFrame employeesDF on column storeId. Identify the error. Code block: storesDF.join(employeesDF, "storeId")

   • AThe default argument to the how parameter is "inner" – an additional argument of "outer" must be specified.
   • BThe key column storeId needs to be wrapped in the col() operation.
   • CThe key column storeId needs to be specified in an expression of both Data Frame columns like storesDF.storeId == employeesDF.storeId.
   • DThe key column storeId needs to be in a list like ["storeId"].
   • EThere is no DataFrame.join() operation – DataFrame.merge() should be used instead.

   Show answer

   Answer: A

   Next question

4. Question 184 · 219 in bank

   If Spark is running in cluster mode, which of the following statements about nodes is incorrect?

   • AThe Spark driver runs in its own node without any executors.
   • BEach executor is a running processing engine inside of a worker node.
   • CThere may be more executors than total nodes or more total nodes than executors.
   • DThere is always more than one node.
   • EThere is a single node that contains the Spark driver and the executors.

   Show answer

   Answer: E

   Next question

5. Question 185 · 219 in bank

   The code block shown below should create a Python UDF assessPerformanceUDF using the Python function assessPerformance() and apply it to column customerSatisfaction in DataFrame storesDF. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: assessPerformanceUDF = _1_(_2_) storesDF.withColumn(“result”, _3_(_4_))

   • A1. udf 2. assessPerformance, IntegerType() 3. assessPerformanceUDF 4. col(“customerSatisfaction”)
   • B1. udf 2. assessPerformance 3. assessPerformanceUDF 4. col(“customerSatisfaction”)
   • C1. spark.register.udf 2. assessPerformanceUDF 3. assessPerformance 4. col(“customerSatisfaction”)
   • D1. spark.register.udf 2. assessPerformance, IntegerType() 3. assessPerformanceUDF 4. col(“customerSatisfaction”)
   • E1. spark.register.udf 2. assessPerformance 3. assessPerformanceUDF 4. “customerSatisfaction”

   Show answer

   Answer: B

   Next question

6. Question 186 · 219 in bank

   A developer needs to produce a Python dictionary using data stored in a small Parquet table, which looks like this:

   

   The resulting Python dictionary must contain a mapping of region –> region_id containing the smallest 3 region_id values. Which code fragment meets the requirements?

   • Aregions = dict( regions_df \ .select(‘region’, ‘region_id’) \ .sort(‘region_id’) \ .take(3) )
   • Bregions = dict( regions_df \ .select(‘region_id’, ‘region’) \ .sort(‘region_id’) \ .take(3) )
   • Cregions = dict( regions_df \ .select(‘region_id’, ‘region’) \ .limit(3) \ .collect() )
   • Dregions = dict( regions_df \ .select(‘region’, ‘region_id’) \ .sort(desc(‘region_id’)) \ .take(3) )

   Show answer

   Answer: B

   Next question

7. Question 187 · 219 in bank

   A Spark engineer is troubleshooting a Spark application that has been encountering out-of-memory errors during execution. By reviewing the Spark driver logs, the engineer notices multiple “GC overhead limit exceeded” messages. Which action should the engineer take to resolve this issue?

   • AOptimize the data processing logic by repartitioning the DataFrame.
   • BModify the Spark configuration to disable garbage collection.
   • CIncrease the memory allocated to the Spark Driver.
   • DCache large DataFrames to persist them in memory.

   Show answer

   Answer: C

   Next question

8. Question 188 · 219 in bank

   A Spark developer wants to improve the performance of an existing PySpark UDF that runs a hash function that is not available in the standard Spark functions library. The existing UDF code is:

   

   The developer wants to replace this existing UDF with a Pandas UDF to improve performance. The developer changes the definition of shake_256_udf to this: shake_256_udf = sf.pandas_udf(shake_256, StringType()) However, the developer receives the error: [UNSUPPORTED_SIGNATURE] Unsupported signature: (raw: str) –> str. What should the signature of the shake_256() function be changed to in order to fix this error?

   • Adef shake 256(df: pd.Series) –> str:
   • Bdef shake_256(df: Iterator[pd.Series]) –> Iterator[pd.Series]:
   • Cdef shake 256(raw: str) –> str:
   • Ddef shake_256(df: pd.Series) –> pd.Series:

   Show answer

   Answer: D

   Next question

9. Question 189 · 219 in bank

   A Developer is working with a pandas DataFrame containing user behaviour data from a web application. Which approach should be used for executing groupby operation in parallel across all workers of Apache Spark 3.5?

   • AUse the applyInPandas API: import pandas as pd def mean_func(key, pdf): return pd.DataFrame([(key + (pdf['value'].mean(),)]) df .groupby (‘user_id') .applyInPandas (mean_func, schema= “user_id long, value double") .show()
   • BUtilize the mapInPandas API: def mean_func(pdf_iter): for pdf in pdf_iter: yield pdf.groupby(‘user_id’).agg({‘value’: ‘mean’}).reset_index() df.mapInPandas(mean_func, schema=”user_id long, value double”).show()
   • CUse a regular Spark UDF (User-Defined Function): from pyspark.sql.functions import men df.groupBy (‘user_id’).agg(mean(‘value’)).show()
   • DCreate a Pandas UDF (User-Defined Function): from pyspark.sql.functions import pandas_udf @pandas_udf(“double”) def mean_func(value: pd.Series) –> float: return value.mean() df.groupby(“user_id”).agg(mean_func(df[‘value’])).show()

   Show answer

   Answer: A

   Next question

10. Question 190 · 219 in bank

    A data engineer has been asked to produce a Parquet table which is overwritten every day with the latest data. The downstream consumer of this Parquet table has a hard requirement that the data in this table is produced with all records sorted by the market_time field. Which line of Spark code will produce a Parquet table that meets these requirements?

    • Afinal_df \ .sort ("market_time") \ .write \ .format ("parquet") \ .mode ("overwrite") \ .saveAsTable ("output .market_events")
    • Bfinal_df \ .orderBy("market_time") \ .write \ .format ("parquet") \ .mode ("overwrite") \ .saveAsTable ("output .market_events")
    • Cfinal_df \ .sort ("market_time") \ .coalesce(1) \ .write \ .format ("parquet") \ .mode ("overwrite") \ .saveAsTable ("output .market_events")
    • Dfinal_df \ .sortWithinPartitions("market_time") \ .write \ .format ("parquet") \ .mode ("overwrite") \ .saveAsTable ("output .market_events")

    Show answer

    Answer: C

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 181. 219 questions available.