SPARK-DEV practice questions 191–200

1. Question 191 · 219 in bank

   In the code block below, aggDf contains aggregations on a streaming DataFrame.

   

   Which output mode at line 3 ensures that the entire result table is written to the console during each trigger execution?

   • Acomplete
   • Bappend
   • Creplace
   • Daggregate

   Show answer

   Answer: A

   Next question

2. Question 192 · 219 in bank

   A data engineer is running a Batch processing job on Spark Cluster with the following configuration: • 10 worker nodes • 16 CPU cores per worker node • 64 GB RAM per node The data engineer wants to allocate four executors per node, each executor using four cores. What is the total number of CPU cores used by the application?

   • A160
   • B64
   • C80
   • D40

   Show answer

   Answer: C

   Next question

3. Question 193 · 219 in bank

   Select 2

   A developer wants to test Spark Connect with an existing Spark application. What are the two alternative ways the developer can start a local Spark Connect server without changing their existing application code? (Choose two.)

   • AExecute their pyspark shell with the option --remote "https: //localhost"
   • BExecute their pyspark shell with the option --remote "sc: //localhost”
   • CSet the environment variable SPARK_REMOTE="sc://localhost” before starting the pyspark shell
   • DAdd .remote ("sc://localhost") to their SparkSession.builder calls in their Spark code
   • EEnsure the spark property spark.connect.grpc.binding.port is set to 15002 in the application code

   Show answer

   Answer: BC

   Next question

4. Question 194 · 219 in bank

   A Spark developer is developing a Spark application to monitor task performance across a cluster. One of the application’s requirements is to track the maximum processing time for tasks on each worker node and consolidate this information on the driver for further analysis. Which technique should the developer use to achieve this?

   • AUse an RDD action like reduce () to compute the maximum time.
   • BUse an accumulator to record the maximum time on the driver.
   • CBroadcast a variable to share the maximum time among workers.
   • DConfigure the Spark UI to automatically collect maximum times.

   Show answer

   Answer: B

   Next question

5. Question 195 · 219 in bank

   Which UDF implementation calculates the length of strings in a Spark DataFrame?

   • Adf.withcolumn("length", spark.udf("len", StringType()))
   • Bdf.select(length (col("stringColumn")) -alias("Length"))
   • Cspark.udf.register("stringLength", lambda s: len(s))
   • Ddf.withcolumn("length", udf(lambda s: len(s), StringType()))

   Show answer

   Answer: D

   Next question

6. Question 196 · 219 in bank

   A Data Analyst is working on the sensor_df; this data frame contains two columns: a record_datetime: timestamp and record: array. Which code fragment returns a DataFrame that splits the record column into separate columns and has one array item per row?

   • Aexploded_df = sensor_df.withColumn (“record_exploded”, explode(“record”)) exploded_df = exploded_df.select( "record datetime", “sensor_id", "status", "health" ) exploded_df = sensor_df.withColumn ("record exploded", col ("record"))
   • Bexploded_df = exploded_df.select( "record datetime", “record_exploded.sensor_id", "record_exploded.status", "record_exploded.health" ) exploded_df = sensor_df.withColumn ("record exploded", explode("record"))
   • Cexploded_df = exploded_df.select( "record datetime", “record_exploded.sensor_id", "record_exploded.status", "record_exploded.health" ) exploded_df = sensor_df.withColumn ("record exploded", explode("record"))
   • Dexploded_df = exploded_df.select(“record_datetime”, “record_exploded”)

   Show answer

   Answer: B

   Next question

7. Question 197 · 219 in bank

   A data engineer is building a Structured Streaming pipeline that streams data from a Kafka topic. While the priority is to have it run at a minimal latency, the engineer also wants to maintain an Exactly-Once processing guarantee in the pipeline. Which trigger mode should the engineer use for their writeStream?

   • A.trigger(processingTime='1 second')
   • B.trigger(continuous=True)
   • C.trigger(continuous='1 second')
   • D.trigger(availableNow=True)

   Show answer

   Answer: A

   Next question

8. Question 198 · 219 in bank

   What is a feature of Spark Connect?

   • AIt supports DataStreamReader,DataStreamWriter,StreamingQuery Streaming APIs.
   • BSupports DataFrame, Functions, Column, SparkContext PySpark APIs.
   • CIt supports only PySpark applications.
   • DIt has built-in authentication.

   Show answer

   Answer: B

   Next question

9. Question 199 · 219 in bank

   A data scientist is working on a large dataset in Apache Spark using PySpark. The data scientist has a DataFrame df with columns ‘user_id’, 'product_id’, and ‘purchase_amount’ and needs to perform some operations on this data efficiently. Which sequence of operations results in transformations that require shuffle followed by transformations that do not?

   • Adf.filter(df.purchase_amount > 100).groupBy("user_id").sum("purchase_amount")
   • Bdf.withcolumn("discount", df.purchase_amount * 0.1).select ("discount")
   • Cdf.withcolumn("purchase_date", current_date()).where("total_purchase > 50")
   • Ddf.groupBy("user_id").agg (sum ("purchase_amount").alias("total_purchase")).repartition(10)

   Show answer

   Answer: A

   Next question

10. Question 200 · 219 in bank

    A Spark engineer must select an appropriate deployment mode for the Spark jobs. What is the benefit of using cluster mode in Apache Spark™?

    • AIn cluster mode, resources are allocated from a resource manager on the cluster, enabling better performance and scalability for large jobs
    • BIn cluster mode, the driver is responsible for executing all tasks locally without distributing them across the worker nodes.
    • CIn cluster mode, the driver runs on the client machine, which can limit the application’s ability to handle large datasets efficiently.
    • DIn cluster mode, the driver program runs on one of the worker nodes, allowing the application to fully utilize the distributed resources of the cluster.

    Show answer

    Answer: D

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 191. 219 questions available.