SPARK-DEV practice questions 201–210

1. Question 201 · 219 in bank

   A data engineer wants to create a Streaming DataFrame that reads from a Kafka topic called feed. 1. spark 2. .readStream 3. .format ("kafka") 4. .option("kafka.bootstrap.servers", "host1:port1,host2:port2") 5. ._____________ 6. .load () Which code fragment should be inserted in line 5 to meet the requirement?

   • A.option("subscribe", "feed")
   • B.option("subscribe.topic", "feed")
   • C.option("kafka.topic", "feed")
   • D.option(“topic”, “feed”)

   Show answer

   Answer: A

   Next question

2. Question 202 · 219 in bank

   A data scientist is working on a project that requires processing large amounts of structured data, performing SQL queries, and applying machine learning algorithms. The data scientist is considering using Apache Spark for this task. Which combination of Apache Spark modules should the data scientist use in this scenario?

   • ASpark DataFrames, Structured Streaming, and Graphx
   • BSpark SQL, Pandas API on Spark, and Structured Streaming
   • CSpark Streaming, GraphX, and Pandas API on Spark
   • DSpark DataFrames, Spark SQL, and MLIib

   Show answer

   Answer: D

   Next question

3. Question 203 · 219 in bank

   A data engineer is implementing a streaming pipeline with watermarking to handle late-arriving records. The engineer has written the following code: inputStream .withWatermark ("event_time", "10 minutes") .groupBy (window ("event_time", "15 minutes")) .count () What happens to data that arrives after the watermark threshold?

   • ARecords that arrive later than the watermark threshold (10 minutes) will automatically be included in the aggregation if they fall within the 15-minute window.
   • BAny data arriving more than 10 minutes after the watermark threshold will be ignored and not included in the aggregation.
   • CData arriving more than 10 minutes after the latest watermark will still be included in the aggregation but will be placed into the next window.
   • DThe watermark ensures that late data arriving within 10 minutes of the latest event_time will be processed and included in the windowed aggregation.

   Show answer

   Answer: B

   Next question

4. Question 204 · 219 in bank

   A data scientist wants to ingest a directory full of plain text files so that each record in the output DataFrame contains the entire contents of a single file and the full path of the file the text was read from. The first attempt at the code does read the text files but each record contains a single line. This code is shown below: raw_txt_path = '/datasets/raw_txt/*' corpus = spark.read.text(raw_txt_path)\ .select ("*', '_metadata.file path") Which code change can be implemented in a DataFrame that meets the data scientist’s requirements?

   • AAdd the option wholetext=True to the text() function
   • BAdd the option linesep="\n" to the text() function
   • CAdd the option wholetext=False to the text() function
   • DAdd the option lineSep=”,” to the text() function

   Show answer

   Answer: A

   Next question

5. Question 205 · 219 in bank

   Which Spark configuration controls the number of tasks that can run in parallel on the executor?

   • Aspark.executor.cores
   • Bspark.task.maxFailures
   • Cspark.driver.cores
   • Dspark.executor.memory

   Show answer

   Answer: A

   Next question

6. Question 206 · 219 in bank

   A developer wants to refactor some older Spark code in order to take advantage of built-in functions introduced in Spark 3.5.0. The developer comes across the following existing DataFrame code: import pyspark.sql.functions as F min_price = 110.50 result_df = prices_df \ .filter(F.col("spot_price") >= F.1it(min_price)) \ .agg(F.count ("*")) Which code block should the developer use to refactor the code?

   • Aresult_df = prices_df \ .withColumn(“valid_price”, F.when(F.col(“spot_price”) > F.lit(min_price), 1).otherwise(0))
   • Bresult_df = prices_df \ .agg(F.count_if(F.col(“spot_price”) >= F.lit(min_price)))
   • Cresult_df = prices_df \ .agg(F.min(“spot_price”), F.max(“spot_price”))
   • Dresult_df = prices_df \ .agg(F.count(“spot_price”).alias(“spot_price”))\ .filter(F.col(“spot_price”) > F.it(“min_price”))

   Show answer

   Answer: B

   Next question

7. Question 207 · 219 in bank

   Select 2

   A data engineer is optimizing a Spark application that performs the following operations in sequence: 1. Reads data from distributed storage into a DataFrame. 2. Applies a filter transformation to remove certain records. 3. Performs a join operation with another large DataFrame. 4. Applies a groupBy and agg to aggregate data. 5. Writes the result to storage. Which two statements describe how Apache Spark constructs the execution hierarchy for this application? (Choose two.)

   • AThe groupBy and agg operations may cause a shuffle, resulting in additional stages.
   • BThe application will execute as a single job initiated by the write action.
   • CThe DataFrame API eliminates all shuffles through automatic optimization.
   • DThe join operation causes a shuffle, leading to a new stage in the execution plan.
   • EThe filter and join transformations, are executed in the same stage because they are narrow transformations.

   Show answer

   Answer: AD

   Next question

8. Question 208 · 219 in bank

   A data analyst at an e-commerce company needs to process daily sales data. The data consists of approximately 50,000 records stored in a single CSV file, totaling about 20 MB. The analyst needs to perform aggregations and generate a summary report. Which approach could the data analyst use in this situation?

   • ADeploy a real-time streaming solution using Spark Streaming to process incoming data.
   • BUse a local Python script with the pandas library to read and analyze the CSV file.
   • CImplement Apache Spark with a distributed cluster to process the data in parallel.
   • DSet up a Hadoop ecosystem with HDFS and MapReduce for distributed processing.

   Show answer

   Answer: B

   Next question

9. Question 209 · 219 in bank

   Given the code fragment: div_by_2 = my_df.filter(“number % 2 = 0”) print (div_by_2) The output ofthe code fragment is: DataFrame[nunber: bigint] What describes the output for this transformation on the Apache Spark engine?

   • AThe filter transformation triggers an immediate computation and action returning only rows that match the condition.
   • BAll transformations in Spark are lazy, in that they do not compute her results right away.
   • CAfter each transformation in Spark the results are processed using an action.
   • DThe filter is an action in Spark it immediately executes the filing operation on my_df and returns the results as div_by_2.

   Show answer

   Answer: B

   Next question

10. Question 210 · 219 in bank

    A data engineer works with streaming data, where records arrive randomly. The engineer needs to de-duplicate records based on order_id while ensuring that the latest records are kept in the stream. Which strategy should the engineer use for de-duplicating records while considering late-arriving data and imaging memory usage?

    • AApply withWatermark() and than use dropDuplicates() on order_id to remove duplicates.
    • BUse dropDuplicates() on timestamp to remove duplicates based on the most recent timestamp.
    • CDeduplication is not supported in streaming while accounting to late-arriving data.
    • DUse dropDuplicates() on order_id without watermarking to remove all duplicates.

    Show answer

    Answer: A

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 201. 219 questions available.