DE-PRO practice questions 41–50

1. Question 41 · 202 in bank

   Which statement regarding stream-static joins and static Delta tables is correct?

   • AThe checkpoint directory will be used to track updates to the static Delta table.
   • BEach microbatch of a stream-static join will use the most recent version of the static Delta table as of the job's initialization.
   • CThe checkpoint directory will be used to track state information for the unique keys present in the join.
   • DStream-static joins cannot use static Delta tables because of consistency issues.

   Show answer

   Answer: B

   Explanation

   When Databricks processes a micro-batch of data in a stream-static join, the latest valid version of data from the static Delta table joins with the records present in the current micro-batch. Because the join is stateless, you do not need to configure watermarking and can process results with low latency. The data in the static Delta table used in the join should be slowly-changing. https://docs.databricks.com/en/transform/join.html#stream-static

   Source: Adapted from community discussion — MDWPartners

   References

   • Databricks documentation

   Next question

2. Question 42 · 202 in bank

   A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Events are recorded once per minute per device. Streaming DataFrame df has the following schema: "device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT" Code block:

   

   Which line of code correctly fills in the blank within the code block to complete this task?

   • Ato_interval("event_time", "5 minutes").alias("time")
   • Bwindow("event_time", "5 minutes").alias("time")
   • C"event_time"
   • Dlag("event_time", "10 minutes").alias("time")

   Show answer

   Answer: B

   Explanation

   This is the standard syntax to do non-overlapping time interval Window-ed grouping by the time field in a dataset in Structured Streaming. .withWatermatk() function defines the staging buffers after which delayed records will be dropped/ignored.

   Source: Adapted from community discussion — m79590530

   Next question

3. Question 43 · 202 in bank

   A Structured Streaming job deployed to production has been resulting in higher than expected cloud storage costs. At present, during normal execution, each microbatch of data is processed in less than 3s; at least 12 times per minute, a microbatch is processed that contains 0 records. The streaming write was configured using the default trigger settings. The production job is currently scheduled alongside many other Databricks jobs in a workspace with instance pools provisioned to reduce start-up time for jobs with batch execution. Holding all other variables constant and assuming records need to be processed in less than 10 minutes, which adjustment will meet the requirement?

   • ASet the trigger interval to 3 seconds; the default trigger interval is consuming too many records per batch, resulting in spill to disk that can increase volume costs.
   • BUse the trigger once option and configure a Databricks job to execute the query every 10 minutes; this approach minimizes costs for both compute and storage.
   • CSet the trigger interval to 10 minutes; each batch calls APIs in the source storage account, so decreasing trigger frequency to maximum allowable threshold should minimize this cost.
   • DSet the trigger interval to 500 milliseconds; setting a small but non-zero trigger interval ensures that the source is not queried too frequently.

   Show answer

   Answer: C

   Explanation

   C, A - incorrect explanation B - trigger once is not correct option here D - 500 miliseconds is already used, it's default trigger interval

   Source: Adapted from community discussion — Isio05

   Next question

4. Question 44 · 202 in bank

   An hourly batch job is configured to ingest data files from a cloud object storage container where each batch represent all records produced by the source system in a given hour. The batch job to process these records into the Lakehouse is sufficiently delayed to ensure no late-arriving data is missed. The user_id field represents a unique key for the data, which has the following schema: user_id BIGINT, username STRING, user_utc STRING, user_region STRING, last_login BIGINT, auto_pay BOOLEAN, last_updated BIGINT New records are all ingested into a table named account_history which maintains a full record of all data in the same schema as the source. The next table in the system is named account_current and is implemented as a Type 1 table representing the most recent value for each unique user_id. Assuming there are millions of user accounts and tens of thousands of records processed hourly, which implementation can be used to efficiently update the described account_current table as part of each hourly batch job?

   • AUse Auto Loader to subscribe to new files in the account_history directory; configure a Structured Streaming trigger once job to batch update newly detected files into the account_current table.
   • BOverwrite the account_current table with each batch using the results of a query against the account_history table grouping by user_id and filtering for the max value of last_updated.
   • CFilter records in account_history using the last_updated field and the most recent hour processed, as well as the max last_iogin by user_id write a merge statement to update or insert the most recent value for each user_id.
   • DUse Delta Lake version history to get the difference between the latest version of account_history and one version prior, then write these records to account_current.
   • EFilter records in account_history using the last_updated field and the most recent hour processed, making sure to deduplicate on username; write a merge statement to update or insert the most recent value for each username.

   Show answer

   Answer: C

   Explanation

   My reasoning is thus: The application is based on batch processes, so A is wrong. Overwriting the table would destroy the Type 1 SCD behavior, so B is wrong. Comparing versions of account_history would not be efficient, as the whole data would be scanned, so D is wrong. 'username' is not a key column, so we have no guarantee that it's unique, thus de-duplicating by it can yield wrongly grouped sets of rows, so E is not a safe bet, with the information we know. C is the best option.

   Source: Adapted from community discussion — RafaelCFC

   Next question

5. Question 45 · 202 in bank

   Which statement characterizes the general programming model used by Spark Structured Streaming?

   • AStructured Streaming leverages the parallel processing of GPUs to achieve highly parallel data throughput.
   • BStructured Streaming is implemented as a messaging bus and is derived from Apache Kafka.
   • CStructured Streaming relies on a distributed network of nodes that hold incremental state values for cached stages.
   • DStructured Streaming models new data arriving in a data stream as new rows appended to an unbounded table.

   Show answer

   Answer: D

   Next question

6. Question 46 · 202 in bank

   Which configuration parameter directly affects the size of a spark-partition upon ingestion of data into Spark?

   • Aspark.sql.files.maxPartitionBytes
   • Bspark.sql.autoBroadcastJoinThreshold
   • Cspark.sql.adaptive.advisoryPartitionSizeInBytes
   • Dspark.sql.adaptive.coalescePartitions.minPartitionNum

   Show answer

   Answer: A

   Next question

7. Question 47 · 202 in bank

   A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to complete a task as roughly the same, but the max duration for a task to be roughly 100 times as long as the minimum. Which situation is causing increased duration of the overall job?

   • ATask queueing resulting from improper thread pool assignment.
   • BSpill resulting from attached volume storage being too small.
   • CNetwork latency due to some cluster nodes being in different regions from the source data
   • DSkew caused by more data being assigned to a subset of spark-partitions.

   Show answer

   Answer: D

   Next question

8. Question 48 · 202 in bank

   Each configuration below is identical to the extent that each cluster has 400 GB total of RAM, 160 total cores and only one Executor per VM. Given an extremely long-running job for which completion must be guaranteed, which cluster configuration will be able to guarantee completion of the job in light of one or more VM failures?

   • A• Total VMs: 8 • 50 GB per Executor • 20 Cores / Executor
   • B• Total VMs: 16 • 25 GB per Executor • 10 Cores / Executor
   • C• Total VMs: 1 • 400 GB per Executor • 160 Cores/Executor
   • D• Total VMs: 4 • 100 GB per Executor • 40 Cores / Executor

   Show answer

   Answer: B

   Next question

9. Question 49 · 202 in bank

   A task orchestrator has been configured to run two hourly tasks. First, an outside system writes Parquet data to a directory mounted at /mnt/raw_orders/. After this data is written, a Databricks job containing the following code is executed:

   

   Assume that the fields customer_id and order_id serve as a composite key to uniquely identify each order, and that the time field indicates when the record was queued in the source system. If the upstream system is known to occasionally enqueue duplicate entries for a single order hours apart, which statement is correct?

   • ADuplicate records enqueued more than 2 hours apart may be retained and the orders table may contain duplicate records with the same customer_id and order_id.
   • BAll records will be held in the state store for 2 hours before being deduplicated and committed to the orders table.
   • CThe orders table will contain only the most recent 2 hours of records and no duplicates will be present.
   • DThe orders table will not contain duplicates, but records arriving more than 2 hours late will be ignored and missing from the table.

   Show answer

   Answer: A

   Explanation

   dropDuplicates only deletes the duplicates of the processed batch. If we have records with same key in different batches, we will have duplicates in the final table. In addition, withWatermark, when is not used in a window, gets the MAX(eventTime) and uses the threshold to define the time range. As the time represent when the data has been queued in the source system, we can get records where the time we get older than 2 hours. pyspark.sql.DataFrame.dropDuplicates — PySpark 3.5.3 documentation pyspark.sql.DataFrame.withWatermark — PySpark 3.5.3 documentation

   Source: Adapted from community discussion — UrcoIbz

   Next question

10. Question 50 · 202 in bank

    A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records. In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?

    • ARely on Delta Lake schema enforcement to prevent duplicate records.
    • BVACUUM the Delta table after each batch completes.
    • CPerform an insert-only merge with a matching condition on a unique key.
    • DPerform a full outer join on a unique key and overwrite existing data.

    Show answer

    Answer: C

    Explanation

    From all the provided options Answer C is the only meaningful and possible one. Also MERGE INTO ... WHEN NOT MATCHED INSERT *; is a standard solution for adding/appending non-existing records (by key) to the target table withOUT duplicating.

    Source: Adapted from community discussion — m79590530

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

DE-PRO practice, question 41. 202 questions available.