DE-PRO practice questions 51–60

1. Question 51 · 202 in bank

   A junior data engineer seeks to leverage Delta Lake's Change Data Feed functionality to create a Type 1 table representing all of the values that have ever been valid for all rows in a bronze table created with the property delta.enableChangeDataFeed = true. They plan to execute the following code as a daily job:

   

   Which statement describes the execution and results of running the above query multiple times?

   • AEach time the job is executed, newly updated records will be merged into the target table, overwriting previous values with the same primary keys.
   • BEach time the job is executed, the entire available history of inserted or updated records will be appended to the target table, resulting in many duplicate entries.
   • CEach time the job is executed, only those records that have been inserted or updated since the last execution will be appended to the target table, giving the desired result.
   • DEach time the job is executed, the differences between the original and current versions are calculated; this may result in duplicate entries for some records.

   Show answer

   Answer: B

   Explanation

   B. This bad effect (many duplicates) happens because the code reads from the starting version 0, appending all changes since the beginning.

   Source: Adapted from community discussion — Adrifersilva

   Next question

2. Question 52 · 202 in bank

   A DLT pipeline includes the following streaming tables: • raw_iot ingests raw device measurement data from a heart rate tracking device. • bpm_stats incrementally computes user statistics based on BPM measurements from raw_iot. How can the data engineer configure this pipeline to be able to retain manually deleted or updated records in the raw_iot table, while recomputing the downstream table bpm_stats table when a pipeline update is run?

   • ASet the pipelines.reset.allowed property to false on raw_iot
   • BSet the skipChangeCommits flag to true on raw_iot
   • CSet the pipelines.reset.allowed property to false on bpm_stats
   • DSet the skipChangeCommits flag to true on bpm_stats

   Show answer

   Answer: B

   Explanation

   B. Set the skipChangeCommits flag to true on raw_iot Let's break down the requirements and explain why this is the best solution: Retain manually deleted or updated records in raw_iot: The skipChangeCommits flag, when set to true, tells Delta Live Tables (DLT) to ignore any manual changes (updates or deletes) made to the table outside of the pipeline. This means that even if records are manually deleted or updated in the raw_iot table, these changes won't be reflected in the table when the pipeline runs again. Recompute downstream bpm_stats table: By default, DLT will recompute downstream tables when their upstream dependencies change. Since bpm_stats is based on raw_iot, it will naturally be recomputed when the pipeline updates, without any special configuration. Why the other options are not correct: A. Setting pipelines.reset.allowed to false on raw_iot would prevent the table from being reset, but it wouldn't address the requirement to retain manually deleted or updated records.

   Source: Adapted from community discussion — pk07

   References

   • Databricks documentation

   Next question

3. Question 53 · 202 in bank

   A data pipeline uses Structured Streaming to ingest data from Apache Kafka to Delta Lake. Data is being stored in a bronze table, and includes the Kafka-generated timestamp, key, and value. Three months after the pipeline is deployed, the data engineering team has noticed some latency issues during certain times of the day. A senior data engineer updates the Delta Table's schema and ingestion logic to include the current timestamp (as recorded by Apache Spark) as well as the Kafka topic and partition. The team plans to use these additional metadata fields to diagnose the transient processing delays. Which limitation will the team face while diagnosing this problem?

   • ANew fields will not be computed for historic records.
   • BSpark cannot capture the topic and partition fields from a Kafka source.
   • CUpdating the table schema requires a default value provided for each field added.
   • DUpdating the table schema will invalidate the Delta transaction log metadata.

   Show answer

   Answer: A

   Explanation

   There is no way to reprocess history/old records to populate these values after 3 months as Kafka does not necessarily preserve them so long. This is the function of the Raw or Bronze table. Also, the other answers just don't make sense.

   Source: Adapted from community discussion — m79590530

   Next question

4. Question 54 · 202 in bank

   A table in the Lakehouse named customer_churn_params is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources. The churn prediction model used by the ML team is fairly stable in production. The team is only interested in making predictions on records that have changed in the past 24 hours. Which approach would simplify the identification of these changed records?

   • AApply the churn model to all rows in the customer_churn_params table, but implement logic to perform an upsert into the predictions table that ignores rows where predictions have not changed.
   • BConvert the batch job to a Structured Streaming job using the complete output mode; configure a Structured Streaming job to read from the customer_churn_params table and incrementally predict against the churn model.
   • CCalculate the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers before making new predictions; only make predictions on those customers not in the previous predictions.
   • DModify the overwrite logic to include a field populated by calling spark.sql.functions.current_timestamp() as data are being written; use this field to identify records written on a particular date.
   • EReplace the current overwrite logic with a merge statement to modify only those records that have changed; write logic to make predictions on the changed records identified by the change data feed.

   Show answer

   Answer: E

   Explanation

   A, B, and C don't make sense. Adding a timestamp with an overwrite logic that overwrites everything does not make sense - all records would have a timestamp from the last night. That would be not helpful in identifying what changed. E is correct. Only write changes, use CDF to identify the changes and apply the model.

   Source: Adapted from community discussion — arekm

   Next question

5. Question 55 · 202 in bank

   A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure. The silver_device_recordings table will be used downstream to power several production monitoring dashboards and a production model. At present, 45 of the 100 fields are being used in at least one of these applications. The data engineer is trying to determine the best approach for dealing with schema declaration given the highly-nested structure of the data and the numerous fields. Which of the following accurately presents information about Delta Lake and Databricks that may impact their decision-making process?

   • AThe Tungsten encoding used by Databricks is optimized for storing string data; newly-added native support for querying JSON strings means that string types are always most efficient.
   • BBecause Delta Lake uses Parquet for data storage, data types can be easily evolved by just modifying file footer information in place.
   • CSchema inference and evolution on Databricks ensure that inferred types will always accurately match the data types used by downstream systems.
   • DBecause Databricks will infer schema using types that allow all observed data to be processed, setting types manually provides greater assurance of data quality enforcement.

   Show answer

   Answer: D

   Next question

6. Question 56 · 202 in bank

   The data engineering team maintains the following code:

   

   Assuming that this code produces logically correct results and the data in the source tables has been de-duplicated and validated, which statement describes what will occur when this code is executed?

   • AA batch job will update the enriched_itemized_orders_by_account table, replacing only those rows that have different values than the current version of the table, using accountID as the primary key.
   • BThe enriched_itemized_orders_by_account table will be overwritten using the current valid version of data in each of the three tables referenced in the join logic.
   • CNo computation will occur until enriched_itemized_orders_by_account is queried; upon query materialization, results will be calculated using the current valid version of data in each of the three tables referenced in the join logic.
   • DAn incremental job will detect if new rows have been written to any of the source tables; if new rows are detected, all results will be recalculated and used to overwrite the enriched_itemized_orders_by_account table.

   Show answer

   Answer: B

   Next question

7. Question 57 · 202 in bank

   The data architect has mandated that all tables in the Lakehouse should be configured as external Delta Lake tables. Which approach will ensure that this requirement is met?

   • AWhenever a database is being created, make sure that the LOCATION keyword is used.
   • BWhen the workspace is being configured, make sure that external cloud object storage has been mounted.
   • CWhenever a table is being created, make sure that the LOCATION keyword is used.
   • DWhen tables are created, make sure that the UNMANAGED keyword is used in the CREATE TABLE statement.

   Show answer

   Answer: C

   Explanation

   CREATE-ing a TABLE with LOCATION key word makes it EXTERNAL TABLE. By CREATE-ing the database/schema with the LOCATION key word we can have specific locations for the schemas/databases containing MANAGED tables when these tables inside these schemas/databases are CREATE-d withOUT the LOCATION key word. This approach allows for configuring MANAGED TABLES at specific locations by fully leveraging Databricks Lakehouse automatic optimizations and performance tunning for them.

   Source: Adapted from community discussion — m79590530

   Next question

8. Question 58 · 202 in bank

   The marketing team is looking to share data in an aggregate table with the sales organization, but the field names used by the teams do not match, and a number of marketing-specific fields have not been approved for the sales org. Which of the following solutions addresses the situation while emphasizing simplicity?

   • ACreate a view on the marketing table selecting only those fields approved for the sales team; alias the names of any fields that should be standardized to the sales naming conventions.
   • BCreate a new table with the required schema and use Delta Lake's DEEP CLONE functionality to sync up changes committed to one table to the corresponding table.
   • CUse a CTAS statement to create a derivative table from the marketing table; configure a production job to propagate changes.
   • DAdd a parallel table write to the current production pipeline, updating a new sales table that varies as required from the marketing table.

   Show answer

   Answer: A

   Next question

9. Question 59 · 202 in bank

   A Delta Lake table representing metadata about content posts from users has the following schema: user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP, date DATE This table is partitioned by the date column. A query is run with the following filter: longitude < 20 & longitude > -20 Which statement describes how data will be filtered?

   • AStatistics in the Delta Log will be used to identify partitions that might Include files in the filtered range.
   • BNo file skipping will occur because the optimizer does not know the relationship between the partition column and the longitude.
   • CThe Delta Engine will scan the parquet file footers to identify each row that meets the filter criteria.
   • DStatistics in the Delta Log will be used to identify data files that might include records in the filtered range.

   Show answer

   Answer: D

   Next question

10. Question 60 · 202 in bank

    A small company based in the United States has recently contracted a consulting firm in India to implement several new data engineering pipelines to power artificial intelligence applications. All the company's data is stored in regional cloud storage in the United States. The workspace administrator at the company is uncertain about where the Databricks workspace used by the contractors should be deployed. Assuming that all data governance considerations are accounted for, which statement accurately informs this decision?

    • ADatabricks runs HDFS on cloud volume storage; as such, cloud virtual machines must be deployed in the region where the data is stored.
    • BDatabricks workspaces do not rely on any regional infrastructure; as such, the decision should be made based upon what is most convenient for the workspace administrator.
    • CCross-region reads and writes can incur significant costs and latency; whenever possible, compute should be deployed in the same region the data is stored.
    • DDatabricks notebooks send all executable code from the user’s browser to virtual machines over the open internet; whenever possible, choosing a workspace region near the end users is the most secure.

    Show answer

    Answer: C

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

DE-PRO practice, question 51. 202 questions available.