PDE practice questions 181–190

1. Question 181 · 340 in bank

   You maintain ETL pipelines. You notice that a streaming pipeline running on Dataflow is taking a long time to process incoming data, which causes output delays. You also noticed that the pipeline graph was automatically optimized by Dataflow and merged into one step. You want to identify where the potential bottleneck is occurring. What should you do?

   • AInsert a Reshuffle operation after each processing step, and monitor the execution details in the Dataflow console.
   • BInsert output sinks after each key processing step, and observe the writing throughput of each block.
   • CLog debug information in each ParDo function, and analyze the logs at execution time.
   • DVerify that the Dataflow service accounts have appropriate permissions to write the processed data to the output sinks.

   Show answer

   Answer: A

   Explanation

   - The Reshuffle operation is used in Dataflow pipelines to break fusion and redistribute elements, which can sometimes help improve parallelization and identify bottlenecks. - By inserting Reshuffle after each processing step and observing the pipeline's performance in the Dataflow console, you can potentially identify stages that are disproportionately slow or stalled. - This can help in pinpointing the step where the bottleneck might be occurring.

   Source: Adapted from community discussion — raaad

   References

   • Google Cloud documentation

   Next question

2. Question 182 · 340 in bank

   You are running your BigQuery project in the on-demand billing model and are executing a change data capture (CDC) process that ingests data. The CDC process loads 1 GB of data every 10 minutes into a temporary table, and then performs a merge into a 10 TB target table. This process is very scan intensive and you want to explore options to enable a predictable cost model. You need to create a BigQuery reservation based on utilization information gathered from BigQuery Monitoring and apply the reservation to the CDC process. What should you do?

   • ACreate a BigQuery reservation for the dataset.
   • BCreate a BigQuery reservation for the job.
   • CCreate a BigQuery reservation for the service account running the job.
   • DCreate a BigQuery reservation for the project.

   Show answer

   Answer: D

   Explanation

   C or D ?? Option C (service account) allows you to target the reservation specifically to the CDC process or any other jobs run by that service account. This is particularly useful if you have multiple processes running in the project with different performance or cost requirements. Option D (project) applies the reservation across all jobs in the project, which is a broader approach. If the CDC process is the primary or sole job running in the project and you want all jobs to share the same reservation, then this option might be more straightforward.

   Source: Adapted from community discussion — raaad

   References

   • Google Cloud documentation
   • Google Cloud documentation
   • Google Cloud documentation

   Next question

3. Question 183 · 340 in bank

   You are designing a fault-tolerant architecture to store data in a regional BigQuery dataset. You need to ensure that your application is able to recover from a corruption event in your tables that occurred within the past seven days. You want to adopt managed services with the lowest RPO and most cost-effective solution. What should you do?

   • AAccess historical data by using time travel in BigQuery.
   • BExport the data from BigQuery into a new table that excludes the corrupted data
   • CCreate a BigQuery table snapshot on a daily basis.
   • DMigrate your data to multi-region BigQuery buckets.

   Show answer

   Answer: A

   Explanation

   - Lowest RPO: Time travel offers point-in-time recovery for the past seven days by default, providing the shortest possible recovery point objective (RPO) among the given options. You can recover data to any state within that window. - No Additional Costs: Time travel is a built-in feature of BigQuery, incurring no extra storage or operational costs. - Managed Service: BigQuery handles time travel automatically, eliminating manual backup and restore processes.

   Source: Adapted from community discussion — raaad

   Next question

4. Question 184 · 340 in bank

   You are building a streaming Dataflow pipeline that ingests noise level data from hundreds of sensors placed near construction sites across a city. The sensors measure noise level every ten seconds, and send that data to the pipeline when levels reach above 70 dBA. You need to detect the average noise level from a sensor when data is received for a duration of more than 30 minutes, but the window ends when no data has been received for 15 minutes. What should you do?

   • AUse session windows with a 15-minute gap duration.
   • BUse session windows with a 30-minute gap duration.
   • CUse hopping windows with a 15-minute window, and a thirty-minute period.
   • DUse tumbling windows with a 15-minute window and a fifteen-minute .withAllowedLateness operator.

   Show answer

   Answer: A

   Explanation

   to detect average noise levels from sensors, the best approach is to use session windows with a 15-minute gap duration (Option A). Session windows are ideal for cases like this where the events (sensor data) are sporadic. They group events that occur within a certain time interval (15 minutes in your case) and a new window is started if no data is received for the duration of the gap. This matches your requirement to end the window when no data is received for 15 minutes, ensuring that the average noise level is calculated over periods of continuous data

   Source: Adapted from community discussion — datapassionate

   References

   • Google Cloud documentation

   Next question

5. Question 185 · 340 in bank

   You are creating a data model in BigQuery that will hold retail transaction data. Your two largest tables, sales_transaction_header and sales_transaction_line, have a tightly coupled immutable relationship. These tables are rarely modified after load and are frequently joined when queried. You need to model the sales_transaction_header and sales_transaction_line tables to improve the performance of data analytics queries. What should you do?

   • ACreate a sales_transaction table that holds the sales_transaction_header information as rows and the sales_transaction_line rows as nested and repeated fields.
   • BCreate a sales_transaction table that holds the sales_transaction_header and sales_transaction_line information as rows, duplicating the sales_transaction_header data for each line.
   • CCreate a sales_transaction table that stores the sales_transaction_header and sales_transaction_line data as a JSON data type.
   • DCreate separate sales_transaction_header and sales_transaction_line tables and, when querying, specify the sales_transaction_line first in the WHERE clause.

   Show answer

   Answer: A

   Explanation

   - In BigQuery, nested and repeated fields can significantly improve performance for certain types of queries, especially joins, because the data is co-located and can be read efficiently. - - This approach is often used in data warehousing scenarios where query performance is a priority, and the data relationships are immutable and rarely modified.

   Source: Adapted from community discussion — raaad

   References

   • Google Cloud documentation

   Next question

6. Question 186 · 340 in bank

   You created a new version of a Dataflow streaming data ingestion pipeline that reads from Pub/Sub and writes to BigQuery. The previous version of the pipeline that runs in production uses a 5-minute window for processing. You need to deploy the new version of the pipeline without losing any data, creating inconsistencies, or increasing the processing latency by more than 10 minutes. What should you do?

   • AUpdate the old pipeline with the new pipeline code.
   • BSnapshot the old pipeline, stop the old pipeline, and then start the new pipeline from the snapshot.
   • CDrain the old pipeline, then start the new pipeline.
   • DCancel the old pipeline, then start the new pipeline.

   Show answer

   Answer: C

   Explanation

   I don't think C is correct, as it will immediately fire the window: "Draining can result in partially filled windows. In that case, if you restart the drained pipeline, the same window might fire a second time, which can cause issues with your data. " https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#effects Maybe "A" means launching a replacement job? https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#Launching

   Source: Adapted from community discussion — AlizCert

   References

   • Google Cloud documentation
   • Google Cloud documentation
   • Google Cloud documentation

   Next question

7. Question 187 · 340 in bank

   Your organization's data assets are stored in BigQuery, Pub/Sub, and a PostgreSQL instance running on Compute Engine. Because there are multiple domains and diverse teams using the data, teams in your organization are unable to discover existing data assets. You need to design a solution to improve data discoverability while keeping development and configuration efforts to a minimum. What should you do?

   • AUse Data Catalog to automatically catalog BigQuery datasets. Use Data Catalog APIs to manually catalog Pub/Sub topics and PostgreSQL tables.
   • BUse Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use Data Catalog APIs to manually catalog PostgreSQL tables.
   • CUse Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use custom connectors to manually catalog PostgreSQL tables.
   • DUse customer connectors to manually catalog BigQuery datasets, Pub/Sub topics, and PostgreSQL tables.

   Show answer

   Answer: C

   Explanation

   Data Catalog is the best choice. But for catalogging PostgreSQL it is better to use a connector when available, instead of using API. https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_unsupported_data_sources

   Source: Adapted from community discussion — datapassionate

   References

   • Google Cloud documentation
   • Google Cloud documentation

   Next question

8. Question 188 · 340 in bank

   You are building a model to predict whether or not it will rain on a given day. You have thousands of input features and want to see if you can improve training speed by removing some features while having a minimum effect on model accuracy. What can you do?

   • AEliminate features that are highly correlated to the output labels.
   • BCombine highly co-dependent features into one representative feature.
   • CInstead of feeding in each feature individually, average their values in batches of 3.
   • DRemove the features that have null values for more than 50% of the training records.

   Show answer

   Answer: B

   Explanation

   B. Combine highly co-dependent features into one representative feature. Combining highly correlated features into a single representative feature can reduce the dimensionality of your dataset, making the training process faster while preserving relevant information. This approach often helps eliminate redundancy in the input data. Option A (eliminating features that are highly correlated to the output labels) can be counterproductive, as you want to maintain features that are informative for your prediction task. Removing features that are correlated with the output may reduce model accuracy. Option C (averaging feature values in batches of 3) is not a common technique for reducing dimensionality, and it could lead to loss of important information. Option D (removing features with null values for more than 50% of training records) can help reduce the dimensionality and may be useful if you have a large number of features with missing data, but it may not necessarily address co-dependency among features.

   Source: Adapted from community discussion — rtcpost

   References

   • Google Cloud documentation
   • Google Cloud documentation

   Next question

9. Question 189 · 340 in bank

   You need to create a SQL pipeline. The pipeline runs an aggregate SQL transformation on a BigQuery table every two hours and appends the result to another existing BigQuery table. You need to configure the pipeline to retry if errors occur. You want the pipeline to send an email notification after three consecutive failures. What should you do?

   • AUse the BigQueryUpsertTableOperator in Cloud Composer, set the retry parameter to three, and set the email_on_failure parameter to true.
   • BUse the BigQueryInsertJobOperator in Cloud Composer, set the retry parameter to three, and set the email_on_failure parameter to true.
   • CCreate a BigQuery scheduled query to run the SQL transformation with schedule options that repeats every two hours, and enable email notifications.
   • DCreate a BigQuery scheduled query to run the SQL transformation with schedule options that repeats every two hours, and enable notification to Pub/Sub topic. Use Pub/Sub and Cloud Functions to send an email after three failed executions.

   Show answer

   Answer: B

   Explanation

   - It provides a direct and controlled way to manage the SQL pipeline using Cloud Composer (Apache Airflow). - The BigQueryInsertJobOperator is well-suited for running SQL jobs in BigQuery, including aggregate transformations and handling of results. - The retry and email_on_failure parameters align with the requirements for error handling and notifications. - Cloud Composer requires more setup than using BigQuery's scheduled queries directly, but it offers robust workflow management, retry logic, and notification capabilities, making it suitable for more complex and controlled data pipeline requirements.

   Source: Adapted from community discussion — raaad

   References

   • Google Cloud documentation
   • Google Cloud documentation

   Next question

10. Question 190 · 340 in bank

    You are monitoring your organization’s data lake hosted on BigQuery. The ingestion pipelines read data from Pub/Sub and write the data into tables on BigQuery. After a new version of the ingestion pipelines is deployed, the daily stored data increased by 50%. The volumes of data in Pub/Sub remained the same and only some tables had their daily partition data size doubled. You need to investigate and fix the cause of the data increase. What should you do?

    • A1. Check for duplicate rows in the BigQuery tables that have the daily partition data size doubled. 2. Schedule daily SQL jobs to deduplicate the affected tables. 3. Share the deduplication script with the other operational teams to reuse if this occurs to other tables.
    • B1. Check for code errors in the deployed pipelines. 2. Check for multiple writing to pipeline BigQuery sink. 3. Check for errors in Cloud Logging during the day of the release of the new pipelines. 4. If no errors, restore the BigQuery tables to their content before the last release by using time travel.
    • C1. Check for duplicate rows in the BigQuery tables that have the daily partition data size doubled. 2. Check the BigQuery Audit logs to find job IDs. 3. Use Cloud Monitoring to determine when the identified Dataflow jobs started and the pipeline code version. 4. When more than one pipeline ingests data into a table, stop all versions except the latest one.
    • D1. Roll back the last deployment. 2. Restore the BigQuery tables to their content before the last release by using time travel. 3. Restart the Dataflow jobs and replay the messages by seeking the subscription to the timestamp of the release.

    Show answer

    Answer: C

    Explanation

    - Detailed Investigation of Logs and Jobs Checking for duplicate rows targets the potential immediate cause of the issue. - Checking the BigQuery Audit logs helps identify which jobs might be contributing to the increased data volume. - Using Cloud Monitoring to correlate job starts with pipeline versions helps identify if a specific version of the pipeline is responsible. - Managing multiple versions of pipelines ensures that only the intended version is active, addressing any versioning errors that might have occurred during deployment. ======= Why not B While it addresses the symptom (excess data), it doesn't necessarily stop the problem from recurring. (The questions asked to investigate and fix)

    Source: Adapted from community discussion — raaad

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

PDE practice, question 181. 340 questions available.