Question 41 · 90 in bank
You work for an online retail company. Your company collects customer purchase data in CSV files and pushes them to Cloud Storage every 10 minutes. The data needs to be transformed and loaded into BigQuery for analysis. The transformation involves cleaning the data, removing duplicates, and enriching it with product information from a separate table in BigQuery. You need to implement a low-overhead solution that initiates data processing as soon as the files are loaded into Cloud Storage. What should you do?
Show answer
Answer: C
Explanation
The best solution is C. Dataflow with Pub/Sub notifications. Dataflow offers low-overhead, real-time streaming, ideal for immediate processing of files as they land in Cloud Storage, and handles the required transformations efficiently before loading into BigQuery. Option A (Composer/Dataproc) is too heavyweight and costly for frequent, small batches. Option B (Composer Hourly Batch) is too slow, introducing delays and data staleness. Option D (Data Fusion/Cloud Run) is more complex and potentially higher overhead than a simpler Dataflow pipeline for this streaming use case.
Source: Adapted from community discussion — n2183712847