Question 11 · 364 in bank
You are experimenting with a built-in distributed XGBoost model in Vertex AI Workbench user-managed notebooks. You use BigQuery to split your data into training and validation sets using the following queries: CREATE OR REPLACE TABLE ‘myproject.mydataset.training‘ AS (SELECT * FROM ‘myproject.mydataset.mytable‘ WHERE RAND() <= 0.8); CREATE OR REPLACE TABLE ‘myproject.mydataset.validation‘ AS (SELECT * FROM ‘myproject.mydataset.mytable‘ WHERE RAND() <= 0.2); After training the model, you achieve an area under the receiver operating characteristic curve (AUC ROC) value of 0.8, but after deploying the model to production, you notice that your model performance has dropped to an AUC ROC value of 0.65. What problem is most likely occurring?
Show answer
Answer: C
Explanation
- Excluding D as RAND() samples 80% for “.training” & 20% for “.validaton”: https://stackoverflow.com/questions/42115968/how-does-rand-works-in-bigquery; - Could be that those 2 samplings share some records since pseudo-randomly sampled over the same “.mytable”, & therefore might not be using all of its data, thus C seems valid; - Excluding B as there is no indication otherwise of insufficient amount of training data, after training AUC ROC was 0.8, that we know; - There could be a training-serving skew occurring in Prod, but “most likely occurring” is C as a result of the selective information presented: https://developers.google.com/machine-learning/guides/rules-of-ml#training-serving_skew
Source: Adapted from community discussion — M25