SPARK-DEV practice questions 121–130

1. Question 121 · 219 in bank

   The code block shown below contains an error. The code block is intended to create a Python UDF assessPerformanceUDF() using the integer-returning Python function assessPerformance() and apply it to column customerSatisfaction in DataFrame storesDF. Identify the error. Code block: assessPerformanceUDF – udf(assessPerformance) storesDF.withColumn("result", assessPerformanceUDF(col("customerSatisfaction")))

   • AThe assessPerformance() operation is not properly registered as a UDF.
   • BThe withColumn() operation is not appropriate here – UDFs should be applied by iterating over rows instead.
   • CUDFs can only be applied vie SQL and not through the DataFrame API.
   • DThe return type of the assessPerformanceUDF() is not specified in the udf() operation.
   • EThe assessPerformance() operation should be used on column customerSatisfaction rather than the assessPerformanceUDF() operation.

   Show answer

   Answer: D

   Explanation

   The right answer is D. pyspark.sql.functions.udf(f=None, returnType=StringType) The default return type is string, but this question requires integer returning. so it should be D. "The return type of the assessPerformanceUDF() is not specified in the udf() operation."

   Source: Adapted from community discussion — ZSun

   Next question

2. Question 122 · 219 in bank

   The code block shown below contains an error. The code block is intended to use SQL to return a new DataFrame containing column storeId and column managerName from a table created from DataFrame storesDF. Identify the error. Code block: storesDF.createOrReplaceTempView("stores") storesDF.sql("SELECT storeId, managerName FROM stores")

   • AThe createOrReplaceTempView() operation does not make a Dataframe accessible via SQL.
   • BThe sql() operation should be accessed via the spark variable rather than DataFrame storesDF.
   • CThere is the sql() operation in DataFrame storesDF. The operation query() should be used instead.
   • DThis cannot be accomplished using SQL – the DataFrame API should be used instead.
   • EThe createOrReplaceTempView() operation should be accessed via the spark variable rather than DataFrame storesDF.

   Show answer

   Answer: B

   Explanation

   Option B is correct because the sql() function is not a method of a DataFrame object. It is actually a method of the SparkSession object spark. Therefore, the correct way to execute a SQL statement using Spark SQL is to call sql() on the SparkSession object as follows: spark.sql("SELECT storeId, managerName FROM stores") In the code block provided in the question, sql() is called on a DataFrame object, which will result in a DataFrame object without executing the SQL statement. Therefore, option B correctly identifies the error in the code block.

   Source: Adapted from community discussion — 4be8126

   Next question

3. Question 123 · 219 in bank

   The code block shown below should create a single-column DataFrame from Python list years which is made up of integers. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: _1_._2_(_3_, _4_)

   • A1. spark 2. createDataFrame 3. years 4. IntegerType
   • B1. DataFrame 2. create 3. [years] 4. IntegerType
   • C1. spark 2. createDataFrame 3. [years] 4. IntegertType
   • D1. spark 2. createDataFrame 3. [years] 4. IntegertType()
   • E1. spark 2. createDataFrame 3. years 4. IntegertType()

   Show answer

   Answer: E

   Explanation

   E is the right answer See code below: # Create a DataFrame from a list of integers from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType spark = SparkSession.builder.appName("MyApp").getOrCreate() years = [2017, 2018, 2019] df = spark.createDataFrame(years, IntegerType()) df.show() df.printSchema()

   Source: Adapted from community discussion — jds0

   Next question

4. Question 124 · 219 in bank

   The code block shown below contains an error. The code block is intended to cache DataFrame storesDF only in Spark’s memory and then return the number of rows in the cached DataFrame. Identify the error. Code block: storesDF.cache().count()

   • AThe cache() operation caches DataFrames at the MEMORY_AND_DISK level by default – the storage level must be specified to MEMORY_ONLY as an argument to cache().
   • BThe cache() operation caches DataFrames at the MEMORY_AND_DISK level by default – the storage level must be set via storesDF.storageLevel prior to calling cache().
   • CThe storesDF DataFrame has not been checkpointed – it must have a checkpoint in order to be cached.
   • DDataFrames themselves cannot be cached – DataFrame storesDF must be cached as a table.
   • EThe cache() operation can only cache DataFrames at the MEMORY_AND_DISK level (the default) – persist() should be used instead.

   Show answer

   Answer: E

   Explanation

   E is correct. You cannot set StorageLevel Memory_only with cache(), if memory available then it keeps everything into memory else it will spill to disk. To keep everything into Memory you need to use Persist() with Storage Level Memory only.

   Source: Adapted from community discussion — singh100

   Next question

5. Question 125 · 219 in bank

   Which of the following operations can be used to return a new DataFrame from DataFrame storesDF without inducing a shuffle?

   • AstoresDF.intersect()
   • BstoresDF.repartition(1)
   • CstoresDF.union()
   • DstoresDF.coalesce(1)
   • EstoresDF.rdd.getNumPartitions()

   Show answer

   Answer: D

   Explanation

   The correct answer is D. coalesce() can be used to return a new DataFrame with a reduced number of partitions, without inducing a shuffle. A shuffle is an expensive operation that involves the redistribution of data across a cluster, so it's important to minimize its use whenever possible. In this case, repartition() and union() both involve shuffles, while intersect() returns only the common rows between two DataFrames, and rdd.getNumPartitions() returns the number of partitions in the RDD underlying the DataFrame.

   Source: Adapted from community discussion — 4be8126

   Next question

6. Question 126 · 219 in bank

   The code block shown below contains an error. The code block is intended to return a new 12-partition DataFrame from the 8-partition DataFrame storesDF by inducing a shuffle. Identify the error. Code block: storesDF.coalesce(12)

   • AThe coalesce() operation cannot guarantee the number of target partitions – the repartition() operation should be used instead.
   • BThe coalesce() operation does not induce a shuffle and cannot increase the number of partitions – the repartition() operation should be used instead.
   • CThe coalesce() operation will only work if the DataFrame has been cached to memory – the repartition() operation should be used instead.
   • DThe coalesce() operation requires a column by which to partition rather than a number of partitions – the repartition() operation should be used instead.
   • EThe number of resulting partitions, 12, is not achievable for an 8-partition DataFrame.

   Show answer

   Answer: B

   Explanation

   The correct answer is B. The coalesce() operation can decrease the number of partitions but cannot increase the number of partitions. It also does not induce a shuffle, and is therefore more efficient when decreasing the number of partitions. If the goal is to increase the number of partitions, repartition() should be used instead.

   Source: Adapted from community discussion — 4be8126

   Next question

7. Question 127 · 219 in bank

   Which of the following Spark properties is used to configure whether DataFrame partitions that do not meet a minimum size threshold are automatically coalesced into larger partitions during a shuffle?

   • Aspark.sql.shuffle.partitions
   • Bspark.sql.autoBroadcastJoinThreshold
   • Cspark.sql.adaptive.skewJoin.enabled
   • Dspark.sql.inMemoryColumnarStorage.batchSize
   • Espark.sql.adaptive.coalescePartitions.enabled

   Show answer

   Answer: E

   Explanation

   The answer is E. spark.sql.adaptive.coalescePartitions.enabled is the Spark property used to configure whether DataFrame partitions that do not meet a minimum size threshold are automatically coalesced into larger partitions during a shuffle. When set to true, Spark automatically coalesces partitions that are smaller than the configured minimum size into larger partitions to optimize shuffles.

   Source: Adapted from community discussion — 4be8126

   Next question

8. Question 128 · 219 in bank

   The code block shown below contains an error. The code block is intended to return a DataFrame containing a column openDateString, a string representation of Java’s SimpleDateFormat. Identify the error. Note that column openDate is of type integer and represents a date in the UNIX epoch format – the number of seconds since midnight on January 1st, 1970. An example of Java’s SimpleDateFormat is "Sunday, Dec 4, 2008 1:05 PM". A sample of storesDF is displayed below:

   

   Code block: storesDF.withColumn("openDateString", from_unixtime(col("openDate"), "EEE, MMM d, yyyy h:mm a", TimestampType()))

   • AThe from_unixtime() operation only accepts two parameters – the TimestampTime() arguments not necessary.
   • BThe from_unixtime() operation only works if column openDate is of type long rather than integer – column openDate must first be converted.
   • CThe second argument to from_unixtime() is not correct – it should be a variant of TimestampType() rather than a string.
   • DThe from_unixtime() operation automatically places the input column in java’s SimpleDateFormat – there is no need for a second or third argument.
   • EThe column openDate must first be converted to a timestamp, and then the Date() function can be used to reformat to java’s SimpleDateFormat.

   Show answer

   Answer: A

   Explanation

   A is correct: from pyspark.sql.functions import from_unixtime, col storesDF = spark.createDataFrame([(0, 1100746394), (1, 1474410343)], ['storeId', 'openDate']) storesDF = storesDF.withColumn("openDateString", from_unixtime(col("openDate"), "EEE, MMM d, yyyy h:mm a")) display(storesDF)

   Source: Adapted from community discussion — juliom6

   Next question

9. Question 129 · 219 in bank

   Which of the following statements about Spark jobs is incorrect?

   • AJobs are broken down into stages.
   • BThere are multiple tasks within a single job when a DataFrame has more than one partition.
   • CJobs are collections of tasks that are divided up based on when an action is called.
   • DThere is no way to monitor the progress of a job.
   • EJobs are collections of tasks that are divided based on when language variables are defined.

   Show answer

   Answer: D

   Explanation

   There are two incorrect answers in the original question. Option D, "There is no way to monitor the progress of a job," is incorrect. As I mentioned earlier, Spark provides various tools and interfaces for monitoring the progress of a job, including the Spark UI, which provides real-time information about the job's stages, tasks, and resource utilization. Other tools, such as the Spark History Server, can be used to view completed job information. Option E, "Jobs are collections of tasks that are divided based on when language variables are defined," is also incorrect. The division of tasks in a Spark job is not based on when language variables are defined, but rather based on when actions are called.

   Source: Adapted from community discussion — 4be8126

   Next question

10. Question 130 · 219 in bank

    Which of the following code blocks returns a DataFrame containing a column dayOfYear, an integer representation of the day of the year from column openDate from DataFrame storesDF? Note that column openDate is of type integer and represents a date in the UNIX epoch format – the number of seconds since midnight on January 1st, 1970. A sample of storesDF is displayed below:

    

    • A(storesDF.withColumn("openTimestamp", col("openDate").cast("Timestamp")) . withColumn("dayOfYear", dayofyear(col("openTimestamp"))))
    • BstoresDF.withColumn("dayOfYear", get dayofyear(col("openDate")))
    • CstoresDF.withColumn("dayOfYear", dayofyear(col("openDate")))
    • D(storesDF.withColumn("openDateFormat", col("openDate").cast("Date")) . withColumn("dayOfYear", dayofyear(col("openDateFormat"))))
    • EstoresDF.withColumn("dayOfYear", substr(col("openDate"), 4, 6))

    Show answer

    Answer: A

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 121. 219 questions available.