SPARK-DEV practice questions 41–50

1. Question 41 · 219 in bank

   Which of the following code blocks returns a new DataFrame where column division is the first two characters of column division in DataFrame storesDF?

   • AstoresDF.withColumn(“division”, substr(col(“division”), 0, 2))
   • BstoresDF.withColumn(“division”, susbtr(col(“division”), 1, 2))
   • CstoresDF,withColumn(“division”, col(“division”).substr(0, 3))
   • DstoresDF.withColumn(“division”, col(“division”).substr(0, 2))
   • EstoresDF.withColumn(“division”, col(“division”).substr(l, 2))

   Show answer

   Answer: D

   Explanation

   D would be correct as it asks first two characters. - substr(startIndex: Int, length: Int): This function takes two arguments: -> startIndex: The starting index of the substring to extract. Indexing starts from 0. -> length: The length of the substring to extract.

   Source: Adapted from community discussion — azure_bimonster

   Next question

2. Question 42 · 219 in bank

   The code block shown below should return a new DataFrame where column division from DataFrame storesDF has been renamed to column state and column managerName from DataFrame storesDF has been renamed to column managerFullName. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: storesDF. __1__(__2__, __2__).__4__(__5__, __6__)

   • A1. withColumnRenamed 2."state" 3."division" 4.withColumnRenamed 5."managerFullName" 6."managerName"
   • B1.withColumnRenamed 2.division 3.col("state") 4. withColumnRenamed 5."managerName" 6.col("managerFullName")
   • C1. WithColumnRenamed 2. "division" 3."state" 4. withColumnRenamed 5. "managerName" 6."managerFullName"
   • D1. withColumn 2. "division" 3. "state" 4.withcolumn 5."managerName" 6."managerFullName
   • E1. withColumn 2. "division" 3. "state" 4. withColumn 5."managerName" 6."managerFullName"

   Show answer

   Answer: C

   Next question

3. Question 43 · 219 in bank

   The code block shown below should return a new DataFrame where rows in DataFrame storesDF with missing values in every column have been dropped. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: storesDF.__1__.2__(3__ = __4__)

   • A1. na 2.drop 3. how 4."any"
   • B1. na 2.drop 3. subset 4."all"
   • C1. na 2.drop 3.subset 4. "any"
   • D1. na 2.drop 3. how 4. "all"
   • E1. drop 2. na 3. how 4. "all"

   Show answer

   Answer: D

   Next question

4. Question 44 · 219 in bank

   The code block shown below should return a collection of summary statistics for column sqft in DataFrame storesDF. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: storesDF.__1__(__2__)

   • A1. summary 2. col("sqft")
   • B1. describe 2. col("sqft")
   • C1. summary 2. "sqft"
   • D1. describe 2. "sqft"
   • E1. summary 2. "all"

   Show answer

   Answer: D

   Next question

5. Question 45 · 219 in bank

   Which of the following code blocks returns a 15 percent sample of rows from DataFrame storesDF without replacement?

   • AstoresDF.sample(True, fraction = 0.15)
   • BstoresDF.sample(fraction = 0.15)
   • CstoresDF.sampleBy(fraction = 0.15)
   • DstoresDF.sample(fraction = 0.10)
   • EstoresDF.sample()

   Show answer

   Answer: B

   Next question

6. Question 46 · 219 in bank

   Of the following situations, in which will it be most advantageous to store DataFrame df at the MEMORY_AND_DISK storage level rather than the MEMORY_ONLY storage level?

   • AWhen all of the computed data in DataFrame df can fit into memory.
   • BWhen the memory is full and it’s faster to recompute all the data in DataFrame df rather than read it from disk.
   • CWhen it’s faster to recompute all the data in DataFrame df that cannot fit into memory based on its logical plan rather than read it from disk.
   • DWhen it’s faster to read all the computed data in DataFrame df that cannot fit into memory from disk rather than recompute it based on its logical plan.
   • EThe storage level MENORY_ONLY will always be more advantageous because it’s faster to read data from memory than it is to read data from disk.

   Show answer

   Answer: D

   Explanation

   All other explanation is either wrong or misleading. To understand the question, you need to understand the difference between Memory_only and Memory_and_Disk 1. Memory_and_Disk, which is the default mode for cache ro persist. That means, if the data size is larger than the memory, it will store the extra data in disk. next time when we n eed to read data, we will read data firstly from memory, and then read from disk. 2. Memory_Only means, if the data size is larger than memory, it will not store the extra data. next time we read data, we will read from memory first and then recompute the extra data which cannot store in memory. PS. Mr. 4be8126 is wrong about raising error when out of memory. Therefore, the difference/balance between Memory_only and memory_and_disk lay in how they handle the extra data out of memory. which is option D, if it is faster to read data from disk is faster than recompute it, then memory_and_disk.

   Source: Adapted from community discussion — ZSun

   Next question

7. Question 47 · 219 in bank

   Which of the following code blocks extracts the value for column sqft from the first row of DataFrame storesDF?

   • AstoresDF.first()[col("sqft")]
   • BstoresDF[0]["sqft"]
   • CstoresDF.collect(l)[0]["sqft"]
   • DstoresDF.first.sqft
   • EstoresDF.first().sqft

   Show answer

   Answer: E

   Next question

8. Question 48 · 219 in bank

   Which of the following code blocks uses SQL to return a new DataFrame containing column storeId and column managerName from a table created from DataFrame storesDF?

   • AstoresDF.createOrReplaceTempView() spark.sql("SELECT storeId, managerName FROM stores")
   • BstoresDF.query(”SELECT storeid, managerName from stores")
   • Cspark.createOrReplaceTempView("storesDF") storesDF.sql("SELECT storeId, managerName from stores")
   • DstoresDF.createOrReplaceTempView("stores") spark.sql("SELECT storeId, managerName FROM stores")
   • EstoresDF.createOrReplaceTempView("stores") storesDF.query("SELECT storeId, managerName FROM stores")

   Show answer

   Answer: D

   Next question

9. Question 49 · 219 in bank

   The code block shown below should adjust the number of partitions used in wide transformations like join() to 32. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: __1__(__2__, __3__)

   • A1. spark.conf.get 2. "spark.sql.shuffle.partitions" 3. "32"
   • B1. spark.conf.set 2. "spark.default.parallelism" 3. 32
   • C1. spark.conf.text 2. "spark.default.parallelism" 3. "32"
   • D1. spark.conf.set 2. "spark.default.parallelism" 3. "32"
   • E1. spark.conf.set 2. "spark.sql.shuffle.partitions" 3. "32"

   Show answer

   Answer: E

   Next question

10. Question 50 · 219 in bank

    Which of the following code blocks returns a DataFrame containing a column openDateString, a string representation of Java’s SimpleDateFormat? Note that column openDate is of type integer and represents a date in the UNIX epoch format — the number of seconds since midnight on January 1st, 1970. An example of Java's SimpleDateFormat is "Sunday, Dec 4, 2008 1:05 pm". A sample of storesDF is displayed below:

    

    • AstoresDF.withColumn("openDatestring", from unixtime(col("openDate“), “EEEE, MMM d, yyyy h:mm a"))
    • BstoresDF.withColumn("openDateString", from_unixtime(col("openDate“), "EEEE, MMM d, yyyy h:mm a", TimestampType()))
    • CstoresDF.withColumn("openDateString", date(col("openDate"), "EEEE, MMM d, yyyy h:mm a"))
    • DstoresDF.newColumn(col("openDateString"), from_unixtime("openDate", "EEEE, MMM d, yyyy h:mm a"))
    • EstoresDF.withColumn("openDateString", date(col("openDate“), "EEEE, MMM d, yyyy h:mm a", TimestampType))

    Show answer

    Answer: A

    Explanation

    A is the right answer # Assuming you have a SparkSession created (replace with your setup if needed) from pyspark.sql import SparkSession from pyspark.sql.functions import from_unixtime # Sample data (replace with your actual DataFrame) data = [("store1", 1668518400), ("store2", 1668432000)] # Sample UNIX epoch timestamps df = spark.createDataFrame(data, ["storeName", "openDate"]) # Convert openDate to formatted string formatted_df = df.withColumn( "openDateString", from_unixtime(col("openDate"), "EEEE, MMM d, yyyy h:mm a") ) # Print the resulting DataFrame formatted_df.show(truncate=False) # Stop the SparkSession (optional) #spark.stop()

    Source: Adapted from community discussion — c3d91ee

    Next question

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 41. 219 questions available.