Which of the following statements about Spark application architecture is correct?
6406532312138.
A Spark program generates a set of tasks that are grouped into stages, each of
which executes concurrently with the driver being responsible for collecting results from each
stage for final output.
6406532312139.
A Spark action generates a job, with each job being broken down into multiple
stages where a stage is a set of tasks that can be executed in parallel and stages themselves are
interdependent, with each task being the unit of work mapped to an executor.
6406532312140.
A Spark program generates a job, with each job being broken down into
multiple stages where a stage is a set of tasks that can be executed in parallel and stages
themselves are interdependent, with each task being the unit of work mapped to an executor.
6406532312141.
A Spark action generates a task that is further broken down into multiple
stages where a stage is a set of jobs that can be executed in parallel and stages themselves are
interdependent, with each job being the unit of work mapped to an executor.
6406532312142.
A Spark program generates a task that is further broken down into multiple
stages where a stage is a set of jobs that can be executed in parallel and stages themselves are
interdependent, with each job being the unit of work mapped to an executor.
A published solution is not available for this question yet.
Consider the problem of sorting a 1 petabyte file of numbers stored in a Hadoop cluster of 10
machines where the size of each HDFS block is 100MB. Which of the below methods is the most
likely to finish soonest in practice?
6406532312143.
Use quicksort on each machine and then merge each machine's sorted data on
the master
6406532312144.
Exploit duality of hashing vs sorting and write a hash function that assigns
each line in the file to a distinct machine, and then sort within each machine
6406532312145.
Write a general-purpose MapReduce / Spark program that picks the Nth
highest descending value from a set, invoke it iteratively from 1 to S (S=Number of lines in file),
and create a new file of sorted data
6406532312146.
Mergesort on each machine by limiting use of memory, and then between all
possible pairs of machines
A published solution is not available for this question yet.
A website sees 1 Billion hits every month. The website owner wants to count average hits per
customer in the latest month, where a customer is denoted by the IP address of the device. The
owner has at his disposal a Hadoop cluster of 5 workers and 2 masters each with 1GB of RAM.
Which of the following methods is the most likely to finish fastest?
6406532312147.
Write a Spark program that forms a Dataframe as grouping by IP address with
count as aggregate, followed by another stage that computes the avg on top of the Dataframe of
the first stage
6406532312148.
Write a MapReduce program where the Map does nothing useful, Combine
computes the aggregated hits per customer, Shuffle combined data based on IP address across
workers, and in the Reduce, build hash table on each machine with hash key = IP address and
hash value = counter, followed by another Reduce that finally computes the avg on top of all hash
values.
6406532312149.
Write a Spark program that forms a Dataframe as grouping by IP address with
count as aggregate, followed by a take into a list in the Spark driver which further computes the
average of all the individual counts in the list
6406532312150.
Write a MapReduce program where the Map does nothing useful, Shuffle data
based on IP address across workers, and in the Reduce, build hash table on each machine with
hash key = IP address and hash value = counter, followed by another Reduce that finally computes
avg on top of all hash values.
A published solution is not available for this question yet.
An enterprise software designer wants to leverage the best of cloud to minimize the number of
administrative overheads associated with her big data pipeline while also getting on-demand
scalability without sacrificing flexibility. What option should she choose to best serve these needs?
6406532312151.
Build the data pipeline using Spark, data storage on HDFS, and deploy both on
Hadoop using IaaS
6406532312152.
Build the data pipeline using Python run on Serverless where the data is stored
on cloud storage
6406532312153.
Build the data pipeline using Spark PaaS option on top of data stored on cloud
storage
6406532312154.
Build the data pipeline using MapReduce, data storage on HDFS, and deploy
both on Hadoop using IaaS
A published solution is not available for this question yet.
Consider an application that can scale from handling 1000 users to handling 100 million users by
simply making copies of itself, is able to seamlessly react to machines going offline by again
making new copies of itself in new machines, and otherwise leaves behind no trace of its
workings. Which of the following statements is true?
6406532312155.
This application has adopted cloud-native design
6406532312156.
This application cannot be called as cloud-native since it is not observable at all
times
6406532312157.
This application cannot be called as cloud-native since it is not manageable
easily
6406532312158.
This application cannot be called as cloud-native since it is not resilient
A published solution is not available for this question yet.
Consider a file “data.bin” which is formatted as follows: every data record is in the form of pairs of
values of the form “column name,column value”, with each pair in its own line. You are asked to
write a data processing script using Python that scales with big data. Which of the following
represents your approach?
6406532312183.
Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to read
the data as is.
6406532312184.
Rename the file data.bin to data.csv to make it compliant with RFC 4180 and
then use PySpark’s read_csv() to read the data
6406532312185.
The problem cannot be solved since the file cannot be converted to a valid
format for reading consistently without additional information
6406532312186.
Write PySpark code to read all lines in data.bin, use string split on “,” as
delimiter, and then collect all column names and corresponding values into a RDD for further
processing
A published solution is not available for this question yet.
Consider the program outline as below running on a Spark cluster of 1 driver and 4 worker nodes
with 2 executors per worker node:
Consider the below mutually-exclusive characterisations about the program:
i. Program will not run since RDD transformation operation map() runs on Executors that is
referencing a variable train_master which is declared in the Spark driver
ii. Program will run successfully and produce the count of bookings that match for trains present
in the train_master
For either of the 2 characterizations, consider the following (one or more) actions you could carry
out in order to improve on the characterizations:
1. Program will need to be changed to bring the train_master into the Closure for executors to pick
it up
2. Program will perform more poorly as the number of executors per worker node increases.
Broadcast of train_master will need to be used to make it perform better.
3. Program will perform more poorly as the number of worker nodes increases (keeping number
of executors per worker node constant). Broadcast of train_master will need to be used to make it
perform better.
Which of the following combinations of statements are correct?
6406532312187.
i & 1
6406532312188.
i & 1 followed by 2
6406532312189.
i & 1 followed by 2 & 3
6406532312190.
ii & 2
6406532312191.
ii & 2 & 3
6406532312192.
ii & 3
A published solution is not available for this question yet.
Would IRCTC's railway ticket booking application be suitable for a serverless implementation?
6406532312159.
No, because each use of the application by a prospective passenger for ticket
search, date choosing, ticket seat blocking and ticket purchase is together a high complexity
operation
6406532312160.
No, because each use of the application by a prospective passenger for ticket
search, date choosing, ticket seat blocking and ticket purchase will take a long time
6406532312161.
No, because a failed step in the use of the application can result in duplicate
payments
6406532312162.
No, because the number of ticket searches cannot be scaled by simply running
the application in multiple machines
6406532312163.
No, because multiple prospective passengers have to use the application on a
single machine (i. e. No embarrassingly parallel computation)
A published solution is not available for this question yet.
You are provided with a Spark program that picks out a list of suspicious transactions based on
the amount of the transaction being higher than a threshold and the geographic location of the
transaction matching a known set of suspected locations provided in a small file. The Spark
program is written to pull all the transactions from the Workers to the Driver and then apply the
logic of threshold filtering and geographic location matching. Which of the following changes will
you make to the program to improve performance?
6406532312172.
Use broadcast variables
6406532312173.
Hardcode threshold value as a filter condition in the Driver program
6406532312174.
Filter for transactions in the Workers before bringing to the Driver
6406532312175.
Hardcode threshold value as a filter condition in the Workers itself
A published solution is not available for this question yet.
Which of the following types of data sources can you read successfully without missing data using
a program that extracts once every day? If no data is missed, then all data in source will match
data extracted successfully.
6406532312205.
A MySQL table where all Creates are new rows and all Updates are in-place,
and these can happen anytime during the day.
6406532312206.
A Linux machine’s system.log file into which all processes append their events
6406532312207.
Share transactions stored as facts & share prices stored as SCD Type II
dimension tables in a PostgreSQL database
6406532312208.
A weather API that provides the temperature & rainfall readings for all its
weather stations in India for the specific time instant when being queried
6406532312209.
Google drive account that stores all historical census data for India
A published solution is not available for this question yet.
YARN provides a centralized service for resource management across a cluster
of machines
6406532312226.
YARN manages distributed computing frameworks whereas Spark is an
example of a distributed computing framework
6406532312227.
Spark can run on YARN, the same way Hadoop Map Reduce can run on YARN
6406532312228.
YARN is a key feature of Spark
**Design Thinking**
**Section Id :** 64065348608
**Section Number :** 15
**Section type :** Online
**Mandatory or Optional :** Mandatory
**Number of Questions :** 27
**Number of Questions to be attempted :** 27
**Section Marks :** 35
**Display Number Panel :** Yes
**Group All Questions :** No
A published solution is not available for this question yet.
