Questions and published explanations below are available without starting a test. Some questions may not have a published solution yet.
17 / 18 published solutions
Question 2 MCQ · 2.0 marks
Which of the following is/are correct and valid reasons to consider sampling actions from a
softmax distribution instead of using an
greedy approach?
1.Under softmax exploration, the probability of selecting an action increases with its estimated
action value, which reduces unnecessary exploration of clearly inferior actions.
2.Unlike the
-greedy method, softmax exploration does not require careful, gradual decay of the
exploration parameter and still yields asymptotically correct behaviour even if the temperature is
reduced sharply.
3.It enables more fine-grained discrimination among actions whose estimated Q-values are close
to the maximum, allowing more nuanced preference for slightly better actions.
Which of the above statements is/are correct?
6406536376298.
1, 2, 3
6406536376299.
only 3
6406536376300.
1, 2
6406536376301.
1, 3
6406536376302.
3, 2
6406536376303.
Only 2
Published solution
**1. Write the softmax probabilities:** \(P(a)=\exp(Q(a)/\tau)/\sum_b\exp(Q(b)/\tau)\), where temperature tau is positive. Higher estimated values therefore receive higher probabilities.
**2. Compare exploration:** Unlike epsilon-greedy’s uniform exploratory choice, softmax distinguishes actions by their value estimates. It can allocate more exploratory mass to near-best actions and less to obviously poor ones, supporting Statements 1 and 3.
**3. Check annealing:** Reducing temperature too quickly can eliminate useful exploration before estimates are accurate. Asymptotic correctness is not guaranteed by a sharp temperature reduction, so Statement 2 is false.
**Answer: D — Selected: 1, 3.**
A: **Incorrect:** This includes the false no-careful-annealing claim in Statement 2.
B: **Incorrect:** Statement 1 is also correct.
C: **Incorrect:** Statement 2 is false and Statement 3 is omitted.
D: **Correct:** Statements 1 and 3 are correct; Statement 2 is false.
E: **Incorrect:** Statement 2 remains false.
F: **Incorrect:** Statement 2 alone is not correct.
and rewards are uniformly bounded, the infinite sum
is always finite.
3.For
, the discounted return
is bounded above in magnitude by
. Here, Rmax be
the maximum reward for any transition.
Which of the above statements is/are correct?
6406536376304.
1, 2, 3
6406536376305.
1, 3
6406536376306.
2, 3
6406536376307.
only 1
6406536376308.
only 3
Published solution
**1. Bound discounted rewards:** If \(|R_t|\le R_{max}\) and \(0\le\gamma<1\), the magnitude of the kth discounted term is at most \(\gamma^kR_{max}\). This decays geometrically when gamma is strictly below one.
**2. Sum the bound:** \[|G_t|\le\sum_{k=0}^{\infty}\gamma^kR_{max}=\frac{R_{max}}{1-\gamma}.\] Thus Statements 1 and 3 hold under the given absolute reward bound.
**3. Test gamma equal to one:** Constant reward 1 is bounded, but its undiscounted infinite sum diverges. Statement 2 is false.
**Answer: B — Selected: 1, 3.**
A: **Incorrect:** Statement 2 is disproved by a continuing stream of unit rewards.
B: **Correct:** Statements 1 and 3 follow from a convergent geometric series.
C: **Incorrect:** Statement 2 is false.
D: **Incorrect:** Statement 3 is also true.
E: **Incorrect:** Statement 1 is also true.
Consider the following assertion and reason pair and select the correct option:
**Assertion:**In the UCB algorithm for multi-armed bandits, replacing the upper confidence bound
with a lower confidence bound (and greedily selecting actions based on that lower bound) would
still promote effective exploration or lead to optimal reward maximisation.
**Reason:**Even when lower confidence bounds are used instead of upper bounds, if the algorithm
greedily selects arms based on these lower bounds, it would still encourage exploration and
ultimately achieve optimal reward maximisation.
6406536376309.
Both Assertion and Reason are true, and Reason is the correct explanation of
Assertion.
6406536376310.
Both Assertion and Reason are true, but Reason is NOT the correct
explanation of Assertion.
6406536376311.
Assertion is true, Reason is false
6406536376312.
Assertion is false, Reason is false
Published solution
**1. Recall optimism in UCB:** An upper confidence index adds an uncertainty bonus to the estimated mean. An under-sampled arm can then become attractive because its true value might be high.
**2. Reverse the sign:** Greedily maximizing a lower confidence bound penalizes uncertain arms. An arm with few observations can be avoided precisely because it is uncertain, preventing discovery of a better reward.
**3. Judge both claims:** Both the assertion and its repeated reason are false as general exploration/optimality claims. Lower confidence bounds can serve other conservative objectives, but do not preserve UCB’s optimistic exploration rule.
**Answer: D — Selected: Assertion is false, Reason is false.**
A: **Incorrect:** Neither claim is true as stated.
B: **Incorrect:** Both claims are false, not merely an incorrect explanation pair.
C: **Incorrect:** The assertion itself is false.
D: **Correct:** Both claims wrongly assert that pessimistic greedy selection preserves effective UCB exploration.
In the context of MDPs, what does it mean for a policy
to be greedy with respect to an action-
value function
?
6406536376313.
For each state
,
selects actions that minimise
6406536376314.
For each state
,
selects actions that maximise
6406536376315.
For each state
,
ignores
and follows a fixed, pre-defined schedule.
6406536376316.
For each state
,
selects actions uniformly at random regardless of
Published solution
**1. Define action value:** q(s,a) estimates expected return after choosing a in s and following the specified continuation policy.
**2. Define a greedy policy:** It puts probability only on actions attaining \(\max_a q(s,a)\). A tie may be broken deterministically or by randomizing among tied maximizers.
**3. Contrast exploration:** Choosing uniformly from all actions or following an unrelated schedule is not greediness with respect to q.
**Answer: B.**
A: **Incorrect:** Minimizing q chooses the least valued action.
B: **Correct:** Greedy actions maximize the supplied action-value function.
C: **Incorrect:** An unrelated fixed schedule does not use q to choose maximizers.
D: **Incorrect:** Uniform choice is greedy only in the special case where every action ties.
In policy iteration for finite MDPs, how are the Bellman equations used during the policy
evaluation step?
6406536376317.
They are ignored; policy iteration does not rely on Bellman equations.
6406536376318.
They are used to compute
exactly (or approximately) for the current policy
6406536376319.
They are used to directly compute the optimal policy without evaluating
intermediate policies.
6406536376320.
They are used to compute immediate rewards without considering transitions.
Published solution
**1. Hold the current policy fixed:** Policy evaluation asks for \(v_\pi\), the return from following the current policy, rather than immediately searching over all policies.
**2. Use Bellman expectation equations:** \[v_\pi(s)=\sum_a\pi(a\mid s)\sum_{s',r}p(s',r\mid s,a)[r+\gamma v_\pi(s')].\] These can be solved as a linear system or approximated by repeated updates.
**3. Improve afterward:** Policy improvement uses the evaluated values to choose a better policy. Both evaluation and improvement are needed in policy iteration.
**Answer: B.**
A: **Incorrect:** Bellman equations are central to policy evaluation.
B: **Correct:** They determine the value of the fixed current policy.
C: **Incorrect:** Direct optimal-action maximization is not fixed-policy evaluation.
D: **Incorrect:** The equations include transitions and future value, not just immediate rewards.
**1. State the standard UCB index:** \[A_t\in\arg\max_a\left[Q_t(a)+c\sqrt{\frac{\ln t}{N_t(a)}}\right].\]
The extra term represents an upper-confidence exploration bonus; untried arms are normally tried first or assigned an infinite index.
**2. Explain the directions:** For fixed time, more samples shrink uncertainty through \(1/\sqrt{N}\). For an ignored arm, increasing time enlarges the bonus logarithmically and encourages another trial.
**3. Match the conventional rule:** A has the standard confidence-radius scaling. B lacks the square root, while C and D can reward frequent sampling or shrink in the wrong way. This question is interpreted as identifying the standard UCB confidence rule, rather than proving every other heuristic must fail on every bandit.
**Answer: A.**
A: **Correct:** This is the usual upper-confidence estimate with the correct square-root sampling scale.
B: **Incorrect:** Its ln(t)/N term is not the standard Hoeffding confidence radius.
C: **Incorrect:** The bonus increases with the arm’s pull count, reversing uncertainty behavior.
D: **Incorrect:** For a fixed under-sampled arm its denominator makes the bonus shrink with time.
E: **Incorrect:** The standard UCB rule is present in A.
**1. Separate reward and optimal continuation:** Bellman optimality chooses the best first action and then uses optimal future value:
\[v_*(s)=\max_aE[R_{t+1}+\gamma v_*(S_{t+1})\mid S_t=s,A_t=a].\]
**2. Express the same return another way:** With optimal continuation policy \(\pi_*\), \(q_*(s,a)=E_{\pi_*}[G_t\mid S_t=s,A_t=a]\), so \(v_*(s)=\max_aq_*(s,a)\). The PDF’s C explicitly uses optimal continuation.
**3. Reject incomplete identities:** A policy-weighted sum of v*(s) is merely a tautology after summing probabilities; it is not a Bellman backup. Immediate reward alone omits all future return.
**Answer: A, C.**
A: **Correct:** This is the one-step Bellman optimality backup.
B: **Incorrect:** The displayed sum repeats the same state value and omits transitions/rewards; it is not the Bellman recurrence.
C: **Correct:** Maximizing expected full return under optimal continuation yields v*.
D: **Incorrect:** Immediate expected reward alone does not generally equal the infinite-horizon value.
The sequence of rewards for a continuing task with
is given below:
Find the return
. Your answer should have exactly two places after the decimal point.
Published solution
**1. State the source’s indexing convention:** Earlier in this paper, return is written \(G_t=\sum_{k\ge0}\gamma^kR_{t+k}\), so G3 begins with R3=3. This differs by one reward index from the common convention beginning with R(t+1).
**2. Expand the tail:** With gamma=0.9 and \(R_t=0.9^{t-4}\) for t≥4,
\[G_3=3+0.9(1)+0.9^2(0.9)+0.9^3(0.9^2)+\cdots.\]
The tail starts at 0.9 and has ratio \(0.9^2=0.81\).
**3. Sum:** \(G_3=3+0.9/(1-0.81)=7.736842\), rounded to 7.74.
**Answer: 7.74.**
You have a 3-armed bandit where each arm, when active, yields a Bernoulli reward with success
probabilities
,
, and
. However, arm 1 is active in any given round with
probability
, and otherwise it yields a reward of 0 regardless of its Bernoulli outcome. Arms 2
and 3 are always active. What is the expected reward from pulling arm 1 in a single round?
Published solution
**1. Condition on activation:** Arm 1 is active with probability 0.8. Conditional on activation, its Bernoulli reward has mean 0.7. Otherwise reward is zero.
**2. Apply total expectation:** \[E[R\mid\text{choose arm 1}]=0.8(0.7)+0.2(0)=0.56.\]
**3. Interpret:** The effective reward probability is lower than the arm’s active-state success probability. Other arms’ probabilities do not enter the reward from this single chosen arm.
**Answer: 0.56.**
Consider a Markov Decision Process (MDP) with three states
,
, and
, where
is a terminal
state and
. Each non-terminal state has a single available action, and the transitions are
deterministic with associated rewards: the agent moves from
to
and receives a reward of 1,
and from
to
and receives a reward of 2.
Let the discount factor be
.
Based on the above data, answer the given subquestions.
Assume the initial value estimates are
,
, and
. After performing
one synchronous policy evaluation update (under the fixed policy that follows the chain),
determine
and
. Compute the ratio
Published solution
**1. Use synchronous evaluation:** Every new value uses the previous sweep’s values. Initially all three state values are zero, with state 3 terminal.
**2. Update the chain:** \(V_1(s_1)=1+0.5V_0(s_2)=1\), and \(V_1(s_2)=2+0.5V_0(s_3)=2\).
**3. Form the ratio:** \(V_1(s_1)/V_1(s_2)=1/2=0.5\). Using the newly updated value of state 2 inside the state-1 update would be an asynchronous sweep, which is not requested.
**Answer: 0.5.**
Consider a Markov Decision Process (MDP) with three states
,
, and
, where
is a terminal
state and
. Each non-terminal state has a single available action, and the transitions are
deterministic with associated rewards: the agent moves from
to
and receives a reward of 1,
and from
to
and receives a reward of 2.
Let the discount factor be
.
Based on the above data, answer the given subquestions.
Starting from
,
, and
. perform two successive synchronous
policy evaluation sweeps (under the fixed policy that follows the chain). What is
after these
two sweeps?
Published solution
**1. Perform the first synchronous sweep:** Starting from zero, the values become \(V_1(s_1)=1\), \(V_1(s_2)=2\), \(V_1(s_3)=0\).
**2. Perform the second:** \(V_2(s_1)=1+0.5(2)=2\), while \(V_2(s_2)=2+0.5(0)=2\). The terminal state remains zero.
**3. Read the requested state:** After two sweeps state 2 still has value 2. The printed question’s superscript refers to V1, but this state already equals 2 after either of the two sweeps, so the numeric result is unaffected.
**Answer: 2.**
, there is a single action that yields an
immediate reward
and transitions back to
.
Assume initial value estimates are
,
Based on the above data, answer the given subquestions.
If you perform a single value iteration Bellman optimality update for
, what is the new value
?
Published solution
**1. Form both action backups at state 1:** Action alpha gives immediate reward 1.5 and leads to state 2; action delta gives reward zero and stays in state 1. Gamma is 0.9.
**2. Use the initial zero values:** The two returns are \(1.5+0.9(0)=1.5\) and \(0+0.9(0)=0\).
**3. Maximize:** \(V_1(s_1)=\max(1.5,0)=1.5\). A value-iteration backup chooses the largest action return.
**Answer: 1.5.**
, there is a single action that yields an
immediate reward
and transitions back to
.
Assume initial value estimates are
,
Based on the above data, answer the given subquestions.
Suppose after some value iterations your current estimates are
,
. If you
perform one value iteration update for
, what is the new value
?
Published solution
**1. Read state 2’s only transition:** It pays 2 immediately and returns to state 1. The current estimate is V(s1)=5 and gamma=0.9.
**2. Apply the backup:** \[V'(s_2)=2+0.9V(s_1)=2+0.9(5)=6.5.\]
**3. Explain the unused value:** The current V(s2)=4 is not used as the continuation because this action goes to state 1, not back to state 2.
**Answer: 6.5.**
**1. Separate correct and faulty activation:** With probability \(1-\epsilon\), the chosen optimal arm executes and expected reward is \(p_*\). With probability epsilon, a uniformly random arm executes and its average reward is \(\bar p=(1/N)\sum_i p_i\).
**2. Average these branches:** \[E[R]=(1-\epsilon)p_*+\frac\epsilon N\sum_{i=1}^N p_i.\]
**3. Check:** With no fault it reduces to p*. With faults every time it reduces to the all-arm average. The uniform fault includes a chance of selecting the optimal arm itself.
**Answer: D.**
A: **Incorrect:** This reverses the correct-activation and noisy-activation weights.
B: **Incorrect:** p* ignores the chance of executing a worse arm.
C: **Incorrect:** The simple average applies only when every activation is randomized.
D: **Correct:** This weights the intended best arm and the uniformly randomized execution correctly.
Consider an N-armed bandit problem in which each arm
, for
, produces a
reward of 1 with probability
and
otherwise, where
denotes the arm’s base success
probability.
Assume the bandit machine suffers from a hardware malfunction: when the agent attempts to pull
arm
, the intended action is not always executed. Instead, with probability
a wiring fault causes
the machine to randomly activate one arm chosen uniformly from the set
,
regardless of the agent’s selection, and the reward is generated according to that arm’s reward
distribution. We refer to this malfunction as
-noise activation
Based on the above data, answer the given subquestions.
Consider a faulty
-armed bandit with
-noise activation and distinct base success probabilities
. Suppose the agent follows a greedy strategy that always chooses the arm with the
highest empirical mean reward. How does the presence of activation noise affect the asymptotic
fraction of time for which the optimal arm (with success probability
) is actually executed?
6406536376338.
Noise has no effect; the optimal arm is activated almost surely in the long run.
6406536376339.
Noise reduces the frequency with which the optimal arm is selected, but once
it is selected, it is always activated.
6406536376340.
Noise eventually causes the greedy policy to oscillate indefinitely between
arms, preventing convergence.
6406536376341.
Even when the greedy policy converges to always selecting the optimal arm,
activation noise prevents the optimal arm from being activated more than a fraction
of the
time.
A published solution is not available for this question yet.
Consider an N-armed bandit problem in which each arm
, for
, produces a
reward of 1 with probability
and
otherwise, where
denotes the arm’s base success
probability.
Assume the bandit machine suffers from a hardware malfunction: when the agent attempts to pull
arm
, the intended action is not always executed. Instead, with probability
a wiring fault causes
the machine to randomly activate one arm chosen uniformly from the set
,
regardless of the agent’s selection, and the reward is generated according to that arm’s reward
distribution. We refer to this malfunction as
-noise activation
Based on the above data, answer the given subquestions.
In a faulty
-armed bandit with
-noise activation, suppose you mistakenly model the
environment as a standard bandit without activation noise. You estimate each arm’s success
probability using sample averages of observed rewards. How does the
-noise affect your
estimates of the true base success probabilities
?
6406536376342.
Your estimates are biased toward the overall average success probability
across all arms.
6406536376343.
Your estimates are biased upward, overestimating all
6406536376344.
Your estimates remain unbiased for
because you still observe correct
rewards for the arm you believe you pulled.
6406536376345.
Your estimates are biased downward, underestimating all
Published solution
**1. Compute the observed mean when choosing arm i:** \(\mu_i^{obs}=(1-\epsilon)p_i+\epsilon\bar p\), where \(\bar p\) is the mean base success rate over all arms.
**2. Compare with the base rate:** \(\mu_i^{obs}-p_i=\epsilon(\bar p-p_i)\). Above-average arms are shifted downward; below-average arms are shifted upward.
**3. Interpret sample averages:** With enough observations they estimate the effective chosen-action reward mean, not the underlying intended arm’s base rate. Hence the estimates are shrunk toward the overall average. Pure greediness alone does not guarantee enough samples of every arm.
**Answer: A — Selected: Your estimates are biased toward the overall average success probability across all arms.**
A: **Correct:** The effective mean is a convex combination of p_i and the all-arm average.
B: **Incorrect:** Above-average arms are shifted down, so not all biases are upward.
C: **Incorrect:** The observed reward can originate from a different arm than the chosen one.
D: **Incorrect:** Below-average arms are shifted up, so not all biases are downward.
Consider a large-scale news recommendation platform that uses a contextual bandit approach to
decide which headline to show to a user visiting the homepage. The context includes user features
(e.g., location, device type, time of day, coarse interest profile) and article features (e.g., topic,
recency, source reputation). Actions correspond to recommending one of the
candidate
headlines, and the reward is
if the user clicks the recommended headline and
otherwise.
Let the context vector
at time
encode both user and article features for the current
recommendation opportunity. There are
discrete actions, each corresponding to selecting one
of the candidate headlines for display.
Based on the above data, answer the given subquestions.
Which of the following best explains why a contextual bandit approach is preferred over a
standard multi-armed bandit in this news recommendation scenario?
6406536376346.
Because user preferences are identical for all users, so context does not
influence click behaviour.
6406536376347.
Because contextual bandits can tailor recommendations to each user and
article pair using the context, thereby improving click-through rates across heterogeneous users.
6406536376348.
Because standard multi-armed bandits cannot perform any exploration at all.
6406536376349.
Because contextual bandits guarantee that the same headline is always
optimal for every user.
Published solution
**1. Identify heterogeneity:** Users differ in interests, location and device, while headlines differ in topic and recency. The same headline can have different click probabilities for different contexts.
**2. Use conditional action values:** A contextual bandit estimates reward for the current context/action pair and learns a policy mapping context to a headline. A standard context-free bandit averages reward per arm across these differences.
**3. Select the advantage:** Context supports personalization and better expected decisions when informative. It does not guarantee perfect predictions or eliminate the need for exploration.
**Answer: B — Selected: Because contextual bandits can tailor recommendations to each user and article pair using the context, thereby improving click-through rates across heterogeneous users.**
A: **Incorrect:** The scenario explicitly involves heterogeneous contexts and user preferences.
B: **Correct:** Contextual conditioning permits different recommendations for different users and articles.
C: **Incorrect:** Ordinary multi-armed bandits can explore.
D: **Incorrect:** The best headline may differ across users/contexts.
Consider a large-scale news recommendation platform that uses a contextual bandit approach to
decide which headline to show to a user visiting the homepage. The context includes user features
(e.g., location, device type, time of day, coarse interest profile) and article features (e.g., topic,
recency, source reputation). Actions correspond to recommending one of the
candidate
headlines, and the reward is
if the user clicks the recommended headline and
otherwise.
Let the context vector
at time
encode both user and article features for the current
recommendation opportunity. There are
discrete actions, each corresponding to selecting one
of the candidate headlines for display.
Based on the above data, answer the given subquestions.
After many rounds of user interaction, what is the main optimality objective of the contextual
bandit algorithm in this recommendation setting?
6406536376350.
Maximise the cumulative expected number of clicks by acting as closely as
possible to the best policy that maps contexts to headlines.
6406536376351.
Ensure that on every single round, it always picks the empirically highest-CTR
headline observed so far, regardless of context.
6406536376352.
Minimise the variance of rewards, even if that substantially reduces the total
number of clicks.
6406536376353.
Focus only on the worst-performing user segment and ignore the rest.
Published solution
**1. Match rewards to the task:** Reward is one for a click and zero otherwise, so cumulative expected reward equals the expected number of clicks.
**2. Choose the comparator:** The ideal reference policy maps each context to the best available headline for that context.
**3. State the objective:** Maximize cumulative expected clicks, equivalently minimize regret against the relevant best context-dependent policy under the model assumptions. Exploration can sacrifice an immediate click opportunity to improve later decisions.
**Answer: A — Selected: Maximise the cumulative expected number of clicks by acting as closely as possible to the best policy that maps contexts to headlines.**
A: **Correct:** This matches cumulative reward and a context-dependent comparator policy.
B: **Incorrect:** Ignoring context and always using the current empirical leader need not maximize long-term reward.
C: **Incorrect:** Variance alone is not the supplied reward objective.
D: **Incorrect:** The stated objective is not restricted to one chosen user segment.
