Questions and published explanations below are available without starting a test. Some questions may not have a published solution yet.
18 / 20 published solutions
Question 2 MCQ · 1.0 marks
What is a key advantage of using n-step TD prediction over one-step TD prediction?
6406536642561.
n-step TD prediction requires less memory and computational resources.
6406536642562.
n-step TD prediction can handle delayed rewards and credit assignment over
multiple time steps
6406536642563.
n-step TD prediction converges faster to the optimal policy.
6406536642564.
n-step TD prediction guarantees convergence to the optimal value function for
any choice of the learning rate.
Published solution
**1. Compare the targets:** One-step TD uses one reward and then bootstraps. An n-step target uses \(R_{t+1}+\gamma R_{t+2}+\cdots+\gamma^{n-1}R_{t+n}+\gamma^nV(S_{t+n})\), ending earlier if the episode terminates.
**2. Explain delayed credit:** Several observed rewards are incorporated before bootstrapping, letting information about a delayed outcome reach earlier states in fewer update stages.
**3. State the tradeoff:** More steps can increase storage and target variance while reducing dependence on immediate bootstrap estimates. They do not guarantee faster policy convergence or convergence for arbitrary learning rates.
**Answer: B — Selected: n-step TD prediction can handle delayed rewards and credit assignment over multiple time steps.**
A: **Incorrect:** Multi-step targets generally need more stored transition information.
B: **Correct:** Observed rewards over several steps improve propagation of delayed reward information.
C: **Incorrect:** Prediction is not itself an optimal-policy convergence guarantee.
D: **Incorrect:** Learning-rate and sampling conditions still matter.
In which of the following scenarios is Expected SARSA a good fit?
6406536642569.
When function approximation is necessary due to large state spaces, requiring
deep networks for learning representations
6406536642570.
When learning needs to prioritize separating state value and advantage
functions for better decision-making
6406536642571.
When the environment has a high degree of stochasticity, making
bootstrapping unstable in value-based methods
6406536642572.
When experience replay is essential for stable learning and sample efficiency
Published solution
**1. Write Expected SARSA’s target:** \[Y=R_{t+1}+\gamma\sum_a\pi(a\mid S_{t+1})Q(S_{t+1},a).\]
It averages over the next policy action instead of sampling a single next action as ordinary SARSA does.
**2. Explain the variance benefit:** This removes the additional variability caused by selecting a random next action, which is useful when stochastic action choice makes targets noisy.
**3. Interpret the closest option:** C describes the intended stability motivation. The method does not eliminate random rewards or state transitions, and it is not defined by deep networks, dueling value/advantage streams or replay buffers.
**Answer: C — Selected: When the environment has a high degree of stochasticity, making bootstrapping unstable in value-based methods.**
A: **Incorrect:** Deep representation learning does not specifically identify Expected SARSA.
B: **Incorrect:** Separating value and advantage describes a dueling architecture.
C: **Correct:** Averaging over next actions can reduce target variance; environmental transition/reward noise can still remain.
D: **Incorrect:** Replay is an independent data-management technique.
Consider the following target equations:
**Equation 1**
**Equation 2**
In both equations,
refers to the shared parameters between the state-value network and
advantage network.
and
refer to the non-shared parameters of the state-value and advantage
network, respectively.
In both equations,
is calculated using the following network.
Note:
are target network parameters for online network parameters
Which among the following are correct statements?
6406536642565.
Target in Equation 1 is characteristic of Dueling DQN algorithm, and the target
in Equation 2 is characteristic of Dueling Double DQN algorithm
6406536642566.
Target in Equation 1 is characteristic of the DQN algorithm, and the target in
Equation 2 is characteristic of the Double DQN algorithm
6406536642567.
Target in Equation 1 is characteristic of the DQN algorithm, and the target in
Equation 2 is characteristic of the Dueling Double DQN algorithm
6406536642568.
Target in Equation 1 is characteristic of the Double Q-learning algorithm, and
the target in Equation 2 is characteristic of Double DQN
Published solution
**1. Identify the architecture:** The diagram has a shared representation followed by separate state-value and advantage streams, combined into Q-values. This is a dueling network.
**2. Read Equation 1:** The target network both selects the maximum and supplies its value, giving the usual DQN target applied to a dueling architecture.
**3. Read Equation 2:** The online network selects the argmax action, while the target network evaluates that chosen action. This is Double DQN action-selection/evaluation separation, again with dueling streams.
**4. Conclude:** The specific names are Dueling DQN and Dueling Double DQN. Dueling describes network structure; Double describes how the bootstrap action/value are computed.
**Answer: A — Selected: Target in Equation 1 is characteristic of Dueling DQN algorithm, and the target in Equation 2 is characteristic of Dueling Double DQN algorithm.**
A: **Correct:** This includes both the shown dueling architecture and the two target rules.
B: **Incorrect:** It recognizes the target-rule distinction but omits the explicitly shown dueling structure; A is the most specific answer.
C: **Incorrect:** Equation 1 also uses the dueling architecture.
D: **Incorrect:** Equation 1 has no separate online-selection/target-evaluation step.
Which of the following methods are a form of Generalized Policy Iteration?
6406536642573.
Policy Improvement
6406536642574.
Value Iteration
6406536642575.
Q-Learning and SARSA
6406536642576.
Monte Carlo control methods
Published solution
**1. Define generalized policy iteration:** Evaluation moves value estimates toward the current policy’s values, while improvement moves the policy toward actions that look better under those values. These processes can be interleaved at many granularities.
**2. Recognize complete interacting methods:** Value iteration combines short evaluation backups with improvement. Q-learning and SARSA update action values while the behavior/target policy is improved. Monte Carlo control estimates returns and improves the policy over episodes.
**3. Distinguish one component:** Policy improvement alone is one side of GPI; without an evaluation process it is not the complete interacting scheme asked here.
**Answer: B, C, D — Selected: Value Iteration; Q-Learning and SARSA; Monte Carlo control methods.**
A: **Incorrect:** Improvement alone supplies only one part of the evaluation–improvement interaction.
B: **Correct:** Optimality backups interleave value estimation and greedy improvement.
C: **Correct:** These control methods update action values and improve action selection.
D: **Correct:** Monte Carlo control alternates/interleaves return estimation and policy improvement.
In value-based deep reinforcement learning, the max operator in Q-learning can lead to
systematic overestimation of Q-values when neural networks are used for function approximation
Which option best explains why vanilla Deep Q-Netwok (DQN) tends to produce overoptimistic Q-
value estimates?
6406536642577.
The same neural approximator
is used to select the best action
and to evaluate that action
, causing the max operator to
preferentially propagate positive approximation errors across correlated action-value predictions
6406536642578.
Delayed target network polyak-averaging
introduces
temporal mismatch in Bellman targets
, causing overestimation
6406536642579.
DQN assumes a learned model of the environment
, and errors in
this model cause biased targets.
6406536642580.
DQN uses Monte Carlo returns
instead of bootstrapped
temporal-difference targets
Published solution
**1. Model approximation errors:** Suppose estimated action values are true values plus noise. Taking a maximum tends to select actions with positive estimation errors, so \(E[\max_a\widehat Q_a]\) can exceed \(\max_aE[\widehat Q_a]\).
**2. Explain vanilla DQN:** Within its target maximum, the same set of target-network estimates both identifies the maximizing action and evaluates its value. A slowly updated target network alone does not remove this selection bias.
**3. Separate alternative causes:** Double DQN reduces this coupling by selecting with the online network and evaluating with the target network. Vanilla DQN is model-free and uses bootstrapped TD targets, not a learned transition model or full Monte Carlo returns.
**Answer: A.**
A: **Correct:** Maximization selects favorable approximation errors when selection and evaluation use the same estimates.
B: **Incorrect:** Target lag can affect learning but is not the stated fundamental max-selection bias.
C: **Incorrect:** DQN does not require a learned environment model.
D: **Incorrect:** Its standard target is bootstrapped, not a full Monte Carlo return.
**1. Start at absorbing state 3:** It receives zero reward forever, so its expected return is \(v_3=0\), even with gamma=1.
**2. Evaluate state 2:** Equiprobable choice gives \(v_2=\tfrac12(-1+v_2)+\tfrac12(-1+v_3)\). With v3=0 this becomes \(v_2=-1+v_2/2\), giving \(v_2=-2\).
**3. Evaluate state 1:** \(v_1=\tfrac12(-2+v_1)+\tfrac12(-1+v_2)\). Thus \(v_1=\tfrac12v_1-2.5\), giving \(v_1=-5\).
**4. Sum:** \(v_1+v_2+v_3=-5-2+0=-7\). Self-loops have finite expected visits before progressing under the random policy.
**Answer: -7.**
Consider a 2D navigation environment (as shown in the figure below) where an autonomous
cleaning robot must locate and collect trash items (T1 and T2) scattered in the environment and
then deliver them to a designated disposal bin (G). The environment is shaped by curved, contour-
like walls, forming nested spaces with only one narrow passage connecting them, which requires
deliberate path planning.
In this environment, the robot takes discrete actions from its action space
,
{ NORTH, SOUTH, EAST, WEST }
Each action attempts to move the robot by 0.1 cm in the corresponding direction. If the action
would cause a collision with a wall boundary, the robot remains in the same position and receives
a penalty.
The grey regions represent potholes. If the robot ends up in one, it cannot escape on its own and
requires human help, leading to a substantial penalty and relocation to the nearest safe position.
When the agent reaches a trash item, it automatically collects it and receives a reward. Likewise,
upon reaching a disposal bin, it automatically unloads the collected trash. Therefore, explicit
pickup and drop actions are not required.
The episode terminates either when the agent reaches the Goal (G) while carrying at least one
piece of trash, or when it has fallen into potholes a total of 10 times.
The position of the Robot (
) can be represented as a tuple
, where
and
are the distances
of the robot from the bottom-left corner of the environment.
For state representation, we would consider state discretisation by performing tile as well as
coarse coding, which converts
into
{
}
.
Observe that there are 3 different grid-tilings here:
Based on the above data, answer the given subquestions.
Using the coded state representation (
) as input to a DQN with two hidden layers and a fully
connected output layer, what should be the dimensionality of the network’s input layer?
Published solution
**1. Count the binary feature positions:** The three tilings have 3-by-2, 6-by-4 and 11-by-7 cells. Their feature lengths are 6,24 and 77.
**2. Concatenate the tiling indicators:** \[d=6+24+77=107.\] A position typically activates one cell per tiling, but the input vector includes every cell’s feature, not only the active entries.
**3. Match network input:** The DQN therefore has 107 input coordinates under the stated positional coding. A fully Markov task representation would additionally need relevant carried-trash/history information; those features are not included in the supplied coding question.
**Answer: 107.**
Consider a 2D navigation environment (as shown in the figure below) where an autonomous
cleaning robot must locate and collect trash items (T1 and T2) scattered in the environment and
then deliver them to a designated disposal bin (G). The environment is shaped by curved, contour-
like walls, forming nested spaces with only one narrow passage connecting them, which requires
deliberate path planning.
In this environment, the robot takes discrete actions from its action space
,
{ NORTH, SOUTH, EAST, WEST }
Each action attempts to move the robot by 0.1 cm in the corresponding direction. If the action
would cause a collision with a wall boundary, the robot remains in the same position and receives
a penalty.
The grey regions represent potholes. If the robot ends up in one, it cannot escape on its own and
requires human help, leading to a substantial penalty and relocation to the nearest safe position.
When the agent reaches a trash item, it automatically collects it and receives a reward. Likewise,
upon reaching a disposal bin, it automatically unloads the collected trash. Therefore, explicit
pickup and drop actions are not required.
The episode terminates either when the agent reaches the Goal (G) while carrying at least one
piece of trash, or when it has fallen into potholes a total of 10 times.
The position of the Robot (
) can be represented as a tuple
, where
and
are the distances
of the robot from the bottom-left corner of the environment.
For state representation, we would consider state discretisation by performing tile as well as
coarse coding, which converts
into
{
}
.
Observe that there are 3 different grid-tilings here:
Based on the above data, answer the given subquestions.
Using the coded state representation (
) as input to a DQN with two hidden layers and a fully
connected output layer, what should be the size of the output layer?
Published solution
**1. Enumerate actions:** The environment has NORTH, SOUTH, EAST and WEST. Pickup and drop are automatic, so they do not add actions.
**2. Match a standard DQN output:** The network returns one scalar action value \(Q(s,a)\) for each discrete action.
**3. Count:** There are four available actions, so the output layer has size 4. The number of tiling features or hidden layers does not change the action count.
**Answer: 4.**
Consider a 2D navigation environment (as shown in the figure below) where an autonomous
cleaning robot must locate and collect trash items (T1 and T2) scattered in the environment and
then deliver them to a designated disposal bin (G). The environment is shaped by curved, contour-
like walls, forming nested spaces with only one narrow passage connecting them, which requires
deliberate path planning.
In this environment, the robot takes discrete actions from its action space
,
{ NORTH, SOUTH, EAST, WEST }
Each action attempts to move the robot by 0.1 cm in the corresponding direction. If the action
would cause a collision with a wall boundary, the robot remains in the same position and receives
a penalty.
The grey regions represent potholes. If the robot ends up in one, it cannot escape on its own and
requires human help, leading to a substantial penalty and relocation to the nearest safe position.
When the agent reaches a trash item, it automatically collects it and receives a reward. Likewise,
upon reaching a disposal bin, it automatically unloads the collected trash. Therefore, explicit
pickup and drop actions are not required.
The episode terminates either when the agent reaches the Goal (G) while carrying at least one
piece of trash, or when it has fallen into potholes a total of 10 times.
The position of the Robot (
) can be represented as a tuple
, where
and
are the distances
of the robot from the bottom-left corner of the environment.
For state representation, we would consider state discretisation by performing tile as well as
coarse coding, which converts
into
{
}
.
Observe that there are 3 different grid-tilings here:
Based on the above data, answer the given subquestions.
You have decided to use the Dueling DQN architecture with the aggregation.
What is the primary purpose of subtracting the term
in this formulation?
6406536642584.
It encourages the network to always choose the action with the highest
advantage, thereby directly implementing a greedy policy inside the Q-network.
6406536642585.
It normalises the advantage values for a given state so that at least one action
has zero advantage, reducing identifiability issues between
and
.
6406536642586.
It rescales the advantages to lie in the range
, making training more
numerically stable.
6406536642587.
It removes the need for a separate value stream
, since the max operator
can recover the state value from the advantages alone.
Published solution
**1. Identify the decomposition ambiguity:** Without a constraint, adding a constant to V and subtracting it from all advantages leaves the same Q-values. Separate streams would not have a unique interpretation.
**2. Normalize advantages:** The supplied form is \(Q(s,a)=V(s)+A(s,a)-\max_b A(s,b)\). The centered advantages have maximum zero, with at least one maximizing action attaining zero.
**3. Explain the result:** Consequently \(\max_aQ(s,a)=V(s)\), reducing the decomposition ambiguity under this chosen convention. This aggregation does not itself execute a greedy policy or restrict all advantages to [0,1].
**Answer: B.**
A: **Incorrect:** The network outputs values; an external action-selection rule still decides what to execute.
B: **Correct:** Subtracting the state’s maximum fixes the maximum centered advantage to zero.
C: **Incorrect:** The centered advantages are nonpositive and need not have a bounded magnitude.
D: **Incorrect:** The separate value stream remains part of the aggregation.
In Deep Q-Networks (DQN), a separate target network is maintained whose weights are periodically
copied from the main (online) network rather than updated at every step.
Based on the above data, answer the given subquestions.
What is the primary reason for using a separate target network with periodically updated weights
in DQN?
6406536642588.
To accelerate convergence by increasing the learning rate.
6406536642589.
To stabilize learning by keeping target values fixed for several steps.
6406536642590.
To ensure the Q-values are always up to date.
6406536642591.
To share weights between the actor and the critic.
Published solution
**1. Identify the moving-target problem:** If every update immediately changes both a Q estimate and the Q-based bootstrap target, training can chase a rapidly changing target.
**2. Freeze a separate target network temporarily:** Its parameters remain fixed for several online updates, making targets less rapidly changing. Periodic copies then refresh it.
**3. Explain the purpose:** This can stabilize bootstrapped learning. It does not guarantee the values are always current, increase the learning rate by definition or turn DQN into an actor-critic algorithm.
**Answer: B — Selected: To stabilize learning by keeping target values fixed for several steps.**
A: **Incorrect:** Target networks do not require a larger learning rate.
B: **Correct:** Holding bootstrap parameters fixed temporarily reduces moving-target instability.
C: **Incorrect:** Target estimates are intentionally delayed rather than always current.
D: **Incorrect:** DQN has no required separate actor whose weights are shared with a critic.
In Deep Q-Networks (DQN), a separate target network is maintained whose weights are periodically
copied from the main (online) network rather than updated at every step.
Based on the above data, answer the given subquestions.
What are the effects of periodically updating the weights of the target network in DQN?
6406536642592.
Provides stable targets to the main network during training.
6406536642593.
Causes the learning targets to be non-stationary.
6406536642594.
Mitigates instabilities in Q-learning updates.
6406536642595.
Avoids the need for a replay buffer.
A published solution is not available for this question yet.
), episodic task that has two non-terminal states, A and B. The
rewards belongs to { 0,1,2,3 } set. The following are some episodes experienced by an agent
following a fixed policy. The terminal state is not explicitly mentioned for any of the episodes.
Based on the above data, answer the given subquestions.
What is the estimate of
returned by first-visit MC?
Published solution
**1. Extract returns from B:** Every episode visits B once immediately before termination. Its observed terminal rewards are 0,1,0,0,1,0,1,3.
**2. Use first-visit Monte Carlo:** Each B return is just that final reward, because no later nonterminal step remains. Their sum is 6 over eight episodes.
**3. Average:** \[\widehat V_{MC}(B)=6/8=0.75.\] The discount factor has no effect on an immediately terminating one-reward return.
**Answer: 0.75.**
), episodic task that has two non-terminal states, A and B. The
rewards belongs to { 0,1,2,3 } set. The following are some episodes experienced by an agent
following a fixed policy. The terminal state is not explicitly mentioned for any of the episodes.
Based on the above data, answer the given subquestions.
What is the estimate of
returned by first-visit MC?
Published solution
**1. Identify episodes containing A:** The first four episodes start at A, with A-to-B rewards 2,0,2,2 and subsequent B rewards 0,1,0,0. Gamma is 0.8.
**2. Compute first-visit returns:** They are \(2+0.8(0)=2\), \(0+0.8(1)=0.8\), 2 and 2.
**3. Average the four:** \[\widehat V_{MC}(A)=\frac{2+0.8+2+2}{4}=1.7.\] Episodes starting at B do not contain A and do not contribute A returns.
**Answer: 1.7.**
), episodic task that has two non-terminal states, A and B. The
rewards belongs to { 0,1,2,3 } set. The following are some episodes experienced by an agent
following a fixed policy. The terminal state is not explicitly mentioned for any of the episodes.
Based on the above data, answer the given subquestions.
What is the estimate of
returned by batch
?
Published solution
**1. Identify all B transitions:** Each of the eight observed B visits goes directly to termination with rewards 0,1,0,0,1,0,1,3.
**2. Form batch TD’s fixed-point equation:** Terminal value is zero, so the mean TD target from B is the mean observed reward: \(\bar r_B+0.8(0)\).
**3. Solve:** \[V(B)=\frac{0+1+0+0+1+0+1+3}{8}=0.75.\] Here batch TD and Monte Carlo agree at B because B has only a one-step terminal continuation in the data.
**Answer: 0.75.**
), episodic task that has two non-terminal states, A and B. The
rewards belongs to { 0,1,2,3 } set. The following are some episodes experienced by an agent
following a fixed policy. The terminal state is not explicitly mentioned for any of the episodes.
Based on the above data, answer the given subquestions.
What is the estimate of
returned by batch
?
Published solution
**1. Estimate A’s transition/reward model from the batch:** Every recorded A transition leads to B. Its mean immediate reward is \((2+0+2+2)/4=1.5\).
**2. Evaluate B from all its observations:** Its eight terminal rewards total 6, so batch TD gives \(V(B)=0.75\).
**3. Apply the empirical Bellman equation:** \[V(A)=1.5+0.8V(B)=1.5+0.8(0.75)=2.1.\]
Unlike first-visit MC, batch TD reuses B’s estimate from all eight episodes, not only the four continuations following A.
**Answer: 2.1.**
Consider the following Deep Q-Network given below that approximates the Q-values for actions
and
for states
. The hidden layer nodes (colored green) employ the
activation
function. The weights of the connections are given directly above the edges connecting the nodes.
For instance, 3 is the weight for edge
and -3 is the weight for edge
.
We are interested in the Q-values calculated by the network for the following states:
Based on the above data, answer the given subquestions.
Recall the linear-approximator view for DQNs (
). Evaluate
i.e. the
state-representation for
. Enter
(the L-1 norm of the state-representation) as your
answer.
Published solution
**1. Read hidden-layer weights:** For input (x,y), the three preactivations are \(3x-3y\), \(3x+3y\) and \(x+2y\). The state representation phi is the hidden activation vector.
**2. Apply ReLU at s1=(1,2):** The preactivations are (-3,9,5). Since ReLU(z)=max(0,z), \(\phi(s_1)=(0,9,5)\).
**3. Compute L1 norm:** \[\|\phi(s_1)\|_1=|0|+|9|+|5|=14.\] Output weights are unnecessary when only the hidden representation is requested.
**Answer: 14.**
Consider the following Deep Q-Network given below that approximates the Q-values for actions
and
for states
. The hidden layer nodes (colored green) employ the
activation
function. The weights of the connections are given directly above the edges connecting the nodes.
For instance, 3 is the weight for edge
and -3 is the weight for edge
.
We are interested in the Q-values calculated by the network for the following states:
Based on the above data, answer the given subquestions.
What is
?
Published solution
**1. Compute the hidden representation:** For s1=(1,2), ReLU of (3-6,3+6,1+4) gives \(\phi(s_1)=(0,9,5)\).
**2. Read output O1’s weights:** From H1,H2,H3 they are (-3,0,-1). No output activation or biases are shown.
**3. Take the linear output:** \[Q(s_1,a_{O1})=(-3)(0)+0(9)+(-1)(5)=-5.\] A negative Q-value is valid even though the hidden activations are nonnegative.
**Answer: -5.**
Consider the following Deep Q-Network given below that approximates the Q-values for actions
and
for states
. The hidden layer nodes (colored green) employ the
activation
function. The weights of the connections are given directly above the edges connecting the nodes.
For instance, 3 is the weight for edge
and -3 is the weight for edge
.
We are interested in the Q-values calculated by the network for the following states:
Based on the above data, answer the given subquestions.
What is
?
Published solution
**1. Compute hidden activations:** At s1=(1,2), the three hidden values after ReLU are (0,9,5).
**2. Read O2’s weights:** They are (-4,5,-1) from H1,H2,H3, respectively.
**3. Calculate:** \[Q(s_1,a_{O2})=(-4)(0)+5(9)-1(5)=45-5=40.\] The final output is linear as drawn.
**Answer: 40.**
Consider the following Deep Q-Network given below that approximates the Q-values for actions
and
for states
. The hidden layer nodes (colored green) employ the
activation
function. The weights of the connections are given directly above the edges connecting the nodes.
For instance, 3 is the weight for edge
and -3 is the weight for edge
.
We are interested in the Q-values calculated by the network for the following states:
Based on the above data, answer the given subquestions.
Suppose you use
-greedy with
selection for balancing exploration with exploitation
during training. What is the probability that you'll pick action O2 given that the observed state is
?
Published solution
**1. Identify the greedy action:** At s1, the network gives Q(O1)=-5 and Q(O2)=40, so O2 is the unique greedy action.
**2. Apply epsilon-greedy:** With probability \(1-\epsilon\) choose the greedy action. With probability epsilon choose uniformly among both actions, which can also choose O2.
**3. Substitute epsilon=0.08:** \[P(O2\mid s_1)=1-0.08+0.08/2=0.92+0.04=0.96.\] Using only 0.92 would omit exploratory selections of the same action.
**Answer: 0.96.**
