Reinforcement Learning · Quiz 2 · Sep 2025

Consider the following assertion reason pair: **Assertion**: In Monte Carlo (MC), the value function converges to the certainty equivalence estimate. **Reason**: Monte Carlo methods use complete trajectories to fit the value function as closely as possible to the sampled returns.

Which of the following is the correct way to represent policy for policy search methods? Assume

represents a real-valued parameter.

Recall the incremental update rule for REINFORCE:

Consider the following binary-bandit problem:

Which of the following expressions are equivalent to

?

In Q-Learning, the update rule for the action-value function is based on bootstrapping from the current estimate. Which of the following correctly represents this update?

Consider an episodic task where the

-step returns follow a decaying exponential pattern:

Assume the episode is sufficiently long so the forward-view infinite sum is valid. Compute the

-return for

. Enter your answer correct to **two decimal places**.

In Deep Q-Networks (DQN), a separate **target network** is maintained, whose weights are periodically copied from the main (online) network instead of being updated at every step. Based on the above data, answer the given subquestions.

What is the primary reason for using a separate target network with periodically updated weights in DQN?

In Deep Q-Networks (DQN), a separate **target network** is maintained, whose weights are periodically copied from the main (online) network instead of being updated at every step. Based on the above data, answer the given subquestions.

What are the effects of periodically updating the weights of the target network in DQN?

Based on the above data, answer the given subquestions.

Find the Q values for all three actions

for the state

. Express your answer as a vector q, where

Based on the above data, answer the given subquestions.

Compute the TD target for this transition using SARSA with

. Use a greedy policy derived from the Q-values at

to select the next action.

Based on the above data, answer the given subquestions.

Perform one Step of semi-gradient TD using this transition. What would be

if

?

Consider the following algorithm for learning action-value estimates in an episodic MDP:

Based on the above data, answer the given subquestions.

Which of the following issues in standard Q-learning does the given algorithm aim to reduce?

Consider the following algorithm for learning action-value estimates in an episodic MDP:

Based on the above data, answer the given subquestions.

In the given pseudocode, the algorithm maintains two value functions,

and

, and updates them alternately. Which of the following best describes the purpose and effect of this design choice?

Recall from the lectures, the forward-view

-**return** is defined as:

where

denotes the

-step return starting at time

.Consider using the TD(

) algorithm for an **episodic** task. Based on the above data, answer the given subquestions.

What is the effect of setting

in TD(

) on the weighting of

-step returns?

Recall from the lectures, the forward-view

-**return** is defined as:

where

denotes the

-step return starting at time

.Consider using the TD(

) algorithm for an **episodic** task. Based on the above data, answer the given subquestions.

As

in an episodic TD(

) task, the

-return

approaches which of the following?

Consider an episodic task with four non-terminal states: A, B, C and D. The following are some episodes experienced by an agent following a fixed policy. The terminal state is not explicitly mentioned for any of the episodes. A,1, C,1, D,0, C,0, A,1, B,1 A,0, C,0, D,1, C,1, B,1 D,1, C,0, A,0, C,1, B,0 C,0, A,1, C,1, B,0 C,1, B,0 D,0, B,1 B,1 B,0 (Enter your answer correct to **three decimal places**.) Based on the above data, answer the given subquestions.

Using every-visit Monte Carlo, estimate

and

. What is the value of

? (use

)