Questions and published explanations below are available without starting a test. Some questions may not have a published solution yet.
21 / 22 published solutions
Question 2 MCQ · 0.0 marks
**INSTRUCTIONS:**
This paper contains bonus questions marked with **[Bonus]** at the beginning of the question. These
questions carry 0 marks toward the regular total of 30 and do not contribute to the End Term
score. However, correct answers will earn additional bonus marks, which will be added to your
final score in accordance with the grading policy. Attempting these questions is optional, and
there is no penalty for incorrect or unattempted bonus questions.
6406536959552.
Instructions has been mentioned above.
6406536959553.
This Instructions is just for a reference & not for an evaluation.
Published solution
**1. Acknowledge the exam instructions:**
This 0-mark administrative question confirms that the candidate has read the bonus-question instructions above.
**Answer: A — Instructions has been mentioned above.**
A: **Correct:** Standard acknowledgment option for the exam instructions.
B: **Incorrect:** Standard instructions require selecting Option A.
**1. Recall the definition of an Option in Hierarchical RL:**
In the Options framework (Sutton, Precup & Singh, 1999), an option \( \omega \) is formally defined as a triple \( \langle \mathcal{I}_\omega,\ \pi_\omega,\ \beta_\omega \rangle \):
1. **Initiation set (\( \mathcal{I}_\omega \subseteq \mathcal{S} \)):** the set of states where the option can be invoked.
2. **Option policy (\( \pi_\omega \)):** the intra-option policy mapping states to actions.
3. **Termination function (\( \beta_\omega \)):** the probability of the option terminating in each state.
**Answer: B — Initiation set, option policy, termination function.**
A: **Incorrect:** State space, action space, and reward function define an MDP, not an option.
B: **Correct:** An option is defined by the triple: initiation set, option policy, and termination function.
C: **Incorrect:** Value function and discount factor are not components of an option tuple.
D: **Incorrect:** Option definitions do not consist of an environment transition/reward model.
In DDPG, target networks are updated via soft updates
with
. The
primary reason for this slow update is:
6406536959558.
To prevent the actor from updating faster than the critic
6406536959559.
To stabilise training by preventing the TD target from changing too rapidly
6406536959560.
To ensure the replay buffer contains on-policy data
6406536959561.
To match the learning rate of the actor and critic networks
Published solution
**1. Understand Polyak soft updates in DDPG:**
Updating target network weights slowly via \( \theta' \leftarrow \tau \theta + (1 - \tau)\theta' \) with \( \tau \ll 1 \) keeps the bootstrap TD target \( y = r + \gamma Q_{\phi'}(s', \mu_{\theta'}(s')) \) nearly stationary across updates, **stabilising critic training by preventing the TD target from changing too rapidly**.
**Answer: B — To stabilise training by preventing the TD target from changing too rapidly.**
A: **Incorrect:** Both actor and critic target networks use soft updates to stabilize TD bootstrapping, not to throttle the online actor relative to the critic.
B: **Correct:** Slowly tracking the online parameters prevents feedback oscillations in the bootstrap TD target.
C: **Incorrect:** The replay buffer stores off-policy transitions regardless of \( \tau \).
D: **Incorrect:** \( \tau \) is the target interpolation rate, separate from the optimizer learning rates.
DDPG uses an experience replay buffer. What is the direct consequence of using an experience
replay buffer
6406536959562.
The policy gradient estimate becomes unbiased
6406536959563.
The algorithm becomes on-policy
6406536959564.
Temporal correlations between consecutive samples are broken
6406536959565.
The critic no longer requires a target network
Published solution
**1. Understand the purpose of the Experience Replay Buffer in DDPG (and DQN):**
Consecutive transitions \( (s_t, a_t, r_t, s_{t+1}) \) collected along a trajectory are strongly correlated in time, which violates the i.i.d. assumption of stochastic gradient descent. Storing transitions in a replay buffer and sampling random mini-batches uniformly **breaks the temporal correlations between consecutive samples**.
**Answer: C — Temporal correlations between consecutive samples are broken.**
A: **Incorrect:** Function approximation in the critic still introduces bias into the deterministic policy gradient.
B: **Incorrect:** Using a replay buffer makes the algorithm *off-policy*, not on-policy.
C: **Correct:** Random uniform sampling from the replay buffer decorrelates sequential trajectory transitions.
D: **Incorrect:** Both experience replay and target networks are needed for stable training in DDPG.
Compared to the policy gradient, the DPG theorem requires integration over:
6406536959566.
Both state and action spaces
6406536959567.
State space only
6406536959568.
Action space only
6406536959569.
Neither — it requires only a single action sample per state
Published solution
**1. Compare Stochastic Policy Gradient (SPG) vs. Deterministic Policy Gradient (DPG):**
- **Stochastic Policy Gradient (SPG):** \( \nabla_\theta J(\theta) = \int_{\mathcal{S}} \rho^\pi(s) \int_{\mathcal{A}} \pi_\theta(a \mid s) \nabla_\theta \ln \pi_\theta(a \mid s) Q^\pi(s, a) \, da \, ds \) integrates over **both state and action spaces**.
- **Deterministic Policy Gradient (DPG):** \( \nabla_\theta J(\theta) = \int_{\mathcal{S}} \rho^\mu(s) \nabla_\theta \mu_\theta(s) \nabla_a Q^\mu(s, a)|_{a=\mu_\theta(s)} \, ds \) integrates over the **state space only**.
**Answer: B — State space only.**
A: **Incorrect:** Integrating over both state and action spaces is required by the stochastic policy gradient, not DPG.
B: **Correct:** Because the policy \( a = \mu_\theta(s) \) is deterministic, the expectation in DPG is taken over the state distribution \( \rho^\mu(s) \) only.
C: **Incorrect:** There is no integral over the action space in DPG.
D: **Incorrect:** DPG still takes an expectation (integral) over the state space.
**[Bonus]** The TRPO surrogate objective is defined as
where
is an importance-sampling ratio.
Why is it justified to optimise
instead of the true objective
6406536959570.
is exactly equal to
for all
6406536959571.
is a first-order approximation of
, accurate when
is
close to
--- precisely the regime enforced by the KL constraint
6406536959572.
is an unbiased estimator of
under any policy
6406536959573.
The surrogate eliminates the need for a value function baseline
Published solution
**1. Understand the TRPO surrogate objective \( L_{\theta_{\text{old}}}(\theta) \):**
The surrogate objective \( L_{\theta_{\text{old}}}(\theta) \) replaces the intractable new-policy state visitation distribution \( \rho^\pi(s) \) with the old policy's state distribution \( \rho^{\pi_{\text{old}}}(s) \). By Kakade & Langford's performance difference lemma and Schulman et al. (2015), \( L_{\theta_{\text{old}}}(\theta) \) matches \( J(\theta) \) to **first order** around \( \theta_{\text{old}} \), making it an accurate local approximation whenever \( \pi_\theta \) remains close to \( \pi_{\text{old}} \) — which is guaranteed by the trust-region KL constraint \( \bar{D}_{\text{KL}}(\pi_{\text{old}} \| \pi_\theta) \le \delta \).
**Answer: B — \( L_{\theta_{\text{old}}}(\theta) \) is a first-order approximation of \( J(\theta) \), accurate when \( \pi_\theta \) is close to \( \pi_{\text{old}} \) --- precisely the regime enforced by the KL constraint.**
A: **Incorrect:** \( L_{\theta_{\text{old}}}(\theta) \) uses the old state distribution \( \rho^{\pi_{\text{old}}} \), so it diverges from \( J(\theta) \) when \( \theta \) moves far from \( \theta_{\text{old}} \).
B: **Correct:** \( L_{\theta_{\text{old}}}(\theta) \) and \( J(\theta) \) share the same value and gradient at \( \theta = \theta_{\text{old}} \), and the KL constraint keeps \( \theta \) inside the region where this first-order approximation is valid.
C: **Incorrect:** Because it uses \( \rho^{\pi_{\text{old}}} \) instead of \( \rho^{\pi_\theta} \), it is a local approximation, not an exact unbiased estimator for arbitrary policies.
D: **Incorrect:** TRPO still estimates advantages \( \hat{A}^{\pi_{\text{old}}}(s, a) \) using a value function baseline.
Which are known limitations or failure modes of DDPG?
6406536959574.
High sensitivity to hyperparameters such as learning rate and architecture
6406536959575.
Inability to handle continuous action spaces
6406536959576.
Brittle training that may diverge without careful tuning
6406536959577.
Deterministic policy requires explicit exploration noise during training
Published solution
**1. Identify the known limitations of DDPG:**
- **Option A:** DDPG is highly sensitive to hyperparameters (learning rates, Polyak \( \tau \), exploration noise, network architecture). **True.**
- **Option B:** False! DDPG was specifically designed to handle continuous action spaces.
- **Option C:** Training can be brittle and unstable (due to critic overestimation and moving targets). **True.**
- **Option D:** Because the actor \( \mu_\theta(s) \) is deterministic, external exploration noise must be added explicitly during training. **True.**
**Answer: A, C, and D.**
A: **Correct:** DDPG is well known for its sensitivity to hyperparameter settings.
B: **Incorrect:** DDPG is designed specifically for continuous action spaces.
C: **Correct:** DDPG training is often brittle and can diverge without careful tuning.
D: **Correct:** A deterministic policy requires explicit external noise to explore.
**1. Check each candidate estimator of the Advantage function \( A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t) \):**
- **Option A (\( r_t + \gamma V(s_{t+1}) - V(s_t) \)):** The 1-step TD residual \( \delta_t \), which is an unbiased estimator of \( A^\pi(s_t, a_t) \) when \( V = V^\pi \). Valid.
- **Option B (\( G_t - V(s_t) \)):** Monte Carlo advantage estimate, which is an unbiased estimator of \( A^\pi(s_t, a_t) \). Valid.
- **Option C (\( Q(s_t, a_t) - V(s_t) \)):** Direct definition of the advantage function. Valid.
- **Option D (\( V(s_t) - Q(s_t, a_t) \) or \( r_t + \gamma V(s_t) - V(s_{t+1}) \)):** Has the **wrong sign** (negates the advantage, or swaps \( s_t \) and \( s_{t+1} \)), so it is **not** a valid estimator of \( A^\pi(s_t, a_t) \).
**Answer: D.**
A: **Incorrect:** The 1-step TD error \( r_t + \gamma V(s_{t+1}) - V(s_t) \) is a standard valid advantage estimator.
B: **Incorrect:** The Monte Carlo return minus baseline \( G_t - V(s_t) \) is a valid advantage estimator.
C: **Incorrect:** \( Q(s_t, a_t) - V(s_t) \) is the exact definition of the advantage function.
D: **Correct:** Reverses the sign/terms of the advantage function and is therefore NOT a valid estimator.
Recall standard control algorithms in RL, like Q-Learning, SARSA, Expected SARSA, etc., involve a
maximisation step in constructing their target policies.
For example:
In Q-Learning, the target is:
Which of the following statements are correct? **Select all that apply.**
**1. Analyze Maximization Bias and Double Q-Learning (same core concepts as Paper 1617 Q#10):**
- **Option B:** Standard Q-learning suffers from maximization (overestimation) bias because the same noisy Q-values are used both to select and to evaluate the maximizing action. **True.**
- **Option C:** Double Q-learning eliminates maximization bias by using one network/table \( Q_1 \) to select the argmax action and an independent network/table \( Q_2 \) to evaluate it. **True.**
- **Option E:** Even with zero-mean noise on true equal action values, \( \mathbb{E}[\max_a Q(s',a)] > \max_a \mathbb{E}[Q(s',a)] \). **True.**
- **Option F:** Clipped Double Q-learning in TD3 addresses this overestimation bias in continuous control. **True.**
**Answer: B, C, E, and F.**
A: **Incorrect:** Q-learning exhibits overestimation bias, not underestimation bias.
B: **Correct:** Reusing the same estimator to select and evaluate the greedy action causes positive maximization bias.
C: **Correct:** Double Q-learning decouples action selection (\( Q_1 \)) from action evaluation (\( Q_2 \)).
D: **Incorrect:** Averaging inside a single `max` operator does not remove maximization bias.
E: **Correct:** By Jensen's inequality, the expectation of the maximum of noisy estimates exceeds the maximum of the expectations.
F: **Correct:** TD3 uses twin critics to mitigate overestimation bias in actor-critic methods.
Ganesh is designing a robot arm controller with a 6-dimensional continuous action space. He
initially uses a stochastic policy
with the standard policy gradient(SPG):
Karthik suggests switching to the Deterministic Policy Gradient (DPG) with policy
:
Karthik argues that DPG is strictly more sample-efficient here. However, Ganesh notices that the
current DPG implementation uses only on-policy data sampled under
, and that the robot
operates in a noisy, stochastic environment where some states are rarely visited under the greedy
policy.
Based on the above data, answer the given subquestions.
Why is DPG generally more sample-efficient than SPG in high-dimensional continuous action
spaces?
6406536959590.
DPG avoids computing
, which is numerically unstable in high
dimensions
6406536959591.
DPG’s gradient integrates over the state space only, whereas SPG integrates
over both state and action spaces - requiring far more samples to estimate accurately when
is large
6406536959592.
DPG uses a target network which reduces gradient variance automatically
6406536959593.
DPG enforces determinism, which eliminates exploration noise and thus
requires fewer environment interactions
Published solution
**1. Compare sample efficiency of DPG vs. SPG in high-dimensional action spaces:**
In a 6-dimensional (or high-dimensional) continuous action space \( \mathcal{A} \subseteq \mathbb{R}^6 \), **Stochastic Policy Gradient (SPG)** must integrate over both the state space \( \mathcal{S} \) and the 6-dimensional action space \( \mathcal{A} \), requiring many action samples to reduce gradient variance. By contrast, **Deterministic Policy Gradient (DPG)** integrates over the **state space only**, making it significantly more sample-efficient when \( \dim(\mathcal{A}) \) is large.
**Answer: B — DPG’s gradient integrates over the state space only, whereas SPG integrates over both state and action spaces - requiring far more samples to estimate accurately when \( \dim(\mathcal{A}) \) is large.**
A: **Incorrect:** Computing \( \nabla_\theta \ln \pi_\theta \) for a Gaussian policy is simple and numerically stable.
B: **Correct:** Avoiding integration over the high-dimensional action space \( \mathcal{A} \) makes DPG much more sample-efficient than SPG.
C: **Incorrect:** Vanilla DPG does not inherently include target networks (those were added in DDPG).
D: **Incorrect:** DPG still requires exploration noise via a behavior policy during training.
Ganesh is designing a robot arm controller with a 6-dimensional continuous action space. He
initially uses a stochastic policy
with the standard policy gradient(SPG):
Karthik suggests switching to the Deterministic Policy Gradient (DPG) with policy
:
Karthik argues that DPG is strictly more sample-efficient here. However, Ganesh notices that the
current DPG implementation uses only on-policy data sampled under
, and that the robot
operates in a noisy, stochastic environment where some states are rarely visited under the greedy
policy.
Based on the above data, answer the given subquestions.
Ganesh’s on-policy DPG implementation is likely to suffer from which of the following problems?
Select all that apply.
6406536959594.
Poor coverage of the state space — the greedy policy may never visit states
needed for accurate
estimation
6406536959595.
Maximization bias, because the greedy policy always selects the action with
highest estimated
6406536959596.
The DPG theorem technically requires an off-policy behaviour policy to ensure
sufficient state coverage; on-policy data violates this
6406536959597.
Insufficient exploration, since
is deterministic and produces no stochastic
variation in actions
6406536959598.
The action-value gradient
cannot be computed on-policy
Published solution
**1. Analyze why purely on-policy DPG fails in a stochastic/sparse environment:**
- **Option A (Poor state-space coverage):** Because \( \mu_\theta(s) \) is greedy/deterministic, rolling out \( \mu_\theta \) on-policy may never visit states and actions needed to learn accurate \( Q_w(s, a) \) gradients. **True.**
- **Option B:** Maximization bias is a critic bootstrapping issue, not the distinct problem of on-policy vs. off-policy DPG exploration.
- **Option C:** To ensure adequate exploration and state-action space coverage, DPG is formulated as an **off-policy** actor-critic algorithm following a stochastic behavior policy \( \beta(a \mid s) \). **True.**
- **Option D (Insufficient exploration):** Because \( \mu_\theta(s) \) is deterministic, it produces zero action variation on its own in a given state. **True.**
- **Option E:** False, \( \nabla_a Q_w(s, a)|_{a=\mu_\theta(s)} \) is computed analytically from the differentiable critic network \( Q_w \).
**Answer: A, C, and D.**
A: **Correct:** A deterministic policy without exploration noise fails to cover the state-action space needed to train the critic \( Q_w \).
B: **Incorrect:** The core failure mode described in the passage is lack of exploration in on-policy deterministic rollouts.
C: **Correct:** Practical DPG requires an off-policy stochastic behavior policy to explore sufficiently.
D: **Correct:** \( \mu_\theta(s) \) is deterministic and generates no exploratory variation in actions.
E: **Incorrect:** \( \nabla_a Q_w(s,a) \) is computed by backpropagating through the critic network \( Q_w \) and does not require off-policy data to evaluate at a point.
Ganesh is designing a robot arm controller with a 6-dimensional continuous action space. He
initially uses a stochastic policy
with the standard policy gradient(SPG):
Karthik suggests switching to the Deterministic Policy Gradient (DPG) with policy
:
Karthik argues that DPG is strictly more sample-efficient here. However, Ganesh notices that the
current DPG implementation uses only on-policy data sampled under
, and that the robot
operates in a noisy, stochastic environment where some states are rarely visited under the greedy
policy.
Based on the above data, answer the given subquestions.
A Bernoulli-logistic unit is a stochastic neuron whose output
follows a Bernoulli
distribution:
The unit receives a feature vector
as input. Let
denote the action preference
for action
in state
under policy parameter
.
Assume that difference in action preferences is a linear function of the input:
The softmax (exponential soft-max) policy converts preferences to probabilities:
The REINFORCE (Monte Carlo policy gradient) update rule is:
where
is the return from time
, and
is called the eligibility
vector.
Based on the above data, answer the given subquestions.
**[Bonus]** The Monte Carlo REINFORCE update for the Bernoulli-logistic unit upon receipt of return
is:
Which of the following statements about this update are correct? Select all that apply.
6406536959604.
When
, the eligibility vector is
, so the update reinforces
in the direction of
when
6406536959605.
When
, the eligibility vector is
, meaning the update
decreases
(i.e. makes action 1 less likely) when
6406536959606.
The magnitude of the update is larger when the selected action is surprising
(low probability), since
is large when
and
is large when
6406536959607.
The update is an unbiased estimate of
only if the episode is generated
on-policy under
6406536959608.
Subtracting a baseline
from
changes the direction of the gradient
update, introducing bias to improve convergence speed
Published solution
**1. Evaluate each statement about the REINFORCE update \( \Delta \theta_t = \alpha G_t (A_t - \sigma(\theta^T \mathbf{x}(S_t))) \mathbf{x}(S_t) \):**
- **Option A:** When \( A_t = 1 \), the eligibility vector is \( (1 - \sigma(\theta^T \mathbf{x}(S_t))) \mathbf{x}(S_t) \), which moves \( \theta \) in the direction of \( +\mathbf{x}(S_t) \) when \( G_t > 0 \). **True.**
- **Option B:** When \( A_t = 0 \), the eligibility vector is \( -\sigma(\theta^T \mathbf{x}(S_t)) \mathbf{x}(S_t) \), which decreases \( \theta^T \mathbf{x}(S_t) \) (making action 1 less likely and action 0 more likely) when \( G_t > 0 \). **True.**
- **Option C:** The factor \( |A_t - \sigma| \) is large when a low-probability ('surprising') action is selected (\( 1 - \sigma \) is large when \( A_t = 1, \sigma \approx 0 \), and \( \sigma \) is large when \( A_t = 0, \sigma \approx 1 \)). **True.**
- **Option D:** The Monte Carlo REINFORCE update is an unbiased estimate of \( \nabla_\theta J(\theta) \) only if the episode is generated on-policy under \( \pi_\theta \). **True.**
- **Option E:** Subtracting a state-dependent baseline \( b(S_t) \) does NOT introduce bias. **False.**
**Answer: A, B, C, and D.**
A: **Correct:** When \( A_t = 1 \) and \( G_t > 0 \), \( (1 - \sigma) > 0 \), reinforcing \( \theta \) along \( +\mathbf{x}(S_t) \).
B: **Correct:** When \( A_t = 0 \) and \( G_t > 0 \), the negative scalar \( -\sigma \) pushes \( \theta \) along \( -\mathbf{x}(S_t) \), reducing \( \sigma(\theta^T \mathbf{x}) \).
C: **Correct:** Surprising actions have a large discrepancy \( |A_t - \sigma| \), producing a larger update magnitude.
D: **Correct:** On-policy sampling under \( \pi_\theta \) is required for unbiasedness.
E: **Incorrect:** A state-only baseline \( b(S_t) \) has zero expectation when multiplied by the score function and introduces no bias.
Vivek implements the REINFORCE (Monte Carlo policy gradient) algorithm for a text-generation
task modelled as an episodic MDP. The policy is a neural network
parameterised by
.
At each episode's end, he collects the full trajectory
and
computes the return:
The REINFORCE gradient estimator is:
After 1000 episodes, he observes that the gradient variance is extremely high and training is
unstable. Neel suggests adding a baseline
:
He claims: "Subtracting
reduces variance without biasing the gradient, as long as
does not
depend on
.''
Based on the above data, answer the given subquestions.
Why does REINFORCE exhibit high gradient variance, particularly in long-horizon tasks like text
generation?
6406536959609.
The log-probability
is numerically unstable for large neural
networks
6406536959610.
accumulates rewards over the entire future horizon; in long episodes with
noisy rewards,
has large variance, which directly scales the gradient estimate
6406536959611.
Monte Carlo rollouts are biased estimators of the true return
6406536959612.
The policy gradient theorem does not hold for episodic tasks with discount
factor
Published solution
**1. Explain why REINFORCE has high gradient variance in long-horizon tasks:**
Because the Monte Carlo return \( G_t = \sum_{k=t}^T \gamma^{k-t} R_k \) sums rewards across the entire remaining trajectory, any randomness in transitions, actions, and rewards over a long horizon accumulates into a very large variance in \( G_t \), which directly scales \( \nabla_\theta \ln \pi_\theta(a_t \mid s_t) \).
**Answer: B — \( G_t \) accumulates rewards over the entire future horizon; in long episodes with noisy rewards, \( G_t \) has large variance, which directly scales the gradient estimate.**
A: **Incorrect:** Log-probabilities are numerically stabilized via log-softmax; the variance comes from \( G_t \).
B: **Correct:** Summing rewards over a long trajectory makes the empirical return \( G_t \) high-variance.
C: **Incorrect:** Monte Carlo returns are *unbiased* (though high-variance) estimators of \( Q^\pi \).
D: **Incorrect:** The policy gradient theorem holds for episodic MDPs with \( \gamma = 1 \).
Vivek implements the REINFORCE (Monte Carlo policy gradient) algorithm for a text-generation
task modelled as an episodic MDP. The policy is a neural network
parameterised by
.
At each episode's end, he collects the full trajectory
and
computes the return:
The REINFORCE gradient estimator is:
After 1000 episodes, he observes that the gradient variance is extremely high and training is
unstable. Neel suggests adding a baseline
:
He claims: "Subtracting
reduces variance without biasing the gradient, as long as
does not
depend on
.''
Based on the above data, answer the given subquestions.
For a single timestep
in an episode, the following values are observed:
Compute the parameter update
and report
.
Round to 4 decimal places.
A published solution is not available for this question yet.
Vivek implements the REINFORCE (Monte Carlo policy gradient) algorithm for a text-generation
task modelled as an episodic MDP. The policy is a neural network
parameterised by
.
At each episode's end, he collects the full trajectory
and
computes the return:
The REINFORCE gradient estimator is:
After 1000 episodes, he observes that the gradient variance is extremely high and training is
unstable. Neel suggests adding a baseline
:
He claims: "Subtracting
reduces variance without biasing the gradient, as long as
does not
depend on
.''
Based on the above data, answer the given subquestions.
Neel claims that any function
not depending on
is a valid zero-bias baseline. Which of the
following statements are correct justifications or counter-examples to this claim?
6406536959614.
for any
independent of
, since
6406536959615.
If
, the baseline perfectly cancels the return and the gradient
becomes zero — a degenerate case
6406536959616.
A baseline that depends on both
and
(e.g.
) still leaves the
gradient unbiased
6406536959617.
Using
introduces bias because
is estimated, not known
exactly
Published solution
**1. Justify why a state-dependent baseline \( b(s_t) \) introduces zero bias:**
Because \( b(s_t) \) does not depend on the action \( a_t \), it factors out of the expectation over \( a_t \sim \pi_\theta(\cdot \mid s_t) \):
\[
\mathbb{E}_{a_t \sim \pi_\theta}\!\left[ b(s_t) \nabla_\theta \ln \pi_\theta(a_t \mid s_t) \right] = b(s_t) \sum_{a} \pi_\theta(a \mid s_t) \frac{\nabla_\theta \pi_\theta(a \mid s_t)}{\pi_\theta(a \mid s_t)} = b(s_t) \nabla_\theta \sum_a \pi_\theta(a \mid s_t) = b(s_t) \nabla_\theta(1) = 0
\]
This matches **Option A**.
**Answer: A.**
A: **Correct:** Uses the identity \( \mathbb{E}_{a_t}[\nabla_\theta \ln \pi_\theta(a_t \mid s_t)] = \nabla_\theta \sum_a \pi_\theta(a \mid s_t) = \nabla_\theta(1) = 0 \).
B: **Incorrect:** \( G_t \) depends on the action \( a_t \) and future trajectory, so \( b(s_t) = G_t \) is not a state-only function and would bias the gradient to zero.
C: **Incorrect:** An action-dependent baseline \( b(s_t, a_t) \) cannot be factored out of the sum over \( a_t \) and generally introduces bias.
D: **Incorrect:** Even an approximate state-value function \( \hat{V}(s_t) \) has zero expectation when multiplied by the score function, so it introduces no bias.
Lalit uses two algorithms to learn option values in a given programming assignment:
Algorithm A — SMDP Q-Learning: Updates
only at option termination using the full option
return:
Algorithm B — Intra-Option Q-Learning: Updates
at every primitive step using the one-step
consistency condition:
where
is the termination probability at the next state.
The agent runs in a corridor with 5 states
. Option
has intra-option policy: move
right always.
if
, else
. Currently:
,
,
,
,
,
, agent moves from
to
.
Based on the above data, answer the given subquestions.
What is the key advantage of Intra-Option Q-Learning (Algorithm B) over SMDP Q-Learning
(Algorithm A)?
6406536959618.
It requires no knowledge of the termination function
6406536959619.
It updates
at every time step rather than only at termination, making
more efficient use of experience — especially for long-duration options
6406536959620.
It converges to a globally optimal policy, whereas SMDP Q-learning only
converges locally
6406536959621.
It eliminates the need for a high-level policy by learning all options
simultaneously
Published solution
**1. Compare Intra-Option Q-Learning (Algorithm B) vs. SMDP Q-Learning (Algorithm A):**
Unlike SMDP Q-Learning (which waits until an option terminates to perform a single update for the state where the option was initiated), **Intra-Option Q-Learning updates \( Q_\Omega \) at every primitive time step** using one-step bootstrapping, making much more efficient use of experience—especially for long-duration options.
**Answer: B — It updates \( Q_\Omega \) at every time step rather than only at termination, making more efficient use of experience — especially for long-duration options.**
A: **Incorrect:** Intra-Option Q-Learning explicitly uses the termination probability \( \beta_\omega(s_{t+1}) \) in \( U_\omega(s_{t+1}) \).
B: **Correct:** Updating option values at every primitive step extracts far more information per transition than waiting for option termination.
C: **Incorrect:** Both SMDP Q-learning and Intra-Option Q-learning converge to the optimal option-value function \( Q^*_\Omega \).
D: **Incorrect:** A policy over options is still used to select options.
