Reinforcement Learning · End Term · Jan 2026 · AN

**1. Acknowledge the exam instructions:** This 0-mark administrative question confirms that the candidate has read the bonus-question instructions above. **Answer: A — Instructions has been mentioned above.** A: **Correct:** Standard acknowledgment option for the exam instructions. B: **Incorrect:** Standard instructions require selecting Option A.

**1. Recall the Options framework in Hierarchical Reinforcement Learning (Sutton, Precup & Singh, 1999):** An option \( \omega = \langle \mathcal{I}_\omega, \pi_\omega, \beta_\omega \rangle \) consists of: - Initiation set \( \mathcal{I}_\omega \subseteq \mathcal{S} \) - **Intra-option policy \( \pi_\omega(a \mid s) \):** specifies the probability distribution over primitive actions (or lower-level options) that the agent executes while option \( \omega \) is active. - Termination condition \( \beta_\omega(s) \). **Answer: C — The distribution over primitive actions the agent executes while option \( \omega \) is running.** A: **Incorrect:** The discount factor \( \gamma \) is a property of the MDP, not the intra-option policy. B: **Incorrect:** Initiation is governed by the initiation set \( \mathcal{I}_\omega \) and the high-level policy over options \( \pi_\Omega \). C: **Correct:** The intra-option policy \( \pi_\omega(a \mid s) \) defines the action selection behavior while option \( \omega \) is executing. D: **Incorrect:** The probability of terminating option \( \omega \) in state \( s \) is specified by the termination function \( \beta_\omega(s) \).

**1. Understand exploration in Deep Deterministic Policy Gradient (DDPG):** Because the DDPG actor \( \mu_\theta(s) \) is a **deterministic** function mapping states to continuous actions, it produces no exploration on its own. Exploration is achieved off-policy by constructing a behavior policy \( \mu'(s_t) = \mu_\theta(s_t) + \mathcal{N}_t \) that **adds time-correlated noise (such as an Ornstein–Uhlenbeck process) or zero-mean Gaussian noise** directly to the actor's continuous action output. **Answer: D — Adding time-correlated noise (e.g., Ornstein-Uhlenbeck) or Gaussian noise to the deterministic policy output.** A: **Incorrect:** Standard \( \varepsilon \)-greedy is used for discrete action spaces, whereas DDPG adds continuous action noise. B: **Incorrect:** Prioritized replay changes sampling weights from the buffer but does not generate exploratory actions in the environment. C: **Incorrect:** Exploration noise in DDPG is added to the actor's actions (or parameter space of the actor), not the critic parameters. D: **Correct:** DDPG explores by adding Ornstein–Uhlenbeck or Gaussian noise to the deterministic actor output \( \mu_\theta(s) \).

**1. Understand the role of the Advantage function \( A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s) \) in A2C:** Subtracting the state-value baseline \( V^\pi(s) \) from \( Q^\pi(s, a) \) centers the action evaluations around zero: - Because \( V^\pi(s) \) depends only on the state \( s \) and not on the action \( a \), \( \mathbb{E}_{a \sim \pi}[\nabla_\theta \log \pi_\theta(a \mid s) V^\pi(s)] = 0 \). Thus, subtracting \( V^\pi(s) \) **introduces zero bias** into the expected policy gradient while **significantly reducing gradient variance**. **Answer: A — Reduces variance without introducing bias.** A: **Correct:** Using \( V(s) \) as a state-dependent baseline yields the advantage \( A(s,a) = Q(s,a) - V(s) \), reducing variance without biasing the gradient. B: **Incorrect:** Bootstrapping \( Q \) and \( V \) with function approximators still has critic approximation bias; the baseline step itself is for variance reduction. C: **Incorrect:** A2C remains an on-policy algorithm. D: **Incorrect:** Computing \( A(s, a) \) explicitly requires a value function approximator \( V_w(s) \).

**1. Recall the Deterministic Policy Gradient (DPG) Theorem (Silver et al., 2014):** For a deterministic policy \( a = \mu_\theta(s) \), applying the chain rule through \( Q^\mu(s, a) \) gives: \[ \nabla_\theta J(\theta) = \mathbb{E}_{s \sim \rho^\mu}\!\left[ \nabla_\theta \mu_\theta(s) \, \nabla_a Q^\mu(s, a)\big|_{a = \mu_\theta(s)} \right] \] **Answer: B — \( \mathbb{E}_{s \sim \rho^\mu}[\nabla_\theta \mu_\theta(s) \nabla_a Q^\mu(s, a)|_{a=\mu_\theta(s)}] \).** A: **Incorrect:** Uses the stochastic log-policy score function \( \nabla_\theta \log \mu_\theta \) instead of the chain-rule gradient \( \nabla_\theta \mu_\theta(s) \nabla_a Q \). B: **Correct:** Matches the exact Deterministic Policy Gradient theorem via the chain rule. C: **Incorrect:** Differentiates \( Q \) with respect to state \( s \) instead of action \( a \). D: **Incorrect:** Missing the Jacobian \( \nabla_\theta \mu_\theta(s) \).

**1. Understand why TRPO uses a backtracking line search:** TRPO computes its search direction using a linear approximation to the surrogate objective and a **quadratic (Fisher Information Matrix) approximation** to the KL-divergence constraint \( \bar{D}_{\text{KL}}(\theta_{\text{old}} \| \theta) \le \delta \). - Because the quadratic Taylor approximation can be inaccurate for finite step sizes, TRPO performs a **backtracking line search** (exponentially shrinking the step size \( \alpha^j \)) to verify that the **true (exact) non-linear KL divergence is \( \le \delta \)** and that the **surrogate objective actually improves**. **Answer: B — To verify that the actual KL divergence (not the quadratic approximation) stays within \( \delta \) and that the surrogate objective improves; if either fails, the step size is halved.** A: **Incorrect:** Line search runs *after* the conjugate gradient solver has already computed the search direction. B: **Correct:** Backtracking line search ensures that the exact KL constraint and surrogate improvement hold despite Taylor approximation errors. C: **Incorrect:** TRPO retains the natural gradient direction and only scales its step length. D: **Incorrect:** The surrogate objective still uses importance sampling ratios.

**1. Evaluate each statement about DDPG:** - **Option A:** DDPG maintains four neural networks: online actor \( \mu_\theta \), online critic \( Q_\phi \), target actor \( \mu_{\theta'} \), and target critic \( Q_{\phi'} \). **True.** - **Option B:** The critic is updated off-policy by minimizing the mean-squared Bellman error (MSBE) on mini-batches sampled from a replay buffer. **True.** - **Option C:** DDPG requires computing \( \nabla_a Q_\phi(s, a) \) with respect to continuous actions \( a \), so it is **not** directly applicable to discrete action spaces without modification (such as Gumbel-Softmax). **False.** - **Option D:** The actor is updated via deterministic policy gradient ascent to maximize \( Q_\phi(s, \mu_\theta(s)) \). **True.** **Answer: A, B, and D.** A: **Correct:** DDPG uses online and target networks for both the actor and the critic (4 networks total). B: **Correct:** The critic minimizes TD/Bellman error over off-policy transitions drawn from the replay buffer. C: **Incorrect:** DDPG relies on action gradients \( \nabla_a Q(s, a) \) and is designed for continuous action spaces, not discrete ones. D: **Correct:** The actor parameters \( \theta \) are updated along \( \nabla_\theta Q_\phi(s, \mu_\theta(s)) \).

**1. Recall the regularity conditions of the DPG Theorem (and note the question asks which are NOT required):** - The DPG theorem requires \( Q^\mu(s, a) \) to be differentiable with respect to \( a \) (Option A — required) and the deterministic policy \( \mu_\theta(s) \) to be differentiable with respect to \( \theta \) (Option B — required), along with regularity of \( p(s' \mid s, a) \) with respect to action \( a \). - **Option C (The policy must be stochastic):** False / **NOT required** — the theorem specifically applies to *deterministic* policies \( \mu_\theta(s) \)! - **Option D (The MDP transition dynamics must be differentiable [w.r.t. state \( s \) / known model]):** In the course quiz key, **Options C and D** are marked as the statements that are **not** valid conditions in that form. **Answer: C and D.** A: **Incorrect:** Differentiability of \( Q^\mu(s,a) \) with respect to \( a \) IS a required condition for DPG. B: **Incorrect:** Differentiability of \( \mu_\theta(s) \) with respect to \( \theta \) IS a required condition for DPG. C: **Correct:** DPG applies to deterministic policies, so requiring the policy to be stochastic is NOT a valid condition. D: **Correct:** Marked in the course answer key as not a valid required condition.

**1. Analyze Maximization Bias and Double Q-Learning:** - Standard Q-learning uses the same Q-estimator both to **select** the maximizing action \( a^* = \arg\max_a Q(s', a) \) and to **evaluate** its value \( Q(s', a^*) \), resulting in positive **maximization (overestimation) bias** (**Option B is true**, **Option A is false**). - **Double Q-Learning** decouples selection and evaluation using two independent estimators \( Q_1 \) and \( Q_2 \): it selects \( a^* = \arg\max_a Q_1(s', a) \) using \( Q_1 \) and evaluates \( Q_2(s', a^*) \) using \( Q_2 \) (**Option C is true**, **Option D is false**). - Even when true action values are identical, noisy estimates cause \( \mathbb{E}[\max_a Q(s', a)] > \max_a \mathbb{E}[Q(s', a)] \) by Jensen's inequality (**Option E is true**). - In continuous actor-critic methods, **TD3 (Twin Delayed DDPG)** addresses this same overestimation bias via clipped double Q-learning (taking the minimum of two critics) (**Option F is true**). **Answer: B, C, E, and F.** A: **Incorrect:** Standard Q-learning suffers from overestimation (positive) bias, not underestimation bias. B: **Correct:** Using the same noisy estimates to both pick and evaluate the maximizing action causes maximization bias. C: **Correct:** Double Q-learning decouples action selection (\( Q_1 \)) from action evaluation (\( Q_2 \)) to eliminate maximization bias. D: **Incorrect:** Averaging the two Q-tables inside a single `max` still suffers from maximization bias. E: **Correct:** By Jensen's inequality, the expectation of the maximum of noisy unbiased estimators exceeds the maximum of their expectations. F: **Correct:** TD3 uses twin critics and takes \( \min(Q_1, Q_2) \) in the target to combat overestimation bias in DDPG.

**1. Set up the cumulative discounted return for the 3-step option:** Given rewards \( r_1 = 2, r_2 = 1, r_3 = 3 \) and discount factor \( \gamma = 0.9 \): \[ G = r_1 + \gamma r_2 + \gamma^2 r_3 \] **2. Calculate:** \[ G = 2 + 0.9(1) + (0.9)^2(3) = 2 + 0.9 + 0.81(3) = 2.9 + 2.43 = 5.33 \] **Answer: 5.33.**

**1. Find the greedy actions under \( Q_1 \) at each possible next state \( s_1, s_2 \):** - At state \( s_1 \) (reached with probability \( p(s_1) = 0.6 \) and reward \( r(s_1) = 1 \)): - \( a^*_1 = \arg\max_a Q_1(s_1, a) \). Evaluating this action using \( Q_2(s_1, a^*_1) \) gives \( Q_2(s_1, a^*_1) = 2.0 \). - Branch target at \( s_1 \): \( r(s_1) + \gamma Q_2(s_1, a^*_1) = 1 + 0.9(2.0) = 2.8 \). - At state \( s_2 \) (reached with probability \( p(s_2) = 0.4 \) and reward \( r(s_2) = 3 \)): - \( a^*_2 = \arg\max_a Q_1(s_2, a) \). Evaluating this action using \( Q_2(s_2, a^*_2) \) gives \( Q_2(s_2, a^*_2) = 1.0 \). - Branch target at \( s_2 \): \( r(s_2) + \gamma Q_2(s_2, a^*_2) = 3 + 0.9(1.0) = 3.9 \). **2. Compute the expected Double Q-learning target \( Y \):** \[ Y = 0.6(2.8) + 0.4(3.9) = 1.68 + 1.56 = 3.24 \] **Answer: 3.24.**

**1. Analyze the effect of a large Polyak averaging parameter \( \tau \) (e.g., \( \tau = 0.5 \)):** In DDPG, target networks are updated via \( \phi' \leftarrow \tau \phi + (1 - \tau)\phi' \) with \( \tau \ll 1 \) (typically \( \tau = 0.001 \) or \( 0.005 \)) so that the TD target \( y_i = r_i + \gamma Q_{\phi'}(s'_i, \mu_{\theta'}(s'_i)) \) changes slowly. - If \( \tau \) is large, \( \phi' \) immediately tracks the rapidly changing online weights \( \phi \), causing the bootstrap target \( y_i \) to shift rapidly at every step—recreating the **moving-target instability** and feedback loops that target networks are meant to prevent. **Answer: B — With large \( \tau \), the target network tracks the online network closely, so the TD target \( y_i \) changes rapidly across consecutive updates -- recreating the moving-target instability that target networks were designed to prevent.** A: **Incorrect:** \( \tau \) controls target network Polyak averaging, not replay buffer sampling. B: **Correct:** A large \( \tau \) makes the target network change nearly as fast as the online network, destroying target stability. C: **Incorrect:** Both target actor and target critic use \( \tau \); the instability comes from the moving bootstrap target. D: **Incorrect:** \( \tau \) does not affect the action exploration noise process.

**1. Identify known issues in DDPG (and Aditi's setup):** - **Option A (Overestimation bias):** Because DDPG uses a single critic \( Q_\phi(s', \mu_\theta(s')) \) maximizing over actions, it suffers from function-approximation overestimation bias (which motivated TD3). **True.** - **Option B (Brittle training):** DDPG is notoriously sensitive to hyperparameters (such as \( \tau \), learning rates, and noise scale) and prone to divergence. **True.** - **Option C (Lack of exploration without explicit noise):** Because \( \mu_\theta(s) \) is deterministic, it provides zero exploration unless noise is explicitly added. **True.** - **Option D:** False! DDPG was specifically designed for continuous action spaces (like 4D drone control) and handles them easily. - **Option E (Temporal correlation):** Consecutive transitions collected from the environment are temporally correlated (which the replay buffer must break by uniform sampling; if buffer size/sampling is poor, correlation destabilizes updates). **True.** **Answer: A, B, C, and E.** A: **Correct:** DDPG's single critic bootstrap target suffers from Q-value overestimation bias. B: **Correct:** DDPG training is brittle and sensitive to hyperparameter choices. C: **Correct:** A deterministic actor requires external noise to explore. D: **Incorrect:** DDPG is explicitly designed to scale to multi-dimensional continuous action spaces. E: **Correct:** Trajectory transitions are temporally correlated.

**1. Set up the Polyak soft update equation:** \[ \phi'_{\text{new}} = \tau \phi + (1 - \tau) \phi' \] **2. Substitute \( \tau = 0.005 \), \( \phi = 1.2 \), and \( \phi' = 0.4 \):** \[ \phi'_{\text{new}} = 0.005(1.2) + (1 - 0.005)(0.4) = 0.006 + 0.995(0.4) = 0.006 + 0.398 = 0.404 \] **Answer: 0.404.**

**1. Analyze the eligibility vector \( \nabla_\theta \ln \pi_\theta(a \mid s) = (a - \sigma(\theta^T \mathbf{x}(s))) \mathbf{x}(s) \) for \( a \in \{0, 1\} \):** - **Option A:** When \( a = 1 \), \( (1 - \sigma(\theta^T \mathbf{x}(s))) \mathbf{x}(s) \). **True.** - **Option B:** When \( a = 0 \), \( (0 - \sigma(\theta^T \mathbf{x}(s))) \mathbf{x}(s) = -\sigma(\theta^T \mathbf{x}(s)) \mathbf{x}(s) \). **True.** - **Option C:** If the feature vector \( \mathbf{x}(s) = \mathbf{0} \), the eligibility vector is \( \mathbf{0} \) (zero). Thus it is not always non-zero. **False.** - **Option D:** The factor \( (a - \sigma(\theta^T \mathbf{x}(s))) \) is the exact residual (prediction error between binary label \( a \) and predicted probability \( \sigma(\theta^T \mathbf{x}) \)) from logistic regression with binary cross-entropy loss. **True.** **Answer: A, B, and D.** A: **Correct:** Substituting \( a = 1 \) into \( (a - \sigma(\theta^T \mathbf{x}(s)))\mathbf{x}(s) \) gives \( (1 - \sigma(\theta^T \mathbf{x}(s)))\mathbf{x}(s) \). B: **Correct:** Substituting \( a = 0 \) gives \( -\sigma(\theta^T \mathbf{x}(s))\mathbf{x}(s) \). C: **Incorrect:** Whenever \( \mathbf{x}(s) = \mathbf{0} \), the eligibility vector equals \( \mathbf{0} \). D: **Correct:** \( a - \sigma(\theta^T \mathbf{x}) \) is the classic gradient residual of binary cross-entropy with a logistic sigmoid.

**1. Evaluate each statement about the REINFORCE update \( \Delta \theta_t = \alpha G_t (A_t - \sigma(\theta^T \mathbf{x}(S_t))) \mathbf{x}(S_t) \):** - **Option A:** When the chosen action \( A_t \) has low predicted probability (e.g., \( A_t = 1 \) when \( \sigma \approx 0 \), or \( A_t = 0 \) when \( \sigma \approx 1 \)), the factor \( |A_t - \sigma| \) is close to 1 (large), giving a larger update for surprising actions. **True.** - **Option B:** When \( G_t > 0 \) and \( A_t = 1 \), \( \Delta \theta_t = c \mathbf{x}(S_t) \) with \( c > 0 \), so \( \theta_{\text{new}}^T \mathbf{x}(S_t) = \theta^T \mathbf{x}(S_t) + c \|\mathbf{x}(S_t)\|^2 \ge \theta^T \mathbf{x}(S_t) \), increasing \( \pi(1 \mid S_t) \). **True.** - **Option C:** When \( G_t > 0 \) and \( A_t = 0 \), \( \Delta \theta_t = -\alpha G_t \sigma \mathbf{x}(S_t) \), which decreases \( \theta^T \mathbf{x}(S_t) \) and increases \( \pi(0 \mid S_t) = 1 - \sigma \). **True.** - **Option D:** The Monte Carlo REINFORCE estimator is an unbiased estimate of \( \nabla_\theta J(\theta) \) only when trajectories are sampled on-policy from \( \pi_\theta \). **True.** - **Option E:** Subtracting a state-dependent baseline \( b(S_t) \) does **not** introduce bias (expected value of baseline term is zero). **False.** **Answer: A, B, C, and D.** A: **Correct:** The factor \( |A_t - \sigma| \) is largest when the action taken had low probability under the current policy. B: **Correct:** A positive return on \( A_t = 1 \) pushes \( \theta \) in the direction of \( +\mathbf{x}(S_t) \), increasing \( \pi(1 \mid S_t) \). C: **Correct:** A positive return on \( A_t = 0 \) pushes \( \theta \) in the direction of \( -\mathbf{x}(S_t) \), decreasing \( \sigma \) and making action 0 more likely. D: **Correct:** On-policy sampling under \( \pi_\theta \) is required for the REINFORCE estimator to be unbiased without importance sampling. E: **Incorrect:** A state-dependent baseline \( b(S_t) \) leaves the expected gradient completely unbiased.

**1. Understand why Monte Carlo REINFORCE has high variance:** In REINFORCE, the score function \( \nabla_\theta \ln \pi_\theta(A_t \mid S_t) \) is multiplied by the full empirical Monte Carlo return \( G_t = \sum_{k=0}^{T-t-1} \gamma^k R_{t+1+k} \). Over a long horizon, \( G_t \) sums many random rewards resulting from stochastic transitions and future actions, giving \( G_t \) very large variance that directly scales the gradient estimate. **Answer: B — \( G_t \) sums rewards over the remaining horizon; in long episodes with stochastic rewards, \( G_t \) has large variance, which directly multiplies \( \nabla_\theta \ln \pi_\theta \) in the gradient estimator.** A: **Incorrect:** High variance persists throughout training due to Monte Carlo return sampling, not just initial weights. B: **Correct:** Summing rewards over many future steps makes the Monte Carlo return \( G_t \) high-variance, which directly scales the policy gradient. C: **Incorrect:** The policy gradient theorem holds for both discrete and continuous action spaces. D: **Incorrect:** Explains what is missing rather than the mathematical cause of high variance in \( G_t \).

**1. Set up the REINFORCE-with-baseline parameter update:** \[ \Delta \theta_t = \alpha (G_t - b(S_t)) \nabla_\theta \ln \pi_\theta(A_t \mid S_t) \] **2. Substitute the given values (\( \alpha = 0.02,\ G_t = 10,\ b(S_t) = 4,\ \nabla_\theta \ln \pi_\theta = [1.5, -2.0]^T \)):** \[ \Delta \theta_t = 0.02 \times (10 - 4) \times \begin{bmatrix} 1.5 \\ -2.0 \end{bmatrix} = 0.12 \times \begin{bmatrix} 1.5 \\ -2.0 \end{bmatrix} = \begin{bmatrix} 0.18 \\ -0.24 \end{bmatrix} \] **3. Compute the \( L_2 \)-norm \( \|\Delta \theta_t\|_2 \):** \[ \|\Delta \theta_t\|_2 = \sqrt{(0.18)^2 + (-0.24)^2} = \sqrt{0.0324 + 0.0576} = \sqrt{0.09} = 0.3 \] **Answer: 0.3.**

**1. Evaluate each statement on baselines in policy gradient:** - **Option A:** Because \( b(S_t) \) does not depend on \( A_t \), we can factor \( b(S_t) \) out of the expectation over \( A_t \): \( \mathbb{E}_{A_t \sim \pi}[b(S_t) \nabla_\theta \ln \pi_\theta(A_t \mid S_t)] = b(S_t) \nabla_\theta \sum_a \pi_\theta(a \mid S_t) = b(S_t) \nabla_\theta (1) = 0 \). **True.** - **Option B:** Even if \( \hat{V}_w(S_t) \) is an imperfect approximation of \( V^\pi(S_t) \), as long as it depends only on \( S_t \) (and not \( A_t \)), the baseline term still has zero expectation and keeps the gradient unbiased. **True.** - **Option C:** False! If the baseline depends on the action \( A_t \), it cannot be factored out of the sum/integral over actions, so \( \sum_a b(s, a) \nabla_\theta \pi_\theta(a \mid s) \neq 0 \), which **biases** the gradient. - **Option D:** The theoretical minimum-variance state-dependent baseline is \( b^*(s) = \frac{\mathbb{E}_a[Q(s,a) \|\nabla_\theta \ln \pi_\theta(a|s)\|^2]}{\mathbb{E}_a[\|\nabla_\theta \ln \pi_\theta(a|s)\|^2]} \), which weights returns by \( \|\nabla_\theta \ln \pi_\theta(a \mid s)\|^2 \). **True.** **Answer: A, B, and D.** A: **Correct:** Factoring \( b(S_t) \) out of the action expectation gives \( b(S_t)\nabla_\theta(1) = 0 \). B: **Correct:** Any state-only function \( \hat{V}(S_t) \) has zero expected product with the score function, regardless of approximation error. C: **Incorrect:** An action-dependent baseline \( b(S_t, A_t) \) cannot be factored out of the sum over actions and introduces bias. D: **Correct:** Setting the derivative of the gradient variance w.r.t. \( b(s) \) to zero yields a weighted average weighted by \( \|\nabla_\theta \ln \pi_\theta\|^2 \).

**1. Compare Intra-Option Q-Learning vs. SMDP Q-Learning:** - **SMDP Q-Learning** treats each option execution as a single indivisible macro-step and only updates \( Q_\Omega(s_t, \omega) \) at the start state \( s_t \) once the option terminates. - **Intra-Option Q-Learning** uses the one-step Bellman consistency of options to update \( Q_\Omega(s, \omega) \) **at every primitive time step** (and for all options consistent with the primitive action), extracting far more learning signal from each transition—especially for long-duration options. **Answer: B — Intra-Option Q-Learning updates \( Q_\Omega \) at every primitive step rather than only at option termination, resulting in more efficient use of experience, especially for long-duration options.** A: **Incorrect:** Both algorithms converge to the same optimal option-value function \( Q^*_\Omega \) under standard conditions. B: **Correct:** Performing one-step TD updates at every primitive transition makes much more efficient use of experience than waiting for option termination. C: **Incorrect:** Intra-Option Q-Learning explicitly uses \( \beta_\omega(s_{t+1}) \) in its bootstrap target \( U_\omega(s_{t+1}) \). D: **Incorrect:** Intra-Option Q-Learning can learn values for an entire set of options simultaneously.

**1. Set up the Intra-Option Q-Learning TD error \( \delta_t \) for \( s_2 \to s_3 \):** Given: - \( r = 1.0 \), \( \gamma = 0.9 \) - \( Q_\Omega(s_2, \omega_{\text{right}}) = 2.5 \), \( Q_\Omega(s_3, \omega_{\text{right}}) = 3.5 \) - At \( s_3 \), option \( \omega_{\text{right}} \) does not terminate, so \( \beta_{\omega_{\text{right}}}(s_3) = 0 \). **2. Compute \( U_{\omega_{\text{right}}}(s_3) \) and \( \delta_t \):** \[ U_{\omega_{\text{right}}}(s_3) = (1 - \beta(s_3)) Q_\Omega(s_3, \omega_{\text{right}}) + \beta(s_3) \max_{\omega'} Q_\Omega(s_3, \omega') = 1 \times 3.5 + 0 = 3.5 \] \[ \delta_t = r + \gamma U_{\omega_{\text{right}}}(s_3) - Q_\Omega(s_2, \omega_{\text{right}}) = 1.0 + 0.9(3.5) - 2.5 = 1.0 + 3.15 - 2.5 = 1.65 \] **Answer: 1.65.**

**1. Apply the Intra-Option Q-Learning update rule:** With current estimate \( Q_\Omega(s_2, \omega_{\text{right}}) = 2.5 \), learning rate \( \alpha = 0.15 \), and TD error \( \delta_t = 1.65 \): \[ Q_\Omega^{\text{new}}(s_2, \omega_{\text{right}}) = Q_\Omega(s_2, \omega_{\text{right}}) + \alpha \delta_t = 2.5 + 0.15(1.65) = 2.5 + 0.2475 = 2.7475 \] **Answer: 2.7475.**