Reinforcement Learning · End Term · Sep 2025 · FN

Consider a discounted return

in an infinite-horizon MDP with bounded rewards

and discount factor

. Which of the following statements is/are correct? 1. For fixed rewards, the contribution of

to

decays geometrically as

. 2. If

and rewards are uniformly bounded, the infinite sum

is always finite. 3. For

, the discounted return

is bounded above in magnitude by

.

In the UCT (Upper Confidence bounds applied to Trees) variant of Monte Carlo Tree Search, the tree policy at a state

typically selects an action

by maximising

Consider the following target equations: **Equation 1**

**Equation 2**

In both equations,

refers to the shared parameters between the state-value network and advantage network.

and

refer to the non-shared parameters of the state-value and advantage network, respectively. In both equations,

is calculated as follows:

Note:

are target network parameters for online network parameters

Which among the following are correct statements?

Consider the REINFORCE update with baseline for a stochastic policy

:

Which of the following statements are true?

Consider following MDP, assume

:

The edges have the value

, where

denotes the transition probability and

is the immediate expected reward.

are non-terminal states and

is a terminal state. Based on the above data, answer the given

What is the probability that the accumulating eligibility trace for state

is more than 3 at the end of a trajectory? **(Write your answer up to 4 decimal places)**

Consider following MDP, assume

:

The edges have the value

, where

denotes the transition probability and

is the immediate expected reward.

are non-terminal states and

is a terminal state. Based on the above data, answer the given

Consider the following trajectory that the agent follows:

What will the eligibility trace of state

just before the episode terminates?**(Write your answer** **up to 4 decimal places)**

Consider a 2D navigation environment (as shown in the figure below) where an autonomous cleaning robot must locate and collect trash items (T1 and T2) scattered in the environment and then deliver them to a designated disposal bin (G). The environment is shaped by curved, contour- like walls, forming nested spaces with only one narrow passage connecting them, which requires deliberate path planning.

In this environment, the robot takes discrete actions from its action space

,

{ NORTH, SOUTH, EAST, WEST } Each action attempts to move the robot by

cm in the corresponding direction. If the action would cause a collision with a wall boundary, the robot remains in the same position and receives a penalty. The grey regions represent potholes. If the robot ends up in one, it cannot escape on its own and requires human help, leading to a substantial penalty and relocation to the nearest safe position. When the agent reaches a trash item, it automatically collects it and receives a reward. Likewise, upon reaching a disposal bin, it automatically unloads the collected trash. Therefore, explicit pickup and drop actions are not required. The episode terminates either when the agent reaches the Goal (G) while carrying at least one piece of trash, or when it has fallen into potholes a total of 10 times. The position of the Robot (

) can be represented as a tuple

, where

and

are the distances of the robot from the bottom-left corner of the environment. For state representation, we would consider state discretisation by performing tile as well as coarse coding, which converts

into

.

Observe that there is 1 grid-tiling and 1 coarse-tiling here:

Based on the above data, answer the given

Using the coded state representation (

) as input to a DQN with two hidden layers and a fully connected output layer, what should be the dimensionality of the network’s input layer?

Consider a 2D navigation environment (as shown in the figure below) where an autonomous cleaning robot must locate and collect trash items (T1 and T2) scattered in the environment and then deliver them to a designated disposal bin (G). The environment is shaped by curved, contour- like walls, forming nested spaces with only one narrow passage connecting them, which requires deliberate path planning.

In this environment, the robot takes discrete actions from its action space

,

{ NORTH, SOUTH, EAST, WEST } Each action attempts to move the robot by

cm in the corresponding direction. If the action would cause a collision with a wall boundary, the robot remains in the same position and receives a penalty. The grey regions represent potholes. If the robot ends up in one, it cannot escape on its own and requires human help, leading to a substantial penalty and relocation to the nearest safe position. When the agent reaches a trash item, it automatically collects it and receives a reward. Likewise, upon reaching a disposal bin, it automatically unloads the collected trash. Therefore, explicit pickup and drop actions are not required. The episode terminates either when the agent reaches the Goal (G) while carrying at least one piece of trash, or when it has fallen into potholes a total of 10 times. The position of the Robot (

) can be represented as a tuple

, where

and

are the distances of the robot from the bottom-left corner of the environment. For state representation, we would consider state discretisation by performing tile as well as coarse coding, which converts

into

.

Observe that there is 1 grid-tiling and 1 coarse-tiling here:

Based on the above data, answer the given

Using the coded state representation (

) as input to a DQN with two hidden layers and a fully connected output layer, what should be the size of the output layer?

Consider a 2D navigation environment (as shown in the figure below) where an autonomous cleaning robot must locate and collect trash items (T1 and T2) scattered in the environment and then deliver them to a designated disposal bin (G). The environment is shaped by curved, contour- like walls, forming nested spaces with only one narrow passage connecting them, which requires deliberate path planning.

In this environment, the robot takes discrete actions from its action space

,

{ NORTH, SOUTH, EAST, WEST } Each action attempts to move the robot by

cm in the corresponding direction. If the action would cause a collision with a wall boundary, the robot remains in the same position and receives a penalty. The grey regions represent potholes. If the robot ends up in one, it cannot escape on its own and requires human help, leading to a substantial penalty and relocation to the nearest safe position. When the agent reaches a trash item, it automatically collects it and receives a reward. Likewise, upon reaching a disposal bin, it automatically unloads the collected trash. Therefore, explicit pickup and drop actions are not required. The episode terminates either when the agent reaches the Goal (G) while carrying at least one piece of trash, or when it has fallen into potholes a total of 10 times. The position of the Robot (

) can be represented as a tuple

, where

and

are the distances of the robot from the bottom-left corner of the environment. For state representation, we would consider state discretisation by performing tile as well as coarse coding, which converts

into

.

Observe that there is 1 grid-tiling and 1 coarse-tiling here:

Based on the above data, answer the given

In the given environment, the state space is continuous while the action space is discrete. Now consider a similar environment where the action space is also continuous, with the action defined as the robot’s rotation angle relative to the positive horizontal axis. You decide to use a **Deterministic Policy Gradient (DPG)** method with a deterministic policy

. Which of the following statements are valid for ensuring adequate exploration and obtaining unbiased policy-gradient estimates?

Consider a 2D navigation environment (as shown in the figure below) where an autonomous cleaning robot must locate and collect trash items (T1 and T2) scattered in the environment and then deliver them to a designated disposal bin (G). The environment is shaped by curved, contour- like walls, forming nested spaces with only one narrow passage connecting them, which requires deliberate path planning.

In this environment, the robot takes discrete actions from its action space

,

{ NORTH, SOUTH, EAST, WEST } Each action attempts to move the robot by

cm in the corresponding direction. If the action would cause a collision with a wall boundary, the robot remains in the same position and receives a penalty. The grey regions represent potholes. If the robot ends up in one, it cannot escape on its own and requires human help, leading to a substantial penalty and relocation to the nearest safe position. When the agent reaches a trash item, it automatically collects it and receives a reward. Likewise, upon reaching a disposal bin, it automatically unloads the collected trash. Therefore, explicit pickup and drop actions are not required. The episode terminates either when the agent reaches the Goal (G) while carrying at least one piece of trash, or when it has fallen into potholes a total of 10 times. The position of the Robot (

) can be represented as a tuple

, where

and

are the distances of the robot from the bottom-left corner of the environment. For state representation, we would consider state discretisation by performing tile as well as coarse coding, which converts

into

.

Observe that there is 1 grid-tiling and 1 coarse-tiling here:

Based on the above data, answer the given

You have decided to use the Dueling DQN architecture with the aggregation.

What is the primary purpose of subtracting the term

in this formulation?

Consider the following Dueling DQN architecture given below that approximates the state-value function (

) and advantage for actions A and B for states

. The hidden layer nodes (grey colored nodes) employ the RELU activation function.

The weight matrices of the network are given by:

Here,

contains the weights connecting the **input layer** to the **first hidden layer**. Similarly,

contains the weights connecting the **first hidden layer**

to the **second hidden** **layer**

.

contains the weights connecting the

to the

.

contains the weights connecting the layer

to the layer

. Lastly,

contains the weights connecting the layer

to the layer

. Each entry in these matrices corresponds to a single scalar weight. For example, the entry

in

denotes the weight from neuron

in the first hidden layer to neuron

in the second hidden layer. At some timestep

, the weight matrices are as follows

Let

denote the set containing the weight matrices

and

. Let

represent the set of weight matrices

and

, and let

denote the set consisting of the weight matrix

. Recall the dueling network formulation used in the programming assignment:

We are interested in computing Q-values for the following state:

Based on the above data, answer the given

Compute the **state-value**

for state

.

Consider the following Dueling DQN architecture given below that approximates the state-value function (

) and advantage for actions A and B for states

. The hidden layer nodes (grey colored nodes) employ the RELU activation function.

The weight matrices of the network are given by:

Here,

contains the weights connecting the **input layer** to the **first hidden layer**. Similarly,

contains the weights connecting the **first hidden layer**

to the **second hidden** **layer**

.

contains the weights connecting the

to the

.

contains the weights connecting the layer

to the layer

. Lastly,

contains the weights connecting the layer

to the layer

. Each entry in these matrices corresponds to a single scalar weight. For example, the entry

in

denotes the weight from neuron

in the first hidden layer to neuron

in the second hidden layer. At some timestep

, the weight matrices are as follows

Let

denote the set containing the weight matrices

and

. Let

represent the set of weight matrices

and

, and let

denote the set consisting of the weight matrix

. Recall the dueling network formulation used in the programming assignment:

We are interested in computing Q-values for the following state:

Based on the above data, answer the given

In order to compute Q-values, we need the advantage of action A and action B for state

? Compute the following advantage vector

What is the value of

+

?

Consider the following Dueling DQN architecture given below that approximates the state-value function (

) and advantage for actions A and B for states

. The hidden layer nodes (grey colored nodes) employ the RELU activation function.

The weight matrices of the network are given by:

Here,

contains the weights connecting the **input layer** to the **first hidden layer**. Similarly,

contains the weights connecting the **first hidden layer**

to the **second hidden** **layer**

.

contains the weights connecting the

to the

.

contains the weights connecting the layer

to the layer

. Lastly,

contains the weights connecting the layer

to the layer

. Each entry in these matrices corresponds to a single scalar weight. For example, the entry

in

denotes the weight from neuron

in the first hidden layer to neuron

in the second hidden layer. At some timestep

, the weight matrices are as follows

Let

denote the set containing the weight matrices

and

. Let

represent the set of weight matrices

and

, and let

denote the set consisting of the weight matrix

. Recall the dueling network formulation used in the programming assignment:

We are interested in computing Q-values for the following state:

Based on the above data, answer the given

Compute the Q-value of action A and action B for state

. What is the value of

+

?

Consider the following navigation problem:

The agent is spawned at the Start state, and has 4 actions on each state (Up, Down, Left, Right). Teleport to X is a special state; if the agent lands on this state, it is **instantly** teleported to the state marked X. Every action from every state, excluding the End state, incurs a reward of -1 (Episode terminates once the agent reaches the End state) A hierarchical learning approach is proposed, with the following hierarchy:

where zone A and B correspond to rows 2-4 and 5-7 on the grid, respectively. Based on the above data, answer the given

Which of the routes illustrated on the grid is taken when the Hierarchically Optimal policy is executed?

Consider the following navigation problem:

The agent is spawned at the Start state, and has 4 actions on each state (Up, Down, Left, Right). Teleport to X is a special state; if the agent lands on this state, it is **instantly** teleported to the state marked X. Every action from every state, excluding the End state, incurs a reward of -1 (Episode terminates once the agent reaches the End state) A hierarchical learning approach is proposed, with the following hierarchy:

where zone A and B correspond to rows 2-4 and 5-7 on the grid, respectively. Based on the above data, answer the given

Which of the routes illustrated on the grid is taken when the Recursively Optimal policy is executed?

Consider the following navigation problem:

The agent is spawned at the Start state, and has 4 actions on each state (Up, Down, Left, Right). Teleport to X is a special state; if the agent lands on this state, it is **instantly** teleported to the state marked X. Every action from every state, excluding the End state, incurs a reward of -1 (Episode terminates once the agent reaches the End state) A hierarchical learning approach is proposed, with the following hierarchy:

where zone A and B correspond to rows 2-4 and 5-7 on the grid, respectively. Based on the above data, answer the given

Calculate the number of time-steps taken to finish the episode when the flat optimal policy is executed.