Reinforcement Learning · Quiz 1 · Sep 2025

Enter the correct answer for Blank (b) ________________ **NOTE:** Enter the exact answer without any extra space in the beginning or at the end.

Consider following assertion reason pair: **Assertion:** In the UCB algorithm for multi-armed bandits, replacing the upper confidence bound with a lower confidence bound (and greedily selecting actions based on that lower bound) would still promote effective exploration or lead to optimal reward maximisation. **Reason:** Even when lower confidence bounds are used instead of upper bounds, if the algorithm greedily selects arms based on these lower bounds, it would still encourage exploration and ultimately achieve optimal reward maximisation.

Which of the following statements correctly describes the differences in computational complexity and convergence behaviour between Policy Iteration and Value Iteration algorithms in solving Markov Decision Processes?

Suppose you face a 2-armed bandit task where, at each time step, the true action values are either (10, 20) with probability 0.7 (case A) or (90, 80) with probability 0.3 (case B). Based on the above data, answer the given subquestions.

If you cannot observe which case you face, what is the best expected reward per step you can achieve, and what strategy should you follow?

Suppose you face a 2-armed bandit task where, at each time step, the true action values are either (10, 20) with probability 0.7 (case A) or (90, 80) with probability 0.3 (case B). Based on the above data, answer the given subquestions.

If, at each time step, you are told whether you face case A or case B (but not the true action values), what is the best expected reward per step you can achieve,and what strategy should you follow?

Based on the above data, answer the given subquestions.

Which of the following best explains why a contextual bandit approach is preferred over a standard multi-armed bandit in this scenario?

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

What is the main optimality goal of a contextual bandit algorithm over many rounds?

Based on the above data, answer the given subquestions.

What is the total reward if the robot collects both treasures and then exits,taking the shortest possible path and never hitting a wall?

Based on the above data, answer the given subquestions.

Consider a pole-balancing task where the goal is to apply forces to a cart moving along a track to keep a hinged pole from falling. A failure event occurs when the pole falls past a certain angle or the cart moves off the track, after which the pole resets to vertical.

The problem can be formulated either as an episodic task (with episodes ending at failure) or as a continuing task (after each failure, pole gets back into its initial position, potentially infinite time horizon with discounting).Rewards can be defined as: **• Episodic:** Reward +1 for each timestamp without failure, so the return is the number of steps until failure (possibly infinite if balanced forever). **• Continuing:** Reward 0 for each timestamp except at each failure, where the reward is −1. Returns are discounted sums of future rewards. Based on the above data, answer the given subquestions.