Reinforcement Learning: A Growing Force in Machine Learning

Reinforcement Learning (RL) has been making waves, particularly with DeepMind's impressive achievements like AlphaZero mastering chess and MuZero excelling in Go and Atari games without prior knowledge of the rules. While these are groundbreaking, the real significance of RL is becoming increasingly apparent in practical, production environments.

RL's Impact at Lyft: A Turning Point

Seeing RL used in production at Lyft, directly impacting the bottom line, sparked a deep dive into the subject. This experience led to a strong conviction: **over the next decade, many machine learning systems will transition from supervised learning to RLbased systems.** This is because the problem statement of RL is so general that, as solutions improve, RL will become the most effective and relevant tool.

From Skepticism to Optimism: The Evolution of RL's Perception

Back in 2018, a Google robotics engineer, Alexander Urban, wrote a blog post highlighting the challenges of RL, noting its data hunger, instability, and the extensive tuning required. He argued it wasn't practical for realworld problems. However, when contacted recently, Urban acknowledged that **RL is being adopted in production at major tech companies**, and he's optimistic about its trajectory. While he doesn't believe RL will be as revolutionary as neural networks, its growing relevance is undeniable.

RL in Action: RealWorld Examples

RL is no longer just a theoretical concept. Recent applications demonstrate its tangible benefits:

• Nvidia uses deep RL to design more efficient arithmetic circuits.
• DeepMind uses RL to assist in nuclear fusion operations, manipulating magnets to contain nuclear plasma.
• Siemens Energy employs RL to manage the energy efficiency and emissions of gas turbines, a solution tested and implemented with customers.

**These examples prove that RL is being used productively, suggesting a continued growth trend that makes knowledge of RL highly valuable.**

Announcing a Reinforcement Learning Series

To help you gain a solid understanding of RL, a sixpart series is being released. This series will cover the fundamentals, drawing heavily from the seminal textbook, Reinforcement Learning: An Introduction by Sutton and Barto. While the book provides a truly deep understanding, the videos offer an efficient way to grasp the core concepts, particularly from Part One and some of Part Two.

The Core of Reinforcement Learning: The Problem Statement

Let's start with the most important aspect: the RL problem statement. It revolves around an agent, which learns and takes actions. At each time step (T), the agent performs an action, which the environment receives. In response, the environment provides a reward and a state at the next time step. This cycle repeats continuously.

Key Components and Notation

• **Time (T):** Discrete, indexed as 0, 1, 2, and so on.
• **State (S):** Represented by 's' (a specific value) from the set 'S' (all possible states).
• **Action (A):** Represented by 'a' (a specific value) from the set 'Cal A(s)' (possible actions in a given state).
• **Reward (R):** Represented by 'r' (a specific value) from a finite subset of the real number line.

The Dynamics of Interaction: Markov Decision Processes (MDPs)

The agentenvironment interaction is defined by a distribution function that specifies the probability of the next state and reward, given the current state and action. **A critical simplification is the Markov property:** the probability of the next state and reward depends only on the current state and action, not on the history. This function, along with the sets of states, actions, and rewards, defines a finite Markov Decision Process (MDP), the fundamental object in RL.

Agent Behavior: The Policy

The agent's behavior is governed by a policy, which dictates the probability of taking each action in a particular state. The agent samples an action based on this probability distribution. If the policy is deterministic, only one action is possible from each state.

The Goal: Maximizing the Return

The aim is to find a policy that maximizes the accumulated reward, called the return (GT). This is a sum of future rewards, potentially discounted to give more weight to immediate rewards. The discount parameter (gamma), between 0 and 1, controls this weighting. A gamma closer to zero emphasizes nearterm rewards.

In the episodic case, the process runs until reaching a terminal state (T), marking the end of an episode. The goal, in essence, is to select a policy that maximizes the expected return – the average return over many episodes.

Visualizing the Process: An Example

Imagine an environment where states are represented by different shades of red, and the agent can move 'up' or 'down'. The MDP probability function dictates the distribution over the next state and reward for each stateaction combination. A starting distribution determines the initial state, and the policy guides action selection. Sampling from the appropriate distributions yields a trajectory of states, actions, and rewards.

In reality, the policy often evolves during the learning process, adding complexity compared to supervised learning.

State Value and Action Value Functions: Key to Optimal Policies

Two crucial functions help determine optimal policies:

• State Value Function (Vπ(s)): The expected return given the agent is in state 's' and follows policy 'π'.
• Action Value Function (Qπ(s, a)): The expected return given the agent is in state 's', takes action 'a', and then follows policy 'π'.

These functions are closely related to the goal of maximizing return and are essential for finding optimal policies, although in practice, they often need to be estimated.

Optimal Policies and Their Value Functions

An optimal policy (π*) achieves the highest possible expected return for all states. Its value functions provide instructions for optimal behavior. By selecting the action that leads to the highest value state (or stateaction pair, for action value functions), an agent can navigate towards the best possible outcome.

Assumptions and Limitations

The RL setup relies on certain assumptions:

• The agent observes the state at each time step.
• The state contains all necessary information to predict the future (Markov property).

**These assumptions rarely hold perfectly in the real world**, where agents have limited observations and incomplete knowledge. Other assumptions, like being able to list all possible states and actions (the tabular case), and having complete knowledge of the MDP's dynamics, are sometimes relaxed.

Despite these limitations, the established problem setup is vital because it provides a solid theoretical foundation for discovering optimal policies. Even when assumptions are relaxed, the underlying theory offers valuable intuition.

Looking Ahead

The journey continues in the next video, which explores clever algorithms that can lead us towards globally optimal solutions.