Rewards and Return

Written by Zhongming Yu

Reward Signal

The reward $r_t = R(s_t, a_t)$ is a scalar signal that tells the agent how good its action was. Rewards can be:

Sparse — only at the end (e.g., win/lose in a game)
Dense — at every step (e.g., distance to goal)
Shaped — hand-crafted to guide learning (careful: can introduce bias)

For language models, the reward typically comes from a reward model trained on human preferences, or from a rule-based function (e.g., format checking, code execution results).

Return: Cumulative Reward

The return $G_t$ is the total reward from timestep $t$ onward:

G_t = r_t + r_{t+1} + r_{t+2} + \cdots = \sum_{k=0}^{\infty} r_{t+k}

But this sum can diverge. Enter the discount factor.

Discount Factor $\gamma$

The discounted return weights future rewards exponentially less:

G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k}

where $\gamma \in [0, 1)$ . The discount factor controls:

$\gamma$	Behavior
$\gamma \to 0$	Myopic — only cares about immediate reward
$\gamma \to 1$	Far-sighted — values future rewards almost equally

With constant reward $r = 1$ , the return converges to $\frac{1}{1 - \gamma}$ .

Worked Example

Sparse-reward episode with $r = [0, 0, 0, 0, 0, 0, 0, 0, 0, +1]$ (reward only at $t = 9$ ):

$\gamma$	$G_0 = \gamma^9$	Effective horizon $1/(1-\gamma)$
0.5	$0.5^9 \approx 0.002$	2 steps
0.9	$0.9^9 \approx 0.387$	10 steps
0.99	$0.99^9 \approx 0.914$	100 steps

At $\gamma = 0.5$ the agent’s gradient signal at $t=0$ is essentially zero — the reward is “out of view.” This is one of the reasons sparse-reward RL is hard: pick $\gamma$ smaller than the actual reward horizon and the problem becomes unlearnable, even with a perfect algorithm.

Interactive: Edit a Reward Sequence

Add/remove timesteps, type rewards, slide $\gamma$ . The three plotted curves are $G_t$ under $\gamma \in \{0.5, \text{your }\gamma, 0.99\}$ . Try a reward at $t=20$ with $\gamma = 0.9$ — see how $G_0$ practically vanishes.

Value Functions

The state-value function $V^\pi(s)$ is the expected return from state $s$ under policy $\pi$ :

V^\pi(s) = \mathbb{E}_\pi \left[ G_t \mid s_t = s \right]

The action-value function $Q^\pi(s, a)$ is the expected return from taking action $a$ in state $s$ , then following $\pi$ :

Q^\pi(s, a) = \mathbb{E}_\pi \left[ G_t \mid s_t = s, a_t = a \right]

The advantage function measures how much better an action is compared to the average:

A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)

This advantage function will be central to policy gradient methods and PPO.