Temporal Difference Learning

Description:

• A model-free way of doing Policy Evaluation, mimicking Bellman updates with running sample averages
• Idea: learn from every experience
  • Update $V(s)$ each time we experience a transition $(s,a,s’,r)$
  • Likely outcome $s^{\prime }$ will contribute updates more often than the other
• TD learning tries to answer the question of how to compute this weighted average without the weights, cleverly doing so with Exponential Moving Average in RL
• Problem:
  • If we want to turn values into a new policy, we are stuck as we dont have $T$ and $R$ so no $Q$
• Solution:
  • Learn Q values instead, Q-learning
  • Make action selection also model-free

Temporal difference of value:

• Policy still fixed, still doing evaluation
• Move values toward value of whatever successor occurs: running average
• Sample of $V(s):\text{sample}=R(s,\pi (s),s^{\prime })+\gamma V^{\pi }(s^{\prime })$
• Update to $V(s)):V^{\pi }(s)\leftarrow (1-\alpha )V^{\pi }(s)+(\alpha )\text{sample}$
  • where $\alpha$ is Learning Rate
• Same update: $V^{\pi }(s)\leftarrow V^{\pi }(s)+\alpha (\text{sample}-(1-V^{\pi }(s))$