Reinforcement Learning: Theory and Algorithms

• It is helpful to overload notation and let P also refer to a matrix of size (S· A)× S where the entry P (s, a), s is equal to P (s| s, a). We also will define Pπ to be the transition matrix on state-action pairs induced by a deterministic policy π. In particular, Pπ (s, a),(s, a)= P (s| s, a) if a= π (s) and Pπ(s, a),(s, a)= 0 if a= π (s). With this notation,