We consider the reinforcement learning problem under partial observability, where observations in the decision process lack the Markov property. To cope with partial observability, first we must detect it. We introduce the $\lambda$-discrepancy: a measure of the degree of non-Markovianity of system dynamics. The $\lambda$-discrepancy is the difference between TD($\lambda$) value functions for two different values of $\lambda$; for example, between 1-step temporal difference learning (TD($0$)), which makes an implicit Markov assumption, and Monte Carlo value estimation (TD($1$)), which does not. We prove that this observable and scalable value-based measure is a reliable signal of partial observability. We then use it as an optimization target for resolving partial observability by searching for memory functions---functions over the agent's history---to augment the agent's observations and reduce $\lambda$-discrepancy. We empirically demonstrate that our approach produces memory-augmented observations that resolve partial observability and improve decision making.
分享
引用
