The Predictron: End-To-End Learning and Planning.

One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple imagined planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function, thereby focusing the model upon the aspects of the environment most relevant to planning. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. predictron yielded significantly more accurate predictions than conventional deep neural network architectures.