PyCRM: A Python Library for Reward Machine-Based Reinforcement Learning

Reinforcement Learning (RL) research often models environments as Markov decision processes. Yet many real-world tasks are non-Markovian, requiring memory of past events. A common approach encodes these history-dependent rewards as finite automata, but these are difficult to integrate into standard RL pipelines. We present PyCRM, a Python framework for non-Markovian RL built on two automata classes: Reward Machines and their Turing-complete generalisation, Counting Reward Machines. PyCRM supports automatic environment construction and integrates seamlessly with popular RL libraries. This lowers the barrier to entry, allowing researchers to focus on task design and theory while exploring more complex, temporally extended problems.