Kiwipy: Robust, High-Volume, Messaging for Big-Data and Computational Science Workflows

In this work we present kiwiPy, a Python library designed to support robustmessage based communication for high-throughput, big-data, applications whilebeing general enough to be useful wherever high-volumes of messages need to becommunicated in a predictable manner. KiwiPy relies on the RabbitMQ protocol,an industry standard message broker, while providing a simple and intuitiveinterface that can be used in both multithreaded and coroutine basedapplications. To demonstrate some of kiwiPy's functionality we give examplesfrom AiiDA, a high-throughput simulation platform, where kiwiPy is used as akey component of the workflow engine.