Parallelization Of Software Pipelines Using The Mpififo Tool

Software pipelines are often used in scientific applications in fields such as bioinformatics. Unix pipes allow processes in the pipeline to concurrently execute and communicate without disk I/O on a single shared-memory node. However, a significant number of CPU-bound, memory-bound, and I/O-bound software pipelines could benefit from the distribution of work across multiple nodes. Rewriting applications to use MPI is often time consuming. mpififo is an MPI-based tool that allows for the easy distribution of the processes in a pipeline across multiple nodes. This paper describes the design of mpififo, and presents an example of its use in a bioinformatics application. mpififo is available for download at https://bitbucket.org/nathanweeks/mpififo.