MEX: A Distributed Computing Framework for Executable Programs

Parallel computing can improve the data-processing efficiency significantly. However, the traditional approaches, such as MPI and MapReduce, need to program in the special environment. In this paper, a new distributed computing framework named MEX is proposed. Users just provides the input files and the name of an executable program to MEX. Then MEX will automatically process these files on a cluster of machines with the executable program. The MEX platform has been designed and implemented based on MapReduce and some key problems are addressed. An improved map function are designed for the start-up of the executable program. To support the improved map function, a data-conversion mechanism is added into MEX which generates the command texts as the parameter of the map function. A process-feedback mechanism is proposed for the fault-tolerance of the executable program. The mechanism also supports the synchronous execution between the map task and the executable program, which can avoid too many processes to be started on the same worknode. Comprehensive experiments are performed to verify the effectiveness of the MEX framework. According to the results, more computing worknodes can result in less job runtime in MEX. When 100 virtual machines are used for an OCR job with 1000 images in 400 dpi, the runtime is reduced 88.6% compared to a single machine.