Characterizing I/O optimization opportunities for array-centric applications on HDFS

Donghe Kang,Vedang Patel, Kalyan Khandrika,Spyros Blanas,Yang Wang,Srinivasan Parthasarathy

2018 IEEE High Performance extreme Computing Conference (HPEC)（2018）

引用 1|浏览31

暂无评分

摘要

An impedance mismatch exists between the increasing sophistication of array-centric analytics and the bytestream-based POSIX interface of parallel file systems. This mismatch is particularly acute in data-intensive scientific applications. This paper examines performance bottlenecks and describes optimizations to alleviate them in the context of computational astronomy pipelines and the Hadoop distributed file system (HDFS). We find that fast data ingestion and intelligent object consolidation promise to accelerate I/O performance by two orders of magnitude.

查看译文

关键词

computational astronomy pipelines,Hadoop distributed file system,HDFS,fast data ingestion,array-centric applications,impedance mismatch,array-centric analytics,bytestream-based POSIX interface,parallel file systems,data-intensive scientific applications,performance bottlenecks,characterizing I/O optimization opportunities,intelligent object consolidation

AI 理解论文

溯源树

样例

生成溯源树，研究论文发展脉络

Chat Paper

正在生成论文摘要