Running the SkyMapper Science Data Pipeline: To be a Big Fish in a Small Pond, or a Small Fish in a Big Ocean?
Astronomical Society of the Pacific Conference Series(2017)
摘要
We review structure and frameworks behind the SkyMapper Science Data Pipeline (SDP), and consider the challenges of deploying on two disparate platforms: a publicly shared, massively parallel, queue-scheduled compute fabric, and a dedicated NUMA-based, multi-core, mini-supercomputer. Concepts reviewed include a) how to impose a layer of central operator control over hundreds of jobs of varying type and CPU/IO profile, all running concurrently and at different stages in their logic, b) how to maintain configuration control in an ever-changing algorithmic environment while not giving up ease of build and deployment, and c) how to configure a NUMA-architected machine for optimal cache buffer usage, process-to-memory locality, and user/system CPU cycle ratio.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要