Scispark: Highly Interactive In-Memory Science Data Analytics

We present further work on SciSpark, a Big Data framework that extends Apache Spark's in-memory parallel computing to scale scientific computations. SciSpark's current architecture and design includes: time and space partitioning of high-resolution geo-grids from NetCDF3/4; a sciDataset class providing N-dimensional array operations in Scala/Java and CF-style variable attributes (an update of our prior sciTensor class); parallel computation of time-series statistical metrics; and an interactive front-end using science (code & visualization) Notebooks. We demonstrate how SciSpark achieves parallel ingest and time/space partitioning of Earth science satellite and model datasets. We illustrate the usability, extensibility, and early performance of SciSpark using several Earth science Use cases, here presenting benchmarks for sciDataset Readers and parallel time-series analytics. A three-hour SciSpark tutorial was taught at an ESIP Federation meeting using a dozen "live" Notebooks.