A Community Cache With Complete Information

Kariz is a new architecture for caching data from datalakes accessed, potentially concurrently, by multiple analytic platforms. It integrates rich information from analytics platforms with global knowledge about demand and resource availability to enable sophisticated cache management and prefetching strategies that, for example, combine historical run time information with job dependency graphs (DAGs), information about the cache state and sharing across compute clusters. Our prototype supportsmultiple analytic frameworks (Pig/Hadoop and Spark), and we show that the required changes aremodest. We have implemented three algorithms in Kariz for optimizing the caching of individual queries (one from the literature, and two novel to our platform) and three policies for optimizing across queries from, potentially, multiple different clusters. With an algorithmthat fully exploits the rich information available from Kariz, we demonstrate major speedups (asmuch as 3x) for TPC-H and TPC-DS.