Big Data Programming with Apache Spark

In this chapter we give an introduction to Apache Spark, a Big Data programming framework. We describe the framework's core aspects as well as some of the challenges that parallel and distributed computing entail. No statistical background is required and neither are any other data analysis skills. It is, however, encouraged for the reader to be familiarized with a functional programming language (e.g. Scala) or with the concept of lambda functions (anonymous functions)-for these are used across most examples. Spark is built on the Scala programming language,(1) hence Scala is the language of choice for the examples given.