I sketch here the components inside Spark and their dependencies so you can have a general overview of Spark.
Each component is in charge of a particular function (of course). Straightforwardly, you can understand most of the components and their functions. I just explain some components that “not easy” to understand.
– repl: the interractive shell for spark, like spark-shell, pyspark…
– bagel: Spark implementation of Google’s Pregel graph processing framework. It will be replaced by GraphX.
– catalyst: a query optimization framework for Spark.
Next “ARCH” posts will focus on the most important components of Spark-Core.
Pingback: [Arch] SparkContext and its components | Quang-Nhat HOANG-XUAN
Pingback: [Sysdeg] Moving to SparkSQL, why not? | Quang-Nhat HOANG-XUAN