After a lot of discussions, we decided to change the design of the system a little bit, so it can be more general and extensible. The new design is described as the figure above. The WorkSharing Detector remains the same as the old design. Its goal is generating bags of DAGs which are labeled with … Continue reading [SysDeg] Worksharing Framework and its design: Some modifications
Long time no see! After one month playing with caching in Spark, I learned many valuable lessons (which will be posted on other blog posts, about Cache Manager and Block Manager of Spark). Our team came back to the design of the system - spark SQL server. To be honest, i spent too much time … Continue reading [SysDeg] Worksharing Framework and its design – Part 3: Prototype for the first version
Maybe you still remember the draft design of the system I proposed here. The reason why I delayed posting the part-2, which mostly focuses on technical details, because Spark is new for me so I need time to dig more into it. However, the design won’t be changed so much, I think. Come back to … Continue reading [Sysdeg] Moving to SparkSQL, why not?