Apache Spark: Cluster Computing with Working Sets

wait, what happens when results are too big for memory?

One more point I would like to add, RDD is made of Trie data structure that makes RDD to get the lineage and evaluate. if you can find out the RDD and Data Frame architecture docs and make a video it would be great help.

Thanks

what makes you say that backprop will benefit by 1000x with spark? can you share the source?

Hi Gaurav can you please do one on Druid

Nice one. Can you also do a deep dive on Google Dataflow and compare with Spark Streaming?

Bro if you have time please make a single video with everything covered up in your channel as a full video to system design

It will be more helpful 🙂

Hi Gaurav, can you please do apache airflow

Thanks Gaurav. I often work with AWS EMR (EC2) along with spark-3. 4.1 in my job. We have large batch processing clusters running on a daily scheduled Airflow DAGs. Acc to me if we use MapReduce with Spark it leverages computation with processing speed at a large scale.

Present Sir :)