Hello

After optimizing a few spark jobs, I realized that calculating the executor memory space is an extremely manual process. I decided to write a simple UI that calculates an executor’s on-heap memory space (e.g. spark/executor/storage memory). I might extend this in the future to also include the executor container’s entire memory space (e.g. executor memory, overhead, and off-heap). You can find the calculator by heading to agbares.com/spark-resource-calculator-ui

Over the past year, I’ve been building a fair amount of Spark ETL pipelines at work (via pyspark). The complexity of the pipelines I build have been growing. Evidently, this complexity required a better understanding in Spark’s inner workings. After a lot of reading, youtube videos, and docs, I think I have a beter grasp on Spark’s memory model. A lot of the information online can be quite confusing. And frankly, incorrect or out of date....