Spark Resource Calculator

After optimizing a few spark jobs, I realized that calculating the executor memory space is an extremely manual process.

I decided to write a simple UI that calculates an executor’s on-heap memory space (e.g. spark/executor/storage memory). I might extend this in the future to also include the executor container’s entire memory space (e.g. executor memory, overhead, and off-heap).

You can find the calculator by heading to agbares.com/spark-resource-calculator-ui