Add the --executor-memory command-line argument while submitting a job to spark-submit to set on-heap memory for the worker node. For example, we could use --executor-memory 4g to allocate 4 GB of memory.
Add the --conf command-line argument to set the off-heap memory for the worker node:
Add the --conf command-line argument to set the off-heap memory for the master node. For example, we could use --conf "spark.driver.memoryOverhead=-Dorg.bytedeco.javacpp.maxbytes=8G" to allocate 8 GB of memory.
Add the --driver-memory command-line argument to specify the on-heap memory for the master node. For example, we could use --driver-memory 4g to allocate 4 GB of memory.
Configure garbage collection for the worker nodes by calling workerTogglePeriodicGC() and workerPeriodicGCFrequency() while you set up the distributed neural network using SharedTrainingMaster:
new SharedTrainingMaster.Builder(voidConfiguration, minibatch) .workerTogglePeriodicGC(true) .workerPeriodicGCFrequency(frequencyIntervalInMs) .build();
Enable Kryo optimization in DL4J by adding the following dependency to the pom.xml file: