DL4J has a dependency with JavaCPP that acts as a bridge between Java and C++: https://github.com/bytedeco/javacpp.

JavaCPP works on the basis of the -Xmx value set on the heap space (off-heap memory) and the overall memory consumption will not exceed this value. DL4J seeks help from the garbage collector and JavaCPP to deallocate memory.

For training sessions with large amounts of data involved, it is important to have more RAM for the off-heap memory space than for on-heap memory (JVM). Why? Because our datasets and computations are involved with INDArrays and are stored in the off-heap memory space.

It is important to identify the memory limits of running applications. The following some instances where the memory limit needs to be properly configured:

• For GPU systems, maxbytes and maxphysicalbytes are the important memory limit settings. We are dealing with off-heap memory here. Allocating reasonable memory to these settings allows us to consume more GPU resources.
• For RunTimeException that refer to memory allocation issues, one possible reason may be the unavailability of off-heap memory spaces. If we don't use the memory limit (off-heap space) settings discussed in the Setting up heap space and garbage collection recipe, the off-heap memory space can be reclaimed by the JVM garbage collector. This can then cause memory allocation issues. 
• If you have limited-memory environments, then it is not recommended to use large values for the -Xmx and -Xms options. For instance, if we use -Xms6G for an 8 GB RAM system, we leave only 2 GB for the off-heap memory space, the OS, and for other processes.