It's difficult to tell from your question, but you may be looking for the -Xmx option in Sun's Oracle's java tool, which sets a maximum size for the Java memory heap.
For details on the various non-standard features offered by that version of the java tool (including several related to the heap), type java -X at a command prompt. For my installed version, that gives me:
java -X
-Xmixed mixed mode execution (default)
-Xint interpreted mode execution only
-Xbootclasspath:<directories and zip/jar files separated by :>
set search path for bootstrap classes and resources
-Xbootclasspath/a:<directories and zip/jar files separated by :>
append to end of bootstrap class path
-Xbootclasspath/p:<directories and zip/jar files separated by :>
prepend in front of bootstrap class path
-Xnoclassgc disable class garbage collection
-Xincgc enable incremental garbage collection
-Xloggc:<file> log GC status to a file with time stamps
-Xbatch disable background compilation
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
-Xss<size> set java thread stack size
-Xprof output cpu profiling data
-Xfuture enable strictest checks, anticipating future default
-Xrs reduce use of OS signals by Java/VM (see documentation)
-Xcheck:jni perform additional checks for JNI functions
-Xshare:off do not attempt to use shared class data
-Xshare:auto use shared class data if possible (default)
-Xshare:on require using shared class data, otherwise fail.
The -X options are non-standard and subject to change without notice.
If you're using a different implementation of Java, those options may vary; and note the warning at the end of the above.