I have a java server (1 process) that spawns a thread for every incoming connection. I know there is a file descriptor limit of 1024 that's compiled into the kernel.
Is there a limit to how many socket connections 1 process can support?
Besides the OS file handle limits mentioned by hroptatyr, you may also want to make sure that you are using a raised value for the connection backlog on the listener socket. This will allow more connections to be queued up before the OS returns "Connection refused", should your server application become momentarily busy and unable to respond to incoming connections quickly enough.
If you're using ServerSocket in Java, you can specify the desired backlog as the argument to the constructor:
ServerSocket server = new ServerSocket(listenPort, 50);