For modern HSM appliances, NTLS uses 2048-bit client/server certificates for client connections, rather than the 1024-bit certs that were considered secure in the past.
This larger certificate size requires more overhead/system resources than before.
For a single connection or just a few simultaneous connection setups, the increased overhead is insignificant.
However, in a stress environment where (say) hundreds of concurrent connections are launched at once, you might see connections fail. The appliance attempts to get to all the incoming requests, but inevitably experiences delay on some. It eventually does get to all the session-open requests, but in a very intense flurry of session-opening, it might be returning responses to a given client after that client has timed out some of its own requests.
Once connections are set up, they can remain open and working with no problem up to the limit allowed by the appliance - 800 concurrent connections.
WORKAROUND: Ensure that your application does not attempt to open hundreds of client connections all at the same time (space the setups over time - the problem is not how many sessions are open, but how many are in the startup process at the same time).
Or if high-volume simultaneous launch of sessions from a single client is unavoidable, then increase the receive timeout value (at the client) from the default 20 seconds to some larger value that eliminates the problem for you.
The obvious trade-off is that, the higher the receive timeout value is set on each client, the longer it takes for failed connection attempts to be recognized and corrective measures to be taken.