VERIFIED SOLUTION i

Resolve "ERROR [STDERR] java.net.BindException: Address already in use: connect" in EngageOne Vault

Issue

Vault Retrieval failures were observed during peak time.

Application server reported the following error during failure periods:
ERROR [STDERR] java.net.BindException: Address already in use: connect

Cause

The error is potentially caused by TCP/IP port exhaustion.

Under very high load a server that repeatedly connects to other services could run out of ephemeral ports.

Each interface is limited to a certain reserved range of ports for outgoing dynamic connections.

Port numbers that have just closed are kept in a wait state for a period of time before it can be reused.

If the rate of connections is high enough all the ports end up in a wait state and the operating system cannot bind your connections to a port anymore.

If there is a high turnover rate, this may be the cause.

Resolution

UPDATED: November 10, 2017
As troubleshooting steps, we can use the followings to ascertain if port exhaustion is the actual bottleneck:
1. Under peak load, running "netstat -ano -p tcp" will list all of the ports in use.
2. We can also monitor the available ports in the range using "netsh int ipv4 show dynamicport tcp".

After following above troubleshooting steps, if port exhaustion is confirmed to be the root cause then in order to permanently fix the issue we need to follow the steps described in http://blogs.msdn.com/b/dgorti/archive/2005/09/18/470766.aspx, for adding extra ports and shorten the life-cycle of ports in the Windows registry settings.

 
Environment Details
Product Feature: General