Issue
Restarting the Vault Server in the DEV environment and the below error was encountered in the e2serverd log when it started up and it started giving the Socket errors when we tried to connect to Vault for retrieving documents from Vault. The server was stopped and started again and it resolved the issue.
20:19:39 open log, date [2014/02/04], file
[/opt/apps/e2vault/server/log/e2serverd.20140204.201939.15599.log]
20:19:39 Vault Server 6.1M0p0114
20:19:39 (C) Copyright 1993-2012 Pitney Bowes Software Inc.
20:19:39 descriptor limits: soft [1024], hard [1024], want [1024]
20:19:39 pid file [/opt/apps/e2vault/server/log/e2serverd.pid]
20:19:39 starting
20:19:39 Vault Server licenced to [Customer]
20:19:40 <log1> start
20:19:40 <manager> start
20:19:40 <cache1> start
20:19:40 <pool1> start
20:19:40 <stat1> start
20:19:40 <authclient1> start
20:19:40 <authserver1> start
20:19:40 <compressed1> start
20:19:40 <connection1> start
20:19:40 <database1> start
20:19:40 <legacyreply1> start
20:19:40 <licence1> start
20:19:40 <resource1> start
20:19:40 <server1> start
20:19:40 <storage1> start
20:19:40 <server1> [1] listening, [0] open
20:19:40 0.0.0.0:6001 ERROR 71101: code [0x02006062] library [system
library] function [bind] reason [Address already in use]
20:19:40 0.0.0.0:6001 ERROR 71101: code [0x20069075] library [BIO routines]
function [BIO_get_accept_socket] reason [unable to bind socket]
20:19:40 0.0.0.0:6001 ERROR 71110: unable to listen on socket
20:19:40 0.0.0.0:6001 <server1> end listen, accepted [0], duration [1] ms
20:19:40 <server1> [0] listening, [0] open, [0] stopping, [0] stopped,
resetlevel [0]
20:19:40 started
The below error was found in the e2loaderd logs:
23:06:00 <connection1> notice.cacheflush request
23:06:00 ERROR 71101: code [0x0200206F] library [system library] function
[connect] reason [Connection refused]
23:06:00 ERROR 71101: code [0x2007306E] library [BIO routines] function
[CONN_STATE] reason [nbio connect error]
23:06:00 ERROR 71117: unable to connect on socket, error [115]
23:06:00 <connection1> ERROR 70310: socket failure connecting to
[xxx.0.0.1:6001], error [71101]
23:06:03 ERROR 71101: code [0x0200206F] library [system library] function
[connect] reason [Connection refused]
23:06:03 ERROR 71101: code [0x2007306E] library [BIO routines] function
[CONN_STATE] reason [nbio connect error]
23:06:03 ERROR 71117: unable to connect on socket, error [115]
23:06:03 <connection1> ERROR 70310: socket failure connecting to
[xxx.0.0.1:6001], error [71101]
23:06:06 ERROR 71101: code [0x0200206F] library [system library] function
[connect] reason [Connection refused]
23:06:06 ERROR 71101: code [0x2007306E] library [BIO routines] function
[CONN_STATE] reason [nbio connect error]
23:06:06 ERROR 71117: unable to connect on socket, error [115]
23:06:06 <connection1> ERROR 70310: socket failure connecting to
[xxx.0.0.1:6001], error [71101]
23:06:06 <connection1> socket failure connecting to [xxx.0.0.1:6001], error
[71101]
23:06:06 <connection1> notice.cacheflush failed, status [70310]
Cause
In this case it is likely caused by a connection that was connected to the server that is still in one of the TCP wait states.
You can see this condition using netstat:
[user@dev server]$ netstat -an | grep :6001
tcp 0 0 161.228.78.78:6001 10.27.24.194:62381 FIN_WAIT2
Resolution
UPDATED: July 26, 2017This is error number 98 which is an address in use error.
Contact support for further assistance.
While there is a connection using this port number that is still waiting, the Vault process won't be able to bind to the port.
You can run into this situation when quickly restarting Vault processes.
Contact support for further assistance.