RabbitMQ - An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

SYMPTOMS

At some point during eDiscovery processing an out of memory error could occur causing the processing progress to halt.

The process is no longer able to connect to the ports of RabbitMQ.

Whenever RabbitMQ hits the predefined limit it will shut down all communication which assures that ediscovery no longer works.
An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

 --

The detailed error that is displayed in the ZyLAB.EDiscovery.DataService.Processing.exe.log:

SERVERNAME [15][4836] ERROR ZyLAB.EDiscovery.ProcessingService.Common.Proxy.ProxyWrapper - Failed to save an error due to following exception:
System.InsufficientMemoryException: Insufficient winsock resources available to complete socket connection initiation. ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 10.0.1.150:4113
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
--- End of inner exception stack trace ---

Server stack trace:
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

RabbitMQ: startup log
RabbitMQ =INFO REPORT
Limiting to approx 8092 file handles (7280 sockets)

INFO REPORT:
Memory limit set to 24571MB of 61429MB total.

CAUSE

Whenever rabbitmq hits the predefined limit it will shut down all communication which assures that ediscovery no longer works.

RabbitMQ reserves memory at start up , our processors and the SQL db will show increasing memory usage.

RESOLUTION

RabbitMQ needs at least 1 gigabyte but preferably more, (about 4 gigabyte).

Reserve sufficient memory for both SQL and RabbitMQ and preserve enough memory for the running system.

In general it is better to split server roles to prevent such memory errors.

Regarding disk/memory management (RabbitMQ documentation):

The RabbitMQ server detects the total amount of RAM installed in the computer on startup and when rabbitmqctl set_vm_memory_high_watermark fraction is executed.
By default, when the RabbitMQ server uses above 40% of the installed RAM, it raises a memory alarm and blocks all connections. Once the memory alarm has cleared (e.g. due to the server paging messages to disk or delivering them to clients) normal service resumes.
When running RabbitMQ in a cluster, the memory alarm is cluster-wide; if one node goes over the limit then all nodes will block connections.
The default memory threshold is set to 40% of installed RAM. Note that this does not prevent the RabbitMQ server from using more than 40%, it is merely the point at which publishers are throttled. Erlang's garbage collector can, in the worst case, cause double the amount of memory to be used (by default, 80% of RAM). It is strongly recommended that OS swap or page files are enabled.

RabbitMQ will periodically check the amount of free disk space.
The frequency with which disk space is checked is related to the amount of space at the last check (in order to ensure that the disk alarm goes off in a timely manner when space is exhausted).
Normally disk space is checked every 10 seconds, but as the limit is approached the frequency increases.
When very near the limit RabbitMQ will check as frequently as 10 times per second. This may have some effect on system load.

APPLIES TO

3.4; 3.5; 3.6