VERIFIED SOLUTION i

Resolve heap space issues using run.sh and purge.sh with multiple concurrent users in EngageOne V3.1.2

Product Feature: Purge
Versions affected: 3.1.2 and 4.4.x
 

Issue

A performance problem occurs in the EngageOne Server with a large number of concurrent users (reported with 250 users).  When run.sh and purge.sh are run with a sixty second wait before running again in a loop, performance issues occur on the EngageOne Server JVM where garbage collection takes an excessive amount of time and users start getting multiple errors and time outs on requests.

Cause

In high load scenarios, the Event Monitor is unable to process items quickly.  The system creates new items as the process continues which can cause slowdowns / higher memory usage.

Resolution

UPDATED: October 10, 2017
Event Monitor database performance was improved, additionally a new configuration setting "eventLoggingLevel" was added to config-settings.xml allowing the user to define the level of EngageOne event creation.

Documentation for 4.4x of EngageOne contains the following information about capturing events:

The Event Monitor processes events asynchronously. Events are received in a queue and are processed sequentially. Some events take longer to process depending on the sink configuration.
If an event is set for delivery to a log, database, SMTP , and JMX, it will take a while for the Event Monitor consumer to process that event. Adding more sinks increases the processing time. The more sinks, the more time is required to process the each event.

Additional information will say that the user can disable putting some type of events to the queue completely (i.e. when they don't need them at all - this is different than filtering, because filtering works on queue) by setting (DEBUG | INFO | WARN | ERROR) for:

<setting>
<key>eventLoggingLevel</key>
<value>DEBUG</value>
</setting>

This setting will be available in EngageOne 3.1.2.3078700132 and later and in EngageOne 4.4.4.