VERIFIED SOLUTION i

Vault Java API java.lang.OutOfMemoryError: Java heap space

Issue

We are working with e2 Vault Server 5.4M4p0221 and the VaultClient API version 5.4M4p0135, we are having an error in the ServiceWeb in a particular Document on our Production Server: 
 
Searching for documents under account AIGE7908259K4 
 
java.lang.OutOfMemoryError: Java Heap Space 
class java.lang.OutOfMemoryError 
 
We increase the heap space to 1 GB but still with this problem, this Client have more than 1000 documents I think this is the problem trying to get all documents. 
 
What can we do in this case? 
 
13:09:25,963 ERROR [[Interface]] Servlet.service() for servlet Interface threw exception 
java.lang.OutOfMemoryError: Java heap space 
at java.util.TreeMap.put(TreeMap.java:554) 
at java.util.TreeSet.add(TreeSet.java:238) 
at com.g1.e2.vault.OutputModesHelper.outputFormatsFromModes(OutputModesHelper.java:34) 
at com.g1.e2.vault.VaultClient$Document.<init>(VaultClient.java:1861) 
at com.g1.e2.vault.VaultClient$DocumentSearchWorker$DocumentSearchResponseListener.dataRow(VaultClient.java:3834) 
at com.g1.e2.vault.protocol.UInfoResponse.readDataBlock(UInfoResponse.java:294) 
at com.g1.e2.vault.protocol.UInfoResponse.readFromStream(UInfoResponse.java:157) 
at com.g1.e2.vault.VaultClient$DocumentSearchWorker.searchForDocumentsContinueFrom(VaultClient.java:3329) 
at com.g1.e2.vault.VaultClient$DocumentSearchWorker.next(VaultClient.java:3654) 
at com.g1.e2.vault.DocumentSearchMatchesIterator.next(DocumentSearchMatchesIterator.java:53) 
at com.g1.e2.vault.VaultClient$Account.getAllDocuments(VaultClient.java:1528)

Cause

When searching for "all documents under account" Java runs out of memory when there are a larg amount of  documents associated with that account under the same date.  e2routerd and e2serverd show that it's looping:
 
171:61319 <connection4> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [16]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [16]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
12:12:49 192.1.2.171:61319 <router1> database.search request, prefix [], database [finafact], index [invlink], first [2013/07/12], max [30], under [AIGE7908259K4]
12:12:49 192.1.2.171:61319 <router1> database.search returned record, [31] rows, [14] columns, more [1], fixed [9], elapsed [0], cached [1]
 

Resolution

UPDATED: October 18, 2017
I checked the render log file for the account AIGE7908259K4 which caused the issue.
And found out that the entries of "AIGE7908259K4 + 2013/07/12" in INVLINK index are more than 30. 
when Java API gets all documents for an account, it actually used Account + Date to search INVLINK index, set maxresults as search window size ( in this case the search window size is 30 ), and use the Account+DATE from the last entry of the last search-window as start/first to get a next search.
 
in this case:
search1: search window is 30, Account is AIGE7908259K4, DATE is "" or some date-string, getting 30 returned;
search2: search window is 30, Account is AIGE7908259K4, DATE is 2013/07/12", getting 30 returned;
search3: search window is 30, Account is AIGE7908259K4, DATE is 2013/07/12", getting 30 returned;
...
but here is a problem, the DATE is always "2013/07/12", so Render/Server is looping, so Java API is getting endless results, so Java client gets OutofMemory error.
 
how it is happening like this, this is because inside INVLINK, the entries of "AIGE7908259K4 + 2013/07/12" is greater than 30.
 
how to find out how many entries for A+D, use indexcheck.exe:
under <Vault-Server-dir>
indexcheck.exe index\finafact\invlink.dri AIGE7908259K4 2013/07/12
 
how to resolve the problem on Java API client side:
use above command to find out the max-entries of AIGE7908259K4 + 2013/07/12 and set maxresults=max-entries + 1, call Account.getAllDocuments(maxresults), then you can skip the Server looping problem.
 
this is a temp solution, if it happens again for same account or other accounts, use the above way to do it again.
 
 
############
the better way, 
 
using Java API 6.0 and later version.
For the newer API, there is an exception of ServerLoopingException or TooManyResultsException, 
 
in your code, catch the exceptions, then you will not get Outofmemory again.
 
or put the max-results into a configuration file, catch the exception, using indexcheck.exe to find out the correct value of max-results, then no need to change your code if the error happens.
 
 
###############
in this case, change the parameter of maxresults in web.xml to a correct value.
 
<context-param>
<description>Maximum number of search results to display.</description>
<param-name>maxresults</param-name>
<param-value>100</param-value>
</context-param>
Environment Details
5.4, 5.5 all platforms