Tuning multi-threaded render

How to I determine how many threads I need in the multi-threaded render to ensure proper performance and concurrency?

Probably the most important settings to look at when configuring a large server are the thread pool and cache settings:
debug=0 set to 1 to periodically log statistics
maxagesuccess=1800 successful results older than this are discarded
maxagefailure=300 unsuccessful results older than this are discarded
size=64 approximate size of the cache store in megabytes
debug=0 set to 1 to periodically log statistics
minthreads=4 the minimum number of threads in the pool
maxthreads=16 the maximum number of threads in the pool
startthreads=8 the initial number of threads in the pool
startinterval=5 the minimum interval that must pass before starting a thread
stopinterval=3600 the minimum interval that must pass before stopping a thread
(keep in mind each server and connection component steals a thread from the pool)
A large server will probably need more cache and threads. Each costs memory though.
In terms of connection limitations, we internally are using a fixed size FD_SET value of 1024 handles.
This generally works well for internal connections between vault processes since they take advantage of the fact you can have multiple commands in flight at once over a single connection. For example, each rendering engine usually has only one connection to the server but can send commands to service many clients at once.
For edge connections from clients, web applications etc. you are generally going to have a lot of connections. In a large environment you could run into the connection limit, particularly on Unix. In those cases, multiple rendering engines make sense.
At some point the memory limitations for 32-bit processes will also interfere with scaling. This would be another reason to use multiple rendering engines. For example a 24GB server running a single rendering engine limited to 2GB of memory might not be the best way to utilize the hardware.
> We have a new customer (a service bureau) that is considering an e2 
> Vault solution to provide document access to their customers clients.
> They have yet to provide us with projected utilization figures, but 
> they are still asking capacity planning questions that require me to 
> have a better understanding of how e2 Render can be tuned.
How is capacity planning usually approached in the wild?
The number of threads is a good question, and yes it can and should be tuned.  Specifically one problem is when a Server is in “passive” mode, and an external system might suddenly change it to “active” mode.  In this scenario it is possible that the component has initially detected continued idleness, and shut down many threads.  If an external system suddenly routes 100% of traffic to this node, it may not start up sufficient threads fast enough to service the load.  
As a result, Min Threads needs to be tuned to be the number of threads required to meet a moderate to high amount of load on your system.
First, add the following to e2xxxxxxd.ini (e2serverd, e2renderd, e2loaderd):
; Debug=1 adds useful detail to logs.
; MaxAgeSuccess (seconds) is the maximum age for a cached result to be considered valid.  Increasing this can have benefit but has the drawback that changes to the system may not become apparent for this period of time
; MaxAgeSuccess (seconds) is the maximum age for a “failure” result to be cached.  Increasing this will reduce load on the system, as it will no longer repeat requests that have no valid answer.  Again, changes to rectify a problem will not be visible until this interval has passed
; Cache Size in MB.  Maximum is 512 MB.
; Debug=1 adds useful detail in logs.
; Minimum number of threads to keep running, even if idle
; Maximum number of threads on the system
; Number of threads to initialize on startup
; Number of seconds to wait before starting each new thread, when high load is detected
; Amount of idle seconds required before shutting down one thread
Second, monitor an active Vault for a period of time.  The new Logging enabled by “debug=1” will return a line as follows in each Log file, each 30 seconds.  
11:22:31 <pool1> threads [8], busy [2], queue [0], through [0]
12:23:32 <pool1> threads [7], busy [2], queue [0], through [0]
13:24:32 <pool1> threads [6], busy [2], queue [0], through [0]
14:25:32 <pool1> threads [5], busy [2], queue [0], through [0]
This will show a value every 30 seconds.
If “busy” equals “threads”, then the Vault is saturated and may be about to start a new thread.  If queue is more than 2 or so, then requests may be backlogged.  Through is the amount of request throughput over the last 30 seconds.
If you chart this “busy” number over a period of time, you can get a sense of how heavily the Vault is loaded.  I would set the minimum threads to be equal to the number ‘busy’ during almost-peak time, e.g. 10am or 2pm on a Monday morning.
The comments are optional and the numbers are tunable.  E2loaderd.ini does not need more than 2-3 threads.  The e2renderd.ini and e2serverd.ini need more and this can be monitored to determine the ideal numbers.
The cache does not need to be huge on each node and it should be determined by the overall RAM available.  For example, 500MB on e2loaderd, 500MB on e2renderd, 500MB on e2serverd and 500MB on e2routerd would be a lot unless there are heavy demands on the environment.
You should definitely add the debug=1 to all though.  This enables debug logging, which does not use excessive disk space.
[Cache1] Debug=1 adds cache time to log entries.
For example, cached [240] means that an entry was returned from a cache entry 240 seconds old.
With the debug on, you can just chart the activity over the course of a day and then run grep (if you have it installed)  “grep 'threads' some_name.log > activity.txt,” import the .txt into Excel and then make a chart from the data.
UPDATED:  September 25, 2017