How to run poolmon to troubleshoot Vault 1450 kernel issues

I'm getting 1450 errors in the Vault log, indicated Windows kernel exhaustion.  How can I use poolmon to determine what is using up all the kernel space?


> ERROR 15115: begin read operation delay, code [1450]

 

A 1450 error is caused when the kernel has insufficient resources to complete commands.

 

Under heavy load Vault will use a lot of system resources and is sensitive to errors like this.

 

This can be made significantly worse by bad drivers or things such as antivirus programs (even when inactive).

 

In theory, something like poolmon.exe can be used to diagnose what memory pool is causing resource exhaustion.

 

Using a 64-version of Windows server can reduce 1450 problems because they have much higher kernel resource limits.

 

This error is quite dangerous as it can cause random I/O failures system wide which can lead to Vault index corruption.

 

> ERROR 17832: attempt to insert index field will cause overflow, record size [14], adding [101], index [index\ncnd\invlink.dri]

 

Make sure you're running the current 5.3 patch level since there are index fixes for some older builds.

 

The kernel has a page and non-page memory pool it uses to operate. These pools have limits which can be tight on 32-bit versions of Windows.

 

Rapidly opening and closing files as Vault does, puts pressure on these resource pools.

 

If something like a bad driver or antivirus program also takes up a lot of pool memory, you could hit the limit and operations would start failing with 1450 errors.

 

The operation delays can make Vault unresponsive and it ultimately it cannot recover it can cause corruption.

 

In general, the best way to avoid this is to use a 64-bit version of Windows.

 

A 1450 error under Windows normally indicates that *kernel* resources are exhausted trying to execute the request. It usually isn't referring to available virtual memory or disk space. Vault will try to back off (the "operation delay" part) in the hope that whatever is causing the system resource problem clears up over time. It won't wait forever and in this case it isn't clearing in time.

 

Things that can cause or increase the likelihood of 1450 errors include:

(1) bad drivers

(2) virus scanners (sometimes even when real time scanning is disabled)

(3) running Vault over redirected storage

(4) large amounts of physical memory (the page tables for the memory take up kernel memory)

(5) loading massive numbers of small files in Vault

(6) file fragmentation

 

Using a 64-bit version of Windows server alleviates most of the 1450 issues. Are they running a 64-bit version?

 

(1) Did any 1450 errors cause corruption?

 

What you should do is search the process logs for any ERROR 15116 messages that did NOT say "code [0]".

 

e.g.

ERROR 15116: end read operation delay, code [1450]

 

If the end code is 0, the operation was retried and succeeded after the delay.

If it still has a non-zero error code if could not recover and that could easily cause corruption.

 

 

(2) The insertion errors.

 

This may be an older version that still has the insertion overflow bug but has insertion guard code that detects it.

 

This will cause keys or ranges of keys to become inaccessible as a result of the insertion failure.

 

Is some cases this could be fixed with just a .index on recent jobs.

 

But if the failure occurred on split (which I think is the case), older data may also be affected.

 

If that's true, reindexing may be the only way to make sure all keys are properly restored.

 

This is pretty important to get fixed since it could cause a reindex to fail or corruption after reindexing.

 

I know Mike Wood recommends 64-bit windows (e.g. Windows Server 2008 R2 64-bit) for 1450 issues.

 

Poolmon may give you some clues. You'll probably need to enable pool tagging with gflags in the older Windows OSes. Then run poolmon and hit 'b' to sort by bytes and see the top consumers. The code on the left is the tag and you can determine what they are in pooltags.txt or by search drivers (*.sys) in system32\drivers.

 

If you can determine a time when the problem started you might be able to link it to some driver or software change on the machine.

 

A tool like poolmon.exe can display tagged memory use in the kernel and can sometimes help point out what in particular is taking up the kernel space.

 

(reference material)

 

How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks

http://support.microsoft.com/kb/177415

 

How to find pool tags that are used by third-party drivers

http://support.microsoft.com/kb/298102

 

Assuming poolmon is already installed and enabled on the Vault server, you can use these steps to collect the relevant information about what programs are using up the kernel memory and send it to us.

 

Click Start, point to Settings, click Control Panel, and then double-click Console.

 

Type Poolmon.exe.

Press P until Poolmon displays the second column "type" and shows the value paged.

Press B to sort the columns from largest to smallest.

Select the whole screen contents, and then press ENTER.

Click Start, point to Programs, point to Accessories, and then click Notepad.

On the Edit menu, click Paste.

Repeat step 7 to look for the value nonpaged.

Repeat steps 8 - 11 to paste.

Some example values: