Remember this Post , here is a quick recap:

Event ID: 2019
The server was unable to allocate from the system nonpaged pool because the pool was empty.

So what does it really mean ?

This error is from  Server Service reporting that when it was trying to satisfy a request, it was not able to find enough free memory of the respective type of pool. Error 2020 indicates Paged Pool and 2019, NonPaged Pool. This doesn’t mean that the Server Service (srv.sys) is broken or the root cause of the problem, more often rather it is the first component to see the resource problem and report it to the Event Log.

I installed Poolmon which told me that Thre  is the largest consumer of memory

 

So what's "Thre" ?

Thre - nt!ps - Thread objects

Note, the nt before the ! means that this is NT or the kernel’s tag for Thread objects. So there has to be a process that is leaking memory.

I got WinDbg running on this machine asap and entered "!proccess 0 0" in the command. Here is the output with all Processes with more than 1000 handle count.

PROCESS 89b9ad88  SessionId: 0  Cid: 0afc    Peb: 7ffd7000  ParentCid: 01c0
    DirBase: dfff07e0  ObjectTable: e628b498  HandleCount: 95969.
    Image: oracle.exe

PROCESS 89b2a690  SessionId: 0  Cid: 0c4c    Peb: 7ffdf000  ParentCid: 01c0
    DirBase: dfff0860  ObjectTable: e63b2358  HandleCount: 2244.
    Image: pinetmgr.exe

PROCESS 8a386698  SessionId: 0  Cid: 0f04    Peb: 7ffd4000  ParentCid: 01c0
    DirBase: dfff0a60  ObjectTable: e17e7408  HandleCount: 2167.
    Image: pimsgss.exe

A handle count of more than 95000 definitely set off alarms. I dug a bit deeper into the Oracle process with

!PROCESS 89b9ad88  4 

The process brought a whole bunch of threads.

!process 89b9ad88 4

PROCESS 89b9ad88  SessionId: 0  Cid: 0afc    Peb: 7ffd7000  ParentCid: 01c0
    DirBase: dfff07e0  ObjectTable: e628b498  HandleCount: 114448.
    Image: oracle.exe

      THREAD 89b97998  Cid 0afc.0b00  Teb: 7ffdf000 Win32Thread: e6355328 WAIT
        THREAD 89b77b78  Cid 0afc.0b44  Teb: 7ffdd000 Win32Thread: 00000000 WAIT
        THREAD 89b64458  Cid 0afc.0b50  Teb: 7ffdc000 Win32Thread: 00000000 WAIT
        THREAD 89a01020  Cid 0afc.1204  Teb: 7ffdb000 Win32Thread: e660b768 WAIT
        THREAD 89a007d0  Cid 0afc.1208  Teb: 7ffd9000 Win32Thread: 00000000 WAIT
        THREAD 89a003b8  Cid 0afc.120c  Teb: 7ffd8000 Win32Thread: 00000000 WAIT
        THREAD 899e5db0  Cid 0afc.1214  Teb: 7ffd6000 Win32Thread: 00000000 WAIT
        THREAD 899fadb0  Cid 0afc.121c  Teb: 7ffd5000 Win32Thread: 00000000 WAIT
        THREAD 899e6db0  Cid 0afc.1220  Teb: 7ffd4000 Win32Thread: 00000000 WAIT

I opened two random threads with

!thread command and this is what it came up with:

THREAD 897ebaf0  Cid 0afc.1b58  Teb: 00000000 Win32Thread: 00000000 TERMINATED
Not impersonating
DeviceMap                 e1000908
Owning Process            89b9ad88       Image:         oracle.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      17893          Ticks: 16701178 (3:00:29:15.906)
Context Switch Count      24            
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0x0040162c
Start Address 0x77e617ec
Stack Init 0 Current b949fba0 Base b94a0000 Limit b949d000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0

THREAD 8969e020  Cid 0afc.08c0  Teb: 00000000 Win32Thread: 00000000 TERMINATED
Not impersonating
DeviceMap                 e1000908
Owning Process            89b9ad88       Image:         oracle.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      45267          Ticks: 16678772 (3:00:23:25.812)
Context Switch Count      27            
UserTime                  00:00:00.000
KernelTime                00:00:00.015
Win32 Start Address 0x0040162c
Start Address 0x77e617ec
Stack Init 0 Current b9c3fba0 Base b9c40000 Limit b9c3d000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr  Args to Child

I could tell that the threads have been terminated and they belonged to Oracle.exe but somehow they have not been cleared from memory.

I opened Task Manager and from the View column option added the Handle count and saw this:

The Handle count was growing at a fair bit of speed.

I have contacted the Oracle boys to check out the issue but I am pretty sure one of the Oracle app on that box or Oracle itself is the cause of the memory leak.

I’ll post back when the Oracle team have come back with their investigation.