Remember this Post , here is a quick recap:
Event ID: 2019
The server was unable to allocate from the system nonpaged pool because the pool was empty.
So what does it really mean ?
This error is from Server Service reporting that when it was trying to satisfy a request, it was not able to find enough free memory of the respective type of pool. Error 2020 indicates Paged Pool and 2019, NonPaged Pool. This doesn’t mean that the Server Service (srv.sys) is broken or the root cause of the problem, more often rather it is the first component to see the resource problem and report it to the Event Log.
I installed Poolmon which told me that Thre is the largest consumer of memory
So what's "Thre" ?
Thre - nt!ps - Thread objects
Note, the nt before the ! means that this is NT or the kernel’s tag for Thread objects. So there has to be a process that is leaking memory.
I got WinDbg running on this machine asap and entered "!proccess 0 0" in the command. Here is the output with all Processes with more than 1000 handle count.
PROCESS 89b9ad88 SessionId: 0 Cid: 0afc Peb: 7ffd7000 ParentCid: 01c0
DirBase: dfff07e0 ObjectTable: e628b498 HandleCount: 95969.
Image: oracle.exePROCESS 89b2a690 SessionId: 0 Cid: 0c4c Peb: 7ffdf000 ParentCid: 01c0
DirBase: dfff0860 ObjectTable: e63b2358 HandleCount: 2244.
Image: pinetmgr.exePROCESS 8a386698 SessionId: 0 Cid: 0f04 Peb: 7ffd4000 ParentCid: 01c0
DirBase: dfff0a60 ObjectTable: e17e7408 HandleCount: 2167.
Image: pimsgss.exeA handle count of more than 95000 definitely set off alarms. I dug a bit deeper into the Oracle process with
!PROCESS 89b9ad88 4
The process brought a whole bunch of threads.
!process 89b9ad88 4
PROCESS 89b9ad88 SessionId: 0 Cid: 0afc Peb: 7ffd7000 ParentCid: 01c0
DirBase: dfff07e0 ObjectTable: e628b498 HandleCount: 114448.
Image: oracle.exe
THREAD 89b97998 Cid 0afc.0b00 Teb: 7ffdf000 Win32Thread: e6355328 WAIT
THREAD 89b77b78 Cid 0afc.0b44 Teb: 7ffdd000 Win32Thread: 00000000 WAIT
THREAD 89b64458 Cid 0afc.0b50 Teb: 7ffdc000 Win32Thread: 00000000 WAIT
THREAD 89a01020 Cid 0afc.1204 Teb: 7ffdb000 Win32Thread: e660b768 WAIT
THREAD 89a007d0 Cid 0afc.1208 Teb: 7ffd9000 Win32Thread: 00000000 WAIT
THREAD 89a003b8 Cid 0afc.120c Teb: 7ffd8000 Win32Thread: 00000000 WAIT
THREAD 899e5db0 Cid 0afc.1214 Teb: 7ffd6000 Win32Thread: 00000000 WAIT
THREAD 899fadb0 Cid 0afc.121c Teb: 7ffd5000 Win32Thread: 00000000 WAIT
THREAD 899e6db0 Cid 0afc.1220 Teb: 7ffd4000 Win32Thread: 00000000 WAIT
I opened two random threads with
!thread command and this is what it came up with:
THREAD 897ebaf0 Cid 0afc.1b58 Teb: 00000000 Win32Thread: 00000000 TERMINATED
Not impersonating
DeviceMap e1000908
Owning Process 89b9ad88 Image: oracle.exe
Attached Process N/A Image: N/A
Wait Start TickCount 17893 Ticks: 16701178 (3:00:29:15.906)
Context Switch Count 24
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address 0x0040162c
Start Address 0x77e617ec
Stack Init 0 Current b949fba0 Base b94a0000 Limit b949d000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
THREAD 8969e020 Cid 0afc.08c0 Teb: 00000000 Win32Thread: 00000000 TERMINATED
Not impersonating
DeviceMap e1000908
Owning Process 89b9ad88 Image: oracle.exe
Attached Process N/A Image: N/A
Wait Start TickCount 45267 Ticks: 16678772 (3:00:23:25.812)
Context Switch Count 27
UserTime 00:00:00.000
KernelTime 00:00:00.015
Win32 Start Address 0x0040162c
Start Address 0x77e617ec
Stack Init 0 Current b9c3fba0 Base b9c40000 Limit b9c3d000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0
ChildEBP RetAddr Args to Child
I could tell that the threads have been terminated and they belonged to Oracle.exe but somehow they have not been cleared from memory.
I opened Task Manager and from the View column option added the Handle count and saw this:
The Handle count was growing at a fair bit of speed.
I have contacted the Oracle boys to check out the issue but I am pretty sure one of the Oracle app on that box or Oracle itself is the cause of the memory leak.
I’ll post back when the Oracle team have come back with their investigation.
Posts
