|
Problem IC70081
|
Status: Closed |
ACCUMULATION OF DB2FMP (JAVA) WITH MEMORY CONSUMPTION WHEN KEEPFENCED=NO |
| product: |
DB2 FOR LUW / DB2FORLUW / 970 - DB2 |
| Problem description: |
Output of "ps -elf | grep db2fmp" shows a large number of db2fmp
(Java) processes running on your system when KEEPFENCED=NO. The
db2fmp processes are not being reused and appear to be stale.
For those db2fmp processes, the TIME value does not change
between runs of the "ps -elf | grep db2fmp" command. The number
of db2fmp processes may grow until virtual memory is exhausted.
Issue "db2pd -fmp" and check the status flag for the 'stale'
db2fmp processes. The flag value 0x80000003 for the problematic
db2fmp processes means:
0x00000001 the db2fmp has a JVM -> used by Java routine
+ 0x00000002 the db2fmp is threaded -> for execution of
threadsafe routines
+ 0x80000000 FMP will no longer be allowed to create new
threads, either due to an explicit thread limit or reaching an
internal resource limit
=====================================
x80000003
Because your db2fmp is flagged as 0x80000003 you may see
messages like the following in the db2diag.log file indicating
that new threads cannot be created:
2010-05-24-16.56.12.520000-420 I8527H330 LEVEL: Warning
PID : 4040 TID : 3552 PROC :
db2fmp.exe
INSTANCE: DB2 NODE : 000
EDUID : 3552
FUNCTION: DB2 UDB, routine_infrastructure,
sqlerMasterThreadListener,
probe:300
MESSAGE : Java heap is nearly full; no thread created
2010-05-24-16.56.12.520000-420 I8859H575 LEVEL: Warning
PID : 2556 TID : 1564 PROC :
db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : TEST
APPHDL : 0-87 APPID: AC13240C.G77B.E57AE0955EF1
AUTHID : NIC
EDUID : 1564 EDUNAME: db2agent (TEST) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq,
probe:
89
MESSAGE : FMP reported it could not create a new thread
DATA #1 : Hexdump, 4 bytes
0x79E7E830 : C80F 0000 ....
2010-05-24-16.56.12.520000-420 I9436H456 LEVEL: Severe
PID : 2556 TID : 1564 PROC :
db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : TEST
APPHDL : 0-87 APPID: AC13240C.G77B.E57AE0955EF1
AUTHID : NIC
EDUID : 1564 EDUNAME: db2agent (TEST) 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerGetFmpThread,
probe:20
RETCODE : ZRC=0xFFFFFBEE=-1042 |
| Problem Summary: |
****************************************************************
* USERS AFFECTED: *
* Java SPs Users running with KEEPFENCED=NO *
****************************************************************
* PROBLEM DESCRIPTION: *
* Output of "ps -elf | grep db2fmp" shows a large number of *
* db2fmp *
* (Java) processes running on your system when KEEPFENCED=NO. *
* The *
* db2fmp processes are not being reused and appear to be *
* stale. *
* For those db2fmp processes, the TIME value does not change *
* *
* between runs of the "ps -elf | grep db2fmp" command. The *
* number *
* of db2fmp processes may grow until virtual memory is *
* exhausted. *
* *
* *
* Issue "db2pd -fmp" and check the status flag for the 'stale' *
* *
* db2fmp processes. The flag value 0x80000003 for the *
* problematic *
* db2fmp processes means: *
* *
* 0x00000001 the db2fmp has a JVM -> used by Java routine *
* *
* + 0x00000002 the db2fmp is threaded -> for execution of *
* *
* threadsafe routines *
* *
* + 0x80000000 FMP will no longer be allowed to create new *
* *
* threads, either due to an explicit thread limit or reaching *
* an *
* internal resource limit *
* *
* ===================================== *
* *
* x80000003 *
* *
* *
* *
* Because your db2fmp is flagged as 0x80000003 you may see *
* *
* messages like the following in the db2diag.log file *
* indicating *
* that new threads cannot be created: *
* *
* *
* *
* 2010-05-24-16.56.12.520000-420 I8527H330 LEVEL: *
* Warning *
* *
* *
* PID : 4040 TID : 3552 PROC : *
* *
* db2fmp.exe *
* *
* INSTANCE: DB2 NODE : 000 *
* *
* *
* *
* EDUID : 3552 *
* *
* *
* *
* FUNCTION: DB2 UDB, routine_infrastructure, *
* *
* sqlerMasterThreadListener, *
* *
* probe:300 *
* *
* *
* *
* MESSAGE : Java heap is nearly full; no thread created *
* *
* *
* *
* 2010-05-24-16.56.12.520000-420 I8859H575 LEVEL: *
* Warning *
* *
* *
* PID : 2556 TID : 1564 PROC : *
* *
* db2syscs.exe *
* *
* INSTANCE: DB2 NODE : 000 DB : TEST *
* *
* *
* *
* APPHDL : 0-87 APPID: *
* AC13240C.G77B.E57AE0955EF1 *
* *
* *
* AUTHID : NIC *
* *
* *
* *
* EDUID : 1564 EDUNAME: db2agent (TEST) 0 *
* *
* *
* *
* FUNCTION: DB2 UDB, routine_infrastructure, *
* sqlerMasterThreadReq, *
* probe: *
* *
* 89 *
* *
* *
* *
* MESSAGE : FMP reported it could not create a new thread *
* *
* *
* *
* DATA #1 : Hexdump, 4 bytes *
* *
* *
* *
* 0x79E7E830 : C80F 0000 .... *
* *
* *
* *
* *
* *
* 2010-05-24-16.56.12.520000-420 I9436H456 LEVEL: *
* Severe *
* *
* *
* PID : 2556 TID : 1564 PROC : *
* *
* db2syscs.exe *
* *
* INSTANCE: DB2 NODE : 000 DB : TEST *
* *
* *
* *
* APPHDL : 0-87 APPID: *
* AC13240C.G77B.E57AE0955EF1 *
* *
* *
* AUTHID : NIC *
* *
* EDUID : 1564 EDUNAME: db2agent (TEST) 0 *
* *
* *
* *
* FUNCTION: DB2 UDB, routine_infrastructure, *
* sqlerGetFmpThread, *
* probe:20 *
* *
* RETCODE : ZRC=0xFFFFFBEE=-1042 *
****************************************************************
* RECOMMENDATION: *
* WORKAROUNDS: *
* *
* 1) Review your Java routines and ensure that the code is not *
* *
* heavily dependent on Garbage Collection. (e.g. close off the *
* *
* objects manually after they are no longer used rather then *
* *
* relying on the JVM to clean them up) *
* *
* *
* *
* From the setting perspective *
* *
* 2) If you use THREADSAFE Java routines (default), then set *
* *
* KEEPFENCED=YES. In that case we will reuse the same thread *
* and *
* reduce the chance of creating new threads. *
* *
* *
* *
* 3) If you declare your routines as NOT THREADSAFE then you *
* can *
* set KEEPFENCED to either YES or NO. When a routine is in a *
* NOT *
* THREADSAFE mode, each routine will run in its own *
* db2fmp/JVM. *
* *
* *
* If KEEPFENCED=NO a new fenced mode process is created and *
* *
* destroyed for each fenced mode invocation. If *
* KEEPFENCED=YES *
* the existing db2fmp will be returned to the pool for reuse. *
**************************************************************** |
| Local Fix: |
WORKAROUNDS:
1) Review your Java routines and ensure that the code is not
heavily dependent on Garbage Collection. (e.g. close off the
objects manually after they are no longer used rather then
relying on the JVM to clean them up)
From the setting perspective
2) If you use THREADSAFE Java routines (default), then set
KEEPFENCED=YES. In that case we will reuse the same thread and
reduce the chance of creating new threads.
3) If you declare your routines as NOT THREADSAFE then you can
set KEEPFENCED to either YES or NO. When a routine is in a NOT
THREADSAFE mode, each routine will run in its own db2fmp/JVM.
If KEEPFENCED=NO a new fenced mode process is created and
destroyed for each fenced mode invocation. If KEEPFENCED=YES
the existing db2fmp will be returned to the pool for reuse. |
| available fix packs: |
DB2 Version 9.7 Fix Pack 4 for Linux, UNIX, and Windows
DB2 Version 9.7 Fix Pack 5 for Linux, UNIX, and Windows
DB2 Version 9.7 Fix Pack 6 for Linux, UNIX, and Windows
DB2 Version 9.7 Fix Pack 7 for Linux, UNIX, and Windows
DB2 Version 9.7 Fix Pack 8 for Linux, UNIX, and Windows
DB2 Version 9.7 Fix Pack 9 for Linux, UNIX, and Windows
DB2 Version 9.7 Fix Pack 9a for Linux, UNIX, and Windows
DB2 Version 9.7 Fix Pack 10 for Linux, UNIX, and Windows
|
| Solution |
|
| Workaround |
not known / see Local fix |
| Timestamps |
Date - problem reported :
Date - problem closed :
Date - last modified :
| 21.07.2010
28.04.2011
28.04.2011 |
| Problem solved at the following versions (IBM BugInfos) |
|
| Problem solved according to the fixlist(s) of the following version(s) |
| 9.7.0.4
|
 |
