Db2 instance crashed on one partition after running out of
memory with the following stack:
CALLSTCK: (Static functions may not be resolved correctly, as
they are resolved to the nearest symbol)
[0] 0x00007FEBD152AEEC sqloPGRPRegisterOneCrash + 0x20C
[1] 0x00007FEBD167EDEF sqloEDUCodeTrapHandler + 0xB2F
[2] 0x00007FEBD8EED7E0 /lib64/libpthread.so.0 + 0xF7E0
[3] 0x00007FEBCFF9F742 _Z8sqlkqopnP17sqlkqBqsOpenParms + 0xC12
[4] 0x00007FEBCFFBC65C
_Z8sqlktopnP8sqeAgentP9sqlri_taoP9SQLE_KRCB + 0x12AC
[5] 0x00007FEBD2213F9B _Z8sqlriopnP8sqlrr_cbP9sqlri_taoPi +
0x44B
[6] 0x00007FEBD2399C2D _Z8sqlritqbP8sqlrr_cb + 0x22D
[7] 0x00007FEBD22A5017
_Z15sqlriSectInvokeP8sqlrr_cbP12sqlri_opparm + 0x4B7
[8] 0x00007FEBD1CE21C9
_Z21sqlrr_subagent_routerP8sqeAgentP12SQLE_DB2RA_T + 0xDE9
[9] 0x00007FEBCF87761F
/db2home/db2edwp1/sqllib/lib64/libdb2e.so.1 + 0x4FDF61F
[10] 0x00007FEBCF875EF0 _Z21sqleProcessSubRequestP8sqeAgent +
0x3E0
[11] 0x00007FEBCF8AA477 _ZN8sqeAgent6RunEDUEv + 0x5D7
[12] 0x00007FEBD30C2FB6 _ZN9sqzEDUObj9EDUDriverEv + 0x116
[13] 0x00007FEBD167B5B8 sqloEDUEntry + 0x578
[14] 0x00007FEBD8EE5AA1 /lib64/libpthread.so.0 + 0x7AA1
[15] 0x00007FEBC9916BCD clone + 0x6D
Messages in db2diag.log will be similar to:
2017-09-04-04.30.39.732652+000 E110068359E706 LEVEL:
Warning
PID : 25249 TID : 140471707887360 PROC :
db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB :
SAMPLE
APPHDL : 0-15638 APPID:
10.55.76.184.51809.170904030603
AUTHID : AUTHID HOSTNAME: partition1
EDUID : 165886 EDUNAME: db2agnts (SAMPLE) 0
FUNCTION: DB2 UDB, SQO Memory Management,
SqloMemController::getPartitionStats, probe:10
DATA #1 : <preformatted>
Instance Memory Controller statistics.
Instance Memory Automatic: No.
Maximum size = 118111600 KB
Current size = 118111424 KB
Usage HWM = 118111552 KB
Cached memory = 216384 KB
Cached DB memory = 0 KB
2017-09-04-04.30.39.748838+000 I110072547E1086 LEVEL:
Warning
PID : 25249 TID : 140471707887360 PROC :
db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB :
SAMPLE
APPHDL : 0-15638 APPID:
10.55.76.184.51809.170904030603
AUTHID : AUTHID HOSTNAME: partition1
EDUID : 165886 EDUNAME: db2agnts (SAMPLE) 0
FUNCTION: DB2 UDB, SQO Memory Management,
SqloMemController::requestMemory, probe:50
MESSAGE : ZRC=0x8B0F0000=-1961951232=SQLO_NOMEM "No Memory
Available"
DIA8300C A memory heap error has occurred.
DATA #1 : String, 36 bytes
OOM - Instance memory request failed
DATA #2 : String, 35 bytes
Logging disabled until next success
DATA #3 : unsigned integer, 8 bytes
1048576
DATA #4 : unsigned integer, 8 bytes
0
DATA #5 : String, 13 bytes
APPL-SAMPLE
DATA #6 : unsigned integer, 8 bytes
9202171904
DATA #7 : unsigned integer, 8 bytes
215547904
DATA #8 : unsigned integer, 8 bytes
120946098176
DATA #9 : unsigned integer, 8 bytes
120946278400
DATA #10: unsigned integer, 8 bytes
215547904
2017-09-04-04.30.42.489568+000 E111194474E766 LEVEL:
Warning
PID : 25249 TID : 140471854688000 PROC :
db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB :
SAMPLE
APPHDL : 0-26595 APPID:
10.120.16.118.50278.171112235219
AUTHID : EDUVDM3P HOSTNAME: partition1
EDUID : 166932 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, SQO Memory Management,
sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Statement Heap on node 0.
Requested block size : 888 bytes.
Physical heap size : 0 bytes.
Configured heap size : 327680000 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set : 0 bytes.
2017-09-04-04.36.13.550198+000 I124014097E1036 LEVEL:
Severe
PID : 25249 TID : 140496517195520 PROC :
db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB :
SAMPLE
APPHDL : 0-46106 APPID:
10.120.16.120.60306.171017052444
AUTHID : EDUCMP1P HOSTNAME: partition1
EDUID : 167545 EDUNAME: db2agnts (SAMPLE) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc,
probe:250
MESSAGE : ZRC=0x87120007=-2028863481=SQLR_SEVERE_PGM_ERROR
"Severe programming error"
DIA8516C A severe internal processing error has
occurred.
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: 0 sqlerrml: 0
sqlerrmc:
sqlerrp : SQL11012
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6)
0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2017-09-04-04.41.09.321155+000 E125395774E610 LEVEL:
Severe
PID : 25242 TID : 139940373456640 PROC :
db2wdog 0 [db2inst1]
INSTANCE: db2inst1 NODE : 000
HOSTNAME: partition1
EDUID : 2 EDUNAME: db2wdog 0 [db2inst1]
FUNCTION: DB2 UDB, base sys utilities, sqleWatchDog, probe:20
MESSAGE : ADM0503C An unexpected internal processing error has
occurred. All
database manager processes associated with this
instance have been
shutdown. Diagnostic information has been recorded.
Contact IBM
Support for further assistance.
This was caused by poor error handling in Db2. |
