db2 agent/db2fmp can hang due to incomplete handshake during
interrupt. This can result in any agent looking to call a
routine to get stuck waiting on the
SQLO_LT_sqlerFmpTable__fmpTableLatch latch result in a hang for
that agent as well.
db2diag.log:
2017-08-23-00.16.02.999306+000 I20742564A998 LEVEL: Error
PID : 2557828 TID : 23697 PROC :
db2sysc 24
INSTANCE: db2inst1 NODE : 024 DB :
DBNAME
APPHDL : 999-5773 APPID:
*N999.db2inst1.170823001545
AUTHID : db2inst1 HOSTNAME: db2host1
EDUID : 23697 EDUNAME: db2agntp (DBNAME) 24
FUNCTION: DB2 UDB, routine_infrastructure,
sqlerInvokeFencedRoutine, probe:1220
MESSAGE : ZRC=0x804B0003=-2142568445=SQLJS_INTRPT "INTERRUPT
DETECTED"
DIA8003C The interrupt has been received.
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: 0 sqlerrml: 0
sqlerrmc:
sqlerrp : SQL11012
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6)
0x00000018
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2017-08-23-00.16.02.689867+000 I20743563A899 LEVEL: Error
PID : 8913656 TID : 772 PROC :
db2fmp (C) 9
INSTANCE: db2inst1 NODE : 009 DB :
DBNAME
APPID : *N999.db2inst1.170823000555
HOSTNAME: db2host
EDUID : 772 EDUNAME: db2fmp (C) 9
FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpReceive,
probe:1259
MESSAGE : ZRC=0x8024006D=-2145124243=SQLER_CA_BUILT
"SQLCA has already been built"
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: -1131 sqlerrml: 0
sqlerrmc:
sqlerrp : SQLERRCV
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0x00000000 (6)
0x00000009
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2017-08-23-00.16.02.691142+000 I20744463A556 LEVEL:
Severe
PID : 8913656 TID : 1 PROC :
db2fmp (C) 9
INSTANCE: db2inst1 NODE : 009
HOSTNAME: db2host
EDUID : 1 EDUNAME: db2fmp (C) 9
FUNCTION: DB2 UDB, trace services, sqlt_logerr_data (secondary
logging function), probe:50
MESSAGE : Library still in use in sqlerLibraryUnloadAll
/db2home/db2inst1/sqllib/bin/routine/db2monudf 3
DATA #1 : Hexdump, 4 bytes
0x0FFFFFFFFFFFDDB0 : 0000 0000
....
2017-08-23-00.16.02.519850+000 E20745020A4253 LEVEL:
Severe
PID : 10159080 TID : 10285 PROC :
db2sysc 57
INSTANCE: db2inst1 NODE : 057 DB :
DBNAME
APPHDL : 999-5773 APPID:
*N999.db2inst1.170823001545
AUTHID : db2inst1 HOSTNAME: db2host
EDUID : 10285 EDUNAME: db2agntp (DBNAME) 57
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq,
probe:910
DATA #1 : String, 23 bytes
Marking fmp as unstable
DATA #2 : String, 8 bytes
Fmp TID:
DATA #3 : Hexdump, 4 bytes
0x0780000002AFFEB8 : 0000 0607
....
DATA #4 : String, 8 bytes
Fmp Row:
DATA #5 : sqlerFmpRow, PD_SQLER_TYPE_FMP_ROW, 856 bytes
fmpPid: 5636556
fmpPoolList Ptr: 0x0780000002a3f740 fmpForcedList Ptr:
0x0000000000000000
nextFmpCB Ptr: 0x0780000000da5360 prevFmpCB Ptr:
0x0000000000000000
fmpIPCList Ptr: 0x0780000002a3ffc0
stateFlags: 0x00000012 numFmp32Attaches: 0
numActiveThreads: 1 numPoolThreads: 3
fmpCodePage: 0 fmpRowUseCount: 1
active: 0x01 rowLoaderValidate: 0x00
ipcLatch:
0x0780000002A3EB64 : 1000 0000 003F 0000
.....?..
rowLatch:
0x0780000002A3EB6C : 1000 0000 01DB 0000
........
fmpAgentList:
2017-08-23-00.16.02.999763+000 I20755577A495 LEVEL:
Severe
PID : 10421710 TID : 1 PROC :
db2fmp (C) 24
INSTANCE: db2inst1 NODE : 024
HOSTNAME: db2host1
EDUID : 1 EDUNAME: db2fmp (C) 24
FUNCTION: DB2 UDB, routine_infrastructure,
sqlerMasterThreadListener, probe:400
RETCODE : ZRC=0xFFFFFB95=-1131
SQL1131N A stored procedure process has been
terminated abnormally.
Routine name: "". Specific name: "".
Agent Stack could look like:
*** Stack ***
Timestamp: 2017-08-23-17.55.48.247784
0x09000000001F5C10 semop + 0xB0
0x090000000E118664 sqloSSemP + 0x1224
0x0900000012C7AD14
IPRA.$sqlccipcWaitSynch__FP18SQLCC_INITSTRUCT_T + 0x254
0x0900000012C74254 sqlccipcinit + 0x8D4
0x090000000E10F94C
sqlccinit__FP18SQLCC_INITSTRUCT_TPP17SQLCC_COMHANDLE_TP12SQLCC_C
OND_TP13SQLO_MEM_POOL + 0x5CC
0x090000000EBF7EBC
IPRA.$sqlerInitCommsLayer__FP14sqlerFmpHandleP8sqeAgentb + 0x11C
0x090000000EBF34D8
sqlerMasterThreadReq__FP17sqlerFmpParmsBaseP13sqlerFmpTableP14sq
lerFmpHandleP18sqlerFmpThreadListP8sqeAgentUicT7P5sqlcab+ 0x858
0x090000000EBF6340
sqlerInterruptThreadedFmp__FP14sqlerFmpHandleP17sqlerFmpParmsBas
e + 0xC0
0x090000000EBE2E1C
sqlerReturnFmpToPool__FcT1P14sqlerFmpHandleP8sqeAgent + 0x1DDC
0x090000000EBDED24 sqlerInvokeFencedRoutine__FP13sqlerFmpParms +
0xB44
0x0900000011D54F54 sqlriInvokeInvoker__FP10sqlri_ufobb + 0x1CF4
0x0900000011D56C9C sqlriutf__FP8sqlrr_cb + 0x13C
0x09000000145156B8 sqlri_tfopn__FP8sqlrr_cbP9sqlri_tao + 0x1F8
0x09000000101296A4 sqlriopn__FP8sqlrr_cbP9sqlri_taoPi + 0x5A4
0x090000001204C750 sqlrita__FP8sqlrr_cb + 0xEF0
0x090000000FCDC898 sqlriSectInvoke__FP8sqlrr_cbP12sqlri_opparm +
0x618
0x090000000F634380 sqlrr_dss_router__FP8sqlrr_cb + 0x9A0
0x090000000F633054
sqlrr_subagent_router__FP8sqeAgentP12SQLE_DB2RA_T + 0xF54 |
