We currently have seen multiple variants of timing issue that
leads to load hang. All cases have mismatched
BeginINDPS/EndINDPS. One or more of the following symptoms may
be observed.
Symptom 1:
The hang was between db2lfrm0 holding serial logging latch and
db2lrid allocating new extent, waiting for serial logging latch
under sqlbDMScheckObjAlloc/sqlpInitBFI.
The stack information will show as following :
-Waiter was db2lrid.
Stack:
======================
0x090000006850626C thread_waitlock@glue881 + 0x8C
0x090000006725FF2C sqloXlatchConflict + 0x54
0x09000000692F42F4
sqlpBeginINDPSLoggingMode__FP8sqeAgentP11SQLP_TENTRY + 0x1EC
0x09000000692F4038
sqlpBeginINDPSLoggingMode__FP8sqeAgentP11SQLP_TENTRY@glue11D2 +
0x74
0x0900000067EE5FC4 sqlpInitBFI__FP8sqeAgentUlPUl + 0x94
0x0900000068567408 sqlpInitBFI__FP8sqeAgentUlPUl@glue1BE +
0x8C
0x09000000684FFD40
sqlbDMScheckObjAlloc__FP16SQLB_OBJECT_DESCUliT3PPUiP12SQLB_GLOBA
LS + 0x64
0x0900000068198918
sqlbDMSCheckAllocation__FP16SQLB_OBJECT_DESCUlUiPUiP12SQLB_GLOBA
LS + 0x3C
0x0900000069B41EA0
sqlbLoadGetExtent__FP12SQLB_GLOBALSP16SQLB_OBJECT_DESCUiPUiPc +
0x2F0
0x090000006A45CD68
sqluMDCRPGetAndInitNewExtent__FP13SQLUCACB_TYPEP8sqeAgentP13SQLO
_QUE_DESCUiUsi + 0xDC
0x090000006A17BEFC
@148@sqluRiddifierProcessRecList__FP13SQLUCACB_TYPEP8sqeAgentP18
SQLU_DMRECORD_LISTP13SQLO_QUE_DESCN24l + 0xBF8
0x090000006A1E4828
@148@sqluMDCRPProcessAnyCachedLists__FP13SQLUCACB_TYPEP14SQLU_RI
DDER_CB + 0x58
0x090000006A1E2C48
@148@sqluGetDMRecLists__FP13SQLUCACB_TYPEP14SQLU_RIDDER_CBl +
0x1A0
0x090000006A1E1E10 iRun__11sqluCRidderFv + 0x514
0x09000000692A4D74 sqluLoadEDUDriver__FPUcUi + 0xF8
0x0900000068429D30 sqloEDUEntry + 0x1C
Latches:
Address Holder Waiter Filename LOC
LatchType HoldCount
0x07000000300512F0 8756 9784 sqlptint.C 785
SQLO_LT_SQLP_TENTRY__seriallogging 1
-Holder was db2lfrm0
Stack:
======================
0x0900000000100158 thread_wait + 0x98
0x0900000068551D20 sqloWaitEDUWaitPost + 0x218
0x090000006A106C70 sqloSyncUp + 0x26C
0x090000006A1266B0
sqluFlushRidderData__FP13SQLUCACB_TYPEP17SQLU_FORMATTER_CBP12SQL
O_SYNCOBJ + 0x14
0x090000006A19A500 iRun__14sqluCFormatterFv + 0x1348
0x09000000692A4D74 sqluLoadEDUDriver__FPUcUi + 0xF8
0x0900000068429D30 sqloEDUEntry + 0x1C
Latches:
Holding Latch type: (SQLO_LT_SQLP_TENTRY__seriallogging) -
Address: (7000000300512f0), Line: 785, File: sqlptint.C
HoldCount: 1
Symptom 2:
db2diag.log entries
2015-05-05-09.29.17.012405-240 E10073839A564 LEVEL:
Severe
PID : 14680198 TID : 11372 PROC :
db2sysc 1
INSTANCE: db2inst1 NODE : 001 DB :
SAMPLE
APPHDL : 0-2706 APPID: *N0.db2inst1.150505132904
AUTHID : db2inst1 HOSTNAME: host1
EDUID : 11372 EDUNAME: db2lfrm0 1
FUNCTION: DB2 UDB, oper system services,
sqloDumpLatchTableToLog, probe:10
DATA #1 : String, 88 bytes
Latch leaked on EDU termination: File sqlptint.C : Line 873
(Identity: 407) HoldCount: 1
2015-05-05-09.29.17.022016-240 E10074404A444 LEVEL:
Severe
PID : 14680198 TID : 11372 PROC :
db2sysc 1
INSTANCE: db2inst1 NODE : 001
HOSTNAME: host1
EDUID : 11372 EDUNAME: db2lfrm0 1
FUNCTION: DB2 UDB, oper system services,
sqloDumpLatchTableToLog, probe:10
DATA #1 : String, 88 bytes
Latch leaked on EDU termination: File sqlptint.C : Line 873
(Identity: 407) HoldCount: 1
Every 5 minutes the following message repeats. However the LIST
INDOUBT TRANASACTIONS command will never show any indoubt
transactions.
2015-05-05-09.34.18.050518-240 I10078272A941 LEVEL:
Warning
PID : 20513220 TID : 4030 PROC :
db2sysc 2
INSTANCE: db2inst1 NODE : 002 DB :
SAMPLE
HOSTNAME: host2
EDUID : 4030 EDUNAME: db2loggr (NVDPFM01) 29
FUNCTION: DB2 UDB, data protection services,
sqpLoggrEdu::sqlpLoggrScanForIndoubts, probe:10
DATA #1 : String, 149 bytes
Lowtranlsn has been in an indoubt state for at least 5 minutes.
Use
the LIST INDOUBT TRANSACTIONS WITH PROMPTING command to resolve
this
transaction.
MESSAGE : Appl Name:
DATA #2 : String, 5 bytes
db2bp
DATA #3 : String, 12 bytes
Coord Node:
DATA #4 : unsigned integer, 2 bytes
0
DATA #5 : String, 19 bytes
Coord Agent Index:
DATA #6 : unsigned integer, 2 bytes
2706
DATA #7 : String, 9 bytes
Appl ID:
DATA #8 : String, 24 bytes
*N0.db2inst1.150505132904
DATA #9 : String, 9 bytes
Auth ID:
DATA #10: String, 8 bytes
DB2INST1
Symptom 3:
db2lrid stack
0x090000004E0DDCF0 sqloXlatchConflict + 0x32C
0x090000004AE108D8
sqlpBeginINDPSLoggingMode__FP8sqeAgentP11SQLP_TENTRY + 0x26C
0x090000004D443FCC
sqlpWriteLR__FP8sqeAgentUlN42P14SQLP_LREC_PARTT2P9SQLP_LSN8PUl +
0x25C
0x090000004ACEFCF8
squLogLoadInxInvalidation__FP8sqeAgentP10sqluCTableib + 0x2C8
0x090000004903666C
squMarkIndexBad__FP13SQLUCACB_TYPEP8sqeAgent + 0x110
0x090000004ADB1A90
squBuildIndexes__FP13SQLULACB_TYPEP13SQLUCACB_TYPEP18SQU_EDU_ESS
ENTIALSiUcP17SQLU_FUNCPTR_LIST + 0x1090
0x090000004C8FC280 iRun__11sqluCRidderFv + 0x7C0
0x090000004AE85584 sqluLoadEDUDriver__FPUcUi + 0x8C8
0x090000004CB1B9D4 sqloEDUEntry + 0x4AC
Waiting on latch type: (SQLO_LT_SQLP_TENTRY__seriallogging) -
Address: (0x700000030fbae58), Line: 875, File: sqlptint.C
db2lfrm0 stack
0x09000000001180B8 thread_wait + 0x98
0x090000004D617B44 sqloWaitEDUWaitPost + 0x260
0x090000004907C56C sqloSyncUp + 0x288
0x090000004A3132AC
sqluTellRidderNoMoreData__FP13SQLUCACB_TYPEP18SQU_EDU_ESSENTIALS
Uc + 0x5E0
0x0900000049009D7C iRun__14sqluCFormatterFv + 0x1AC0
0x090000004AE85584 sqluLoadEDUDriver__FPUcUi + 0x8C8
0x090000004CB1B9D4 sqloEDUEntry + 0x4AC
Holding Latch type: (SQLO_LT_SQLP_TENTRY__seriallogging) -
Address: (0x700000030fbae58), Line:875, File: sqlptint.C
HoldCount: 1 |
