(This issue affects DB2 9.7 fixpack 6 only. It should not
affect AIX and Windows platforms. Any other "load hanging"
issues that do not match the diagnostics below, or are not with
DB2 9.7 fixpack 6, are not related to this APAR.)
A load hangs at startup time with the following symptoms:
(1) in db2diag.log there are error messages indicating a
SQLO_QUE_NOT_SENT error, followed by messages pointing to the
sqluAllocTBufs() function, like these:
2012-06-04-16.26.29.483991-240 E473381E3213 LEVEL:
Error (OS)
PID : 25108 TID : 46931657091392 KTID :
29706
PROC : db2sysc 0
INSTANCE: eunicec NODE : 000 DB :
CORAL
APPHDL : 0-82 APPID: *N0.eunicec.120604202625
AUTHID : EUNICEC HOSTNAME: coralm221
EDUID : 143 EDUNAME: db2agent (CORAL) 0
FUNCTION: DB2 UDB, oper system services, sqlowqueInternal,
probe:40
MESSAGE : ZRC=0x870F0041=-2029060031=SQLO_QUE_NOT_SENT "Message
Not Sent"
DIA8557C No message was sent using the message queue.
CALLED : OS, -, msgsnd OSERR: EAGAIN
(11)
DATA #1 : system V message queue identifier.,
PD_TYPE_SYSV_QUEUE_ID, 4 bytes
0x35270044
DATA #2 : Pointer, 8 bytes
0x00002aaf20beb0a8
DATA #3 : unsigned integer, 8 bytes
48
CALLSTCK: (Static functions may not be resolved correctly, as
they are resolved to the nearest symbol)
[0] 0x00002AAAACDF8EF2 pdLogSysRC + 0x22C
[1] 0x00002AAAB074EE77
_Z16sqlowqueInternalP13SQLO_QUE_DESCP12SQLO_MSG_HDRib + 0x549
[2] 0x00002AAAB074FDA7
_Z15sqlowqueWrapperP13SQLO_QUE_DESCP12SQLO_MSG_HDRib + 0x59
[3] 0x00002AAAB075030A sqlowqueNoBlock + 0x10E
[4] 0x00002AAAB255BA0A
_Z14sqluWriteQueueP13SQLO_QUE_DESCP12SQLO_MSG_HDRii + 0xE6
[5] 0x00002AAAB26C2330 _Z14sqluAllocTBufsP13SQLUCACB_TYPEPii +
0x1E8A
[6] 0x00002AAAB26226D7 /home/eunicec/sqllib/lib64/libdb2e.so.1
+ 0x79426D7
[7] 0x00002AAAB26099A8
_Z9sqluvloadPhS_P9SQLP_LSN8P7sqlubmhPP15sqlu_media_listS6_hP26sq
luCLoadRequestDescriptorP8sqluvlcbPi + 0x3A82
[8] 0x00002AAAB25208F2 /home/eunicec/sqllib/lib64/libdb2e.so.1
+ 0x78408F2
[9] 0x00002AAAB25284C4
_Z17sqluLoadPartitionP8sqeAgentP26sqluCLoadRequestDescriptoriiiP
iS3_ + 0x11EE
[10] 0x00002AAAB251EA23
_Z24sqlu_register_table_loadP26sqluCLoadRequestDescriptorP24sqlu
sCLoadMPPCoordinatorP15sql_static_dataPb + 0x1BC9
[11] 0x00002AAAB24F2777 _Z17sqluvtld_route_inP12SQLE_DB2RA_T +
0x2255
[12] 0x00002AAAADD958AA _Z8sqlesrvrP14db2UCinterface + 0x1336
[13] 0x00002AAAADDFE9BC _Z19sqleMappingFnServerP5sqldaP5sqlca
+ 0x15EC
[14] 0x00002AAAAE3F3EC9
_Z19sqlerKnownProcedureiPcPiP5sqldaS2_P13sqlerFmpTableP8sqeAgent
P5sqlca + 0x18C9
[15] 0x00002AAAAE3FB397
_Z11sqlerCallDLP14db2UCinterfaceP9UCstpInfo + 0x248F
[16] 0x00002AAAAE6D5D27
_Z19sqljs_ddm_excsqlsttP14db2UCinterfaceP13sqljDDMObject + 0x4EB
[17] 0x00002AAAAE6A4FC2
_Z21sqljsParseRdbAccessedP13sqljsDrdaAsCbP13sqljDDMObjectP14db2U
Cinterface + 0x25C
[18] 0x00002AAAAE6A6249
_Z10sqljsParseP13sqljsDrdaAsCbP14db2UCinterfaceP8sqeAgentb +
0x239
[19] 0x00002AAAAE698899
/home/eunicec/sqllib/lib64/libdb2e.so.1 + 0x39B8899
[20] 0x00002AAAAE69AADB
/home/eunicec/sqllib/lib64/libdb2e.so.1 + 0x39BAADB
[21] 0x00002AAAAE695FDF
/home/eunicec/sqllib/lib64/libdb2e.so.1 + 0x39B5FDF
[22] 0x00002AAAAE69584C
_Z17sqljsDrdaAsDriverP18SQLCC_INITSTRUCT_T + 0x29A
[23] 0x00002AAAADE086E6 _ZN8sqeAgent6RunEDUEv + 0x4F8
[24] 0x00002AAAB3094260 _ZN9sqzEDUObj9EDUDriverEv + 0x17C
[25] 0x00002AAAB3093E88 _Z10sqlzRunEDUPcj + 0x26
[26] 0x00002AAAB08E8BB7 sqloEDUEntry + 0x971
..................
2012-06-04-16.26.40.830447-240 I510014E1061 LEVEL:
Error
PID : 25108 TID : 46931657091392 KTID :
29706
PROC : db2sysc 0
INSTANCE: eunicec NODE : 000 DB :
CORAL
APPHDL : 0-82 APPID: *N0.eunicec.120604202625
AUTHID : EUNICEC HOSTNAME: coralm221
EDUID : 143 EDUNAME: db2agent (CORAL) 0
FUNCTION: DB2 UDB, database utilities, sqluAllocTBufs,
probe:2599
MESSAGE : ZRC=0x8015006D=-2146107283=SQLU_CA_BUILT
"SQLCA has already been built"
DATA #1 : LOADID, PD_TYPE_LOADID, 47 bytes
LOADID: 143.2012-06-04-16.26.25.889421.0 (22;4)
DATA #2 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: -2044 sqlerrml: 1
sqlerrmc: 3
sqlerrp : SQLUBUF2
sqlerrd : (1) 0x8015006D (2) 0x00000000 (3)
0x00000000
(4) 0x00000000 (5) 0x00000000 (6)
0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
2012-06-04-16.26.40.834263-240 I511076E555 LEVEL:
Severe
PID : 25108 TID : 46931657091392 KTID :
29706
PROC : db2sysc 0
INSTANCE: eunicec NODE : 000 DB :
CORAL
APPHDL : 0-82 APPID: *N0.eunicec.120604202625
AUTHID : EUNICEC HOSTNAME: coralm221
EDUID : 143 EDUNAME: db2agent (CORAL) 0
FUNCTION: DB2 UDB, database utilities, sqluAllocTBufs,
probe:2599
MESSAGE : DIA0001E An internal error occurred. Report the
following error code
: "".
(2) the hanging DB2 agent's stack trace shows load terminating
functions (sqlulTermEDU() / sqlulTerminate()) trying to write to
a message queue (sqlowque()) and is stuck, like this:
0x00002AAAB8B3BD8F
_Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm +
0x046f
(/home/eunicec/sqllib/lib64/libdb2osse.so.1)
0x00002AAAB8B3C6DF ossDumpStackTraceV98 + 0x0081
(/home/eunicec/sqllib/lib64/libdb2osse.so.1)
0x00002AAAB8B34BE7 _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm +
0x0377
(/home/eunicec/sqllib/lib64/libdb2osse.so.1)
0x00002AAAB074A8B6 sqlo_trce + 0x0678
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB08EC534 sqloDumpDiagInfoHandler + 0x0212
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAABD4D60 address: 0x00002AAAAABD4D60 ; dladdress:
0x00002AAAAABC7000 ; offset in lib: 0x000000000000DD60 ;
(/lib64/libpthread.so.0)
0x00002AAAB954A368 msgsnd + 0x0058
(/lib64/libc.so.6)
0x00002AAAB0752747 _Z21uninterruptableMsgSndiPKvmi + 0x0037
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB074EC87
_Z16sqlowqueInternalP13SQLO_QUE_DESCP12SQLO_MSG_HDRib + 0x0359
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB074FDA7
_Z15sqlowqueWrapperP13SQLO_QUE_DESCP12SQLO_MSG_HDRib + 0x0059
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB07501F0 sqlowque + 0x0036
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB255B9D8
_Z14sqluWriteQueueP13SQLO_QUE_DESCP12SQLO_MSG_HDRii + 0x00b4
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB261970E
_Z15sqlulSndTermMsgP13SqluQueueInfoP13SQLUCACB_TYPEN22SqluTermin
ationMessage21LoadTerminationReasonE + 0x0230
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB2618AC4
_Z14sqlulTerminateP13SQLUCACB_TYPEPsN22SqluTerminationMessage21L
oadTerminationReasonEPi + 0x0550
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB260CB34 _Z12sqlulTermEDUP13SQLUCACB_TYPEiPi + 0x0144
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB260B99B
_Z9sqluvloadPhS_P9SQLP_LSN8P7sqlubmhPP15sqlu_media_listS6_hP26sq
luCLoadRequestDescriptorP8sqluvlcbPi + 0x5a75
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB25208F2 address: 0x00002AAAB25208F2 ; dladdress:
0x00002AAAAACE0000 ; offset in lib: 0x00000000078408F2 ;
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB25284C4
_Z17sqluLoadPartitionP8sqeAgentP26sqluCLoadRequestDescriptoriiiP
iS3_ + 0x11ee
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB251EA23
_Z24sqlu_register_table_loadP26sqluCLoadRequestDescriptorP24sqlu
sCLoadMPPCoordinatorP15sql_static_dataPb + 0x1bc9
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB24F2777 _Z17sqluvtld_route_inP12SQLE_DB2RA_T + 0x2255
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAADD958AA _Z8sqlesrvrP14db2UCinterface + 0x1336
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAADDFE9BC _Z19sqleMappingFnServerP5sqldaP5sqlca +
0x15ec
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE3F3EC9
_Z19sqlerKnownProcedureiPcPiP5sqldaS2_P13sqlerFmpTableP8sqeAgent
P5sqlca + 0x18c9
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE3FB397 _Z11sqlerCallDLP14db2UCinterfaceP9UCstpInfo +
0x248f
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE6D5D27
_Z19sqljs_ddm_excsqlsttP14db2UCinterfaceP13sqljDDMObject +
0x04eb
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE6A4FC2
_Z21sqljsParseRdbAccessedP13sqljsDrdaAsCbP13sqljDDMObjectP14db2U
Cinterface + 0x025c
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE6A6249
_Z10sqljsParseP13sqljsDrdaAsCbP14db2UCinterfaceP8sqeAgentb +
0x0239
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE698899 address: 0x00002AAAAE698899 ; dladdress:
0x00002AAAAACE0000 ; offset in lib: 0x00000000039B8899 ;
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE69AADB address: 0x00002AAAAE69AADB ; dladdress:
0x00002AAAAACE0000 ; offset in lib: 0x00000000039BAADB ;
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE695FDF address: 0x00002AAAAE695FDF ; dladdress:
0x00002AAAAACE0000 ; offset in lib: 0x00000000039B5FDF ;
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAE69584C _Z17sqljsDrdaAsDriverP18SQLCC_INITSTRUCT_T +
0x029a
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAADE086E6 _ZN8sqeAgent6RunEDUEv + 0x04f8
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB3094260 _ZN9sqzEDUObj9EDUDriverEv + 0x017c
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB3093E88 _Z10sqlzRunEDUPcj + 0x0026
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAB08E8BB7 sqloEDUEntry + 0x0971
(/home/eunicec/sqllib/lib64/libdb2e.so.1)
0x00002AAAAABCD193 address: 0x00002AAAAABCD193 ; dladdress:
0x00002AAAAABC7000 ; offset in lib: 0x0000000000006193 ;
(/lib64/libpthread.so.0)
0x00002AAAB95490DD __clone + 0x006d
(/lib64/libc.so.6) |
Workarounds include
1) Don't use CPU_PARALLELISM 1 (implicitly or explicitly), or
explicitly use a higher value
2) If you cannot change from CPU_PARALLELISM 1 (e.g. if table
has LOBs), lower the DATA BUFFER value (explicitly, or
implicitly via lowering the UTIL_HEAP_SZ database configuration
parameter) |
