This issue is caused by wrongly checking the node failure with
ROCM as to the status of that failed node.
It may cause -1229 error message logs in db2diag and
XAER_RMERR (-3) error in xa_end transaction.
The following messages are logged in db2diag.log during this
problem reproduction.
2016-08-04-17.54.11.466295+540 I18546175A594 LEVEL: Error
PID : 3735956 TID : 63498 PROC :
db2sysc 1
INSTANCE: db2inst1 NODE : 001 DB :
DBNAME
APPHDL : 1-17964 APPID:
10.5.123.148.34932.160803121313
AUTHID : DB2INST1 HOSTNAME: HOSTNAME
EDUID : 63498 EDUNAME: db2agent (DBNAME) 1
FUNCTION: DB2 UDB, buffer dist serv, sqlkdDispatchRequest,
probe:250
RETCODE : ZRC=0x81590016=-2124873706=SQLKF_NODE_FAILED "Node
Recovery"
DATA #1 : Codepath, 8 bytes
1:3:4:6:9:11:17:18:22:23:26:27:29:30:33:43:50:52:53
2016-08-04-17.54.11.467412+540 I18546770A1199 LEVEL:
Warning
PID : 3735956 TID : 63498 PROC :
db2sysc 1
INSTANCE: db2inst1 NODE : 001 DB :
DBNAME
APPHDL : 1-17964 APPID:
10.5.123.148.34932.160803121313
AUTHID : DB2INST1 HOSTNAME: HOSTNAME
EDUID : 63498 EDUNAME: db2agent (DBNAME) 1
FUNCTION: DB2 UDB, buffer dist serv, sqlkdDispatchRequest,
probe:300
DATA #1 : Hexdump, 125 bytes
0x0A00000022BF50F0 : 8000 0000 0000 0000 0000 0000 111D B511
................
0x0A00000022BF5100 : 0000 0020 0000 0000 0000 0000 0000 0000
... ............
0x0A00000022BF5110 : 0000 0001 15E5 E2E0 0780 0000 0739 0080
.............9..
0x0A00000022BF5120 : 0780 0000 0865 77E0 0000 0000 0028 00C2
.....ew......(..
0x0A00000022BF5130 : 2020 2020 0000 0000 2020 2020 0000 0000
.... ....
0x0A00000022BF5140 : 0000 0000 2020 2020 0A00 0000 22BF FED8
.... ...."...
0x0A00000022BF5150 : 0A00 0000 22BF FED8 2020 2020 2020 2020
...."...
0x0A00000022BF5160 : 0000 0000 0000 0000 0A00 0000 22
............"
DATA #2 : Codepath, 8 bytes
1:3:4:6:9:11:17:18:22:23:26:27:29:30:33:43:50:52:53
:
2016-08-04-17.54.11.467855+540 I18548857A1562 LEVEL:
Warning
PID : 3735956 TID : 63498 PROC :
db2sysc 1
INSTANCE: db2inst1 NODE : 001 DB :
DBNAME
APPHDL : 1-17964 APPID:
10.5.123.148.34932.160803121313
AUTHID : DB2INST1 HOSTNAME: HOSTNAME
EDUID : 63498 EDUNAME: db2agent (DBNAME) 1
FUNCTION: DB2 UDB, buffer dist serv, sqlkdDispatchRequest,
probe:4070
DATA #1 : String, 48 bytes
Request type encountering communication failure:
DATA #2 : Hex integer, 4 bytes
0x80000017
DATA #3 : String, 34 bytes
Did we attempt the send operation?
DATA #4 : Boolean, 1 bytes
true
DATA #5 : String, 22 bytes
Node failure handling:
DATA #6 : signed integer, 4 bytes
1
DATA #7 : String, 33 bytes
Number of node failures detected:
DATA #8 : signed integer, 4 bytes
1
DATA #9 : String, 19 bytes
BDS Failure Bitmap:
DATA #10: Hexdump, 125 bytes
0x078000000AD8AC58 : 8000 0000 0000 0000 0000 0000 0000 0000
................
0x078000000AD8AC68 : 0000 0000 0000 0000 0000 0000 0000 0000
................
0x078000000AD8AC78 : 0000 0000 0000 0000 0000 0000 0000 0000
................
0x078000000AD8AC88 : 0000 0000 0000 0000 0000 0000 0000 0000
................
0x078000000AD8AC98 : 0000 0000 0000 0000 0000 0000 0000 0000
................
0x078000000AD8ACA8 : 0000 0000 0000 0000 0000 0000 0000 0000
................
0x078000000AD8ACB8 : 0000 0000 0000 0000 0000 0000 0000 0000
................
0x078000000AD8ACC8 : 0000 0000 0000 0000 0000 0000 00
.............
2016-08-04-17.54.11.471762+540 I18551955A950 LEVEL: Error
PID : 3735956 TID : 63498 PROC :
db2sysc 1
INSTANCE: db2inst1 NODE : 001 DB :
DBNAME
APPHDL : 1-17964 APPID:
10.5.123.148.34932.160803121313
AUTHID : DB2INST1 HOSTNAME: HOSTNAME
EDUID : 63498 EDUNAME: db2agent (DBNAME) 1
FUNCTION: DB2 UDB, relation data serv, sqlrkend, probe:100
MESSAGE : ZRC=0x81580016=-2124939242=SQLKD_NODE_FAILURE
"Mapping for SQLKF_NODE_FAILED"
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: -1229 sqlerrml: 0
sqlerrmc:
sqlerrp : SQLRR06A
sqlerrd : (1) 0x81580016 (2) 0x00000016 (3)
0x00000000
(4) 0x00000000 (5) 0xFFFFFA24 (6)
0x00000001
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate: |
