In temp file ibm.db2.cluster.* , we can see timeout when run
that command "db2cluster -cfs -network_resiliency -add -all" ,
and retry every 20 mins , the fixpack installation could
complete successfully though there is such timeout error, while
the timeout extend the installation to 1h20min(normal situation
is 30-40mins) for each node , this increased the maintenance
time window for the whole process .
2018-11-20-16.14.23.196861+480 I1751E310 LEVEL: Info
PID : 18702 TID : 140737352718112 PROC : db2cluster
INSTANCE: db2sdin1 NODE : 001
HOSTNAME: db2dev2-m01
FUNCTION: , , , probe:2681
DATA #1 : String, 46 bytes
db2cluster -cfs -network_resiliency -add -all
2018-11-20-16.36.17.347424+480 I2062E627 LEVEL: Error
PID : 18702 TID : 140737352718112 PROC : db2cluster
INSTANCE: db2sdin1 NODE : 001
HOSTNAME: db2dev2-m01
FUNCTION: DB2 UDB, oper system services, sqloReadNamedPipe,
probe:1277
MESSAGE : ZRC=0x870F00B3=-2029059917=SQLO_NPIPE_TIMEOUT
"Operation on named pipe timed out."
DATA #1 : String, 55 bytes
Timeout error occurred with the following timeout value
DATA #2 : String, 18 bytes
timeout (seconds):
DATA #3 : signed integer, 8 bytes
0
DATA #4 : String, 23 bytes
timeout (microseconds):
DATA #5 : signed integer, 8 bytes
0
2018-11-20-16.36.17.350992+480 E5256E1480 LEVEL: Severe
PID : 18702 TID : 140737352718112 PROC : db2cluster
INSTANCE: db2sdin1 NODE : 001
HOSTNAME: db2dev2-m01
FUNCTION: DB2 UDB, high avail services, sqlhaExecuteCommandSet,
probe:579
RETCODE : ZRC=0x827300D3=-2106392365=HA_ZRC_CLUSTER_TIMEOUT
"Cluster operation time out"
DATA #1 : SQLHA Remote Command Set, PD_TYPE_SQLHA_COMMAND_SET,
293272 bytes
commandSet->numCommands: 1
commandSet->options: NO THREAD
commandSet->options: CLUSTER REMOTE COMMAND
commandSet->previousDb2RshCmd:
DATA #2 : unsigned integer, 8 bytes
0
DATA #3 : SQLHA Remote Command, PD_TYPE_SQLHA_COMMAND, 2008
bytes
command->commandData->execName:
/opt/ibm/db2/V10.5SP38004/bin/db2cluster
command->commandData->arguments: -cfs -add -network_resiliency
-gpfsadapter -noop -rocmout
command->commandData->hostname: db2dev2-cf2
command->commandData->username: root
command->options: NO THREAD
command->options: CLUSTER REMOTE COMMAND
command->pResponse: 0x00007fffffff7c30
CALLSTCK: (Static functions may not be resolved correctly, as
they are resolved to the nearest symbol)
[0] 0x0000000000523A4D
_Z22sqlhaExecuteCommandSetP17SQLHA_COMMAND_SET + 0x2FD
[1] 0x00000000004978DA db2cluster + 0x978DA
[2] 0x000000000049567B _Z24sqlhaUICFSCreateCondRespP9sqlhaUICBb
+ 0x12B
[3] 0x000000000042734D main + 0x195D
[4] 0x00007FFFF34D1C36 __libc_start_main + 0xE6
[5] 0x0000000000425959 db2cluster + 0x25959
The root cause is : db2cluster command spawns the db2havend
process from the current $HOME/sqllib/adm path which at the time
it is the fp8 version. This is wrong, db2cluster with fp10
version must spawn the fp10 version of db2havend.
This behaviour always existed . However, there were changes to
read/write pipe behaviour in fp10 which caused the issue |
