This error only exists in 11.5.6 and 11.5.7.
On HADR standby startup, standby scans log files in active log
path to see if log files needed for recovery exist. However,
this scan may delete log file that was previously retrieved from
archive, which may cause subsequent standby replay fail to find
it. In that case, replay cannot continue as the log file is
needed for recovery. Eventually, standby will be forced to shut
down due to replay error.
If this were to happen, on the standby one may see messages in
the db2diag.log as follows:
1. Standby retrieved log file (S0000002.LOG) from archive:
2022-02-17-00.27.47.956835-300 I739230E579 LEVEL: Info
PID : 26608 TID : 139992252802816 PROC :
db2sysc
INSTANCE: myinstance NODE : 000 DB : MYDB
HOSTNAME: myhost
EDUID : 82 EDUNAME: db2logmgr (MYDB)
FUNCTION: DB2 UDB, data protection services,
sqpLogMgrEdu::sqlpgRetrieveLogFile, probe:4148
DATA #1 :
Completed retrieve for log file S0000001.LOG on chain 1 to
/home/myinstance/myinstance/NODE0000/SQL00001/LOGSTREAM0000/LOGS
TREAM0000/. The newly retrieved log will be verified.
2. Standby replay cannot find log file S0000002.LOG:
2022-02-17-00.28.37.253042-300 I1629509E537 LEVEL: Info
PID : 4946 TID : 140631208879872 PROC :
db2sysc
INSTANCE: myinstance NODE : 000 DB : MYDB
HOSTNAME: myhost
EDUID : 28 EDUNAME: db2lfr.0 (MYDB)
FUNCTION: DB2 UDB, data protection services, sqlpgOpenLogExtent,
probe:40
MESSAGE : ZRC=0x860F000A=-2045837302=SQLO_FNEX "File not found."
DIA8411C A file "" could not be found.
DATA #1 :
Error opening database log extent 2.
2022-02-17-00.28.37.332256-300 I1634260E596 LEVEL:
Error
PID : 4946 TID : 140631208879872 PROC :
db2sysc
INSTANCE: myinstance NODE : 000 DB : MYDB
HOSTNAME: myhost
EDUID : 28 EDUNAME: db2lfr.0 (MYDB)
FUNCTION: DB2 UDB, recovery manager, sqlplfrDoScanNext,
probe:1100
MESSAGE : hit EOL for hadr
DATA #1 : SQLPLFR_SCAN_ID, PD_TYPE_SQLPLFR_SCAN_ID, 8 bytes
LFR Scan Num = 5
LFR Scan Caller's EDUID = 79
DATA #2 : Hex integer, 8 bytes
0x0000000004031008
DATA #3 : unsigned integer, 4 bytes
2249129994
3. Standby replay failed due to missing log file:
2022-02-17-00.28.37.601962-300 I1669869E1100 LEVEL: Info
PID : 4946 TID : 140631234045696 PROC :
db2sysc
INSTANCE: myinstance NODE : 000 DB : MYDB
APPHDL : 0-8 APPID: *LOCAL.DB2.220217052837
HOSTNAME: myhost
EDUID : 22 EDUNAME: db2agent (MYDB)
FUNCTION: DB2 UDB, recovery manager, sqlpReplayMaster,
probe:2460
MESSAGE : ZRC=0x860F000A=-2045837302=SQLO_FNEX "File not found."
DIA8411C A file "" could not be found.
DATA #1 : String, 88 bytes
Replay master encoutered some error. Try to restart standby to
handle recoverable error.
DATA #2 : unsigned integer, 8 bytes
0
DATA #3 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: -1042 sqlerrml: 0
sqlerrmc:
sqlerrp : sqlpRepl
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3)
0x00000000
(4) 0x00000000 (5) 0x00000000 (6)
0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:
4. Standby get forced shut down due to replay failure:
2022-02-17-00.28.44.160556-300 I2122852E622 LEVEL:
Severe
PID : 4946 TID : 140628885235456 PROC :
db2sysc
INSTANCE: myinstance NODE : 000 DB : MYDB
APPHDL : 0-14 APPID: *LOCAL.DB2.220217052843
HOSTNAME: myhost
EDUID : 163 EDUNAME: db2agent (MYDB)
FUNCTION: DB2 UDB, recovery manager, sqlpReplayMaster,
probe:2500
MESSAGE : ZRC=0x860F000A=-2045837302=SQLO_FNEX "File not found."
DIA8411C A file "" could not be found.
DATA #1 :
Fatal error during HADR replay. Forcing the database to shut
down.
2022-02-17-00.28.44.161403-300 I2123475E1339 LEVEL: Info
PID : 4946 TID : 140628889429760 PROC :
db2sysc
INSTANCE: myinstance NODE : 000 DB : MYDB
APPHDL : 0-15 APPID: *LOCAL.DB2.220217052844
HOSTNAME: myhost
EDUID : 204 EDUNAME: db2agent (MYDB)
FUNCTION: DB2 UDB, base sys utilities,
sqeLocalDatabase::ForceDBShutdown, probe:17516
MESSAGE : Regular agent EDU doing ForceDBShutdown. Force DB
shutdown agent ID
is:
DATA #1 : APPHDL, PD_TYPE_SQLZ_APPHDL, 4 bytes
0-15
DATA #2 : sqeApplication_acbInfo,
PD_TYPE_sqeApplication_acbInfo, 4 bytes
x0
CALLSTCK: (Static functions may not be resolved correctly, as
they are resolved to the nearest symbol)
[0] 0x00007FE7592FE7BF
_ZN16sqeLocalDatabase15ForceDBShutdownEi + 0x18F
[1] 0x00007FE752FEAB02
_ZN21HDR_FORCE_DB_SHUTDOWN16hdrExecOperationEP8sqeAgent + 0x12
[2] 0x00007FE752FEB27C
_ZN18HDR_ICOORD_COMMAND19hdrICoordCmdExecuteEP8sqeAgent + 0xDC
[3] 0x00007FE75914B9EF
_Z26sqleIndCoordProcessRequestP8sqeAgent + 0x1BDF
[4] 0x00007FE75917DC78 _ZN8sqeAgent6RunEDUEv + 0x538
[5] 0x00007FE75D445D94 _ZN9sqzEDUObj9EDUDriverEv + 0x1A4
[6] 0x00007FE75B6D05D9 sqloEDUEntry + 0x2A9
[7] 0x00007FE762317EA5 /lib64/libpthread.so.0 + 0x7EA5
[8] 0x00007FE74FA518DD clone + 0x6D |
