Db2 may panic during a ForceDBShutdown process after the db logs
have been marked bad.
One scenario is a HADR forced takeover:
The primary was handling a forced takeover, took the poison pill
and has marked the db logs bad, then a page cleaner tried to
flush log records, saw the bad logs and then panicked.
This works as design to prevent page cleaners from proceeding
further when db shut down is already in progress.
The workaround is to restart db2.
2020-01-08-19.06.42.880714+480 I1691215E440 LEVEL: Info
PID : 20252 TID : 140736708208384 PROC : db2sysc 0
EDUID : 62 EDUNAME: db2hadrp.0.1 (XXXX) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrPriTkHandleInitialRequest, probe:56020
MESSAGE : Primary has started a forced takeover request.
2020-01-08-19.06.42.881791+480 I1691656E528 LEVEL: Info
PID : 20252 TID : 140731289167616 PROC : db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
APPHDL : 0-44774 APPID: *LOCAL.DB2.200108110642
EDUID : 776 EDUNAME: db2hdrptkpp (XXXX) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrPoisonLocalMember, probe:50
MESSAGE : Received poison pill!
DATA #1 : db2LogStreamIDType, PD_TYPE_DB2_LOG_STREAM_ID, 2 bytes
0
2020-01-08-19.06.42.890366+480 I1696940E553 LEVEL: Error
PID : 20252 TID : 140731289167616 PROC : db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
APPHDL : 0-44774 APPID: *LOCAL.DB2.200108110642
EDUID : 776 EDUNAME: db2hdrptkpp (XXXX) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrPoisonLocalMember, probe:41160
DATA #1 :
HADR marking logs bad; database should shut down to avoid split
brain; standby is taking over.
2020-01-08-19.06.42.891073+480 E1697494E1265 LEVEL:
Severe
PID : 20252 TID : 140731289167616 PROC :
db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
APPHDL : 0-44774 APPID: *LOCAL.DB2.200108110642
EDUID : 776 EDUNAME: db2hdrptkpp (XXXX) 0
FUNCTION: DB2 UDB, data protection services,
SQLP_DBCB::setLogState, probe:5000
DATA #1 :
Database error has been detected. As a result, for
precautionary reasons
all logging services have been stopped.
CALLSTCK: (Static functions may not be resolved correctly, as
they are resolved to the nearest symbol)
[0] 0x00007FFFEA7B562D pdLogPrintf + 0x8D
[1] 0x00007FFFF050279C
_Z18sqlpSetLogStateBadP16sqeLocalDatabase + 0x6C
[2] 0x00007FFFEA47D2EE
_Z20hdrPoisonLocalMemberP9sqeBsuEduP8HDR_DBCB + 0x5BE
[3] 0x00007FFFEA47CC7B _Z22hdrPoisonPrimaryIcoordP8sqeAgent +
0xBB
[4] 0x00007FFFEE2C4D8B
_Z26sqleIndCoordProcessRequestP8sqeAgent + 0xAFB
[5] 0x00007FFFEE2EB111 _ZN8sqeAgent6RunEDUEv + 0x5A1
[6] 0x00007FFFF1BAD84E _ZN9sqzEDUObj9EDUDriverEv + 0x1BE
[7] 0x00007FFFF010EDB8 sqloEDUEntry + 0x578
[8] 0x00007FFFF7BC6DD5 /usr/lib64/libpthread.so.0 + 0x7DD5
[9] 0x00007FFFE78F8B3D clone + 0x6D
2020-01-08-19.06.42.899762+480 I1699319E1233 LEVEL: Info
PID : 20252 TID : 140731289167616 PROC :
db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
APPHDL : 0-44774 APPID: *LOCAL.DB2.200108110642
EDUID : 776 EDUNAME: db2hdrptkpp (XXXX) 0
FUNCTION: DB2 UDB, base sys utilities,
sqeLocalDatabase::ForceDBShutdown, probe:16912
MESSAGE : Regular agent EDU doing ForceDBShutdown. Force DB
shutdown agent ID
is:
DATA #1 : APPHDL, PD_TYPE_SQLZ_APPHDL, 4 bytes
0-44774
DATA #2 : sqeApplication_acbInfo,
PD_TYPE_sqeApplication_acbInfo, 4 bytes
x0
CALLSTCK: (Static functions may not be resolved correctly, as
they are resolved to the nearest symbol)
[0] 0x00007FFFEE4149DF
_ZN16sqeLocalDatabase15ForceDBShutdownEi + 0x87F
[1] 0x00007FFFEA47CCA0 _Z22hdrPoisonPrimaryIcoordP8sqeAgent +
0xE0
[2] 0x00007FFFEE2C4D8B
_Z26sqleIndCoordProcessRequestP8sqeAgent + 0xAFB
[3] 0x00007FFFEE2EB111 _ZN8sqeAgent6RunEDUEv + 0x5A1
[4] 0x00007FFFF1BAD84E _ZN9sqzEDUObj9EDUDriverEv + 0x1BE
[5] 0x00007FFFF010EDB8 sqloEDUEntry + 0x578
[6] 0x00007FFFF7BC6DD5 /usr/lib64/libpthread.so.0 + 0x7DD5
[7] 0x00007FFFE78F8B3D clone + 0x6D
|
| Page cleaner tries to flush log records and then see
SQLP_BADLOG and panic.
|
V
2020-01-08-19.06.44.609247+480 I2093709E483 LEVEL:
Error
PID : 20252 TID : 140736817260288 PROC :
db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
EDUID : 36 EDUNAME: db2pclnr (XXXX) 0
FUNCTION: DB2 UDB, data protection services, sqlpgild, probe:310
MESSAGE : ZRC=0x8610000D=-2045771763=SQLP_BADLOG "Log File
cannot be used"
DIA8414C Logging can not continue due to an error.
2020-01-08-19.06.44.623804+480 E2095416E548 LEVEL:
Error
PID : 20252 TID : 140736817260288 PROC :
db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
EDUID : 36 EDUNAME: db2pclnr (XXXX) 0
FUNCTION: DB2 UDB, data protection services, sqlpflog, probe:480
MESSAGE : ZRC=0x8610000D=-2045771763=SQLP_BADLOG "Log File
cannot be used"
DIA8414C Logging can not continue due to an error.
DATA #1 : SQLP_LSN8, PD_TYPE_SQLP_LSN8, 8 bytes
000000000470184E
2020-01-08-19.06.44.624422+480 I2095965E493 LEVEL:
Error
PID : 20252 TID : 140736817260288 PROC :
db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
EDUID : 36 EDUNAME: db2pclnr (XXXX) 0
FUNCTION: DB2 UDB, buffer pool services,
sqlbClnrAsyncWriteSetup, probe:80
MESSAGE : ZRC=0x8610000D=-2045771763=SQLP_BADLOG "Log File
cannot be used"
DIA8414C Logging can not continue due to an error.
2020-01-08-19.06.44.641776+480 I2096459E4034 LEVEL:
Severe
PID : 20252 TID : 140736817260288 PROC :
db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
EDUID : 36 EDUNAME: db2pclnr (XXXX) 0
FUNCTION: DB2 UDB, buffer pool services,
sqlbClnrAsyncWriteSetup, probe:300
DATA #1 : Buffer page descriptor, PD_TYPE_SQLB_BPD, 256 bytes
Pagekey: {pool:0;obj:13;type:0} PPNum:39
2020-01-08-19.06.44.661377+480 E2100494E896 LEVEL: Critical
PID : 20252 TID : 140736817260288 PROC : db2sysc 0
INSTANCE: XXXXXX NODE : 000 DB : XXXX
EDUID : 36 EDUNAME: db2pclnr (XXXX) 0
FUNCTION: DB2 UDB, RAS/PD component, pdStartFODC, probe:10
MESSAGE : ADM14001C An unexpected and critical error has
occurred: "Panic".
The instance may have been shutdown as a result.
"Automatic" FODC
(First Occurrence Data Capture) has been invoked and
diagnostic
information has been recorded in directory
"/db2_home/db2instance/sminst/sqllib/db2dump/FODC_Panic_2020-01-
08-19
.06.44.645897_0000/". Please look in this directory for
detailed
evidence about what happened and contact IBM support if
necessary to
diagnose the problem.
Stack #1 Signal #6 Timestamp
2020-01-08-19.06.54.998654
0 ossDumpStackTraceInternal
1 ossDumpStackTraceV98
2 OSSTrapFile::dumpEx
3 sqlo_trce
4 sqloEDUCodeTrapHandler
5 pthread_mutexattr_getprioceiling
6 gsignal
7 abort
8 sqloExitEDU
9 sqle_panic
10 sqlb_panic
11 sqlbClnrAsyncWriteSetup
12 SQLO_LIO_HANDLE_DATA::sqloDispatchNBlocks
13 sqlbClnrDispatchSomeAIO
14 sqlbClnrWritePages
15 sqlbClnrEntryPoint
16 sqbPgClnrEdu::RunEDU
17 sqzEDUObj::EDUDriver
18 sqloEDUEntry
19 __pthread_mutex_lock_full
20 clone |
