DB2 - Problem description
Problem IT31479 | Status: Closed |
DB2 MAY PANIC DURING ForceDBShutdown | |
product: | |
DB2 FOR LUW / DB2FORLUW / B10 - DB2 | |
Problem description: | |
Db2 may panic during a ForceDBShutdown process after the db logs have been marked bad. One scenario is a HADR forced takeover: The primary was handling a forced takeover, took the poison pill and has marked the db logs bad, then a page cleaner tried to flush log records, saw the bad logs and then panicked. This works as design to prevent page cleaners from proceeding further when db shut down is already in progress. The workaround is to restart db2. 2020-01-08-19.06.42.880714+480 I1691215E440 LEVEL: Info PID : 20252 TID : 140736708208384 PROC : db2sysc 0 EDUID : 62 EDUNAME: db2hadrp.0.1 (XXXX) 0 FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrPriTkHandleInitialRequest, probe:56020 MESSAGE : Primary has started a forced takeover request. 2020-01-08-19.06.42.881791+480 I1691656E528 LEVEL: Info PID : 20252 TID : 140731289167616 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX APPHDL : 0-44774 APPID: *LOCAL.DB2.200108110642 EDUID : 776 EDUNAME: db2hdrptkpp (XXXX) 0 FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrPoisonLocalMember, probe:50 MESSAGE : Received poison pill! DATA #1 : db2LogStreamIDType, PD_TYPE_DB2_LOG_STREAM_ID, 2 bytes 0 2020-01-08-19.06.42.890366+480 I1696940E553 LEVEL: Error PID : 20252 TID : 140731289167616 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX APPHDL : 0-44774 APPID: *LOCAL.DB2.200108110642 EDUID : 776 EDUNAME: db2hdrptkpp (XXXX) 0 FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrPoisonLocalMember, probe:41160 DATA #1 : HADR marking logs bad; database should shut down to avoid split brain; standby is taking over. 2020-01-08-19.06.42.891073+480 E1697494E1265 LEVEL: Severe PID : 20252 TID : 140731289167616 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX APPHDL : 0-44774 APPID: *LOCAL.DB2.200108110642 EDUID : 776 EDUNAME: db2hdrptkpp (XXXX) 0 FUNCTION: DB2 UDB, data protection services, SQLP_DBCB::setLogState, probe:5000 DATA #1 : Database error has been detected. As a result, for precautionary reasons all logging services have been stopped. CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol) [0] 0x00007FFFEA7B562D pdLogPrintf + 0x8D [1] 0x00007FFFF050279C _Z18sqlpSetLogStateBadP16sqeLocalDatabase + 0x6C [2] 0x00007FFFEA47D2EE _Z20hdrPoisonLocalMemberP9sqeBsuEduP8HDR_DBCB + 0x5BE [3] 0x00007FFFEA47CC7B _Z22hdrPoisonPrimaryIcoordP8sqeAgent + 0xBB [4] 0x00007FFFEE2C4D8B _Z26sqleIndCoordProcessRequestP8sqeAgent + 0xAFB [5] 0x00007FFFEE2EB111 _ZN8sqeAgent6RunEDUEv + 0x5A1 [6] 0x00007FFFF1BAD84E _ZN9sqzEDUObj9EDUDriverEv + 0x1BE [7] 0x00007FFFF010EDB8 sqloEDUEntry + 0x578 [8] 0x00007FFFF7BC6DD5 /usr/lib64/libpthread.so.0 + 0x7DD5 [9] 0x00007FFFE78F8B3D clone + 0x6D 2020-01-08-19.06.42.899762+480 I1699319E1233 LEVEL: Info PID : 20252 TID : 140731289167616 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX APPHDL : 0-44774 APPID: *LOCAL.DB2.200108110642 EDUID : 776 EDUNAME: db2hdrptkpp (XXXX) 0 FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::ForceDBShutdown, probe:16912 MESSAGE : Regular agent EDU doing ForceDBShutdown. Force DB shutdown agent ID is: DATA #1 : APPHDL, PD_TYPE_SQLZ_APPHDL, 4 bytes 0-44774 DATA #2 : sqeApplication_acbInfo, PD_TYPE_sqeApplication_acbInfo, 4 bytes x0 CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol) [0] 0x00007FFFEE4149DF _ZN16sqeLocalDatabase15ForceDBShutdownEi + 0x87F [1] 0x00007FFFEA47CCA0 _Z22hdrPoisonPrimaryIcoordP8sqeAgent + 0xE0 [2] 0x00007FFFEE2C4D8B _Z26sqleIndCoordProcessRequestP8sqeAgent + 0xAFB [3] 0x00007FFFEE2EB111 _ZN8sqeAgent6RunEDUEv + 0x5A1 [4] 0x00007FFFF1BAD84E _ZN9sqzEDUObj9EDUDriverEv + 0x1BE [5] 0x00007FFFF010EDB8 sqloEDUEntry + 0x578 [6] 0x00007FFFF7BC6DD5 /usr/lib64/libpthread.so.0 + 0x7DD5 [7] 0x00007FFFE78F8B3D clone + 0x6D | | Page cleaner tries to flush log records and then see SQLP_BADLOG and panic. | V 2020-01-08-19.06.44.609247+480 I2093709E483 LEVEL: Error PID : 20252 TID : 140736817260288 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX EDUID : 36 EDUNAME: db2pclnr (XXXX) 0 FUNCTION: DB2 UDB, data protection services, sqlpgild, probe:310 MESSAGE : ZRC=0x8610000D=-2045771763=SQLP_BADLOG "Log File cannot be used" DIA8414C Logging can not continue due to an error. 2020-01-08-19.06.44.623804+480 E2095416E548 LEVEL: Error PID : 20252 TID : 140736817260288 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX EDUID : 36 EDUNAME: db2pclnr (XXXX) 0 FUNCTION: DB2 UDB, data protection services, sqlpflog, probe:480 MESSAGE : ZRC=0x8610000D=-2045771763=SQLP_BADLOG "Log File cannot be used" DIA8414C Logging can not continue due to an error. DATA #1 : SQLP_LSN8, PD_TYPE_SQLP_LSN8, 8 bytes 000000000470184E 2020-01-08-19.06.44.624422+480 I2095965E493 LEVEL: Error PID : 20252 TID : 140736817260288 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX EDUID : 36 EDUNAME: db2pclnr (XXXX) 0 FUNCTION: DB2 UDB, buffer pool services, sqlbClnrAsyncWriteSetup, probe:80 MESSAGE : ZRC=0x8610000D=-2045771763=SQLP_BADLOG "Log File cannot be used" DIA8414C Logging can not continue due to an error. 2020-01-08-19.06.44.641776+480 I2096459E4034 LEVEL: Severe PID : 20252 TID : 140736817260288 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX EDUID : 36 EDUNAME: db2pclnr (XXXX) 0 FUNCTION: DB2 UDB, buffer pool services, sqlbClnrAsyncWriteSetup, probe:300 DATA #1 : Buffer page descriptor, PD_TYPE_SQLB_BPD, 256 bytes Pagekey: {pool:0;obj:13;type:0} PPNum:39 2020-01-08-19.06.44.661377+480 E2100494E896 LEVEL: Critical PID : 20252 TID : 140736817260288 PROC : db2sysc 0 INSTANCE: XXXXXX NODE : 000 DB : XXXX EDUID : 36 EDUNAME: db2pclnr (XXXX) 0 FUNCTION: DB2 UDB, RAS/PD component, pdStartFODC, probe:10 MESSAGE : ADM14001C An unexpected and critical error has occurred: "Panic". The instance may have been shutdown as a result. "Automatic" FODC (First Occurrence Data Capture) has been invoked and diagnostic information has been recorded in directory "/db2_home/db2instance/sminst/sqllib/db2dump/FODC_Panic_2020-01- 08-19 .06.44.645897_0000/". Please look in this directory for detailed evidence about what happened and contact IBM support if necessary to diagnose the problem. Stack #1 Signal #6 Timestamp 2020-01-08-19.06.54.998654 0 ossDumpStackTraceInternal 1 ossDumpStackTraceV98 2 OSSTrapFile::dumpEx 3 sqlo_trce 4 sqloEDUCodeTrapHandler 5 pthread_mutexattr_getprioceiling 6 gsignal 7 abort 8 sqloExitEDU 9 sqle_panic 10 sqlb_panic 11 sqlbClnrAsyncWriteSetup 12 SQLO_LIO_HANDLE_DATA::sqloDispatchNBlocks 13 sqlbClnrDispatchSomeAIO 14 sqlbClnrWritePages 15 sqlbClnrEntryPoint 16 sqbPgClnrEdu::RunEDU 17 sqzEDUObj::EDUDriver 18 sqloEDUEntry 19 __pthread_mutex_lock_full 20 clone | |
Problem Summary: | |
**************************************************************** * USERS AFFECTED: * * This problem may occur when db is already in shutdown * * process. * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Restart database manager. * **************************************************************** | |
Local Fix: | |
There is no fix yet. | |
Solution | |
Workaround | |
**************************************************************** * USERS AFFECTED: * * This problem may occur when db is already in shutdown * * process. * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Restart database manager. * **************************************************************** | |
Timestamps | |
Date - problem reported : Date - problem closed : Date - last modified : | 10.01.2020 19.02.2021 19.02.2021 |
Problem solved at the following versions (IBM BugInfos) | |
Problem solved according to the fixlist(s) of the following version(s) |