DB2 can panic and crash in scenarios where FMPs are frequently
interrupted before completing.
This issue has been most commonly seen when DSM (Data Server
Manager) is issuing various monitoring queries using the
built-in monitoring table functions; however it is possible that
any monitoring behaviour can cause this issue.
The problem will manifest in the form of a DB2 panic crash
showing symptoms of a memory corruption in the
communication areas between the db2fmp process and the engine.
Since the problem is a memory corruption, the symptoms can vary,
but the main flavour of this problem that as been
observed is an error when trying to free memory. The
db2diag.log in this example case may show something like this:
2016-02-12-07.32.33.091656+000 I345230267E7964 LEVEL:
Error
PID : 28801 TID : 139840351889152 PROC :
db2sysc 2
INSTANCE: inst1 NODE : 002 DB :
BIGSQL
APPHDL : 0-22249 APPID:
10.164.5.240.24422.160212073321
AUTHID : CLUSTERUSER02 HOSTNAME:
EDUID : 72 EDUNAME: db2agntp
FUNCTION: DB2 UDB, routine_infrastructure,
sqlerInvokeFencedRoutine, probe:1200
DATA #1 : String, 37 bytes
Routine failed with termination error
DATA #2 : Hexdump, 1456 bytes
0x00007F2EE6CDA878 : 5546 4F42 0000 0000 9057 0000 0000 0000
UFOB.....W......
0x00007F2EE6CDAA28 : 6400 0000 0000 1800 0000 0000 0000 0000
d...............
0x00007F2EE6CDAA38 : 0000 B804 0000 2000 0A00 0F00 5359 5350
...... .....SYSP
0x00007F2EE6CDAA48 : 524F 432E 534E 4150 5F47 4554 5F44 4200
ROC.SNAP_GET_DB.
2016-02-12-07.32.33.092854+000 E345238232E1040 LEVEL:
Critical
PID : 28801 TID : 139840351889152 PROC :
db2sysc 2
INSTANCE: inst1 NODE : 002 DB : mydb
APPHDL : 0-22249 APPID:
10.222.1.111.24422.160212073321
AUTHID : CLUSTERUSER02 HOSTNAME:
EDUID : 72 EDUNAME: db2agntp
FUNCTION: DB2 UDB, SQO Memory Management,
sqloDiagnoseFreeBlockFailure, probe:10
In the case of a trap, the stack may be similar to:
Stack #1 Signal #11
0 ossDumpStackTraceInternal
1 ossDumpStackTraceV98
2 OSSTrapFile::dumpEx
3 sqlo_trce
4 sqloEDUCodeTrapHandler
5 _IO_ftrylockfile
6 sqloCrashOnCriticalMemoryValidationFailure
7 SQLO_MEM_POOL::diagnoseMemoryCorruptionAndCrash
8 SQLO_MEM_POOL::MemTreeGet
9 SQLO_MEM_POOL::allocateMemoryBlock
10 sqloGetMemoryBlockExtended
11 sqlerGetFMPIPC
12 sqlccGetIPCs
13 sqlerAllocFmpIPC
14 sqlerGetFmpThreadEntry
15 sqlerGetFmpFromPool
16 sqlerInvokeFencedRoutine
17 sqlriInvokeInvoker |
