In a HDR pair, if the servers disconnect and reconnect, and at
the reconnect point, the primary determines it needs to do
logical recovery from the secondary server which switched up
into standard mode, under some conditions if the read ahead
thread runs at that point, it can free up memory for rpartn
structures but leave pointers to that memory behind. This can
lead to various memory corruption issues with the RSAM pool, or
invalid mutex assertion failures when the left over pointers are
referenced as rpartn structures, but the memory is now in use as
something else.
Two different invalid mutex af's enountered shown:
1)
10:02:40 DR: Turned off on primary server
10:02:40 DR: Cannot connect to secondary server
10:03:31 DR: Primary server connected
10:03:31 DR: Primary server needs failure recovery
10:03:43 Physical Recovery Started at Page (1:1126).
10:03:43 Recovery Mode
10:03:43 Physical Recovery Complete: 0 Pages Examined, 0 Pages
Restored.
10:03:44 DR: Failure recovery from disk in progress ...
10:03:44 Logical Recovery Started.
10:03:44 10 recovery worker threads will be started.
10:03:44 Start Logical Recovery - Start Log 4, End Log ?
10:03:44 Starting Log Position - 4 0x5d018
...
10:13:55 Assert Failed: Invalid Mutex Type
10:13:55 IBM Informix Dynamic Server Version 12.10.F
10:13:55 Who: Session(5, informix@vox, 0, 0x44d79148)
Thread(19, btscanner_0, 44d357a8, 1)
10:13:55 Stack for thread: 19 btscanner_0
afstack
afhandler
afcrash_interface
mt_slock
btc_create_hot_list
btscanner_loop
th_init_initgls
startup
2)
03:43:36 DR_ERR set to -1
03:43:39 DR: Turned off on primary server
03:43:39 DR: Cannot connect to secondary server
03:44:04 DR: Primary server connected
03:44:04 SCHAPI: thread dbWorker2 task
post_alarm_message(19-30087) shutting down
03:44:04 SCHAPI: thread dbWorker1 task
post_alarm_message(19-30088) shutting down
03:44:04 SCHAPI: thread dbScheduler(116) shutting down
03:44:04 DR: Primary server needs failure recovery
03:44:04 Physical Recovery Started at Page (2:92306).
03:44:05 Physical Recovery Complete: 593 Pages Examined, 593
Pages Restored.
03:44:05 Recovery Mode
03:44:05 DR: Failure recovery from disk in progress ...
03:44:06 Logical Recovery Started.
03:44:06 10 recovery worker threads will be started.
03:44:06 Start Logical Recovery - Start Log 14455, End Log ?
03:44:06 Starting Log Position - 14455 0x7a9018
03:44:06 DR: Cleared 6076 KB of logical log in 0 seconds.
...
03:44:13 Assert Failed: Invalid Mutex Type
03:44:13 Who: Session(94022, informix@machine, 0, 0x1523b46d8)
Thread(44159, xchg_1.3, 14db3bdf8, 1)
03:44:13 Stack for thread: 44159 xchg_1.3
base: 0x0000000152bb1000
len: 69632
pc: 0x0000000001369653
tos: 0x0000000152bbfa70
state: running
vp: 1
afstack
afhandler
afcrash_interface
mt_lock
ptalloc
flalloc
rspnopen
pntorsfd
plogredo
rlogm_redo
next_recvr
producer_thread
startup |
