A HDR Secondary thread can loop endlessly when running a stress
test that executes two transactions by many threads. The first
transaction creates a raw table, various indexes on this table
and loads a small amount of data. The second transaction drops
the table.
This will ultimately lead to a "DR: ping timeout" on the primary
and then on the HDR secondary you will soon
see messages like:
09:44:02 DR: Received connection request from remote server
when DR is not Off
[Local type: Secondary, Current state: ?]
[Remote type: Primary]
09:44:06 DR: ping timeout
09:44:15 DR: Received connection request from remote server
when DR is not Off
[Local type: Secondary, Current state: FAILED]
[Remote type: Primary]
Usually one of three things occurs:
1. Most frequently, you can check to see that there will be an
xchg_ thread running
continually on a cpu vp. It's spinning endlessly in
ixrfind().
$ rasu pstack 632
#0 ixrfind ()
#1 rspnlockid ()
#2 plogredo ()
#3 rlogm_redo ()
#4 next_recvr ()
#5 prod_loop2 ()
#6 producer_thread ()
#7 startup ()
#8 ?? ()
2. Sometimes, after the "DR: ping timeout" on the HDR secondary,
an xchg thread
will segv in ixrfind() and assert
10:10:35 Stack for thread: 41 xchg_1.1
(../bin/oninit) afstack
(../bin/oninit) mt_ex_throw_sig
(../bin/oninit) afsig_handler
(Linux)
(../bin/oninit) ixrfind
(../bin/oninit) rspnlockid
(../bin/oninit) plogredo
(../bin/oninit) rlogm_redo
(../bin/oninit) next_recvr
(../bin/oninit) producer_thread
(../bin/oninit) startup
3. Less often, whatever causes the the bad list in ixrfind()
'corrects' itself and
the thread comes out of its loop and continues on. I say
'corrects' loosely because
it's just a second dose of corruption that causes the list
to terminate rather
than endlessly loop. |
