|
Problem IT16755
|
Status: Closed |
ADDING A READAHEAD THREAD DURING TRANSITION FROM SECONDARY TO PRIMARY CAN HANG THE SERVER
|
| product: |
INFORMIX SERVER / 5725A3900 / C10 - IDS 12.10 |
| Problem description: |
With a bit of readahead activity going on, i.e. a number of
readahead requests pending, a readahead thread might decide to
get some company and spawn another readahead thread.
If this happens during transition from secondary to primary,
e.g. for a 'make primary', the allocation of this new thread
will be blocked in txalloc() (no new transactions allowed as
long as log stream not yet owned by new primary). If on the
other hand one of the outstanding readahead requests is for a
thread 'in critical section' and that thread eventually abandons
the request, and if our readahead thread still is the only one
of its kind, that readahead request could not be served and not
be abandoned and the next checkpoint - required to procede with
the transition - would be stuck.
Symptoms seen:
- engine still in 'Fast Recovery (CKPT REQ)' -> Blocked:CKPT
- main_loop in wait4critex()
- lots of threads (mostly sqlexec) in ra_terminate_req()
- RSS_onmode thread in this func as well - and 'X'-flagged
(critical section)
Threads:
tid tcb rstcb prty status
vp-class name
...
7 700000026fa6248 7000000260b9028 1 sleeping secs:
1 12cpu main_loop()
...
828 70000002c4e5bf8 7000000260d50a8 1 join wait 829
1cpu sqlexec
829 70000002c22d028 7000000260d8468 1 join wait 830
1cpu Priority Make Primary
830 70000002c22d2b0 7000000260c67c8 1 sleeping secs:
1 12cpu RSS_onmod
...
Userthreads
address flags sessid user tty wait
tout locks nreads nwrites
...
7000000260c67c8 --R-XR- 31 informix - 0
0 0 8 1
...
Stack for thread: 7 main_loop()
base: 0x0700000026fb0000
len: 69632
pc: 0x0000000100062610
tos: 0x0700000026fbcb60
state: sleeping
vp: 8
yield_processor_mvp
mt_yield
wait4critex
checkpoint
main_loop
th_init_initgls
startup
Stack for thread: 830 RSS_onmod
base: 0x070000002c7c1000
len: 69632
pc: 0x0000000100062610
tos: 0x070000002c7cfff0
state: sleeping
vp: 10
yield_processor_mvp
mt_yield
ra_terminate_req
ra_free_req
rollback
rsrollback
txcleanup
rsclose_lgr
dr_lgr_end
dr_finish_recovery
sdcloneAssumePrimaryActivate
sdcloneAssumePrimary
cloneOnmodeSetSDCloneThread
th_init_initgls
startup
Stack for thread: 33 readahead_0
base: 0x070000002762e000
len: 69632
pc: 0x0000000100062610
tos: 0x070000002763e310
state: sleeping
vp: 10
yield_processor_mvp
mt_yield
txalloc
init_rstcb
rstcb_alloc
fork_be_session
add_readahead_thread
readahead_daemon
th_init_initgls
startup
Many other threads in stack like these:
yield_processor_mvp
mt_yield
ra_terminate_req
ra_init_leafscan
...
yield_processor_mvp
mt_yield
ra_terminate_req
ra_free_req
opfree
... |
| Problem Summary: |
****************************************************************
* USERS AFFECTED: *
* Informix users using MACH11 SDS *
****************************************************************
* PROBLEM DESCRIPTION: *
* During transition from SDS to primary role, a readahead *
* thread might hang in the following stack and block the *
* transition: *
* *
* yield_processor_mvp *
* mt_yield *
* txalloc *
* init_rstcb *
* rstcb_alloc *
* fork_be_session *
* add_readahead_thread *
* readahead_daemon *
* th_init_initgls *
* startup *
****************************************************************
* RECOMMENDATION: *
* Update to IBM Informix Server 12.10.xC8 *
**************************************************************** |
| Local Fix: |
Bring down and restart either old or new primary. |
| Solution |
|
| Workaround |
not known / see Local fix |
| Timestamps |
Date - problem reported :
Date - problem closed :
Date - last modified :
| 24.08.2016
09.12.2016
09.12.2016 |
| Problem solved at the following versions (IBM BugInfos) |
12.10.xC8
|
| Problem solved according to the fixlist(s) of the following version(s) |
