IBM IDS EXPRESS / 5724L2305 / B15 - IDS 11.50

Informix's JSON database service can suffer an outage with the
symptom that a client cannot connect to the database.

Analysis shows that all sqlexec threads are in "mutex wait
session" status except for one, which holds the "session" mutex
for thousands of seconds.  The server permitts no new
connections.
$ onstat -g wmx

IBM Informix Dynamic Server Version 12.10.FC2 -- On-Line -- Up 5
days 19:39:13 -- 18821440 Kbytes

Mutexes with waiters:
mid      addr             name               holder   lkcnt
waiter   waittime
17       440ca838         session            1687439  0
1687441  6527

1687442  6527
                                                             14
6527

1687443  6527

1687444  6527
                                                             12
6158

1652248  5875

1652249  5875

1666664  3015

1666665  3015

1669536  2373

1669534  2373

1669538  2373

1669537  2373

1669535  2373

1600443  590

1687432  590

1687434  590

1687411  590

1687306  590

1687316  590

1687312  590

1687422  590

1687329  590

1687413  590

1687300  590

1687320  590

Holder 1687439 corresponds to a sqlexec thread doing this query
against sysmaster:sysdbopen:
...
Stored procedure stack :

   context            proc-counter       opcode name

----------------------------------------------------------------
--
   0x000000008e712058 0x6522b2a8+0x0010 EXPR
o_170488:dbs_check_user_limit
   0x0000000080b4ec58 0x93752b90+0x0010 EXPR
o_170488:sysdbopen

Current SQL statement (5) in procedure
o_170488:dbs_check_user_limit
   proc-counter 0x0x6522b2a8 opcode EXPR

  (select (nvl (count *), 0)
    from sysmaster:sysopendb
    where (= odb_dbname, (: 136 )))

Last parsed SQL statement :
  DATABASE o_170488

...
OS tools (pstack) were used to get the running thread's stack
multiple times, which showed

sometimes #-1  in lock_held ()
sometimes #0  in mt_lock ()
sometimes #1  in mt_nxtscb ()

---- this part of the stack down never changed ----
#2  in propendb ()
#3  in pstread ()
#4  in pst_rsread ()
#5  in rsread ()
#6  in fmread ()
#7  in readseq_single ()
#8  in gettupl ()
#9  in scan_next ()
#10 in next_row ()
#11 in get_first_row_from_producer ()
#12 in group_open ()
#13 in prepselect ()
#14 in subqprep ()
#15 in exsubq ()
#16 in ev_cb ()
#17 in new_eval ()
#18 in ip_evalexpr ()
#19 in runproc ()
#20 in udrlm_spl_execute ()
#21 in udrlm_exec_routine ()
#22 in udr_execute ()
#23 in exroutine ()
#24 in geval ()
#25 in ev_geval ()
#26 in new_eval ()
#27 in ip_evalexpr ()
#28 in runproc ()
#29 in udrlm_spl_execute ()
#30 in udrlm_exec_routine ()
#31 in udr_execute ()
#32 in exec_sysdbproc ()
#33 in sqopendb ()
#34 in excommand ()
#35 in sq_execute ()
#36 in sqmain ()
#37 in spawn_thread ()
#38 in th_init_initgls ()
#39 in startup ()
#40 in ?? ()

which indicates a problem traversing the mcb's session list
(ses_scblist).

****************************************************************
* USERS AFFECTED:                                              *
* IDS 11.50 users running queries on sysmaster:sysopendb with  *
* an unusually volatile session list.                          *
****************************************************************
* PROBLEM DESCRIPTION:                                         *
* See Error Description                                        *
****************************************************************
* RECOMMENDATION:                                              *
* Update to latest version of IBM Informix Server 11.50.xC10   *
****************************************************************

Bounce the database server.

not known / see Local fix

Date  - problem reported    :
Date  - problem closed      :
Date  - last modified       :

17.05.2016
15.06.2017
15.06.2017

11.50.xC10