Queries on sysmaster's syscdr_ats, syscdr_atsdir, syscdr_ris or
syscdr_risdir SMI tables, esp. when done in parallel by multiple
sessions, have a potential to crash the database server, in
threads named ATSt*, e.g. "ATSt536".
Most likely source of such queries would be certain ats/ris
related 'cdr view' commands, possibly originating from OAT.
Stacks seen on Linux were of little help for diagnosing this as
they only showed top part which usually was in some glibc
malloc() or free() related functions, typically with
malloc_printerr() and/or __libc_message() followed by
__GI_abort() indicating a glibc forced shutdown using signal 6
-- SIGABRT (hence some OS patch level dependency might exist).
The condition for this kind of crash is the existence of more
than one 'cdrsmi' VP (with the crash occurring on one of them).
A 'cdrsmi' VP typically gets auto-created for first such query
and any subsequent query would be using this VP, with the code
depending on always running on same VP. Unfortunately such
thread isn't kept from migrating between such VPs and it's not
guaranteed that auto-creation of such VP doesn't happen multiple
times in parallel, so there's a small chance for ending up with
two such VPs and threads eventually migrating. |
