In a cluster with a Primary, HDR & RSS running on IDS 12.10.FC9,
these are the steps to reproduce issue:
-force a checkpoint on the Primary
-shutdown the RSS and upgrade from FC9 to FC10
-next shutdown/upgrade the HDR Secondary
-with cluster online and HDR/RSS running FC10, run the onmode -d
make primary command on the HDR Secondary
-upgrade the old Primary to FC10
-after successful upgrade, attempt to connect the old Primary as
an HDR Secondary to the newly promoted Primary and this will end
up crashing in the process
The log of the old Primary will look like this when trying to
connect to the cluster as an HDR Secondary:
12:11:46 DR: Reservation of the last logical log for log backup
turned on
12:11:46 Data replication type and state information reset. To
start DR, use
the 'onmode -d' command and wait for the pair to be
operational,
before shutting down the database server
12:11:46 Warning: Invalid (non-existent/blobspace/disabled)
dbspace listed
in DBSPACETEMP: 'tmpdbspace'
12:11:46 Physical Recovery Started at Page (2:4781).
12:11:46 Physical Recovery Complete: 0 Pages Examined, 0 Pages
Restored.
12:11:46 Dataskip is now OFF for all dbspaces
12:11:46 Restartable Restore has been ENABLED
12:11:46 Recovery Mode
12:13:04 smx creates 1 transports to server pri_root
12:13:18 DR: Reservation of the last logical log for log backup
turned off
12:13:18 DR: new type = secondary, primary server name =
sec_root
12:13:18 DR: Trying to connect to primary server = sec_root
12:13:20 DR: Secondary server connected
12:13:20 DR: Secondary server needs failure recovery
12:13:20 DR: Failure recovery from disk in progress ...
12:13:20 Logical Recovery Started.
12:13:20 11 recovery worker threads will be started.
12:13:21 Warning: Invalid (non-existent/blobspace/disabled)
dbspace listed
in DBSPACETEMP: 'tmpdbspace'
12:13:21 Start Logical Recovery - Start Log 89, End Log ?
12:13:21 Starting Log Position - 89 0xaf0018
12:13:21 DR: Cleared 542 KB of logical log in 0 seconds.
12:13:21 Warning: Some logical logs are not backed up. Do not
proceed. The secondary server might hang or the logs
might be
lost.
12:13:21 Defragmenter cleaner thread now running
12:13:21 Defragmenter cleaner thread cleaned:0 partitions
12:13:23 DR_ERR set to -25
12:13:23 Assert Failed: DR: Log Record Apply Thread Exited
Abnormally. Internal Error.
A restart of the database server shall be required
to correct
this problem.
12:13:23 IBM Informix Dynamic Server Version 12.10.FC10
12:13:23 Who: Session(34, informix@, 0, 0x45142bc8)
Thread(42, dr_secapply, 451024c8, 1)
File: hdr/hdrApply.c Line: 580
12:13:23 Results: Dynamic Server must abort
12:13:23 Action: Reinitialize shared memory
12:13:23 stack trace for pid 25248 written to
/opt/informix/12.10.FC10/tmp/af.412bf43
12:13:24 See Also: /opt/informix/12.10.FC10/tmp/af.412bf43
12:13:27 Thread ID 42 will NOT be suspended because
it is a daemon.
12:13:27 See Also: /opt/informix/12.10.FC10/tmp/af.412bf43,
shmem.412bf43.0
12:13:28 Starting crash time check of:
12:13:28 1. memory block headers
12:13:28 2. stacks
12:13:28 Crash time checking found no problems
12:13:28 hdr/hdrApply.c, line 580, thread 42, proc id 25248,
DR: Log Record Apply Thread Exited Abnormally. Internal Error.
A restart of the database server shall be required
to correct
this problem.
.
12:13:28 The Master Daemon Died
12:13:28 PANIC: Attempting to bring system down
12:16:06 Parameter's user-configured value was adjusted.
(DS_MAX_SCANS)
12:16:06 Parameter's user-configured value was adjusted.
(ONLIDX_MAXMEM)
12:16:06 IBM Informix Dynamic Server Started.
12:16:06 Requested shared memory segment size rounded from
8308KB to 8840KB
Subsequent restarts will allow the new HDR to connect to the
cluster for a short time, but will crash again with a log replay
error.
Initial crash will have this stack:
12:13:23 shmcb = 0x000000004401a418
12:13:23 rhead = 0x0000000044086800
12:13:23 pool list = 0x000000004401a4f0
12:13:23 block pool list = 0x00000000440810f8
12:13:23 TRANSP = 0x0000000045142bc8
12:13:23 PARTP = 0x0000000000000000
12:13:23 PARTNP = 0x0000000000000000
12:13:23 OPENP = 0x0000000000000000
12:13:23 FILEP = 0x0000000000000000
12:13:23 Raw hex dump of stack located in
/opt/informix/12.10.FC10/tmp/af.412bf43.rawstk
12:13:23 Stack for thread: 42 dr_secapply
base: 0x000000004677e000
len: 69632
pc: 0x00000000013f01a9
tos: 0x000000004678de80
state: running
vp: 1
(oninit) afstack
(oninit) afhandler
(oninit) afcrash_interface
(oninit) dr_secondary_apply
(oninit) startup
siginfo: |
****************************************************************
* USERS AFFECTED: *
* Users of Informix Server prior to 12.10.xC11. *
****************************************************************
* PROBLEM DESCRIPTION: *
* SWITCH PRIMARY ROLE TO SECONDARY CAUSES SERVER CRASH *
****************************************************************
* RECOMMENDATION: *
* Upgrade to Informix Server 12.10.xC11+. *
****************************************************************
The fix for this APAR corrects the error message in server
message log file and assertion file to show more details on the
problem and solution details.
Log message before this fix:
14:32:21 Start Logical Recovery - Start Log 67, End Log ?
14:32:21 Starting Log Position - 67 0xa2b018
14:32:21 DR: Cleared 4792 KB of logical log in 0 seconds.
14:32:21 Warning: Some logical logs are not backed up. Do not
proceed. The secondary server might hang or the logs
might be
lost.
14:32:22 Defragmenter cleaner thread now running
14:32:22 Defragmenter cleaner thread cleaned:0 partitions
14:32:22 DR_ERR set to -25
14:32:22 Assert Failed: DR: Log Record Apply Thread Exited
Abnormally. Internal Error.
A restart of the database server shall be required
to correct
this problem.
14:32:22 IBM Informix Dynamic Server Version 12.10.F
14:32:22 Who: Session(31, informix@, 0, 0x44d1ba98)
Thread(42, dr_secapply, 44cd7bd0, 1)
File: hdr/hdrApply.c Line: 580
14:32:22 Results: Dynamic Server must abort
14:32:22 Action: Reinitialize shared memory
14:32:22 stack trace for pid 12064 written to
/work/.../af.412ed56
14:32:22 See Also: /work/.../af.412ed56
14:32:24 Thread ID 42 will NOT be suspended because
it is a daemon.
14:32:24 See Also: /work/.../af.412ed56
14:32:24 Starting crash time check of:
14:32:24 1. memory block headers
14:32:24 2. stacks
14:32:24 Crash time checking found no problems
14:32:24 hdr/hdrApply.c, line 580, thread 42, proc id 12064,
DR: Log Record Apply Thread Exited Abnormally. Internal Error.
A restart of the database server shall be required
to correct
this problem.
.
14:32:24 The Master Daemon Died
14:32:24 PANIC: Attempting to bring system down
After this:
=========
11:12:12 DR_ERR set to -25
11:12:12 Assert Failed: Some logical logs are not backed up.
The secondary server might hang or the
logs might be lost.
Primary server was started with the 'oninit -PHY' command and
then it was
attempted to be converted into a secondary server.
If you want to back up the logs, restart the primary server as a
standard
server and then back up the logs.
If you want to ignore the logs, restart the server with the
'oninit -PHY
--forget_logs' command.
11:12:12 HCL Informix Dynamic Server Version 12.10.F
11:12:12 Who: Session(52, informix@, 0, 0x44d19e08)
Thread(63, dr_secapply, 44cd64f8, 1)
File: hdr/hdrApply.c Line: 586
11:12:12 Results: Dynamic Server must abort
11:12:12 Action: Reinitialize shared memory
11:12:12 stack trace for pid 11150 written to
/work/.../af.427616c
11:12:12 See Also: /work/.../af.427616c
*** WARNING: HCL Informix Dynamic Server is no longer running.
***
This provides a better explanation in the online.log and can be
worked around using the command oninit -PHY --forget_logs |
