rac dg启动时报错ORA-10456、ORA-01153、ORA-16016

问题描述：rac dg启动时报错ORA-10456、ORA-01153、ORA-16016，如下所示：
环境介绍：主库11.2.0.4 rac + 双节点，备库11.2.0.4 rac + 双节点
问题说明：首先启动的是备库的rac两个节点，但启动出现异常，在备库并没有恢复正常的情况下开启主库两个节点，此时主库也遇到问题，主库通过重启解决问题，不过备库异常依然存在，主要体现在以下告警.
1、问题重现
备库节点1：
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-10456: cannot open standby database; media recovery session may be in progress

SQL> alter database recover managed standby database using current logfile disconnect from session;
alter database recover managed standby database using current logfile disconnect from session
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active
备库节点2 ：
--trace告警如下.
Mon Nov 28 10:26:52 2022
Standby crash recovery failed to bring standby database to a consistent
point because needed redo hasn't arrived yet.
MRP: Wait timeout: thread 2 sequence# 0
Standby Crash Recovery aborted due to error 16016.
Errors in file /u01/app/oracle/diag/rdbms/healdg/healdg2/trace/healdg2_ora_3169.trc:
ORA-16016: archived log for thread 2 sequence# 15 unavailable
Recovery interrupted!
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail
2、异常解决
2.1、日志应用情况
主库查询：
SQL> select thread#,sequence#,to_char(first_time,'yyyy-mm-dd hh24:mi:ss'),to_char(next_time,'yyyy-mm-dd hh24:mi:ss'),to_char(completion_time,'yyyy-mm-dd hh24:mi:ss'),applied from v$archived_log where first_time>sysdate-1 order by 1,2

   THREAD#  SEQUENCE# TO_CHAR(FIRST_TIME, TO_CHAR(NEXT_TIME,' TO_CHAR(COMPLETION_ APPLIED
---------- ---------- ------------------- ------------------- ------------------- ---------
         1         10 2022-11-27 14:08:24 2022-11-27 14:34:09 2022-11-27 14:34:09 NO
         1         11 2022-11-27 14:34:09 2022-11-27 14:34:24 2022-11-27 14:34:25 NO
         1         12 2022-11-27 14:34:24 2022-11-27 14:39:24 2022-11-27 14:39:24 YES
         1         12 2022-11-27 14:34:24 2022-11-27 14:39:24 2022-11-27 14:39:24 NO
         1         13 2022-11-27 14:39:24 2022-11-27 16:54:57 2022-11-27 16:54:58 YES
         1         13 2022-11-27 14:39:24 2022-11-27 16:54:57 2022-11-27 16:54:58 NO
         1         14 2022-11-27 16:54:57 2022-11-27 19:33:43 2022-11-27 19:33:44 YES
         1         14 2022-11-27 16:54:57 2022-11-27 19:33:43 2022-11-27 19:33:44 NO
         1         15 2022-11-27 19:33:43 2022-11-27 20:33:53 2022-11-27 20:33:54 NO
         1         15 2022-11-27 19:33:43 2022-11-27 20:33:53 2022-11-27 20:33:56 YES
         1         16 2022-11-27 20:33:53 2022-11-27 20:33:58 2022-11-27 20:33:59 YES

   THREAD#  SEQUENCE# TO_CHAR(FIRST_TIME, TO_CHAR(NEXT_TIME,' TO_CHAR(COMPLETION_ APPLIED
---------- ---------- ------------------- ------------------- ------------------- ---------
         1         16 2022-11-27 20:33:53 2022-11-27 20:33:58 2022-11-27 20:33:58 NO
         1         17 2022-11-27 20:33:58 2022-11-27 20:34:37 2022-11-27 20:34:37 YES
         1         17 2022-11-27 20:33:58 2022-11-27 20:34:37 2022-11-27 20:34:37 NO
         1         18 2022-11-27 20:34:37 2022-11-27 20:36:16 2022-11-27 20:36:16 YES
         1         18 2022-11-27 20:34:37 2022-11-27 20:36:16 2022-11-27 20:36:16 NO
         1         19 2022-11-27 20:36:16 2022-11-27 20:36:20 2022-11-27 20:36:20 NO
         1         19 2022-11-27 20:36:16 2022-11-27 20:36:20 2022-11-27 20:36:20 YES
         1         20 2022-11-27 20:36:20 2022-11-27 20:36:33 2022-11-27 20:36:33 YES
         1         20 2022-11-27 20:36:20 2022-11-27 20:36:33 2022-11-27 20:36:33 NO
         1         21 2022-11-27 20:36:33 2022-11-27 23:00:22 2022-11-27 23:00:22 NO
         1         21 2022-11-27 20:36:33 2022-11-27 23:00:22 2022-11-27 23:00:22 NO

   THREAD#  SEQUENCE# TO_CHAR(FIRST_TIME, TO_CHAR(NEXT_TIME,' TO_CHAR(COMPLETION_ APPLIED
---------- ---------- ------------------- ------------------- ------------------- ---------
         1         22 2022-11-27 23:00:22 2022-11-28 09:52:19 2022-11-28 09:52:21 NO
         1         23 2022-11-28 09:52:19 2022-11-28 09:52:24 2022-11-28 09:52:24 NO
         1         24 2022-11-28 09:52:24 2022-11-28 10:18:06 2022-11-28 10:18:06 NO
         1         25 2022-11-28 10:18:06 2022-11-28 10:22:42 2022-11-28 10:22:42 NO
         1         26 2022-11-28 10:22:42 2022-11-28 10:25:51 2022-11-28 10:25:51 NO
         2          6 2022-11-27 14:00:27 2022-11-27 14:34:09 2022-11-27 14:34:09 NO
         2          7 2022-11-27 14:34:09 2022-11-27 14:34:24 2022-11-27 14:34:25 NO
         2          8 2022-11-27 14:34:24 2022-11-27 14:39:54 2022-11-27 14:39:54 NO
         2          8 2022-11-27 14:34:24 2022-11-27 14:39:54 2022-11-27 14:39:54 YES
         2          9 2022-11-27 14:39:54 2022-11-27 16:54:57 2022-11-27 16:54:58 NO
         2          9 2022-11-27 14:39:54 2022-11-27 16:54:57 2022-11-27 16:54:58 YES

   THREAD#  SEQUENCE# TO_CHAR(FIRST_TIME, TO_CHAR(NEXT_TIME,' TO_CHAR(COMPLETION_ APPLIED
---------- ---------- ------------------- ------------------- ------------------- ---------
         2         10 2022-11-27 16:54:57 2022-11-27 19:33:43 2022-11-27 19:33:43 NO
         2         10 2022-11-27 16:54:57 2022-11-27 19:33:43 2022-11-27 19:33:43 YES
         2         11 2022-11-27 19:33:43 2022-11-27 20:33:58 2022-11-27 20:33:59 YES
         2         11 2022-11-27 19:33:43 2022-11-27 20:33:58 2022-11-27 20:33:58 NO
         2         12 2022-11-27 20:33:58 2022-11-27 20:34:25 2022-11-27 20:34:27 NO
         2         12 2022-11-27 20:33:58 2022-11-27 20:34:25 2022-11-27 20:34:27 YES
         2         13 2022-11-27 20:34:25 2022-11-27 20:34:31 2022-11-27 20:34:32 YES
         2         13 2022-11-27 20:34:25 2022-11-27 20:34:31 2022-11-27 20:34:32 NO
         2         14 2022-11-27 20:34:31 2022-11-27 20:36:26 2022-11-27 20:36:26 NO
         2         14 2022-11-27 20:34:31 2022-11-27 20:36:26 2022-11-27 20:36:26 YES
         2         15 2022-11-27 20:36:26 2022-11-28 09:52:19 2022-11-28 10:17:03 NO

   THREAD#  SEQUENCE# TO_CHAR(FIRST_TIME, TO_CHAR(NEXT_TIME,' TO_CHAR(COMPLETION_ APPLIED
---------- ---------- ------------------- ------------------- ------------------- ---------
         2         16 2022-11-28 09:52:19 2022-11-28 10:17:03 2022-11-28 10:17:03 NO
         2         17 2022-11-28 10:17:58 2022-11-28 10:17:58 2022-11-28 10:17:59 NO
         2         18 2022-11-28 10:17:58 2022-11-28 10:18:04 2022-11-28 10:18:04 NO
         2         19 2022-11-28 10:18:04 2022-11-28 10:25:49 2022-11-28 10:25:49 NO
         2         20 2022-11-28 10:25:49 2022-11-28 10:25:55 2022-11-28 10:25:55 NO

49 rows selected.
备库查询.
SQL> select thread#,sequence#,to_char(first_time,'yyyy-mm-dd hh24:mi:ss'),to_char(next_time,'yyyy-mm-dd hh24:mi:ss'),to_char(completion_time,'yyyy-mm-dd hh24:mi:ss'),applied from v$archived_log where first_time>sysdate-1 order by 1,2

   THREAD#  SEQUENCE# TO_CHAR(FIRST_TIME, TO_CHAR(NEXT_TIME,' TO_CHAR(COMPLETION_ APPLIED
---------- ---------- ------------------- ------------------- ------------------- ---------
         1         10 2022-11-27 14:08:24 2022-11-27 14:34:09 2022-11-27 14:34:49 YES
         1         11 2022-11-27 14:34:09 2022-11-27 14:34:24 2022-11-27 14:34:49 YES
         1         12 2022-11-27 14:34:24 2022-11-27 14:39:24 2022-11-27 14:39:26 YES
         1         13 2022-11-27 14:39:24 2022-11-27 16:54:57 2022-11-27 16:55:00 YES
         1         14 2022-11-27 16:54:57 2022-11-27 19:33:43 2022-11-27 19:33:46 YES
         1         15 2022-11-27 19:33:43 2022-11-27 20:33:53 2022-11-27 20:33:58 YES
         1         16 2022-11-27 20:33:53 2022-11-27 20:33:58 2022-11-27 20:34:01 YES
         1         17 2022-11-27 20:33:58 2022-11-27 20:34:37 2022-11-27 20:34:39 YES
         1         18 2022-11-27 20:34:37 2022-11-27 20:36:16 2022-11-27 20:36:18 YES
         1         19 2022-11-27 20:36:16 2022-11-27 20:36:20 2022-11-27 20:36:22 YES
         1         20 2022-11-27 20:36:20 2022-11-27 20:36:33 2022-11-27 20:36:35 YES

   THREAD#  SEQUENCE# TO_CHAR(FIRST_TIME, TO_CHAR(NEXT_TIME,' TO_CHAR(COMPLETION_ APPLIED
---------- ---------- ------------------- ------------------- ------------------- ---------
         1         21 2022-11-27 20:36:33 2022-11-27 23:00:22 2022-11-27 23:00:25 NO
         2          6 2022-11-27 14:00:27 2022-11-27 14:34:09 2022-11-27 14:34:49 YES
         2          7 2022-11-27 14:34:09 2022-11-27 14:34:24 2022-11-27 14:34:49 YES
         2          8 2022-11-27 14:34:24 2022-11-27 14:39:54 2022-11-27 14:39:57 YES
         2          9 2022-11-27 14:39:54 2022-11-27 16:54:57 2022-11-27 16:55:00 YES
         2         10 2022-11-27 16:54:57 2022-11-27 19:33:43 2022-11-27 19:33:45 YES
         2         11 2022-11-27 19:33:43 2022-11-27 20:33:58 2022-11-27 20:34:01 YES
         2         12 2022-11-27 20:33:58 2022-11-27 20:34:25 2022-11-27 20:34:29 YES
         2         13 2022-11-27 20:34:25 2022-11-27 20:34:31 2022-11-27 20:34:34 YES
         2         14 2022-11-27 20:34:31 2022-11-27 20:36:26 2022-11-27 20:36:28 YES

21 rows selected.

说明：此时可以看到备库端未应用thread# 1的21至26号，thread# 2的15至20号归档日志.并查询到主库除thread# 1的21号归档日志外，其余在备库端均没有发现.
2.2、日志传输
主库传输归档日志到备库端.
ASMCMD> cp thread_1_seq_22.464.1121939541 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_1_seq_22.bak
copying +fra/heal/archivelog/2022_11_28/thread_1_seq_22.464.1121939541 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_1_seq_22.bak
ASMCMD> cp thread_1_seq_23.465.1121939545 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_1_seq_23
copying +fra/heal/archivelog/2022_11_28/thread_1_seq_23.465.1121939545 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_1_seq_23
ASMCMD> cp thread_1_seq_24.470.1121941087 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_1_seq_24.470
copying +fra/heal/archivelog/2022_11_28/thread_1_seq_24.470.1121941087 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_1_seq_24.470
ASMCMD> cp thread_1_seq_25.471.1121941363 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_1_seq_25.471
copying +fra/heal/archivelog/2022_11_28/thread_1_seq_25.471.1121941363 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_1_seq_25.471
ASMCMD> cp thread_1_seq_26.473.1121941551 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_1_seq_26.473
copying +fra/heal/archivelog/2022_11_28/thread_1_seq_26.473.1121941551 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_1_seq_26.473
ASMCMD> cp thread_2_seq_15.466.1121941023 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_2_seq_15.466
copying +fra/heal/archivelog/2022_11_28/thread_2_seq_15.466.1121941023 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_2_seq_15.466
ASMCMD> cp thread_2_seq_16.467.1121941023 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_2_seq_16.467
copying +fra/heal/archivelog/2022_11_28/thread_2_seq_16.467.1121941023 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_2_seq_16.467
ASMCMD> cp thread_2_seq_17.468.1121941079 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_2_seq_17.468
copying +fra/heal/archivelog/2022_11_28/thread_2_seq_17.468.1121941079 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_2_seq_17.468
ASMCMD> cp thread_2_seq_18.469.1121941085 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_2_seq_18.469
copying +fra/heal/archivelog/2022_11_28/thread_2_seq_18.469.1121941085 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_2_seq_18.469
ASMCMD> cp thread_2_seq_19.472.1121941549 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_2_seq_19.472
copying +fra/heal/archivelog/2022_11_28/thread_2_seq_19.472.1121941549 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_2_seq_19.472
ASMCMD>cp thread_2_seq_20.474.1121941555 sys/oracle_4U@192.168.133.220.+ASM1:+fra/healdg/archivelog/2022_11_28/thread_2_seq_20.474copying
copying +fra/heal/archivelog/2022_11_28/thread_2_seq_20.474.1121941555 -> 192.168.133.220:+fra/healdg/archivelog/2022_11_28/thread_2_seq_20.474

说明：
a、此时备库两个节点均处于mount阶段；
b、主备库直接传输磁盘组文件时两端文件名需不同，否则报错.
2.3、注册日志
备库节点1注册归档日志.
SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_1_seq_22.bak';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_1_seq_23';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_1_seq_24.470';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_1_seq_25.471';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_1_seq_26.473';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_2_seq_15.466';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_2_seq_16.467';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_2_seq_17.468';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_2_seq_18.469';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_2_seq_19.472';

Database altered.

SQL> alter database register logfile '+fra/healdg/archivelog/2022_11_28/thread_2_seq_20.474';

Database altered.

--查未应用的日志.
SQL> select name from v$archived_log where applied='NO';

NAME
-------------------------------------------------------------------------------
+FRA/healdg/archivelog/2022_11_27/thread_1_seq_21.277.1121900425
+FRA/healdg/archivelog/2022_11_28/thread_1_seq_22.bak
+FRA/healdg/archivelog/2022_11_28/thread_1_seq_23
+FRA/healdg/archivelog/2022_11_28/thread_1_seq_24.470
+FRA/healdg/archivelog/2022_11_28/thread_1_seq_25.471
+FRA/healdg/archivelog/2022_11_28/thread_1_seq_26.473
+FRA/healdg/archivelog/2022_11_28/thread_2_seq_15.466
+FRA/healdg/archivelog/2022_11_28/thread_2_seq_16.467
+FRA/healdg/archivelog/2022_11_28/thread_2_seq_17.468
+FRA/healdg/archivelog/2022_11_28/thread_2_seq_18.469
+FRA/healdg/archivelog/2022_11_28/thread_2_seq_19.472
+FRA/healdg/archivelog/2022_11_28/thread_2_seq_20.474

12 rows selected.
2.4、应用日志
--此后停止两个处在mount阶段的备库节点
[root@hisdb3 ~]# ps -ef|grep ora_smon
oracle     2912      1  0 10:25 ?        00:00:00 ora_smon_healdg1
root       7731   3168  0 13:33 pts/1    00:00:00 grep --color=auto ora_smon
[root@hisdb3 ~]# kill -9 2912
[root@hisdb4 ~]# ps -ef |grep ora_smon
oracle     3133      1  0 10:25 ?        00:00:00 ora_smon_healdg2
root       9070   8890  0 13:34 pts/1    00:00:00 grep --color=auto ora_smon
[root@hisdb4 ~]# kill -9 3133
--备库节点1 到mount阶段应用日志.
[oracle@hisdb3 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Nov 28 13:35:42 2022

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area  855982080 bytes
Fixed Size                  2258040 bytes
Variable Size             348130184 bytes
Database Buffers          499122176 bytes
Redo Buffers                6471680 bytes
SQL> alter database mount;

Database altered.

SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> alter database open;

Database altered.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY

SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

SQL> select process,status from v$managed_standby;

PROCESS   STATUS
--------- ------------
ARCH     CONNECTED
ARCH     CONNECTED
ARCH     CLOSING
ARCH     CLOSING
RFS       IDLE
RFS       IDLE
RFS       IDLE
RFS       IDLE
RFS       IDLE
RFS       IDLE
MRP0     APPLYING_LOG

11 rows selected.
--open备库节点2.
[oracle@hisdb4 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Nov 28 13:37:58 2022

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area  855982080 bytes
Fixed Size                  2258040 bytes
Variable Size             348130184 bytes
Database Buffers          499122176 bytes
Redo Buffers                6471680 bytes
Database mounted.
Database opened.
SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY WITH APPLY

说明：此后主备库均恢复正常.