有个11204的rac环境，没应用补丁，5号突然报归档满，登录环境后发现奇怪，一个1T磁盘建成的DATA磁盘组使用了近800G，读写正常，一个1.5T磁盘建成的FRA磁盘组，目前还剩余729551M，无法写入归档
手动启停实例，报错如下
SQL> startup;
ORACLE instance started.
Total System Global Area 8.5516E+10 bytes
Fixed Size 2262656 bytes
Variable Size 1.8522E+10 bytes
Database Buffers 6.6840E+10 bytes
Redo Buffers 151117824 bytes
Database mounted.
ORA-16038: log 8 sequence# 10572 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 8 thread 2:
'+DATA/orcl/onlinelog/group_8.358.1128304555'
SQL>
alert日志报错
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 179 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845657
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_arc0_456667.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 164 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845642
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_arc0_456667.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 110 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845588
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_arc0_456667.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 95 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845573
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_arc0_456667.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
ORA-19502: write error on file "+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713", block number 67584 (block size=512)
WARNING: failed to write mirror side 1 of virtual extent 48 logical extent 0 of file 128205 in group 1 on disk 0 allocation unit 845525
ARC0: I/O error 19502 archiving log 7 to '+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10575.128205.1165411713'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance orcl2 - Archival Error
ORA-16038: log 7 sequence# 10575 cannot be archived
ORA-19502: write error on file "", block number (block size=)
Thu Apr 04 12:47:37 2024
WARNING: Write Failed. group:1 disk:0 AU:831879 offset:0 size:1048576
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc1_318526.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: failed to write mirror side 1 of virtual extent 166 logical extent 0 of file 129834 in group 1 on disk 0 allocation unit 831879
ARC1: Encountered disk I/O error 19502
ARC1: Closing local archive destination LOG_ARCHIVE_DEST_1: '+FRA/orcl/archivelog/2024_04_04/thread_1_seq_14255.129834.1165409257' (error 19502) (orcl1)
ARC1: I/O error 19502 archiving log 3 to '+FRA/orcl/archivelog/2024_04_04/thread_1_seq_14255.129834.1165409257'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance orcl1 - Archival Error
ORA-16038: log 3 sequence# 14255 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 3 thread 1: '+DATA/orcl/onlinelog/group_3.353.1128304551'
Thu Apr 04 12:47:37 2024
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance orcl1 - Archival Error
ORA-16014: log 3 sequence# 14255 not archived, no available destinations
-----------------------------------------------------
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
WARNING: Write Failed. group:1 disk:0 AU:831879 offset:0 size:1048576
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_211812.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: failed to write mirror side 1 of virtual extent 166 logical extent 0 of file 129834 in group 1 on disk 0 allocation unit 831879
ARCH: Encountered disk I/O error 19502
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10572.129834.1165409485' (error 19502) (orcl1)
ARCH: I/O error 19502 archiving log 8 to '+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10572.129834.1165409485'
Abort recovery for domain 0
WARNING: Write Failed. group:1 disk:0 AU:844803 offset:0 size:1048576
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc2_212060.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: failed to write mirror side 1 of virtual extent 5 logical extent 0 of file 129836 in group 1 on disk 0 allocation unit 844803
ARC2: Encountered disk I/O error 19502
ARC2: Closing local archive destination LOG_ARCHIVE_DEST_1: '+FRA/orcl/archivelog/2024_04_04/thread_2_seq_10573.129836.1165409485' (error 19502) (orcl1)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_211812.trc:
ORA-16038: log 8 sequence# 10572 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 8 thread 2: '+DATA/orcl/onlinelog/group_8.358.1128304555'
ORA-16038 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:33866:2} */...
[root@oracle1 trace]# strings /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc2_212060.trc |more
Trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_arc2_212060.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1
System name: Linux
Node name: oracle1
Release: 3.10.0-957.el7.x86_64
Version: #1 SMP Thu Nov 8 23:39:32 UTC 2018
Machine: x86_64
Instance name: orcl1
Redo thread mounted by this instance: 1
Oracle process number: 49
Unix process pid: 212060, image: oracle@oracle1 (ARC2)
*** 2024-04-04 12:51:24.838
*** SESSION ID:(9458.19) 2024-04-04 12:51:24.838
*** CLIENT ID:() 2024-04-04 12:51:24.838
*** SERVICE NAME:() 2024-04-04 12:51:24.838
*** MODULE NAME:() 2024-04-04 12:51:24.838
*** ACTION NAME:() 2024-04-04 12:51:24.838
Initial buffer sizes: read 1024K, overflow 832K, change 805K
*** 2024-04-04 12:51:25.696
WARNING: Write Failed. group:1 disk:0 AU:844803 offset:0 size:1048576
path:/dev/asmfra1
incarnation:0xd36c3ae2 asynchronous result:'I/O error'
subsys:System iop:0x7fd3d9ae1178 bufp:0x7fd3d95de000 osderr:0x0 osderr1:0x0
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: failed to write mirror side 1 of virtual extent 5 logical extent 0 of file 129836 in group 1 on disk 0 allocation unit 844803
*** 2024-04-04 12:51:25.699 4329 krsh.c
ARC2: Encountered disk I/O error 19502
[root@oracle1 trace]#
因业务紧急，将部分归档拷贝到节点一的/home/grid/目录下，在rman中crosscheck archivelog all;后，业务恢复正常。
存储端反馈正常，截图如下
