Symptoms:
alert_+ASM.log
Tue Oct 25 13:53:04 2016
WARNING: Library 'ASM Library - Generic Linux, version 2.0.4 (KABI_V2)' does not support advanced format disks
Tue Oct 25 13:53:04 2016
ORA-15186: ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:DATA_0001"
Tue Oct 25 13:53:04 2016
ORA-15186: ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:INDX"
Tue Oct 25 13:53:04 2016
ORA-15186: ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:REDO"
Tue Oct 25 13:53:04 2016
WARNING: Read Failed. group:0 disk:0 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xecc6a988 asynchronous result:'I/O error'
subsys:Unknown library krq:0x2b38f4156350 bufp:0x2b38f3d33000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:0 disk:2 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xecc6a98a asynchronous result:'I/O error'
subsys:Unknown library krq:0x2b38f3d301a0 bufp:0x2b38f3ad8000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:0 disk:3 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xecc6a98b asynchronous result:'I/O error'
subsys:Unknown library krq:0x2b38f4155b10 bufp:0x2b38f3ad6000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 1/0xA1065978 (DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image: oracle@testdb.ipx.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 1/0xA1065978 (DATA)
NOTE: cache ending mount (fail) of group DATA number=1 incarn=0xa1065978
NOTE: cache deleting context for group DATA 1/0xa1065978
Tue Oct 25 13:53:04 2016
GMON dismounting group 1 at 2 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup DATA was not mounted
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 2/0xA1165979 (INDX)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image: oracle@testdb.ipx.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 2/0xA1165979 (INDX)
NOTE: cache ending mount (fail) of group INDX number=2 incarn=0xa1165979
NOTE: cache deleting context for group INDX 2/0xa1165979
Tue Oct 25 13:53:04 2016
GMON dismounting group 2 at 4 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup INDX was not mounted
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 3/0xA116597A (REDO)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image: oracle@testdb.ipx.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 3/0xA116597A (REDO)
NOTE: cache ending mount (fail) of group REDO number=3 incarn=0xa116597a
NOTE: cache deleting context for group REDO 3/0xa116597a
Tue Oct 25 13:53:04 2016
GMON dismounting group 3 at 6 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup REDO was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "REDO" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15017: diskgroup "INDX" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15080: synchronous I/O operation failed to read block 0 of disk 3 in disk group
ORA-15080: synchronous I/O operation failed to read block 0 of disk 2 in disk group
ORA-15080: synchronous I/O operation failed to read bloc
Cause of the Problem:
All the asmlib disks appear in ASM's v$asm_disk view, however their HEADER_STATUS=UNKNOWN
However, ASMLIB sees the disks:
[root@testdb ~]# /etc/init.d/oracleasm listdisks
DATA_0001
FLASH001
INDX
REDO
A list of the asmlib disks, show their major and minor numbers (first set of numbers from the left):
[root@testdb ~]# ls -l /dev/oracleasm/disks
total 0
brw-rw---- 1 grid asmadmin 8, 17 Oct 21 16:06 DATA_0001
brw-rw---- 1 grid asmadmin 252, 1 Oct 21 16:06 FLASH001
brw-rw---- 1 grid asmadmin 8, 1 Oct 21 16:06 INDX
brw-rw---- 1 grid asmadmin 8, 33 Oct 21 16:06 REDO
A 'cat /proc/partitions', show the asmlib devices are linked to the 'sd' devices instead the 'multipath (dm)' devices. These asmlib devices used to be linked to the multipath devices before the problem:
[root@testdb ~]# cat /proc/partitions
major minor #blocks name
105 0 1172057816 cciss/c1d0
.
.
252 0 156250976 fioa
252 1 156248158 fioa1
253 8 1757614382 dm-8
253 9 292935982 dm-9
253 10 292935824 dm-10
253 11 1757610944 dm-11
8 0 2147483648 sda
8 1 2147480811 sda1
8 16 2147483648 sdb
8 17 2147480811 sdb1
8 32 209715200 sdc
8 33 209712478 sdc1
8 48 314612736 sdd
.
.
Solution:
The asmlib file (/etc/sysconfig/oracleasm) to do with mapping asmlib devices to multipath devices, seems is not configure for multipathing:
[root@testdb ~]# cat /etc/sysconfig/oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver. It is generated
# By running /etc/init.d/oracleasm configure. Please use that method
# to modify this file
#
# ORACLEASM_ENABELED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true
# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid
# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=asmadmin
# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER=""
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=""
After ORACLEASM_SCANORDER and ORACLEASM_EXCLUDE were updated in /etc/sysconfig/oracleasm, ASM was able to display the asmlib disks with v$asm_disk.header_status=MEMBER. Thereafter, ASM was able to mount the diskgroup.
ORACLEASM_SCANORDER="mpath dm"
ORACLEASM_SCANEXCLUDE="sd"
That’s it – now simply restart ASMLib and it should be able to mount the disk group.