Symptoms:

alert_+ASM.log

Tue Oct 25 13:53:04 2016
WARNING: Library 'ASM Library - Generic Linux, version 2.0.4 (KABI_V2)' does not support advanced format disks
Tue Oct 25 13:53:04 2016
ORA-15186: ASMLIB error function = [asm_open(global)],  error = [1],  mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:DATA_0001"
Tue Oct 25 13:53:04 2016
ORA-15186: ASMLIB error function = [asm_open(global)],  error = [1],  mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:INDX"
Tue Oct 25 13:53:04 2016
ORA-15186: ASMLIB error function = [asm_open(global)],  error = [1],  mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:REDO"
Tue Oct 25 13:53:04 2016
WARNING: Read Failed. group:0 disk:0 AU:0 offset:0 size:4096
path:Unknown disk
         incarnation:0xecc6a988 asynchronous result:'I/O error'
         subsys:Unknown library krq:0x2b38f4156350 bufp:0x2b38f3d33000 osderr1:0x0 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:0 disk:2 AU:0 offset:0 size:4096
path:Unknown disk
         incarnation:0xecc6a98a asynchronous result:'I/O error'
         subsys:Unknown library krq:0x2b38f3d301a0 bufp:0x2b38f3ad8000 osderr1:0x0 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:0 disk:3 AU:0 offset:0 size:4096
path:Unknown disk
         incarnation:0xecc6a98b asynchronous result:'I/O error'
         subsys:Unknown library krq:0x2b38f4155b10 bufp:0x2b38f3ad6000 osderr1:0x0 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 1/0xA1065978 (DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image: oracle@testdb.ipx.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 1/0xA1065978 (DATA)
NOTE: cache ending mount (fail) of group DATA number=1 incarn=0xa1065978
NOTE: cache deleting context for group DATA 1/0xa1065978
Tue Oct 25 13:53:04 2016
GMON dismounting group 1 at 2 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup DATA was not mounted
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 2/0xA1165979 (INDX)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image: oracle@testdb.ipx.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 2/0xA1165979 (INDX)
NOTE: cache ending mount (fail) of group INDX number=2 incarn=0xa1165979
NOTE: cache deleting context for group INDX 2/0xa1165979
Tue Oct 25 13:53:04 2016
GMON dismounting group 2 at 4 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup INDX was not mounted
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 3/0xA116597A (REDO)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image: oracle@testdb.ipx.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 3/0xA116597A (REDO)
NOTE: cache ending mount (fail) of group REDO number=3 incarn=0xa116597a
NOTE: cache deleting context for group REDO 3/0xa116597a
Tue Oct 25 13:53:04 2016
GMON dismounting group 3 at 6 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup REDO was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "REDO" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15017: diskgroup "INDX" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15080: synchronous I/O operation failed to read block 0 of disk 3 in disk group
ORA-15080: synchronous I/O operation failed to read block 0 of disk 2 in disk group
ORA-15080: synchronous I/O operation failed to read bloc

Cause of the Problem:

All the asmlib disks appear in ASM's v$asm_disk view, however their HEADER_STATUS=UNKNOWN

However, ASMLIB sees the disks:

[root@testdb ~]#  /etc/init.d/oracleasm listdisks

DATA_0001

FLASH001

INDX

REDO

A list of the asmlib disks, show their major and minor numbers (first set of numbers from the left):

 [root@testdb ~]# ls -l /dev/oracleasm/disks

total 0

brw-rw---- 1 grid asmadmin   8, 17 Oct 21 16:06 DATA_0001

brw-rw---- 1 grid asmadmin 252,  1 Oct 21 16:06 FLASH001

brw-rw---- 1 grid asmadmin   8,  1 Oct 21 16:06 INDX

brw-rw---- 1 grid asmadmin   8, 33 Oct 21 16:06 REDO

 

A 'cat /proc/partitions', show the asmlib devices are linked to the 'sd' devices instead the 'multipath (dm)' devices. These asmlib devices used to be linked to the multipath devices before the problem:

[root@testdb ~]# cat /proc/partitions

major minor  #blocks  name

 

 105     0 1172057816 cciss/c1d0

 .

.

 252     0  156250976 fioa

 252     1  156248158 fioa1

 253     8 1757614382 dm-8

 253     9  292935982 dm-9

 253    10  292935824 dm-10

 253    11 1757610944 dm-11

   8     0 2147483648 sda

   8     1 2147480811 sda1

   8    16 2147483648 sdb

   8    17 2147480811 sdb1

   8    32  209715200 sdc

   8    33  209712478 sdc1

   8    48  314612736 sdd

.

.

Solution:

The asmlib file (/etc/sysconfig/oracleasm) to do with mapping asmlib devices to multipath devices, seems is not configure for multipathing:

[root@testdb ~]# cat /etc/sysconfig/oracleasm

#

# This is a configuration file for automatic loading of the Oracle

# Automatic Storage Management library kernel driver.  It is generated

# By running /etc/init.d/oracleasm configure.  Please use that method

# to modify this file

#

 

# ORACLEASM_ENABELED: 'true' means to load the driver on boot.

ORACLEASM_ENABLED=true

 

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

ORACLEASM_UID=grid

 

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

ORACLEASM_GID=asmadmin

 

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.

ORACLEASM_SCANBOOT=true

 

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER=""

 

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE=""

 

After ORACLEASM_SCANORDER and ORACLEASM_EXCLUDE were updated in /etc/sysconfig/oracleasm, ASM was able to display the asmlib disks with v$asm_disk.header_status=MEMBER. Thereafter,  ASM was able to mount the diskgroup.

 

ORACLEASM_SCANORDER="mpath dm"
ORACLEASM_SCANEXCLUDE="sd"

 

That’s it – now simply restart ASMLib and it should be able to mount the disk group.