Symptoms:
alert_+ASM.log
Tue Oct 25 13:53:04 2016
WARNING: Library 'ASM Library - Generic Linux, version 2.0.4 (KABI_V2)' does not
support advanced format disks
Tue Oct 25 13:53:04 2016
ORA-15186:
ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not
permitted]
ORA-15025: could not open
disk "ORCL:DATA_0001"
Tue Oct 25 13:53:04 2016
ORA-15186:
ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not
permitted]
ORA-15025: could not open
disk "ORCL:INDX"
Tue Oct 25 13:53:04 2016
ORA-15186:
ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not
permitted]
ORA-15025: could not open
disk "ORCL:REDO"
Tue Oct 25 13:53:04 2016
WARNING: Read Failed. group:0 disk:0 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xecc6a988
asynchronous result:'I/O error'
subsys:Unknown library krq:0x2b38f4156350
bufp:0x2b38f3d33000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time
waited on I/O: 0 usec
WARNING: Read Failed. group:0 disk:2 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xecc6a98a
asynchronous result:'I/O error'
subsys:Unknown library krq:0x2b38f3d301a0
bufp:0x2b38f3ad8000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time
waited on I/O: 0 usec
WARNING: Read Failed. group:0 disk:3 AU:0 offset:0 size:4096
path:Unknown disk
incarnation:0xecc6a98b
asynchronous result:'I/O error'
subsys:Unknown library krq:0x2b38f4155b10
bufp:0x2b38f3ad6000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time
waited on I/O: 0 usec
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 1/0xA1065978 (DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image:
oracle@testdb.ipx.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 1/0xA1065978 (DATA)
NOTE: cache ending mount (fail) of group DATA number=1 incarn=0xa1065978
NOTE: cache deleting context for group DATA 1/0xa1065978
Tue Oct 25 13:53:04 2016
GMON dismounting group 1 at 2 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup DATA was not mounted
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 2/0xA1165979 (INDX)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image: oracle@testdb.ipx.com
(TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 2/0xA1165979 (INDX)
NOTE: cache ending mount (fail) of group INDX number=2 incarn=0xa1165979
NOTE: cache deleting context for group INDX 2/0xa1165979
Tue Oct 25 13:53:04 2016
GMON dismounting group 2 at 4 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup INDX was not
mounted
Tue Oct 25 13:53:04 2016
ERROR: no read quorum in group: required 2, found 0 disks
Tue Oct 25 13:53:04 2016
NOTE: cache dismounting (clean) group 3/0xA116597A (REDO)
NOTE: messaging CKPT to quiesce pins Unix process pid: 25451, image:
oracle@testdb.ipx.com (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 3/0xA116597A (REDO)
NOTE: cache ending mount (fail) of group REDO number=3 incarn=0xa116597a
NOTE: cache deleting context for group REDO 3/0xa116597a
Tue Oct 25 13:53:04 2016
GMON dismounting group 3 at 6 for pid 20, osid 25451
Tue Oct 25 13:53:04 2016
ERROR: diskgroup REDO was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "REDO" cannot be mounted
ORA-15040:
diskgroup is incomplete
ORA-15017: diskgroup "INDX" cannot be mounted
ORA-15040:
diskgroup is incomplete
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15040:
diskgroup is incomplete
ORA-15080:
synchronous I/O operation failed to read block 0 of disk 3 in disk group
ORA-15080:
synchronous I/O operation failed to read block 0 of disk 2 in disk group
ORA-15080: synchronous I/O operation failed to read bloc
Cause
of the Problem:
All the asmlib disks appear in ASM's v$asm_disk view, however their HEADER_STATUS=UNKNOWN
However, ASMLIB sees the disks:
[root@testdb ~]#
/etc/init.d/oracleasm listdisks
DATA_0001
FLASH001
INDX
REDO
A list of the asmlib disks, show their major and minor numbers (first
set of numbers from the left):
[root@testdb ~]# ls -l /dev/oracleasm/disks
total 0
brw-rw---- 1 grid
asmadmin 8, 17 Oct 21 16:06 DATA_0001
brw-rw---- 1 grid
asmadmin 252, 1 Oct 21 16:06 FLASH001
brw-rw---- 1 grid
asmadmin 8, 1 Oct 21 16:06 INDX
brw-rw---- 1 grid
asmadmin 8, 33 Oct 21 16:06 REDO
A 'cat /proc/partitions', show the asmlib devices are linked to the 'sd' devices
instead the 'multipath (dm)' devices. These asmlib devices used to be linked to
the multipath devices before the problem:
[root@testdb
~]# cat /proc/partitions
major minor #blocks
name
105
0 1172057816 cciss/c1d0
.
.
252
0 156250976 fioa
252 1
156248158 fioa1
253
8 1757614382 dm-8
253
9 292935982 dm-9
253
10 292935824 dm-10
253
11 1757610944 dm-11
8
0 2147483648 sda
8 1 2147480811 sda1
8
16 2147483648 sdb
8 17 2147480811 sdb1
8
32 209715200 sdc
8 33 209712478 sdc1
8
48 314612736 sdd
.
.
Solution:
The asmlib file (/etc/sysconfig/oracleasm) to do with mapping asmlib devices to multipath devices,
seems is not configure for multipathing:
[root@testdb
~]# cat /etc/sysconfig/oracleasm
#
# This is a
configuration file for automatic loading of the Oracle
# Automatic
Storage Management library kernel driver.
It is generated
# By running
/etc/init.d/oracleasm configure. Please
use that method
# to modify this
file
#
#
ORACLEASM_ENABELED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true
# ORACLEASM_UID:
Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid
# ORACLEASM_GID:
Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=asmadmin
#
ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true
#
ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER=""
#
ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=""
After ORACLEASM_SCANORDER and ORACLEASM_EXCLUDE were updated in
/etc/sysconfig/oracleasm, ASM was able to display the asmlib disks with
v$asm_disk.header_status=MEMBER. Thereafter, ASM was able to mount the
diskgroup.
ORACLEASM_SCANORDER="mpath dm"
ORACLEASM_SCANEXCLUDE="sd"
That’s it – now simply restart ASMLib and it should be
able to mount the disk group.