SQL> startup
ORACLE instance started.
Total System Global Area 285212672 bytes
Fixed Size 1218992 bytes
Variable Size 79693392 bytes
Database Buffers 201326592 bytes
Redo Buffers 2973696 bytes
ORA-01102: cannot mount database in EXCLUSIVE mode
Cause: There are some client shadow processes hanging. Although the lk<SID> file is deleted the hanging processes still have a lock on the open file handle. This prevents the database to startup although a new lk<SID> file can be created successfully.
An oracle process (background or shadow process) that exists while the instance is not started (crashed or not cleanly stopped) can have a lock on a file while this file is actually removed from the system. This is because on UNIX there is still a lock on the open file handle.
Solution:
Verify if there are existing oracle processes for the database:
[oracle@servername ~]$ ps -ef | grep $ORACLE_SID
oracle 1525 1 0 10:13 ? 00:00:03 ora_j000_GRIDDB3
oracle 1527 1 0 10:13 ? 00:00:01 ora_j001_GRIDDB3
oracle 4198 1 0 10:27 ? 00:00:00 oracleGRIDDB3 (LOCAL=NO)
oracle 7479 1 0 May30 ? 00:00:00 oracleGRIDDB3 (LOCAL=NO)
.
.
.
oracle 21246 1 0 May30 ? 00:00:09 ora_pmon_GRIDDB3
oracle 21248 1 0 May30 ? 00:00:11 ora_psp0_GRIDDB3
oracle 21250 1 0 May30 ? 00:12:29 ora_vktm_GRIDDB3
oracle 21254 1 0 May30 ? 00:00:01 ora_gen0_GRIDDB3
oracle 21256 1 0 May30 ? 00:00:02 ora_diag_GRIDDB3
oracle 21258 1 0 May30 ? 00:00:02 ora_dbrm_GRIDDB3
oracle 21260 1 0 May30 ? 00:01:00 ora_dia0_GRIDDB3
oracle 21262 1 0 May30 ? 00:00:01 ora_mman_GRIDDB3
oracle 21264 1 0 May30 ? 00:00:11 ora_dbw0_GRIDDB3
oracle 21266 1 0 May30 ? 00:00:24 ora_lgwr_GRIDDB3
oracle 21268 1 0 May30 ? 00:00:16 ora_ckpt_GRIDDB3
oracle 21270 1 0 May30 ? 00:00:07 ora_smon_GRIDDB3
oracle 21272 1 0 May30 ? 00:00:00 ora_reco_GRIDDB3
oracle 21274 1 0 May30 ? 00:00:16 ora_mmon_GRIDDB3
.
.
Kill the hanging processes to release the lock on the file handle:
[oracle@servername ~]$ kill -9 `ps -ef | grep $ORACLE_SID | grep -v grep | awk '{print $2}'`
[oracle@sservername ~]$ ps -ef | grep $ORACLE_SID
oracle 11990 11927 0 10:50 pts/1 00:00:00 grep GRIDDB3