Email: service@parnassusdata.com 7 x 24 online support!

Oracle WHILE OPENING DATABASE INSTANCE CRASH WITH ORA-00240

Oracle WHILE OPENING DATABASE INSTANCE CRASH WITH ORA-00240

PROBLEM:
--------
- This is three node RAC system.
- CRS & ASM is running on 11.1.0.7.
- Database is running on 10.2.0.4.
 
While trying to start the third database instance on any of the node, it is 
failing with ORA-240 error.
 
Thu Jun  2 06:04:40 2011
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Thu Jun  2 06:43:09 2011
System State dumped to trace file 
/u01/app/oracle/admin/gbsg3/bdump/gbsg33_diag_15760.trc
Thu Jun  2 06:43:15 2011
Errors in file /u01/app/oracle/admin/gbsg3/bdump/gbsg33_diag_15760.trc:
ORA-494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 3, 
osid 15788'
Thu Jun  2 06:43:20 2011
System State dumped to trace file 
/u01/app/oracle/admin/gbsg3/bdump/gbsg33_diag_15760.trc
System State dumped to trace file 
/u01/app/oracle/admin/gbsg3/bdump/gbsg33_diag_15760.trc
 
- It is a dbwr process which is hitting with [CF] enqueue.
- dbwr trace file shows below message:
 
CONTROL FILE ENQUEUE HELD FOR TOO LONG
 
      holding mode : S
    enqueue holder : 'inst 3, osid 23178'
 enqueue held time : 120 seconds
 
- dbwr process waiting for "Asm file metadata operation".
 
- Increased _controlfile_enqueue_holding_time=300 with no luck.