Oracle OPEN DATABASE FILED WITH ORA-600 [2662]

PROBLEM:
--------
Customer is cloing a database from production.
Placing all the database in begin backup mode then take a snapshot of all the
datafiles and redo logs. Copy all these files to clone server.
Recrate controlfiles, recover by entering redo log.
Recovery shows 'media recovery complete'.
alter database open resetlogs failed with ORA-600 [2662]
 
DIAGNOSTIC ANALYSIS:
--------------------
Customer has tried many times and got the same results.
Also tried to applied archivelog instead of redo log but failed with same 
error.
V$datafile and v$datafile_header shows the SCNs of all the datafiles are
the same.
Customer also tried setting event 10015 to adjust SCN but didn't work.
 

DATA COLLECTED
================
 
Customer is cloing a database from production.
 
1. Take all the tablespace of prod database into 'begin backup' mode.
2. Take a snapshop of the prod database array.
3. Copy all the datafiles and redo logs to clone server.
4. Recreate controlfiles with NOARCHIVELOG option. (Prod database is in 
archvielog mode)
5. recover database using backup controfile until cancel;
6. enter pathname of redo log file.
	--> it shows media recovery complete
 
---------------------
SQL> ed cr_cntrl.sql
 
"cr_cntrl.sql" 718 lines, 24060 characters
STARTUP NOMOUNT
CREATE CONTROLFILE SET DATABASE "ERP1" RESETLOGS NOARCHIVELOG
    MAXLOGFILES 32
    MAXLOGMEMBERS 5
    MAXDATAFILES 1022
    MAXINSTANCES 1
    MAXLOGHISTORY 1588
LOGFILE
  GROUP 10 '/m006/oradata/ERP1/redo10.log'  SIZE 25M,
  GROUP 11 '/m006/oradata/ERP1/redo11.log'  SIZE 25M,
  GROUP 12 '/m006/oradata/ERP1/redo12.log'  SIZE 25M,
  GROUP 13 '/m006/oradata/ERP1/redo13.log'  SIZE 25M,
  GROUP 14 '/m006/oradata/ERP1/redo14.log'  SIZE 25M
DATAFILE
'/m006/oradata/ERP1/abmd01.dbf',
'/m006/oradata/ERP1/ahmd01.dbf',
'/m006/oradata/ERP1/akdr01.dbf',
'/m006/oradata/ERP1/amwx01.dbf',
'/m006/oradata/ERP1/apdb02.dbf',
'/m006/oradata/ERP1/apdb03.dbf',
'/m006/oradata/ERP1/apxr01.dbf',
'/m006/oradata/ERP1/arc_data02.dbf',
'/m006/oradata/ERP1/asfd01.dbf',
'/m006/oradata/ERP1/asgd01.dbf',
'/m006/oradata/ERP1/asod01.dbf',
'/m006/oradata/ERP1/axdr01.dbf',
'/m006/oradata/ERP1/azx01.dbf',
'/m006/oradata/ERP1/bisd05.dbf',
'/m006/oradata/ERP1/bisd09.dbf',
'/m006/oradata/ERP1/bisx02.dbf',
'/m006/oradata/ERP1/bivd01.dbf',
'/m006/oradata/ERP1/bivx01.dbf',
'/m006/oradata/ERP1/bomxb02.dbf',
'/m006/oradata/ERP1/bscaix01.dbf',
'/m006/oradata/ERP1/bsd01.dbf',
'/m006/oradata/ERP1/cscx01.dbf',
'/m006/oradata/ERP1/csfx01.dbf',
'/m006/oradata/ERP1/cufd01.dbf',
'/m006/oradata/ERP1/cuid01.dbf',
'/m006/oradata/ERP1/czd01.dbf',
'/m006/oradata/ERP1/eaad01.dbf',
 
SQL> @cr_cntrl.sql
ORACLE instance started.
 
Total System Global Area 1152349480 bytes
Fixed Size                   738600 bytes
Variable Size             973078528 bytes
Database Buffers          167772160 bytes
Redo Buffers               10760192 bytes
 
Control file created.
 
SQL> recover database until cancel using backup controlfile;
ORA-279: change 13818598237 generated at 03/23/2006 21:21:28 needed for
thread 1
ORA-289: suggestion : /m001/app/oracle/product/9.2.0/dbs/arch1_255784.dbf
ORA-280: change 13818598237 for thread 1 is in sequence #255784
 
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/bcm011/oradata/PROD/redo19.log
ORA-279: change 13818628587 generated at 03/23/2006 21:28:11 needed for
thread 1
ORA-289: suggestion : /m001/app/oracle/product/9.2.0/dbs/arch1_255785.dbf
ORA-280: change 13818628587 for thread 1 is in sequence #255785
ORA-278: log file '/bcm011/oradata/PROD/redo19.log' no longer needed for 
this
recovery
 
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/bcm011/oradata/PROD/redo10.log
Log applied.
Media recovery complete.
 
--------------------
 
 
7. alter database open resetlogs failed with ORA-600 [2662] error.
--------------
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
 
ORA-3113: end-of-file on communication channel
----------------
 
Before issued 'alter database open resetlogs', query checkpoint_change# from 
v$datafile and v$datafile_header. The outputs shows the SCNs of the all the 
datafiles are the same for both views.
 
SQL> select distinct  CHECKPOINT_CHANGE# from v$datafile_header UNION select 
distinct  CHECKPOINT_CHANGE# from v$datafile;
 
CHECKPOINT_CHANGE#
------------------
       13818655769
 
- Customer continued recovery to apply the online redo logs and get out of 
hot backup mode.