Copying an Oracle Database to another environment does not work - oracle

Question

I am trying to restore a database from one environment to another. I backed up all the files from machineA and copied them to machineB.
The directory structures and file locations of both machines is the same and both are running Oracle 10.2.0.3.0.
I have done this several times before and it has always worked fine but this time i seem to be struggling and i appear to be stuck. After restoring all files into machineB, i startup oracle and it is showing that it has started.
SQL> startup
ORACLE instance started.
Total System Global Area 1610612736 bytes
Fixed Size 2030456 bytes
Variable Size 234882184 bytes
Database Buffers 1358954496 bytes
Redo Buffers 14745600 bytes
Database mounted.
Database opened.
A few minutes later it just terminates. I look at the alert log and this is what it is showing
ALTER DATABASE MOUNT
Wed Nov 23 11:16:14 2011
Setting recovery target incarnation to 1
Wed Nov 23 11:16:14 2011
Successful mount of redo thread 1, with mount id 4202976378
Wed Nov 23 11:16:14 2011
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Wed Nov 23 11:16:14 2011
ALTER DATABASE OPEN
Wed Nov 23 11:16:15 2011
Beginning crash recovery of 1 threads
parallel recovery started with 2 processes
Wed Nov 23 11:16:15 2011
Started redo scan
Wed Nov 23 11:16:15 2011
Completed redo scan
22887 redo blocks read, 29 data blocks need recovery
Wed Nov 23 11:16:15 2011
Started redo application at
Thread 1: logseq 29229, block 72
Wed Nov 23 11:16:15 2011
Recovery of Online Redo Log: Thread 1 Group 3 Seq 29229 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo03.log
Wed Nov 23 11:16:15 2011
Completed redo application
Wed Nov 23 11:16:16 2011
Completed crash recovery at
Thread 1: logseq 29229, block 22959, scn 10603747634124
29 data blocks read, 29 data blocks written, 22887 redo blocks read
Wed Nov 23 11:16:17 2011
Thread 1 advanced to log sequence 29230
Thread 1 opened at log sequence 29230
Current log# 1 seq# 29230 mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Successful open of redo thread 1
Wed Nov 23 11:16:17 2011
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Nov 23 11:16:17 2011
SMON: enabling cache recovery
Wed Nov 23 11:16:18 2011
Successfully onlined Undo Tablespace 1.
Wed Nov 23 11:16:18 2011
SMON: enabling tx recovery
Wed Nov 23 11:16:18 2011
Database Characterset is WE8ISO8859P1
Wed Nov 23 11:16:18 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_smon_13515.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=16, OS id=13532
Wed Nov 23 11:16:20 2011
Doing block recovery for file 2 block 972
Block recovery from logseq 29230, block 56 to scn 10603747634191
Wed Nov 23 11:16:20 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery stopped at EOT rba 29230.66.16
Block recovery completed at rba 29230.66.16, scn 2468.3768347663
Doing block recovery for file 2 block 25
Block recovery from logseq 29230, block 56 to scn 10603747634177
Wed Nov 23 11:16:20 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.58.16, scn 2468.3768347651
Wed Nov 23 11:16:20 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_smon_13515.trc:
ORA-01595: error freeing extent (3) of rollback segment (2))
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
Wed Nov 23 11:16:20 2011
Completed: ALTER DATABASE OPEN
Wed Nov 23 11:16:21 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_mmon_13521.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
Wed Nov 23 11:16:22 2011
Doing block recovery for file 2 block 972
Block recovery from logseq 29230, block 56 to scn 10603747634191
Wed Nov 23 11:16:22 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.66.16, scn 2468.3768347664
Doing block recovery for file 2 block 25
Block recovery from logseq 29230, block 56 to scn 10603747634208
Wed Nov 23 11:16:23 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.88.16, scn 2468.3768347681
Wed Nov 23 11:18:27 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_m000_13538.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
Wed Nov 23 11:18:28 2011
Flush retried for xcb 0x3ddf94728, pmd 0x3dc32cc30
Doing block recovery for file 2 block 972
Block recovery from logseq 29230, block 56 to scn 10603747634191
Wed Nov 23 11:18:28 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.66.16, scn 2468.3768347664
Wed Nov 23 11:18:28 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_m000_13538.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
Flush retried for xcb 0x3ddf94728, pmd 0x3dc32cc30
Doing block recovery for file 2 block 972
Block recovery from logseq 29230, block 56 to scn 10603747634191
Wed Nov 23 11:18:30 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.66.16, scn 2468.3768347664
Wed Nov 23 11:18:30 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_m000_13538.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
Flush retried for xcb 0x3ddf94728, pmd 0x3dc32cc30
Doing block recovery for file 2 block 972
Block recovery from logseq 29230, block 56 to scn 10603747634191
Wed Nov 23 11:18:32 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.66.16, scn 2468.3768347664
Wed Nov 23 11:18:32 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_m000_13538.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
Flush retried for xcb 0x3ddf94728, pmd 0x3dc32cc30
Doing block recovery for file 2 block 972
Block recovery from logseq 29230, block 56 to scn 10603747634191
Wed Nov 23 11:18:34 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.66.16, scn 2468.3768347664
Wed Nov 23 11:18:34 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_m000_13538.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-06512: at "SYS.PRVT_ADVISOR", line 4896
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-06512: at line 1
Flush retried for xcb 0x3ddf94728, pmd 0x3dc32cc30
Doing block recovery for file 2 block 972
Block recovery from logseq 29230, block 56 to scn 10603747634191
Wed Nov 23 11:18:35 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.66.16, scn 2468.3768347664
Wed Nov 23 11:18:35 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_m000_13538.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-06512: at "SYS.PRVT_ADVISOR", line 4896
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
ORA-06512: at line 1
Wed Nov 23 11:18:36 2011
Flush retried for xcb 0x3ddf94728, pmd 0x3dc32cc30
Doing block recovery for file 2 block 972
Block recovery from logseq 29230, block 56 to scn 10603747634191
Wed Nov 23 11:18:36 2011
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Mem# 0: /u/db1/app/oracle/oradata/mydb/redo01.log
Block recovery completed at rba 29230.66.16, scn 2468.3768347664
Wed Nov 23 11:18:36 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_pmon_13503.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
Wed Nov 23 11:18:37 2011
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_pmon_13503.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
PMON: terminating instance due to error 472
Instance terminated by PMON, pid = 13503
The only difference this time (compared from last time i restored the database) is that this time i cleared the trace files from the bdump directory before i started up the directory(not the alert log). Could this have caused this problem?
Here is an example of one of the trace file mentioned in one of the alert log
/u/db1/app/oracle/admin/ccsbill/bdump/mydb_pmon_13503.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME = /u/db1/app/oracle/product/10.2.0/db
System name: SunOS
Node name: myPC
Release: 5.9
Version: Generic_122300-13
Machine: sun4u
Instance name: mydb
Redo thread mounted by this instance: 1
Oracle process number: 2
Unix process pid: 13503, image: oracle#myPC (PMON)
*** 2011-11-23 11:18:36.626
*** SERVICE NAME:(SYS$BACKGROUND) 2011-11-23 11:18:36.625
*** SESSION ID:(170.1) 2011-11-23 11:18:36.625
Flush retried for xcb 0x3ddf94728, pmd 0x3dc32cc30
DEBUG: Reconstructing undo block 0x8003cc for xcb 0x3ddf94728
Doing block recovery for file 2 block 972
Block header before block recovery:
buffer tsn: 1 rdba: 0x008003cc (2/972)
scn: 0x09a4.e09bc65c seq: 0x01 flg: 0x04 tail: 0xc65c0201
frmt: 0x02 chkval: 0x409e type: 0x02=KTU UNDO BLOCK
Block recovery from logseq 29230, block 56 to scn 10603747634191
*** 2011-11-23 11:18:36.641
Recovery of Online Redo Log: Thread 1 Group 1 Seq 29230 Reading mem 0
Block recovery completed at rba 29230.66.16, scn 2468.3768347664
----- Redo read statistics for thread 1 -----
Read rate (ASYNC): 383Kb in 0.06s => 6.24 Mb/sec
Total physical reads: 4096Kb
Longest record: 0Kb, moves: 0/11 (0%)
Longest LWN: 1Kb, moves: 0/7 (0%), moved: 0Mb
Last redo scn: 0x09a4.e09c6c0e (10603747634190)
----------------------------------------------
Block image after block recovery:
buffer tsn: 1 rdba: 0x008003cc (2/972)
scn: 0x09a4.e09bc65c seq: 0x01 flg: 0x04 tail: 0xc65c0201
frmt: 0x02 chkval: 0x409e type: 0x02=KTU UNDO BLOCK
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x00000003D13FA000 to 0x00000003D13FC000
3D13FA000 02A20000 008003CC E09BC65C 09A40104 [...........\....]
3D13FA010 409E0000 00020018 00065BF2 C49F1515 [#.........[.....]
3D13FA020 00001FE8 1F641ED8 1E041D7C 1CF81C5C [.....d.....|...\]
3D13FA030 1BC41B40 1ADC1A88 1A1C1998 192818A8 [...#.........(..]
3D13FA040 1810172C 16C4166C 15E8156C 14D40000 [...,...l...l....]
3D13FA050 00000000 00000000 00000000 00000000 [................]
Repeat 328 times
3D13FB4E0 00000000 00000000 000C0048 0020001D [...........H. ..]
3D13FB4F0 00020000 000000ED 000000ED 00000000 [................]
3D13FB500 00000000 0B011800 04080001 008003CC [................]
3D13FB510 C49F1300 E09BC00B 09A40000 E09BC011 [................]
3D13FB520 09A40001 000A0024 E09BC647 09A4FFFF [.......$...G....]
3D13FB530 008003C5 00000000 00000000 04010000 [................]
3D13FB540 00000000 00070010 0004935C 00800580 [...........\....]
3D13FB550 9B5B1600 800009A4 E09BC643 0040067A [.[.........C.#.z]
3D13FB560 00400679 12FF0501 020076C0 2C000100 [.#.y......v.,...]
3D13FB570 00001301 FFF90100 00000000 00050000 [................]
...skipping...
child# table reference handle
------ -------- --------- --------
0 3dac7ffd8 3dac7fc48 3de7ab528
DATA BLOCKS:
data# heap pointer status pins change whr
----- -------- -------- --------- ---- ------ ---
0 3de7abae8 3dac80628 I/P/A/-/- 0 NONE 00
----------------------------------------
SO: 3df696d30, type: 12, owner: 3df4091d8, flag: -/-/-/0x00
KSV Slave Class State
--------------
slave num 0, incarnation 1, KSV Context 3df694da0, creator: 3df2f5ff8
slave flags: 0x102
ksvcctx: 3df694da0 dpptr: 3df696d30 exitcond: 0 class#: 5
active: 1 spawned: 1 max: a flags: 0x2 enqueue: 0
directmsghdl: 3df4678b8 workmsghdl: 3df467928
ksvwqlr: 3df694da0 latch 3df694da0
ksvrecv: 3df694e40 op: 0x0 ro = 0 owner = 0
Queue (0)
ksvmqd: 3df694e90 count : 0
ksvwqlr: 3df694e90 latch 3df694e90
ksvrecv: 3df694f30 op: 0x0 ro = 0 owner = 0
Queue messages 3df694f50 Is Empty [3df694f50,3df694f50]
Queue (1)
ksvmqd: 3df694f68 count : 0
ksvwqlr: 3df694f68 latch 3df694f68
ksvrecv: 3df695008 op: 0x0 ro = 0 owner = 0
Queue messages 3df695028 Is Empty [3df695028,3df695028]
Queue (2)
ksvmqd: 3df695040 count : 0
ksvwqlr: 3df695040 latch 3df695040
ksvrecv: 3df6950e0 op: 0x0 ro = 0 owner = 0
Queue messages 3df695100 Is Empty [3df695100,3df695100]
dmsg: sendq: 3df696dc0 Is Empty [3df696dc0,3df696dc0]
dmsg: recvq: 3df696dd0 Is Empty [3df696dd0,3df696dd0]
dmsg: doneq: 3df696de0 Is Empty [3df696de0,3df696de0]
wmsg: workq: 3df696df0 Is Empty [3df696df0,3df696df0]
wmsg: doneq: 3df696e00 Is Empty [3df696e00,3df696e00]
Class Context: active: 1, spawned: 1, max: 10
Context Flags: 0x2, Work Queue: 3df694e90, Class Num: 5
----------------------------------------
SO: 3ddfcbbe8, type: 41, owner: 3df4091d8, flag: INIT/-/-/0x00
(dummy) nxc=0, nlb=1
----------------------------------------
SO: 3ddf46648, type: 39, owner: 3ddfcbbe8, flag: -/-/-/0x00
(List of Blocks) next index = 5
index itli buffer hint rdba savepoint
-----------------------------------------------------------
0 1 0x3d0fa00a8 0xc05534 0x6b69
1 2 0x3d0f9f2d8 0xc002ee 0x6b6b
2 2 0x3d0f97ce8 0xc002f8 0x6b6d
3 2 0x3d0f97ac8 0xc00300 0x6b6f
4 2 0x3d0f97578 0xc0894a 0x6b71
----------------------------------------
SO: 3df43ad08, type: 3, owner: 3df2f9f38, flag: INIT/-/-/0x00
(call) sess: cur 3df4091d8, rec 3df4053c8, usr 3df4091d8; depth: 0
(k2g table)
error 600 detected in background process
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []

Answer 1

Did you see the ORA-00600 [4194] errors? They look like this:
Errors in file /u/db1/app/oracle/admin/mydb/bdump/mydb_smon_13515.trc:
ORA-00600: internal error code, arguments: [4194], [21], [21], [], [], [], [], []
That's your problem.
ORA-00600 always means to work with Oracle Support on the problem. I did a quick lookup, and the 4194 error means you have undo segment corruption.
You may try redoing the clone, assuming the source database itself isn't corrupted. If the source has this problem too, you'll probably need to restore/recover the UNDO tablespace, at a minimum.
I strongly suggest you login to MOS support site, and look closely at this document:
ORA-600 [4194] "Undo Record Number Mismatch While Adding Undo Record"
[ID 39283.1]
Hope that helps.