Oracle generates lots of .dbf files - oracle

Question

When I arrived at the office this morning, our Oracle 10.2 server was out of disk space. On closer inspection I found that about 1 to 4 or more .dbf files are generated once a minute (e.g. 1_1278092_658232789.dbf, 1_1278093_658232789.dbf, etc.). I created a bit of space, but Oracle still creates these files without deleting the old ones. It seems to have started about 35 hours ago. How do I restore the server to normal. Please note that I am not an Oracle DBA and have limited Oracle knowledge.
Edit 1:
First, I manage to clear about 270GB of space with the following, which allowed the server to keep running:
RMAN> CROSSCHECK BACKUP;
RMAN> DELETE ARCHIVELOG ALL;
To answer ora-600's questions:
In which path does Oracle create those files?
/home/oracle/archive/
(which is also the value of log_archive_dest_1)
DB_CREATE_FILE_DEST (parameter for datafiles)
This does not seem to have been set ("show parameter DB_CREATE_FILE_DEST" shows no value), but the database files are in
/home/oracle/app/oracle/product/oradata/irs3
DB_RECOVERY_FILE_DEST (parameter for FRA) -- which sub directory?
sys#iris > show parameter DB_RECOVERY_FILE_DEST
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest string /home/backup
db_recovery_file_dest_size big integer 2500G
I suspect that these are flashback logs. If so you should limit the flash recovery area (FRA) by setting the parameter DB_RECOVERY_FILE_DEST_SIZE to a smaller value. Oracle keeps writing flashback logs until the FRA is out of space... then it stats removing/overwriting old files.
Wel, the previous DBA did set this to a very high value and now it is full. E.g. look at:
sys#iris > SELECT NAME, (SPACE_LIMIT/1024/1024) || 'MB' AS SPACE_LIMIT,
((SPACE_LIMIT - SPACE_USED + SPACE_RECLAIMABLE)/1024/1024) || 'MB' AS SPACE_AVAILABLE,
ROUND((SPACE_USED - SPACE_RECLAIMABLE)/SPACE_LIMIT * 100, 1)
AS PERCENT_FULL
FROM V$RECOVERY_FILE_DEST;
NAME SPACE_LIMIT SPACE_AVAILABLE PERCENT_FULL
/home/backup 2560000MB 940MB 100
But RMAN now spits errors like these in its log files:
....
input archive log thread=1 sequence=1278543 recid=1271197 stamp=866048159
input archive log thread=1 sequence=1278544 recid=1271198 stamp=866048232
channel ORA_DISK_1: starting piece 1 at 11-DEC-14
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 12/11/2014 22:07:20
ORA-19809: limit exceeded for recovery files
ORA-19804: cannot reclaim 2691888128 bytes disk space from 2684354560000 limit
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting archive log backupset
channel ORA_DISK_1: specifying archive log(s) in backup set
input archive log thread=1 sequence=1278907 recid=1271561 stamp=866062135
....
Even though there is space on the drive:
-bash-3.2$ df -h
Filesystem Size Used Avail Use% Mounted on
....
/dev/vg01/lvol1 684G 365G 317G 54% /home
Why does the query above give the space as full, even though there are space available on the drive?
Below is more info, if needed.
Thanks.
Nico
RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/home/backup/%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/home/oracle/app/oracle/product/10/dbs/snapcf_irs3.f'; # default

Answer 1

Thanks for the details that helps to identify the problem.
I think you have 2 problems.
1st problem is the database keeps creating theese small .dbf files. This is not a problem but the files need to be dealt with correctly. These files are called "archivelogs". When a database is in archivelog mode (required for online backup) it creates a copy of a redolog every time is full. During your daily backup you should backup and delete archivelogs.
2nd problem lots of reclaimable space in the FRA.
The FRA has a logical limit which is expressed by DB_RECOVERY_FILE_DEST_SIZE. When oracle creates a file in the FRA it is registered in the controlfile as well. This means you have to delete files from the FRA always with rman. I think you know this since you deleted archivelogs with rman and not with "rm -f".
Your query showed 100% as a result of: (SPACE_USED - SPACE_RECLAIMABLE)/SPACE_LIMIT * 100
This means all files in the FRA are reclaimable. They might don't even exist physically what means they are expired. 2nd option is they exist but they are obsolete according to the "RETENTION POLICY TO REDUNDANCY 1" rule.
Solution:
I think you should adjust the backup concept a bit.
a) First of all run the following rman commands:
crosscheck archivelog all;
crosscheck backup;
delete noprompt expired archivelog all;
delete noprompt expired backup;
delete obsolete;
b) Configure the parameter DB_RECOVERY_FILE_DEST_SIZE to an appropriate value. It depends on how many databases you have on the server and how much space is used from the /home directory for other stuff. I would say choose a value between 300GB and 600GB.
c) Adjust the backup scripts:
RMAN should run the commands mentioned in a) in the daily backup job.
With this setup you never should have much reclaimable space in the FRA (except you enabled flashback functionality -- check with "select flashback_on from v$database;").
Maybe you have to adjust some of the following commands but this is a default rman script which includes self cleaning:
crosscheck archivelog all;
backup database;
backup archivelog all delete input;
crosscheck backup;
delete noprompt expired archivelog all;
delete noprompt expired backup;
delete obsolete;
This backup script cleans up expired entries from the controlfile, backs up archivelogs + deletes them and deletes old backups which are no longer needed.
To tell rman which backups are no longer needed configure the RETENTION POLICY. I prefer a recovery window than redundancy:
RMAN> CONFIGURE RETENTION POLICY TO recovery window of 2 days;

Answer 2

In which path does Oracle create those files?
- DB_CREATE_FILE_DEST (parameter for datafiles)
- DB_RECOVERY_FILE_DEST (parameter for FRA)
-- which sub directory?
I suspect that these are flashback logs. If so you should limit the flash recovery area (FRA) by setting the parameter DB_RECOVERY_FILE_DEST_SIZE to a smaller value. Oracle keeps writing flashback logs until the FRA is out of space... then it stats removing/overwriting old files.