I am trying to take a level 0 incremental backup of my database on a linux server.
backup incremental level 0 database;
When I run it I get
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 06/03/2020 10:52:40
ORA-19502: write error on file "/opt/oracle/product/12.2.0/rman/full_1jv1ql2l_51_1", block number 1195008 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 1195008
Additional information: 995328
Checking the disk space returns
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 7.9G 2.2M 7.9G 1% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 30G 20G 9.2G 68% /
/dev/sda3 19G 249M 18G 2% /home
/dev/sda4 430G 255G 154G 63% /oracle
All the research I've done is saying that I'm running out of space but clearly I'm not. What else am I missing to get this up and running?
Output of
show parameter reco;
This is my RMAN configuration
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 2 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/opt/oracle/product/12.2.0/rman/full_%u_%s_%p';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 2 DAYS;
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/opt/oracle/product/12.2.0/dbs/snapcf_orcl.f'; # default
Regarding to your output of "df -h" you are trying to take a backup under file system /opt which is located under / the free space is only 9G , the /Oracle having 154G and you need to allocate channel inside your backup script to determine the location to be /Oracle
Thank you :)
regarding to my information "ORA-19502" is related to space or usually "The filesystem has insufficient disk space available" did you check that, or can you post the output of df -h so we can see it ?
Related
I'm trying to expand more space for my virtual server (Homestead) using Parallels provider on Macbook.
The default disk size is 18GB
vagrant#homestead:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 964M 0 964M 0% /dev
tmpfs 199M 7.7M 192M 4% /run
/dev/mapper/homestead--vg-root 18G 11G 5.9G 65% /
tmpfs 994M 8.0K 994M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 994M 0 994M 0% /sys/fs/cgroup
/dev/mapper/homestead--vg-mysql--master 9.8G 234M 9.1G 3% /homestead-vg/master
10.211.55.2:/Users/orange/code 234G 234G 165G 59% /home/vagrant/code
vagrant 234G 69G 165G 30% /vagrant
tmpfs 199M 0 199M 0% /run/user/1000
I don't know why the default space of VM is 64G but actually Homestead server is just 18GB
☁ homestead-7.pvm prl_disk_tool resize --info --units G --hdd harddisk1.hdd
Operation progress 100 %
Disk information:
SectorSize: 512
Size: 64G
Minimum: 64G
Minimum without resizing the last partition: 64G
Maximum: 2047G
Warning! The last partition cannot be resized because its file system is either not supported or damaged.
Make sure that the virtual HDD is not used by another process.
Warning! The disk image you specified has snapshots.
You need to delete all snapshots using the prlctl command line utility before resizing the disk.
I've so many searched but it still not solve.
How can I solve it?
(Sorry my bad English)
Based on the discussion in the reported issue on GitHub the following command will help:
lvextend -r -l +100%FREE /dev/mapper/homestead--vg-root
There's a commit that should handle the issue, but it wasn't released yet within tagged Vagrant version.
The reason for this whole dance is that Vagrant packages the VirtualBox disk as .vmdk which doesn't have the same resizability options as .vdi does.
I am trying to generate an AWR report in Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production (using Oracle Database Server Docker Image).
I am connected to the DB as sysdba.
In sqlplus I run: #$ORACLE_HOME/rdbms/admin/awrrpt.sql, after answering output format(html), starting and ending snapshot, and output file name I got an output:
Using the report name awrrpt_1_1_4.html
SP2-0606: Cannot create SPOOL file "awrrpt_1_1_4.html"
Obviously I can not find the file awrrpt_1_1_4.html on path /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/
I got enough space:
Filesystem Size Used Avail Use% Mounted on
overlay 59G 13G 43G 23% /
tmpfs 64M 0 64M 0% /dev
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda1 59G 13G 43G 23% /ORCL
shm 64M 0 64M 0% /dev/shm
tmpfs 2.0G 0 2.0G 0% /proc/acpi
tmpfs 2.0G 0 2.0G 0% /sys/firmware
For the output directory $ORACLE_HOME/rdbms/admin/ I gave chmod 777, I can also create a file and put some data into it and save it with vi editor.
What can be the problem?
why are you changing permissions of Oracle DB's own directories? Just do it in your home dir
create directory /home/oracle/myWorkspace
then execute your script using spool, and there you go.
Cause: The STORE command was unable to create the specified file. There may be insufficient disk space, too many open files, or read-only protection on the output directory.
Action: Check that there is sufficient disk space and that the protection on the directory allows file creation.
We just switched from using datapumps over to using incremental backups with RMAN. I've been having issues with the incremental backup pieces rolling forward. Here are my scripts for the backup:
BACKUP device type disk incremental level=1 tag='T4PRO' section size 1g database plus archivelog;
backup current controlfile format='&1CONTROLFILE_%T.BKP';
and the recover:
RECOVER COPY OF DATABASE WITH TAG 'T4PRO' UNTIL TIME 'SYSDATE-3';
DELETE NOPROMPT OBSOLETE DEVICE TYPE DISK;
Backup completes successfully, although the size of the backup copies seems larger than I had expected, but that's an aside for this discussion. My real issue is that we're on day 5 and are still getting this when I run the recover command:
RMAN> RECOVER COPY OF DATABASE WITH TAG 'T4PRO' UNTIL TIME 'SYSDATE-3';
Starting recover at 17-JAN-18
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=133 device type=DISK
no copy of datafile 1 found to recover
no copy of datafile 2 found to recover
no copy of datafile 3 found to recover
no copy of datafile 4 found to recover
no copy of datafile 5 found to recover
Finished recover at 17-JAN-18
RMAN> DELETE NOPROMPT OBSOLETE DEVICE TYPE DISK;
RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 7 days
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=133 device type=DISK
no obsolete backups found
The backup pieces are not rolling forward as I had expected them to two days ago. Here are the results of a SHOW ALL command if it provides any additional information:
RMAN> show all;
RMAN configuration parameters for database with db_unique_name T4PRO are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'C:\APP\ITWORKS\PRODUCT\11.2.0\DBHOME_1\D
ATABASE\SNCFT4PRO.ORA'; # default
Check if datafile copies exist:
RMAN> LIST COPY OF DATABASE;
Check if datafile copies exist with TAG 'T4PRO':
RMAN> LIST COPY OF DATABASE TAG 'T4PRO';
You must take a "image" backup using BACKUP AS COPY... before you can recover that image with an incremental backup.
I have Oracle Database 11g Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management.
OS is Linux Red Hat 2.6.18-348.12.1.el5.
Database name: dbname
Database size is approx 92 GB.
The execution of backups using RMAN it is taking about 4 hours 45 minutes to complete the task, and damage the correct execution of other processes that interact with the database.
The tables have so many DML (udpate/insert/delete) every moment every day.
I see in rman log two critics moment:
Starting backup at 04-04-2017 04:33:59
channel dbname_backup_disk1: starting incremental level 1 datafile backup set
channel dbname_backup_disk1: specifying datafile(s) in backup set
input datafile file number=00017 name=file_name ....
... "here all files"
channel dbname_backup_disk1: starting piece 1 at 04-04-2017 04:34:00
channel dbname_backup_disk1: finished piece 1 at 04-04-2017 07:22:47
piece handle=+RECOVERY/dbname/backupset/2017_04_04/nnndn1_dbname_level_0_0.2614.940394043 tag=DBNAME_LEVEL_0 comment=NONE
channel dbname_backup_disk1: backup set complete, ***elapsed time: 02:48:48***
and:
Starting backup at 04-04-2017 07:46:20
backup will be obsolete on date 04-07-2017 07:46:20
archived logs required to recover from this backup will expire when this
backup expires
channel dev1: starting compressed full datafile backup set
channel dev1: starting piece 1 at 04-04-2017 07:46:21
channel dev1: finished piece 1 at 04-04-2017 09:22:07
piece handle=/backup/oracle/backup/DBNAME_940405581_6656_1 tag=TAG20170404T074620 comment=NONE
channel dev1: backup set complete, ***elapsed time: 01:35:46***
Finished backup at 04-04-2017 09:22:07
How can I decrease the time taken?
The lines below are a portion of main script called backup.pl. It is executed every day at 02:30am local (UTC -3), from crontab.
RMAN> run {
show all;
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 1 DAYS;
configure controlfile autobackup on;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+RECOVERY/DBTARGET/%F';
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '+RECOVERY/DBTARGET/%d_%t_%s_%p';
allocate channel dbname_backup_disk1 device type disk;
REPORT OBSOLETE;
DELETE OBSOLETE;
recover copy of database with tag 'dbname_LEVEL_0' until time 'sysdate-1';
backup incremental level 1 cumulative copies=1 for recover of copy with tag 'dbname_LEVEL_0' database include current controlfile;
backup archivelog all not backed up format '+RECOVERY/DBTARGET/%d_%t_s%s_s%p';
backup current controlfile for standby;
delete archivelog until time='sysdate-3';
release channel dbtarget_backup_disk1;
sql "create pfile=''/backup/oracle/backup/ros1or01-initdbtarget1-20170404.ora'' from spfile";
}
allocate channel for maintenance type disk;
delete noprompt obsolete device type disk;
CROSSCHECK BACKUP;
CROSSCHECK DATAFILECOPY ALL;
CHANGE ARCHIVELOG ALL CROSSCHECK;
DELETE EXPIRED ARCHIVELOG ALL;
REPORT OBSOLETE;
DELETE OBSOLETE;
release channel;
run
{
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/backup/oracle/backup/%F';
allocate channel dev1 device type disk format '/backup/oracle/backup/%d_%t_%s_%p';
backup backupset completed after 'sysdate-3/24';
backup as compressed backupset datafilecopy all noduplicates KEEP UNTIL TIME 'SYSDATE+3' logs;
release channel dev1;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+RECOVERY/DBTARGET/%F';
}
quit
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name DBTARGET are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 1 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+RECOVERY/DBTARGET/%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '+RECOVERY/DBTARGET/%d_%t_%s_%p';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/cots/oracle/11.2.0/dbhome_1/dbs/snapcf_dbtarget1.f'; # default
Could you please tell me what the 92GB express?
Is it the backup size?
Is it the real data size?
Is it the size of all datafiles?
I suspect that it is the size of the backup or the real data size.
I suspect that you have very fragmented tables. This means, many empty blocks are read from disk. The output looks like very little work but in fact there is much more to do (applys to compressed backup).
Besides that your backup script is crazy! I would recommend to first simplify it and then try to improve performance.
Try the following script. It does more or less the same as yours:
RMAN>
#Do all the CONFIGURE... stuff here if you do not want to rely on the current controlfile config.
# Configure the default channel to disk and configure a format so the output goes to the filesystem not to your FRA (ASM).
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/backup/DBTARGET/%d_%t_%s_%p';
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO COMPRESSED BACKUPSET;
# Place your controlfile autobackup to disk not asm. In case of a recovery szenario you will be glad to have it in your filesystem.
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/backup/DBTARGET/%F';
BACKUP DATABASE;
CROSSCHECK ARCHIVELOG ALL;
BACKUP ARCHIVELOG ALL DELETE ALL INPUT;
CROSSCHECK BACKUP;
DELETE NOPROMPT EXPIRED BACKUP;
DELETE NOPROMPT EXPIRED ARCHIVELOG ALL;
DELETE NOPROMPT OBSOLETE;
run {
allocate channel d1 device type disk;
recover copy of database with tag 'dbname_LEVEL_0';
backup incremental level 1 for recover of copy with tag 'dbname_LEVEL_0' database;
}
I'm running spark job consuming 50GB+, my guess is that shuffle operations written to disk are causing space to run out.
I'm using the current Spark 1.6.0 EC2 script to build my cluster, close to finishing I get this error:
16/03/16 22:11:16 WARN TaskSetManager: Lost task 29948.1 in stage 3.0 (TID 185427, ip-172-31-29-236.ec2.internal): java.io.FileNotFoundException: /mnt/spark/spark-86d64093-d1e0-4f51-b5bc-e7eeffa96e82/executor-b13d39ba-0d17-428d-846a-b1b1f69c0eb6/blockmgr-12c0d9df-3654-4ff8-ba16-8ed36ca68612/29/shuffle_1_29948_0.index.3065f0c8-2511-48ab-8bf0-d0f40ab524ba (No space left on device)
I've tried using various EC2 types, but they all seem to just have the 8GB mounted for / when they start. Doing a df -h doesn't show any other storage mounted for /mnt/spark so does that mean it's only using the little bit of space left?
My df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 4.1G 3.7G 53% /
devtmpfs 30G 56K 30G 1% /dev
tmpfs 30G 0 30G 0% /dev/shm
How do you expand the disk space? I've created my own AMI for this based off the Amazon default Spark one, because of extra packages I need.