Oracle Backup with RMAN take a long time - oracle

I have Oracle Database 11g Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management.
OS is Linux Red Hat 2.6.18-348.12.1.el5.
Database name: dbname
Database size is approx 92 GB.
The execution of backups using RMAN it is taking about 4 hours 45 minutes to complete the task, and damage the correct execution of other processes that interact with the database.
The tables have so many DML (udpate/insert/delete) every moment every day.
I see in rman log two critics moment:
Starting backup at 04-04-2017 04:33:59
channel dbname_backup_disk1: starting incremental level 1 datafile backup set
channel dbname_backup_disk1: specifying datafile(s) in backup set
input datafile file number=00017 name=file_name ....
... "here all files"
channel dbname_backup_disk1: starting piece 1 at 04-04-2017 04:34:00
channel dbname_backup_disk1: finished piece 1 at 04-04-2017 07:22:47
piece handle=+RECOVERY/dbname/backupset/2017_04_04/nnndn1_dbname_level_0_0.2614.940394043 tag=DBNAME_LEVEL_0 comment=NONE
channel dbname_backup_disk1: backup set complete, ***elapsed time: 02:48:48***
and:
Starting backup at 04-04-2017 07:46:20
backup will be obsolete on date 04-07-2017 07:46:20
archived logs required to recover from this backup will expire when this
backup expires
channel dev1: starting compressed full datafile backup set
channel dev1: starting piece 1 at 04-04-2017 07:46:21
channel dev1: finished piece 1 at 04-04-2017 09:22:07
piece handle=/backup/oracle/backup/DBNAME_940405581_6656_1 tag=TAG20170404T074620 comment=NONE
channel dev1: backup set complete, ***elapsed time: 01:35:46***
Finished backup at 04-04-2017 09:22:07
How can I decrease the time taken?
The lines below are a portion of main script called backup.pl. It is executed every day at 02:30am local (UTC -3), from crontab.
RMAN> run {
show all;
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 1 DAYS;
configure controlfile autobackup on;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+RECOVERY/DBTARGET/%F';
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '+RECOVERY/DBTARGET/%d_%t_%s_%p';
allocate channel dbname_backup_disk1 device type disk;
REPORT OBSOLETE;
DELETE OBSOLETE;
recover copy of database with tag 'dbname_LEVEL_0' until time 'sysdate-1';
backup incremental level 1 cumulative copies=1 for recover of copy with tag 'dbname_LEVEL_0' database include current controlfile;
backup archivelog all not backed up format '+RECOVERY/DBTARGET/%d_%t_s%s_s%p';
backup current controlfile for standby;
delete archivelog until time='sysdate-3';
release channel dbtarget_backup_disk1;
sql "create pfile=''/backup/oracle/backup/ros1or01-initdbtarget1-20170404.ora'' from spfile";
}
allocate channel for maintenance type disk;
delete noprompt obsolete device type disk;
CROSSCHECK BACKUP;
CROSSCHECK DATAFILECOPY ALL;
CHANGE ARCHIVELOG ALL CROSSCHECK;
DELETE EXPIRED ARCHIVELOG ALL;
REPORT OBSOLETE;
DELETE OBSOLETE;
release channel;
run
{
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/backup/oracle/backup/%F';
allocate channel dev1 device type disk format '/backup/oracle/backup/%d_%t_%s_%p';
backup backupset completed after 'sysdate-3/24';
backup as compressed backupset datafilecopy all noduplicates KEEP UNTIL TIME 'SYSDATE+3' logs;
release channel dev1;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+RECOVERY/DBTARGET/%F';
}
quit
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name DBTARGET are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 1 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '+RECOVERY/DBTARGET/%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '+RECOVERY/DBTARGET/%d_%t_%s_%p';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/cots/oracle/11.2.0/dbhome_1/dbs/snapcf_dbtarget1.f'; # default

Could you please tell me what the 92GB express?
Is it the backup size?
Is it the real data size?
Is it the size of all datafiles?
I suspect that it is the size of the backup or the real data size.
I suspect that you have very fragmented tables. This means, many empty blocks are read from disk. The output looks like very little work but in fact there is much more to do (applys to compressed backup).
Besides that your backup script is crazy! I would recommend to first simplify it and then try to improve performance.
Try the following script. It does more or less the same as yours:
RMAN>
#Do all the CONFIGURE... stuff here if you do not want to rely on the current controlfile config.
# Configure the default channel to disk and configure a format so the output goes to the filesystem not to your FRA (ASM).
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/backup/DBTARGET/%d_%t_%s_%p';
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO COMPRESSED BACKUPSET;
# Place your controlfile autobackup to disk not asm. In case of a recovery szenario you will be glad to have it in your filesystem.
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/backup/DBTARGET/%F';
BACKUP DATABASE;
CROSSCHECK ARCHIVELOG ALL;
BACKUP ARCHIVELOG ALL DELETE ALL INPUT;
CROSSCHECK BACKUP;
DELETE NOPROMPT EXPIRED BACKUP;
DELETE NOPROMPT EXPIRED ARCHIVELOG ALL;
DELETE NOPROMPT OBSOLETE;
run {
allocate channel d1 device type disk;
recover copy of database with tag 'dbname_LEVEL_0';
backup incremental level 1 for recover of copy with tag 'dbname_LEVEL_0' database;
}

Related

RMAN backup is failing with write error on file

I am trying to take a level 0 incremental backup of my database on a linux server.
backup incremental level 0 database;
When I run it I get
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 06/03/2020 10:52:40
ORA-19502: write error on file "/opt/oracle/product/12.2.0/rman/full_1jv1ql2l_51_1", block number 1195008 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 1195008
Additional information: 995328
Checking the disk space returns
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 7.9G 2.2M 7.9G 1% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 30G 20G 9.2G 68% /
/dev/sda3 19G 249M 18G 2% /home
/dev/sda4 430G 255G 154G 63% /oracle
All the research I've done is saying that I'm running out of space but clearly I'm not. What else am I missing to get this up and running?
Output of
show parameter reco;
This is my RMAN configuration
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name ORCL are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 2 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/opt/oracle/product/12.2.0/rman/full_%u_%s_%p';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 2 DAYS;
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/opt/oracle/product/12.2.0/dbs/snapcf_orcl.f'; # default
Regarding to your output of "df -h" you are trying to take a backup under file system /opt which is located under / the free space is only 9G , the /Oracle having 154G and you need to allocate channel inside your backup script to determine the location to be /Oracle
Thank you :)
regarding to my information "ORA-19502" is related to space or usually "The filesystem has insufficient disk space available" did you check that, or can you post the output of df -h so we can see it ?

Installing ceph using kolla-ansible for all-in-one setup

I am trying to deploy the all-in-one configuration using kolla-ansible with ceph enabled
enable_ceph: "yes"
#enable_ceph_mds: "no"
enable_ceph_rgw: "yes"
#enable_ceph_nfs: "no"
enable_ceph_dashboard: "{{ enable_ceph | bool }}"
#enable_chrony: "yes"
enable_cinder: "yes"
enable_cinder_backup: "yes"
glance_backend_ceph: "yes"
gnocchi_backend_storage: "{{ 'ceph' if enable_ceph|bool else 'file' }}"
cinder_backend_ceph: "{{ enable_ceph }}"
cinder_backup_driver: "ceph"
nova_backend_ceph: "{{ enable_ceph }}"
And, my setup consists of a Virtual Box VM with Ubuntu 18.04.4 desktop version with 2 CPU cores, 30 GB Disk (single disk), 2GB RAM, the partitioning type is msdos.
ansible version==2.9.7
kolla-ansible version==9.1.0
In order to install ceph OSD using kolla-ansible i read that a partition should have the name KOLLA_CEPH_OSD_BOOTSTRAP_BS.
Hence, i created root partition with 20 GB i.e /dev/sda1 and then an extended partition /dev/sda2 for the rest 20GB and followed by two logical partitions (/dev/sda5 and /dev/sda6) each of 10GB for OSD. But in msdos type partitioning there is no feature to allocate name to partitions.
So my questions are:
How do I go about labeling the partition in case of msdos type partition in order for kolla-ansible to recognize that /dev/sda5 and /dev/sda6 is designated for Ceph-OSD ?
Is it a must to have a separate storage drive than the one containing Operating System for Ceph OSD (i know its not recommended to have all in single disk) ?
How do I have to provision my single drive HD space in order to install Ceph-OSD using kolla-ansible?
P.S: I also tried to install ceph using kolla-ansible using an OpenStack VM (4 CPU cores, 80GB disk space - single drive, as i didn"t install Cinder in my OpenStack infra.) and Ubuntu 18.04.4 Cloud image, which has GPT partition type and supports naming partitions. The partitions were as follows:
/dev/vda1 for root partition
/dev/vda2 for ceph OSD
/dev/vda3 for ceph OSD
But the drawback was that, kolla-ansible wiped out complete disk and resulted in failure in installation.
Any help is highly appreciated. Thanks a lot in advance.
I also had installed an Kolla-Ansible single-node all-in-one with Ceph as storage backend, so I had the same problem.
Yes, the bluestore installation of the ceph doesn't work with a single partition. I had also tried different ways of labeling, but for me it only worked with a whole disk, instead of a partition. So for your virtual setup create a whole new disk, for example /dev/vdb.
For labeling, I used the following as bash-script:
#!/bin/bash
DEV="/dev/vdb"
(
echo g # create GPT partition table
echo n # new partiton
echo # partition number (automatic)
echo # start sector (automatic)
echo +10G # end sector (use 10G size)
echo w # write changes
) | fdisk $DEV
parted $DEV -- name 1 KOLLA_CEPH_OSD_BOOTSTRAP_BS
Be aware, that DEV at the beginning is correctly set for your setup. This creates a new partiton table and one partition on the new disc with 10GB size. The kolla-ansible deploy-run register the label and wipe the whole disc, so the size-value has nothing to say and is only for the temporary partition on the disc.
One single disc is enough for the Ceph-OSD in kolla-ansible. You don't need a second OSD. For this, add the following config-file in your kolla-ansible setup in this path, when you used the default kolla installation path: /etc/kolla/config/ceph.conf with the content:
[global]
osd pool default size = 1
osd pool default min size = 1
This is to make sure, that there is only one OSD requested by kolla-ansible. If your kolla-directory with the globals.yml is not under /etc/kolla/, you have to change the path of the config-file as well.
Solution for setup with one single disc with multiple partitions is to switch the storage-type of the ceph-storage in the kolla-ansible setup from bluestore to the older filestore OSD type. This also requires different partition-labels like written here: https://docs.openstack.org/kolla-ansible/rocky/reference/ceph-guide.html#using-an-external-journal-drive .
With filestore you need one parition with the label KOLLA_CEPH_OSD_BOOTSTRAP_FOO and a small journal-partition with label KOLLA_CEPH_OSD_BOOTSTRAP_FOO_J (the FOO in the name is really required...). To be able to switch your kolla-installation to filestore OSD type, edit all-in-one file's [storage] section by adding ceph_osd_store_type=filestore next to the host as follows, to override the default bluestore.
[storage]
localhost ansible_connection=local ceph_osd_store_type=filestore
The above method has been tested with ansible==2.9.7 and kolla-ansible==9.1.0 and OpenStack Train release and prior releases.

incremental RMAN backups not rolling forward

We just switched from using datapumps over to using incremental backups with RMAN. I've been having issues with the incremental backup pieces rolling forward. Here are my scripts for the backup:
BACKUP device type disk incremental level=1 tag='T4PRO' section size 1g database plus archivelog;
backup current controlfile format='&1CONTROLFILE_%T.BKP';
and the recover:
RECOVER COPY OF DATABASE WITH TAG 'T4PRO' UNTIL TIME 'SYSDATE-3';
DELETE NOPROMPT OBSOLETE DEVICE TYPE DISK;
Backup completes successfully, although the size of the backup copies seems larger than I had expected, but that's an aside for this discussion. My real issue is that we're on day 5 and are still getting this when I run the recover command:
RMAN> RECOVER COPY OF DATABASE WITH TAG 'T4PRO' UNTIL TIME 'SYSDATE-3';
Starting recover at 17-JAN-18
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=133 device type=DISK
no copy of datafile 1 found to recover
no copy of datafile 2 found to recover
no copy of datafile 3 found to recover
no copy of datafile 4 found to recover
no copy of datafile 5 found to recover
Finished recover at 17-JAN-18
RMAN> DELETE NOPROMPT OBSOLETE DEVICE TYPE DISK;
RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 7 days
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=133 device type=DISK
no obsolete backups found
The backup pieces are not rolling forward as I had expected them to two days ago. Here are the results of a SHOW ALL command if it provides any additional information:
RMAN> show all;
RMAN configuration parameters for database with db_unique_name T4PRO are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'C:\APP\ITWORKS\PRODUCT\11.2.0\DBHOME_1\D
ATABASE\SNCFT4PRO.ORA'; # default
Check if datafile copies exist:
RMAN> LIST COPY OF DATABASE;
Check if datafile copies exist with TAG 'T4PRO':
RMAN> LIST COPY OF DATABASE TAG 'T4PRO';
You must take a "image" backup using BACKUP AS COPY... before you can recover that image with an incremental backup.

oracle database 12cR1: Tablespace point in time recovery is failing on the export metadata level

I have a problem using TSPITR to restore a single tablespace.
the error stack showed the following:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 06/15/2017 20:09:35
RMAN-06962: Error received during export of metadata
RMAN-06960: EXPDP> ORA-31626: job does not exist
ORA-31637: cannot create job TSPITR_EXP_yhqe_Erry for user SYS
ORA-39062: error creating master process DM00
ORA-31613: Master process DM00 failed during startup.
I'm using the fully automated TSPITR method and it is failing on the level when rman initiate export metadata after full recovery of the auxiliary db and creating the datapump directory. please help.
we solved the issue. we increased the number of processes of the auxiliary instance and then the export started and the tablespace is successfully restored.

Change HBase WAL location

I'm planning to use HBase with gs:// scheme (google buckets), but gs:// fs can't work with WAL files.
Cause:
java.io.IOException: cannot get log writer
Caused by: java.io.IOException: createNonRecursive unsupported for this filesystem class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
As I found it should way to store WAL files separate from a HBase root dir, eg: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/images/hbase_s3.png
So, my question is - how to separate data files and WAL files. To store data in gs:// and WALs in hdfs:// ... Unfortunately still can't find it by myself ...
Many thanks in advance
Current stable version doesn't support storing WALs in a different location outside of hbase.rootdir.
It's planned in future releases, ref https://issues.apache.org/jira/browse/HBASE-17437

Resources