I am new to AWS DMS service. Plans are to migrate on-prem Oracle to Redshift. Before going into production environment, currently trying out a test Oracle RDS in AWS which is a small subset of actual database as source. So far have been successful in the bulk load and incremental migration from RDS to Redshift.
When it comes to on-prem oracle , particularly for the incremental load
1) As per document : http://docs.aws.amazon.com/dms/latest/sbs/CHAP_On-PremOracle2Aurora.Steps.ConfigureOracle.html, the on-prem needs to be enabled with supplemental logging. Plans are to use the following two commands.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;
The production database has multiple logging locations. Is there any other log settings other than the above two that I should be looking into for DMS to pick up multiple log locations?
2) In the same link given, point 4 says 'Create or configure a database account to be used by AWS DMS.'
Where should I create this user? on-prem oracle or AWS?
How do I configure DMS to use this user?
You need to read this documentation:
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html
For your second question; You need to create a user in the Oracle source database, the section 'Working with a Self-Managed Oracle Database as a Source for AWS DMS' tells you all of the grants you need to give.
For your first question, if you look at the SQL Server documentation;
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html
It specifies the limitation of; 'SQL Server backup to multiple disks isn't supported. If the backup is defined to write the database backup to multiple files over different disks, AWS DMS can't read the data and the AWS DMS task fails.'
I can't see a similar stipulation in the oracle documentation, first link, I would hazard a guess that DMS is able, in the case of oracle, to determine and cope with multiple logging locations from a configuration value inside the database.
Related
I have a couple of questions on AWS RDS service for migrating an oracle 19c server on RHEL with 3 standalone instances and databases:
Can RDS instance support multiple oracle standalone instances/databases or only one instance?
If I have an existing RDS service running in AWS, can I migrate another on-premise oracle database to the RDS database as another oracle schema?
Have not tried it yet.
RDS for Oracle limits one instance to a single database. However you can have multiple schemas in one database.
An account can have up to 40 BYOL type Oracle RDS instances, or up to 10 instances where license is included. You can also increase these limits by contacting AWS support.
See here for more details.
Each RDS instance equals one database. However, you can set up multiple RDS instances, if you want to.
For migrations, please have a look at the Database Migration Service (DMS). Regarding your schema in particular, please check the Schema Conversion Tool which is a part of the DMS.
Sorry I can’t be more specific as the questions are rather vague themselves.
We have a custom inventory system, for tracking all of our database, including Oracle v12 and v19 based databases. But as our DBA team has work with Container Databases (CDB), we need to identify, which one is a standalone or container, but we don't have any option to connection to the, due to our security policy.
Is there any way to Identify oracle Container Databases (CDB), without connecting to them?
As after reviewing Oracle official documentation: https://docs.oracle.com/database/121/ADMIN/cdb_mon.htm#ADMIN14174.
It seems that it's not possible, as the documented clearly mentioning, that first, it's necessary to connect to the database, and only then to run the following command: SELECT CDB FROM V$DATABASE;
And if the CDB column returns YES, the current database is a CDB. and NO, if the current database is a non-CDB.
My problem:
I need a data pipeline created from my organization’s Oracle DB (Oracle Cloud Infrastructure) to an AWS S3 bucket. Ideally, I would love for there to be some mechanism for oracle to push new data that has entered the database to be pushed to an S3 bucket as it is added (in whatever format).
Question:
Is this possible with Oracle native, specifically Oracle Cloud Infrastructure?
Or would is there a better solution you have seen?
Note:
I have seen AWS has the Data Sync product, this seems like it could facilitate with this problem, however I am not sure if it is suitable for this specific problem.
An S3 bucket is object storage; it can only hold complete files. You cannot open and update an existing file like you would in a normal file system, even just to add new rows. You will need to construct your whole file outside of Oracle and then push it to S3 with some other mechanism.
You may want to consider the following steps:
Export your data from Oracle Cloud into Oracle Object Storage (similar to S3) using the Oracle Cloud's integration with their object storage. (https://blogs.oracle.com/datawarehousing/the-simplest-guide-to-exporting-data-from-autonomous-database-directly-to-object-storage)
THEN:
Let the customer access the Oracle Object Store as they normally would access S3, using Oracle's Amazon S3 Compatibility API. (https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
OR:
Use an externally driven script to download the data - either from Oracle Object Store or directly from the database - to a server, then push the file up to Amazon S3. The server could be local, or hosted in either Oracle OCI or in AWS, as long as it has access to both object stores. (https://blogs.oracle.com/linux/using-rclone-to-copy-data-in-and-out-of-oracle-cloud-object-storage)
OR:
You may be able to use AWS Data Sync to move data directly from Oracle Object Storage to S3, depending on networking configuration requirements. (https://aws.amazon.com/blogs/aws/aws-datasync-adds-support-for-on-premises-object-storage/)
Is there are way to reverse engineer CLI for an existing Oracle RDS Database ?
I already have an existing Oracle RDS Database and want to create another one with exact same parameters ( except database name ).
Instead of using GUI, I want to use CLI ( "aws rds create-db-instance" ).
You could use describe-db-instances to get detailed information about your existing db instance. Then, based on the information obtained, you could create new db instance using create-db-instance.
If this is something that you will be doing often (i.e. creating same instances), it would be better to look into CloudFormation. This would allow you to provision reproducible db instances across different regions and accounts.
The question sums it up basically.
I have an external (not in AWS) MySQL v5.7 database that is 40Gb (it has one table of 16Gb). This database will be imported to RDS MySQL by another team.
I will then need to have a copy of it in my microservice but not the whole DB, just some tables (among them, the 16Gb one).
The optimal solution would be a "read replica" of just some columns of various tables.
It will need to be in constant sync with the "Master" table.
I would also like to use Aurora instead of MySQL (reason: speed & cost).
I googled for information but couldn't find anything helpful besides
AWS Data migration service.
Anyone here has experience with this? And what would you suggest?
Yes you can definetly use DMS service. You can use transformation rule and put db name and table name , which will help in selective migration.
Note :- Binlog_format has to be set true for any kind of source of RDS . They should also have backup enabled.