I have an instance of Oracle on PROD VPC (in AWS) and a postgreSQL on DEV VPC (in AWS). I want to copy all tables from Oracle to postgreSQL. As they are on different VPC, I am currently using the below process:
Using data pump, export the tables from Oracle PROD and import them to Oracle DEV (on DEV VPC)
Use python to extract data from Oracle DEV in CSV format and load them to postgreSQL using COPY
Is there any other efficient way to do this copy? I am trying to copy all work schemas (DDL and data) from source to destination.
Thanks in advance!
You could use VPC peering to get rid of the copy.What is VPC peering
Next, the oracle foreign data wrapper is a marvel to use for this kind of work. It makes your oracle tables accessible from the postgres database and makes copying data a piece of cake. A nice little python script can always help if there are many tables to handle.
Related
I am wandering around and didn't get an answer to this question.
is there any way to import Oracle .dmp file stored on a local machine to RDS Oracle?
If Yes how to do it?
Else why it's not possible to do so as other databases gave the flexibility to do these kinds of imports through more than one way.
You can't do that. When you import data with Oracle Data Pump, you must transfer the dump file that contains the data from the source database to the target database**. You can transfer the dump file using an Amazon S3 bucket or by using a database link between the two databases.
If your local machine contains a database and you have a network connection between your on-premises database and your Oracle RDS , then you can use NETWORK_LINK, although I don't recommend it. It is much better to tranfer the file using S3 bucket.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html
For a POC, we need 10 GB of data to be available in Oracle RDS instance. Any test data is ok ( like TPC for benchmarking ) , for this requirement is there any specific way to create database and pump the sample data ?
Assuming you have opened the firewall port 1521 to the RDS server, you should be able to use sqlplus, sqlci, sqlldr to connect to the RDS instance as follows:
USER/PASSWORD#//hostName:port/SID
e.g.
sqlplus scott/tigerD#//myhost.this.that.amazon.com:1521/THE_SID
Once you prove this works, use this in the tools to squirt data into the RDS instance.
E.g. .sql INSERT scripts, sqlldr scripts etc.
We have RDS (Oracle) instance, I need to export specific Schema into dumpfile. Export works and copies dump file into DATA_PUMP_DIR. Issue is that RDS do not have file directory access.
I need exported DMP file either on S3 or copy to another EC2 instance.
The article: LINK talks about copying data dump file between two RDS instances but not to S3 or EC2.
Third option. I am using it.
Take a look at alexandria-plsql-utils project, and especially look at: amazon_aws_auth_pkg, amazon_aws_s3_pkg and ftp_util_pkg packages.
Install required packages and dependencies.
Do your dump, then with such example code below you can copy file from Amazon RDS Oracle into S3 bucket.
declare
b_blob blob;
begin
b_blob := file_util_pkg.get_blob_from_file ('DATA_PUMP_DIR', 'my_dump.dmp');
amazon_aws_auth_pkg.init ('aws_key_id','aws_secret', p_gmt_offset => 0);
amazon_aws_s3_pkg.new_object('my-bucket-name', 'my_dump.dmp', b_blob, 'application/octet-stream');
end;
`
There are several ways to solve this problem.
First option.
Install a free database version of the Oracle XE version on EC2
instance(It is very easy and fast)
Export a schema from the RDS instance to DATA_PUMP_DIR
directory. Use DBMS_DATAPUMP package or run expdp user/pass#rds on EC2 to create a dump file.
Create database link on RDS instance between RDS DB and Oracle XE
DB.
If you are creating a database link between two DB instances inside
the same VPC or peered VPCs the two DB instances should have a valid
route between them.
Adjusting Database Links for Use with DB Instances in a VPC
Copy the dump files from RDS instance to Oracle XE DB on EC2 uses
the DBMS_FILE_TRANSFER.PUT_FILE via database link
Copy files from the DATA_PUMP_DIR directory Oracle XE on EC2 instance to the S3.
Second option.
Use the obsolete utility exp to export. It has restrictions on the export of certain types of data and is slower.
Run exp user/password#rds on EC2 instance.
Copy files from the directory Oracle XE on EC2 instance to the S3
Original export is desupported for general use as of Oracle Database
11g. The only supported use of Original Export in 11g is backward
migration of XMLType data to a database version 10g release 2 (10.2)
or earlier. Therefore, Oracle recommends that you use the new Data
Pump Export and Import utilities, except in the following situations
which require Original Export and Import:
Original Export and Import
It's now possible to directly access a S3 bucket from a Oracle database. Please have a look at the following documentation: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/oracle-s3-integration.html
And here the official news that this is supported: https://aws.amazon.com/about-aws/whats-new/2019/02/Amazon-RDS-for-Oracle-Now-Supports-Amazon-S3-Integration/?nc1=h_ls
It seems that the first post was a little bit to early to get this news. But anyway this post lists further good solutions like the database link.
I need to migrate an existing application's database into oracle RDS database in Amazon Web services.
I have the dump file which is residing on an EC2 instance. The dump has not been taken by me.Also I would like to know how can I take the dump so that it can be imported successfully. The EC2 instance has an oracle regular client.
I have set up the oracle RDS instance in AWS and I am able to connect to the server.
I would like to know how can I import the database dump on RDS.
I am using this command :
imp rdsuser#oracledb FILE=fulldb.dmp TOUSER=rdsuser FROMUSER=SYSTEM log=test.log buffer=100000
Any lead is appreciated.
Also I would like to know what is the best method to import an existing database:
1. to take dump.
2. Or to take the clone of all files of database ( that will require the downtime in the server).
Best strategy is to take dump then import it into RDS . If your DB size is too big then contact AWS guys for help .
I have an Oracle view containing very large amount of data in it and I want to migrate this data in a table in Greenplum database. Is there any way I can write any query in Postgresql to fetch that Oracle view's data?
If not possible by query in Postgresql, kindly suggest me some way to access Oracle view from Linux server, so that I can create data file from that Oracle view to my Linux server and load that file via gpfdist to a Greenplum table.
NOTE: an Oracle view is from third party, I only have an access to view that data (I have all the connection info) I can access that view via SQL Developer
NOTE: Exporting data from SQL Developer to my local machine is not feasible here as the data is very large
Thanks,
Sunny
The last time I used Greenplum (3 years ago) I don't think there were any untrusted languages like plperlu, so fetching directly from Oracle from within Greenplum might not be possible. If the data has a primary key, are you able to fetch in batches, compress it, then ship it to Greenplum?
Do you have a Greenplum support contract? If so, you could also try them if you haven't already: https://sso.emc.com/sso/login.htm
I recall that gpfdist can be configured to fetch from remote servers with a bit of fiddling, so if you are able to copy out the Oracle data to disk, you could fetch it using gpfdist without any intermediary steps.