Is there any way we can read data from Greenplam database in Singlestore database over the network without physically copying the data from Greenplum to Singlestore.
yes you can create a PXF external table using JDBC connector to Single Store and query it via Greenplum.
The usql tools support 'copy' command for a lot of databases, including SingleStore and Postgres. you can try to copy from SingleStore to Greenplum directly.
I don't have Singlesotre but I have tried with mysql.
https://github.com/xo/usql
Related
Trying to connect database sql warehouse from postgres db but the problem I am facing is that it shows me that data source is not supported.
Please see screenshot below.
enter image description here
There is no source type postgresql in databricks. The only supported types are TEXT, AVRO, CSV, JSON, JDBC, PARQUET, ORC, DELTA, LIBSVM (https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-ddl-create-table-using.html)
Now, for your case, it is typical external data sourcing from a remote server or data location. For postgres, you can use the JDBC type. The implementation of that can be found in the relative documentation here https://docs.databricks.com/external-data/jdbc.html#language-sql
I have an instance of Oracle on PROD VPC (in AWS) and a postgreSQL on DEV VPC (in AWS). I want to copy all tables from Oracle to postgreSQL. As they are on different VPC, I am currently using the below process:
Using data pump, export the tables from Oracle PROD and import them to Oracle DEV (on DEV VPC)
Use python to extract data from Oracle DEV in CSV format and load them to postgreSQL using COPY
Is there any other efficient way to do this copy? I am trying to copy all work schemas (DDL and data) from source to destination.
Thanks in advance!
You could use VPC peering to get rid of the copy.What is VPC peering
Next, the oracle foreign data wrapper is a marvel to use for this kind of work. It makes your oracle tables accessible from the postgres database and makes copying data a piece of cake. A nice little python script can always help if there are many tables to handle.
I have to pull some data from oracle and update the data in snowflake. And ofcourse the size of the data is 5gb.
Is there any procedure to connect to oracle database from snowflake? OR
Do I need to connect them using a programming language as python?
You'll need to unload the data from Oracle and load into Snowflake, as there are no "direct connect" options I've ever heard about.
I'd use SQL*Loader to unload, push the files to AWS S3 (or your cloud vendor's storage), and issue Snowflake COPY INTO TABLE commands, it should be fairly straightforward.
There is no equivalent to Oracle database links in Snowflake. You would need an external process to move the data from Oracle to S3. Then you can configure a Snowpipe task to load from S3 into Snowflake. See Loading Continuously Using Snowpipe for more information.
I would suggest to use python programming to extract and load data from oracle to snowflake. Since your oracle table is being updated daily write python program to generate merge statement dynamically to load your incremental data from oracle to snowflake.
Snowflake supports Java script based stored procedure so you can use stored procedure to generate merge statement dynamically by passing table name as parameter and you can call it via python.
Initial load from oracle to snowflake may take time as you have 5GB data from your source system.
I have a PostgreSql database and I need to connect it to read data from oracle view and store that data in custom table
The PostgreSql database will connect to oracle everyday automatically to read the latest updates from oracle view
How to create it?
It sounds like you probably want a SQL/MED foreign data wrapper. Check out oracle_fdw. You could also use the generic odbc_fdw or jdbc_fdw wrappers via Oracle's ODBC or JDBC drivers.
Another option is DBI-Link.
Combine these with a cron job if you want to copy to a local view.
I have an Oracle view containing very large amount of data in it and I want to migrate this data in a table in Greenplum database. Is there any way I can write any query in Postgresql to fetch that Oracle view's data?
If not possible by query in Postgresql, kindly suggest me some way to access Oracle view from Linux server, so that I can create data file from that Oracle view to my Linux server and load that file via gpfdist to a Greenplum table.
NOTE: an Oracle view is from third party, I only have an access to view that data (I have all the connection info) I can access that view via SQL Developer
NOTE: Exporting data from SQL Developer to my local machine is not feasible here as the data is very large
Thanks,
Sunny
The last time I used Greenplum (3 years ago) I don't think there were any untrusted languages like plperlu, so fetching directly from Oracle from within Greenplum might not be possible. If the data has a primary key, are you able to fetch in batches, compress it, then ship it to Greenplum?
Do you have a Greenplum support contract? If so, you could also try them if you haven't already: https://sso.emc.com/sso/login.htm
I recall that gpfdist can be configured to fetch from remote servers with a bit of fiddling, so if you are able to copy out the Oracle data to disk, you could fetch it using gpfdist without any intermediary steps.