How to load oracle table data into kafka topic? i did some research and got to know,i should use CDC tool,but all CDC tools are paid version ,can anyone suggest me how to achieve this ?
You'll find this article useful: No More Silos: How to Integrate your Databases with Apache Kafka and CDC
It details all of your options and currently-available tools. In short, you can do bulk (or query-based CDC) with the Kafka Connect JDBC Connector, or you can use a log-based CDC approach with one of several CDC tools that support Oracle as a source, including Attunity, GoldenGate, SQ Data, and IBM's IIDR.
You'll generally find that if you've paid for your database (e.g. Oracle, DB2, etc) you're going to have to pay for a log-based CDC tool. Open source CDC tools are available for open source databases. For example, Debezium is open source and works great with MongoDB, MySQL, and PostgreSQL.
You might be interested in the Debezium project, which provides open-source CDC connectors for a variety of databases. Amongst others, we provide one for Oracle DB. Note that this connector currently is based on the XStream API of Oracle, which itself requires a separate license, but we hope to add a fully free alternative soon.
Disclaimer: I'm the lead of Debezium
Please refer to kafka jdbc source connector . Below is link
https://docs.confluent.io/current/connect/connect-jdbc/docs/index.html
You don't need a Change Data Capture (CDC) tool in order to load data from Oracle Table into a Kafka topic.
You can use Kafka Confluent's JDBC Source Connector in order to load the data.
However, if you need to capture deletes and updates you must use a CDC tool for which you need to pay a licence. Confluent has certified the following CDC tools (Source connectors):
Attunity
Dbvisit
Striim
Oracle GoldenGate
As others have mentioned, CDC requires paid products. If you'd just like to try something out, Striim is available for free for the first 30 days.
https://www.striim.com/instant-download/
The 'free' options which include JDBC..but you would be introducing a significant load on your database if you actually want to use triggers to capture changes.
disclaimer: i work at striim
There's a custom Kafka source connector for Oracle database which is based on logminer here:
https://github.com/erdemcer/kafka-connect-oracle
This project is in development.
You might be interested in OpenLogReplicator. It is an open source GPL-licensed tool written completely in C++. It reads binary format of Oracle Redo logs and sends them to Kafka.
It is very fast - you can achieve low latency without much effort, since it operates fully in memory. It supports all Oracle database versions since 11.2.0.1 and requires no additional licensing.
It can work on the database host, but you can also configure it to read the redo logs using sshfs from another host - with minimal load of the database.
disclaimer: I am the author of this solution
Related
I'm trying to find a tool that will help me to build a documentation for a database.
I need to write comment on column / table and to have a summary of data flow that creates/modifies data and most important, at a conceptual level what data means. I'm also looking for a free tool
After a long search, I found out http://databasenotetaker.com/ which seems to be suitable for my purposes.
My database is Oracle.
From the documentation they say it potentially support every database but I can't connect to Oracle.
I don't know how to specify the driver in the connection window:
According to their documentation
**
Can Database Note Taker connect to other database platforms?
**
Yes and we will add support for other platforms in the future. So long as it's a relational database and there are freely available connectors, then Database Note Taker will be able to work with it. Out of the box you can connect to Microsoft Sql Server, and MySql via ODBC."
Out of the box you can't connect to Oracle, therefore you need a driver. Check with them to understand which driver they mean when they said "freely available connectors". Out-of-the-box in this context refers to the drivers that appear in the selection list.
My advice: Ask them which kind of driver do you need. If standard ODBC is ok, you must download the ODBC driver from Oracle.
https://www.oracle.com/database/technologies/instant-client/winx64-64-downloads.html
Important Check whether this product supports 64 bits or it is 32, in order to select the right ODBC driver.
In one of the use cases in my application there is a requirement to publish neo4j transaction data to oracle database in real time. I did google on it, but couldn't find a tool or plug-in which can help. Everywhere on internet talks about rdbms to neo4j sync. So I am planning to do this by manually invoking jdbc commands.
Can you please suggest something?
Had to write my own jdbc code.
TL;DR:
Is there a way to connect to an Oracle DB via JDBC without the Oracle JDBC driver? Or any other way (e.g. hibernate)?
Full story:
We have an existing Spring Boot app, which connects to an Oracle 11c database instance using OJDBC driver. Spring Boot internally uses JdbcTemplate to execute a few inserts and selects, basically 5 or 6 type of statements. Technically it is working fine.
However we are in an interesting situation, the code we built has to pass through a series of checks (we are working as subcontractors under a larger international company), and part of these checks is a license inspection. BlackDuck (https://www.blackducksoftware.com) is used to report every single library which is linked in the solution (around 280 in our case), and we had to manually add the copyright statement to each (for example: "Copyright © 2005-2019 The Apache Software Foundation. All Rights Reserved."). We added the Oracle Technology Network license to the Oracle Driver, and it was rejected by the legal team of the international company:
"Unless we are actually licensed to use an Oracle database (which you are not), we cannot allow the use and distribution of the Oracle JDBC drivers. Those drivers are licensed under the Oracle Technology network license and it obligates XXX [company name] to designate Oracle as a 3rd party beneficiary, which is not something our legal department allows. You must therefore find a compatible replacement that is licensed under open source."
Now, we are obviously trying to get around this thing (seeing as the end client who is receiving the solution has a licensed Oracle DB installed, and we can see earlier versions of the OJDBC driver in the company's Nexus repository), but I wanted to ask, is there a way to connect to an Oracle DB via JDBC without the Oracle JDBC driver? Or any other way (e.g. hibernate)? To my knowledge, Oracle DB communication protocol is not fully standard, that's why you need the Oracle drivers. Is there an open source alternative that I don't know of?
Oracle DB communication protocol is not only "not fully standard", there is simply no SQL or JDBC standard for any on-the-wire communication protocol for SQL databases, they are all proprietary.
There is no open-source alternative that I'm aware of, and there couldn't be without Oracle's help as the protocol is not publicly specified as far as I know. We solved a similar problem by having our customers with an Oracle license add the driver to their deployment of our app, that way we didn't redistribute it, and they were the ones who used it.
Yes, use someone else's driver. Progress makes one for example.
I'm not speaking on behalf of Oracle in terms of the quality of THIS driver or the legal terms concerning your decision to not use our driver. However, there are alternatives to our driver for Java applications, and this is one of them.
(I do not code on my own, to make things clear)
I am looking for a solution that would allow to replicate data between a, master, Oracle 11g DB and a new PostgreSQL DB. Those are 2 different applications but the need to exchange data in real-time. There are some trigger-based ways but there is quite a big concern that this can affect the master DB efficiency - which we can't do.
I have also come across some log-based solutions, like HVR, but the cost is way too high for 500MB of data to be replicated.
Maybe anyone of You had a similar issue and found a way to deal with it?
Any kind of tips and help will be really appreciated as I am quite short on time
Oracle Archive Logs have different format than Postgres Write Ahead Logs. Despite the general similarity in concept of Oracle Streams, SQL Log Shipping, Postgres Streaming Replication etc, transaction logs <> redo logs <> xlogs and you can't use one provider logs to roll on the other provider engine.
Moreover you can't roll logs over same DB provider different version because of difference in binary format.
Something alike logical replication you can get with Postgres Logical Decoding, Oracle GoldenGate, Heterogeneous Database Replication, AWS DMS. But none of above gives you "Log-Based replication" between different db vendors
You can use a product that specializes in change data capture based data integration. Striim, GoldenGate, Attunity allow you to do CDC from Oracle. Striim also allows you to do CDC from PostgreSQL and write to Oracle as well.
https://striim.com
https://attunity.com
I'm trying to move snapshots of data from our MongoDB into our Oracle BI data store.
From the BI team I've been asked to make the data available for ODI, but I haven't been able to find an example of that being done.
Is it possible and what do I need to implement it?
If there is a more generic way of getting MongoDB data into Oracle then I'm happy to propose that as well.
Versions
MongoDB: 2.0.1
ODI: 11.1.1.5
Oracle: 11.2g
Edit:
This is something that will be queried once a day, maybe twice but at this stage the BI report granularity is daily
In ODI, under the Topology tab and Physical Architecture sub-tab, you can see all technologies that are supported out of the box. MongoDB is not one of them. There are also no Knowledge Modules available for importing/exporting from/to MongoDB.
ODI supports implementing your own technologies and your own Knowledge Modules.
This manual will get you started with developing your won Knowledge module, and in one of the other manuals i'm sure you can find an explanation on how to implement your own technologies. (Ctrl-F for "Data integrator")
If you're lucky, you might find someone else who has already implemented it. Your best places to look would be The Oracle Technology Network Forum, or a forum related to MongoDB.
Instead of creating a direct link, you could also take an easier workaround. Export the data from the MongoDB to a format that ODI supports, and MongoDB can extract to. CSV or XML maybe? Then load the data trough ODI into the oracle database. I think... that will be the best option, unless you have to do this frequently...
Look at the blog post below for an option;
https://blogs.oracle.com/dataintegration/entry/odi_mongodb_and_a_java
Cheers
David