There are several similar Oracle technologies - Oracle Streams, Oracle Change Data Capture and Database Change Notification.
What are the differences between these?
Are they related to Oracle Advanced Queueing?
Oracle CDC is all about capturing changes in DB tables and the changes are stored in special Oracle tables. There are two modes of CDC operation: asynchronous (based on Java) or synchronous (based on DB triggers, more performance overhead).
Oracle Streams sits on top of Oracle CDC and it's a full transport mechanism (over e.g. HTTP) for data synchronization between 2 servers. It's based on Oracle Advanced Queues technology and it's designed for high performance and reliability.
Both Oracle CDC and Streams are generally used for data synchronization between Oracle DB servers... With Oracle CDC, you don't have to use Oracle Streams for, e.g. you could write your own data export routines which create flat files for the purpose of synchronization between 2 DB servers, whereas with Streams you must have a network link between the 2 servers.
Database Change Notification is something else again, it's not used for server-to-server synch but instead more for server notification of resultset changes to clients, mostly in the context of data caches on the client side.
I would add that for synchronizing between systems, you can achieve asynchronous mode by combining Streams and the CDC publishing mechanism. If you choose not to use Streams for this purpose, you would end up using a synchronous mode (I think it's via triggers) putting a little extra overhead on each transaction.
Related
Is there a tool to synchronise an Oracle and a Grakn database in real time? I couldn't find any information online so any help would be greatly appreciated.
I know Grakn offers GRPC through its client drivers which I'm assuming is the way to go to push things into Grakn.
And I'm aware there are triggers in the Oracle tables, but not sure if will slow down the application layer on the Oracle DB. How would this impact the performance of the BD?
Thanks!
There aren't any such tools that you can use OOTB. If you need to keep the Oracle system running then I think you've got two options:
Intercept the data as it comes in and dispatch to both Oracle and Grakn.
Find a triggering mechanism, and as you outline, read from Oracle and push the data to Grakn through its driver(s).
GRPC is built in to Grakn's drivers, and is the only way to communicate with a Grakn server. These drivers are optimised as such!
I'm actually working in a situation where a .NET stack is managing an Oracle Database. Now, because the legacy code is consistently based on PL SQL stored procedures that deal with the majority of the work, the correct choice of driver to connect to the database is of primary importance.
For this reason, knowing that Oracle provides a large number of driver for the most known programming languages, I was trying to find a documented benchmark (even with all the problem and the influence of the context in which the tests are made) that could compare the different Oracle drivers for the different programming languages, just to support the hypothesis that the best choice in terms of performance for an isolated test microservice would be to use the Java driver in combination with Scala (Java is now property of Oracle now, after all).
Are there any on the internet that could support (or not) this hypothesis?
EDIT
I didn’t provide all the information. What I’m trying to achieve is develop a series of microservices focused on fetching data from the database and convert them into json to be consumed by the front end. .NET driver behaves seamlessly until the numbers become really overwhelming (> 1000 rows).
That’s why I was wondering if it could make any sense using JDBC to increase performance, having heard that, for instance, .NET driver for SQL server, both made by the same Company, performs 5 times better than the oracle one when it comes to gathering data from a cursor.
Your choice of drives may not give you the milage if most of the work is in PL/SQL or stored procedures. Having said that, jdbc is a good choice. However do not fight if developers are more familiar with other drivers like Oracle ODBC, oracle provider for .NET, ADO,etc.. all protocols have a Oracle drive to access Oracle DB.
You don’t have to convert to json. Oracle DB can convert it. Your network latency is more impacted by how big the pipe is ,array size,and packet size than protocols.
Given a pre-production oracle database and a production oracle database and if around 300K records need to be transferred from the former to the latter, would using a messaging system such as an ESB/JMS/TIBCO be a good option?
I don't know Oracle, but if I was trying to asynchronously replicate data with SQL Server, I would use their own internal tools to accomplish it. I would imagine Oracle has similar tools to run jobs to copy between two Oracle databases.
However, I do have quite a bit of experience using an ESB (Mule) with ActiveMQ to replicate data across database technologies. Specifically I've done SQL Server->Mongo and MySQL->Mongo with Mule and ActiveMQ.
So far I've found Mule to be a wonderful solution - especially coupled with ActiveMQ. I've been able to replicate about 400k Wordpress blog posts (from MySQL) to Mongo in about 20 minutes. To transfer 100k articles from a CMS system we were able to get it done in about 30 minutes.
I figured I'd weigh in because you mentioned and ESB and messaging. I would go that route if the integration points are heterogenous. If you do go down that route, Mule is awesome.
If you are trying to move data from an old database to a new one instead of doing it asynchronously, possibly a simpler method would be sql injection. Assuming your old database allows you to "export" your database, when you export it you will download a sql file. Then you can just open that sql file in a program like notepad and copy-paste that code in the sql executor at your new database and it will re-create all your tables and populate them with the old data.
Actually using the database tools will be the recommended method for replicating data between databases.
When using messaging, one does not get the guarantee that the data will be transferred in the same sequence as it was sent and honor relationships between tables, potentially resulting in replication errors, unless one builds up some mechanism on the JMS receiver side to maintain the sequence. But that looks rather like an overhead.
I have multiple data sources inserting and updating to an oracle10g database. I also have multiple clients that would like to know when an insert or update has occured. I would like to know if oracle has some kind of native ability for my clients to realize that an insert or update has occured? I feel like the answer is no, but wanted to ask just in case.
Thanks!
edit: ideally push notification
There are a few different mechanisms depending on your goals.
You can use Oracle Asynchronous Queues (AQ) to publish the changes and then have your applications subscribe to the changes either via OCI or through one of the higher level Oracle AQ APIs (such as having AQ feed a JMS queue for a J2EE application).
If you are writing an OCI application, Oracle provides support for continuous query notification which is another method of pushing data to the client.
Finally, your applications could use Oracle Change Data Capture (CDC) to subscribe to changes. This last option would be more of a poll option (i.e. your application would ask every few minutes if there had been any interesting changes). The other two would be push options.
We've used DBMS_ALERT quite successfully; you can use a trigger to send an alert to a client registered to receive notifications. Effectiveness probably depends somewhat on the number and types of changes you're looking for.
http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14258/d_alert.htm
We have an application provided by a third party which takes a stream of market data (provided by said third party), and writes it into a JDBC compatible database.
The only configuration parameters it has are the JDBC connection string, plus settings allowing us to pick what pieces of data we'd like to be stored in this database.
This is very good for static data, but we'd like to feed this data into our internal ActiveMQ messaging fabric (in addition to writing it into the DB).
The database updates are triggered by pushes of market data to us. I'd like to have this application write the data directly to a set of MQ topics by implementing some kind of jdbc "facade" that would re-route the data directly into MQ.
What I don't want to do is poll the database for new information - as I want to keep the same fluidity of the data (e.g. fast moving stocks will generate a lot more data than slow moving - and we'd want to retain this).
Advice and pointers are very much welcome!
Camel is the answer, but potentially only if you're ok with polling the database. It's great for integration issues like this. If there was some other trigger that you could work with, you could use that to cause the database to be read.