Is there any tutorial or guide to migrate data from MSSQL to MySQL or any other database?
I am trying to migrate data from MSSQL to MySQL using apache NiFi with some modification in data + schema.
Check this SO post for my answer for migrating between two different RDBMS in Apache NiFi. You may need some extra processors such as a JoltTransformJSON after the ConvertAvroToJSON processor, to make the schema/data changes.
Related
I am trying to export data from Kafka to Oracle db. I've searched related questions and web but could not understand that we need a platform (confluent etc.. ) or not. I'd been read the link below but it's not clear enough.
https://docs.confluent.io/3.2.2/connect/connect-jdbc/docs/sink_connector.html
So, what we actually need to export data without 3rd party platform? Thanks in advance.
It's not clear what you mean by "third-party" here
What you linked to is Kafka Connect, which is Apache 2.0 Licensed and open source.
Kafka Connect is a plugin ecosystem, you install connectors individually, written by anyone, or write your own, just like any other Java dependency (i.e. a third-party)
The JDBC connector just happens to be maintained by Confluent. and you can configure the Confluent Hub CLI
to install within any Kafka Connect distribution (or use Kafka Connect Docker images from Confluent)
Alternatively, you use Apache Spark, Flink, Nifi, and many other Kafka Consumer libraries to read data and then start an Oracle transaction per record batch
Or you can explore non-JVM kafka libraries as well and use a language you're more familiar with doing Oracle operations with
Is there any way to fetch incremental data from an Oracle database using user-defined query using JDBC?
We are ok to use Spark, Kafka or plain JDBC.
The only thing it should be able to support heavy load.
You've not specified the destination. If it's a Kafka topic then using Apache Kafka makes sense to do the extract too, using Kafka Connect.
In which case, you can use the Kafka Connect JDBC connector to do this. See here for the specifics on using incremental mode with a custom query.
++ EDIT ++
If your final target is BigQuery then you can use Kafka Connect for that too with the appropriate BigQuery connector. You can see an example of it in action here.
I have few tables in Hive and my goal is to create a view over them and then publish it over a topic in Kafka through Apache NiFi.
What are the options to get it done?
I am planning to do it through Nifi .
I'm sure Nifi would work,
see PutHiveStreaming processor, but sounds like a lot of effort.
Kafka Connect HDFS is able to consume Kafka data and automatically register a Hive table for you.
And if I misunderstood that, and you're trying to query Hive and publish it into a Kafka topic, then sure - Nifi is perfectly capable of that
Use SelectHiveQL and PublishKafka, however Kafka Connect JDBC Source should be able to query Hive and write to Kafka as well
I have the following setup :
Mysql RDBMS server
Elastic Search Server
My requirement is to copy data periodically from MYSQL RDBMS and update Elastic server with it.Currently i am following the approach below :
A Batch Job which reads all data from MYSQL using Spring Data Jpa
It then pushes all data to elastic server using spring data elastic
This approach is very cumbersome and not efficient.Is there a way where i can read only the updated values using spring data and update the index accordingly in elastic.
Using jdbc-river etc is not an option for me as the application uses Spring data elastic to get data and search over elastic search,with jdbc-river it will not be able to function properly i think.
I want to expose Oracle database data with an Odata endpoint.I tried using JayData server on node.js but it currently supports only mongo db but not oracle.So before I start trying connecting Oracle with Apache Olingo I would like to know if someone has already been this path.Please advise
I have successfully used Oracle with Olingo from java JPA and EclipseLink