not able to figure out how to configure this processor to convert the incoming JSON twitter data to Avro to Put that data into a hive table
plz help
Thank you in advance.
#RishiPandey
ConvertJsonToAvro is the processor you need.
Related
I am creating a data pipeline with Apache Ni-Fi to copy data from a remote MySQL database into InfluxDB.
I use QueryDatabaseTable processor to extract the data from the MySQL database, then I use UpdateRecord to do some data transformation and I would like to use PutInfluxDB to insert the time series into my local Influx instance in Linux.
The data coming from the QueryDatabaseTable processor uses AVRO scheme and I need to convert it into line protocol by configuring which are the tags and which are the measurement values.
However, I do not find any processor that allows doing this conversion.
Any hints?
Thanks,
Bernardo
There is no built-in processor for InfluxDB Line Protocol conversions - you could write a ScriptedRecordWriter if you wanted to do it yourself, however there is a project that already implements a Line Protocol reader for NiFi here by InfluxData that seems to be active & up-to-date.
See the documentation for adding it into NiFi here
I am trying to use the NiFi MergeContent to merge multiple parquet files together.
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.MergeContent/index.html
The parquet files I have obtained via the ConvertRecord operator.
Please let me know if you have an example on how to solve this.
If you are working with Records, as you have used ConvertRecord, consider using MergeRecord instead of MergeContent.
I'm trying to create a Spark Streaming application using fileStream(). The document guide specified:
streamingContext.fileStream[KeyClass, ValueClass, InputFormatClass](dataDirectory)
I need to pass KeyClass, ValueClass, InputFormatClass. My main question is what can I use for these parameters for Avro formatted data?
Note: that my Avro data already have schema embedded in the data.
I found a related question here. However theirs input is in Parquet format.
I am using 2.5 HDP. I got issue in ConvertJSONToSQL processor I try to convert bulk(1GB) size of Avro file into SQL format,for that first I convert Avro format into JSON format (by using ConvertAvroToJSON processor) after that I convert JSON format into SQL format (by using ConvertJSONToSQL processor) that time I got issue that I mention below.
First you try to split bulk size of Avro format file into some small size by using SplitAvro processor which is present in Nifi and then you convert it into json
Configure processor of SplitAvro processor
For more information follow this link
Just started working with Apache Nifi. I am trying to fetch data from oracle and place it in HDFS then build an external hive table on top of it. The problem is ExecuteSQL processor returns data in avro format. Is there anyway I can get this data in a readable format?
apache nifi also has an 'ConvertAvroToJSON' processor. That might help you get it into a readable format. We also really need to just knock out the ability for our content viewer to nicely render avro data which would help as well.
Thanks
joe