I Have an HTTP response as a list of JSON records [{},{},{}] within those records we have some dates in UNIX format "1651030211980" but we need them in timestamp format.
[ {
"username_user":"json",
"surname":"file",
"creationDate":"1651030211980",
"modificationDate":"1651030211980"
},
{},
{}
]
The result needs to look like that:
[ {
"username":"json",
"surname":"file",
"creationDate":"YYYY-MM-DD HH:MM:SS",
"modificationDate":"YYYY-MM-DD HH:MM:SS"
},
{},
{}
]
You could use ConvertRecord (JsonTreeReader, JsonRecordSetWriter) where you could redefine fields
here is not bad example with json-to-csv but idea is the same: https://rihab-feki.medium.com/converting-json-to-csv-with-apache-nifi-a9899ca3f24b
as alternative you could use ExecuteGroovyScript
import groovy.json.*
def ff = session.get()
if(!ff)return
def formatDate={ s-> new Date(s as Long).format('yyyy-MM-dd HH:mm:ss') }
ff.write{rawIn,rawOut->
def data = new JsonSlurper().parse(rawIn)
data.each{obj->
if(obj.creationDate) obj.creationDate = formatDate(obj.creationDate)
if(obj.modificationDate) obj.modificationDate = formatDate(obj.modificationDate)
}
rawOut.withWriter{ w-> w << new JsonBuilder(data).toPrettyString() }
}
REL_SUCCESS << ff
Related
I have json with dozens of fields, how can I easily convert it to nifi attribute?
I used EvaluateJsonPath but it is necessary to enter the values one by one.
I will use these attributes in Phoenix DB, When I use ConvertJsontoSQL it doesn't work...
Can you help with this issue?
JoltTransformJSON Sample Content as follows ;
{
"AAAA": "AAAA",
"BBBB": "BBBB",
"CCCC": "CCCC",
"DDDD": "DDDD",
"EEEE": "EEEE",
"FFFF": "FFFF",
"GGGG": "GGGG",
"HHHH": "HHHH",
...
...
...
}
I want to define json fields to nifi Attributes. I don't want to enter one by one with EvaluateJsonPath.
Edit : I found the this Script for ExecuteGroovyScript and handle it.
import org.apache.commons.io.IOUtils
import java.nio.charset.*
def flowFile = session.get();
if (flowFile == null) {
return;
}
def slurper = new groovy.json.JsonSlurper()
def attrs = [:] as Map<String,String>
session.read(flowFile,
{ inputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = slurper.parseText(text)
obj.each {k,v ->
attrs[k] = v.toString()
}
} as InputStreamCallback)
flowFile = session.putAllAttributes(flowFile, attrs)
session.transfer(flowFile, REL_SUCCESS)
I have multiple systems that send data as JSON Request Body. This is my simple config file.
input {
http {
port => 5001
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}
In most cases this works just fine. I can look at the json data with kibana.
In some cases the JSON will not be processed. It hase something to do with the JSON escaping. For example: If a key contains a '.', the JSON will not be processed.
I can not control the JSON. Is there a way to escape these characters in a JSON key?
Update: As mentioned in the comments I'll give an example of a JSON String (Content is altered. But I,ve tested the JSON String. It has the same behavior as the original.):
{
"http://example.com": {
"a": "",
"b": ""
}
}
My research brings me back to my post, finally.
Before Elasticsearch 2.0 dots in the key were allowed. Since version 2.0 this is not the case anymore.
One user in the logstash forum developed a ruby script that takes care of the dots in json keys:
filter {
ruby {
init => "
def remove_dots hash
new = Hash.new
hash.each { |k,v|
if v.is_a? Hash
v = remove_dots(v)
end
new[ k.gsub('.','_') ] = v
if v.is_a? Array
v.each { |elem|
if elem.is_a? Hash
elem = remove_dots(elem)
end
new[ k.gsub('.','_') ] = elem
} unless v.nil?
end
} unless hash.nil?
return new
end
"
code => "
event.instance_variable_set(:#data,remove_dots(event.to_hash))
"
}
}
All credits go to #hanzmeier1234 (Field name cannot contain ‘.’)
I'm trying to insert an alert in elasticsearch from bosun but I don't know how to fill the variable $timestamp (Have a look at my example) with the present time. Can I use functions in bosun.conf? I'd like something like now().
Can anybody help me, please?
This is an extract of an example configuration:
macro m1
{
$timestamp = **???**
}
notification http_crit
{
macro = m1
post = http://xxxxxxx:9200/alerts/http/
body = {"#timestamp":$timestamp,"level":"critical","alert_name":"my_alert"}
next = http_crit
timeout = 1m
}
alert http
{
template = elastic
$testHTTP = lscount("logstash", "", "_type:stat_http,http_response:200", "1m", "5m", "")
$testAvgHTTP = avg($testHTTP)
crit = $testAvgHTTP < 100
critNotification = http_crit
}
We use .State.Touched.Format which was recently renamed to .Last.Time.Format in the master branch. The format string is a go time format, and you would have to get it to print the correct format that elastic is expecting.
template elastic {
subject = `Time: {{.State.Touched.Format "15:04:05UTC"}}`
}
//Changed on 2016 Feb 01 to
template elastic {
subject = `Time: {{.Last.Time.Format "15:04:05UTC"}}`
}
Which when rendered would look like:
Time: 01:30:13UTC
I have a ruby code block, as follows:
require "elasticsearch"
require "json"
search_term = "big data"
city = "Hong Kong"
client = Elasticsearch::Client.new log: true
r = client.search index: 'candidates', body:
{
query: {
bool: {
must: [
{
match: {
tags: search_term
}
},
{
match: {
city: city
}
}
]
}
}
}
It produces multiple returns like this one:
{"_index":"candidates","_type":"data",
"_id":"AU3DyAmvtewNSFHuYn88",
"_score":3.889237,
"_source":{"first":"Kota","last":"Okayama","city":"Tokyo","designation":"Systems Engineer","email":"user#hotmail.co.jp","phone":"phone","country":"Japan","industry":"Technology","tags":["remarks","virtualization big data"]}}
I want to iterate through it and extract various elements. I have tried
data = JSON.parse(r)
data.each do |row|
puts row["_source"]["first"]
end
and the error is:
no implicit conversion of Hash into String (TypeError)
What's the best way forward on this chaps?
I have the solution, I hope it helps somebody else. It took me hours of fiddling and experimentation. Here it is:
require "elasticsearch"
require "json"
search_term = "big data"
city = "Tokyo"
client = Elasticsearch::Client.new log: true
h = client.search index: 'swiss_candidates', body:
{
query: {
bool: {
must: [
{
match: {
tags: search_term
}
},
{
match: {
city: city
}
}
]
}
}
}
data = JSON.parse(h.to_json)
data["hits"]["hits"].each do |r|
puts r["_id"]
puts r["_source"]["first"]
puts r["_source"]["tags"][1]
puts r["_source"]["screened"][0]
end
The important thing seems to be to convert the elasticsearch result into something ruby friendly.
JSON.parse expects a String containing a JSON document, but you are passing it the Hash which was returned from client.search.
I'm not entirely sure what you are trying to achieve with that, why you want to parse something which is already a Ruby Hash into a Ruby Hash.
I currently have some Ruby code that creates output like this (after conversion to JSON):
"days": [
{
"Jul-22": ""
},
{
"Aug-19": ""
}
],
What I want is output like this:
"days": {
"Jul-22": "",
"Aug-19": ""
},
Here is my code:
CalendarDay.in_the_past_30_days(patient).select(&:noteworthy?).collect do |noteworthy_day|
{ noteworthy_day.date.to_s(:trends_id) => "" }
end
In other words I want a hash instead of an array of hashes. Here's my ugly solution:
days = {}
CalendarDay.in_the_past_30_days(patient).select(&:noteworthy?).each do |noteworthy_day|
days[noteworthy_day.date.to_s(:trends_id)] = ""
end
days
That seems very unrubylike, though. Can someone help me do this more efficiently?
Hash[
CalendarDay.in_the_past_30_days(patient).select(&:noteworthy?).collect { |noteworthy_day|
[noteworthy_day.date.to_s(:trends_id), ""]
}
]
Or...
CalendarDay.in_the_past_30_days(patient).select(&:noteworthy?).each_with_object(Hash.new) { |noteworthy_day, ndays|
ndays[noteworthy_day] = ""
}
This is a problem tailor made for Enumerable#inject
CalendarDay.in_the_past_30_days(patient).select(&:noteworthy?).inject({}) do |hash, noteworthy_day|
hash[noteworthy_day.date.to_s(:trends_id)] = ''
hash
end