I have multiple systems that send data as JSON Request Body. This is my simple config file.
input {
http {
port => 5001
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}
In most cases this works just fine. I can look at the json data with kibana.
In some cases the JSON will not be processed. It hase something to do with the JSON escaping. For example: If a key contains a '.', the JSON will not be processed.
I can not control the JSON. Is there a way to escape these characters in a JSON key?
Update: As mentioned in the comments I'll give an example of a JSON String (Content is altered. But I,ve tested the JSON String. It has the same behavior as the original.):
{
"http://example.com": {
"a": "",
"b": ""
}
}
My research brings me back to my post, finally.
Before Elasticsearch 2.0 dots in the key were allowed. Since version 2.0 this is not the case anymore.
One user in the logstash forum developed a ruby script that takes care of the dots in json keys:
filter {
ruby {
init => "
def remove_dots hash
new = Hash.new
hash.each { |k,v|
if v.is_a? Hash
v = remove_dots(v)
end
new[ k.gsub('.','_') ] = v
if v.is_a? Array
v.each { |elem|
if elem.is_a? Hash
elem = remove_dots(elem)
end
new[ k.gsub('.','_') ] = elem
} unless v.nil?
end
} unless hash.nil?
return new
end
"
code => "
event.instance_variable_set(:#data,remove_dots(event.to_hash))
"
}
}
All credits go to #hanzmeier1234 (Field name cannot contain ‘.’)
Related
I have quite an extensive Logstash pipeline ending in a Json as such:
{
"keyA": 1,
"keyB": "sample",
"arrayKey": [
{
"key": "data"
},
{
"key": null
}
]
}
What I want to achieve is to filter "arrayKey" and remove objects within with value for "key" is null.
Tried this to no luck:
filter {
ruby {
code => "
event.get('arrayKey').each do |key|
[key].delete_if do |keyCandidate|
if [keyCandidate][key] != nil
true
end
end
end
"
}
}
This gives no implicit converter found from |hash|:|Int| error. How do I achieve this? Is there and easier way to do this?
As Aleksei pointed out, you can create a copy of the array that does not contain entries where [key] is null using reject. You have to use event.set to overwrite the inital value of [arrayKey]
ruby {
code => '
a = event.get("arrayKey")
if a
event.set("arrayKey", a.reject { |x| x["key"] == nil })
end
'
}
I'm receving events in Logstash containing measurement, values and tags. I do not know ahead of time what field there are and what tags. So i wanted to do something like this:
input {
http {}
}
filter {
ruby {
code => '
tags = event.get("stats_tags").split(",")
samples = event.get("stats_samples").split(" ")
datapoints = {}
samples.each {|s|
splat = s.split(" ")
datapoints[splat[0]] = splat[1]
}
event.set("[#metadata][stats-send-as-tags]", tags)
event.set("[#metadata][stats-datapoints]", datapoints)
'
}
}
output {
influxdb {
host => "influxdb"
db => "events_db"
measurement => measurement
send_as_tags => [#metadata][stats-send-as-tags]
data_points => [#metadata][stats-datapoints]
}
}
But this produce error. After much googling to no avail i'm starting to think this is imposible.
Is there a way to pass hash and array from event field to output/filter configuration?
EDIT: If i doublequote it, the error i'm getting is
output {
influxdb {
# This setting must be a hash
# This field must contain an even number of items, got 1
data_points => "[#metadata][stats-datapoints]"
...
}
}
MongoDb doesn't support keys with dot. I have a Ruby nested hash that has many keys with dot (.) character. Is there a configuration that can be used to specify a character replacement for . like an underscore _ while inserting such data to MongoDb
I'm using MongoDB with Ruby & mongo gem.
example hash is like below
{
"key.1" => {
"second.key" => {
"third.key" => "val"
}
}
}
If it isn't possible to use keys with . in Mongodb, you'll have to modify the input data :
hash = {
'key.1' => {
'second.key' => {
'third.key' => 'val.1',
'fourth.key' => ['val.1', 'val.2']
}
}
}
Transforming string keys
This recursive method transforms the keys of a nested Hash :
def nested_gsub(object, pattern = '.', replace = '_')
if object.is_a? Hash
object.map do |k, v|
[k.to_s.gsub(pattern, replace), nested_gsub(v, pattern, replace)]
end.to_h
else
object
end
end
nested_gsub(hash) returns :
{
"key_1" => {
"second_key" => {
"third_key" => "val.1",
"fourth_key" => [
"val.1",
"val.2"
]
}
}
}
Transforming keys and values
It's possible to add more cases to the previous method :
def nested_gsub(object, pattern = '.', replace = '_')
case object
when Hash
object.map do |k, v|
[k.to_s.gsub(pattern, replace), nested_gsub(v, pattern, replace)]
end.to_h
when Array
object.map { |v| nested_gsub(v, pattern, replace) }
when String
object.gsub(pattern, replace)
else
object
end
end
nested_gsub will now iterate on string values and arrays :
{
"key_1" => {
"second_key" => {
"third_key" => "val_1",
"fourth_key" => [
"val_1",
"val_2"
]
}
}
}
In mongoDB, there is no configuration to support dot in the key. You need to preprocess the JSON before inserting to MongoDB collection.
One approach is that you can replace the dot with its unicode equivalent U+FF0E before insertion.
Hope this helps.
I am doing an mongodb aggregation using mongoid, using ModleName.collection.aggregate(pipeline) . The value returned is an array and not a Mongoid::Criteria, so if a do a first on the array, I get the first element which is of the type BSON::Document instead of ModelName. As a result, I am unable to use it as a model.
Is there a method to return a criteria instead of an array from the aggregation, or convert a bson document to a model instance?
Using mongoid (4.0.0)
I've been struggling with this on my own too. I'm afraid you have to build your "models" on your own. Let's take an example from my code:
class Searcher
# ...
def results(page: 1, per_page: 50)
pipeline = []
pipeline <<
"$match" => {
title: /#{#params['query']}/i
}
}
geoNear = {
"near" => coordinates,
"distanceField" => "distance",
"distanceMultiplier" => 3959,
"num" => 500,
"spherical" => true,
}
pipeline << {
"$geoNear" => geoNear
}
count = aggregate(pipeline).count
pipeline << { "$skip" => ((page.to_i - 1) * per_page) }
pipeline << { "$limit" => per_page }
places_hash = aggregate(pipeline)
places = places_hash.map { |attrs| Offer.new(attrs) { |o| o.new_record = false } }
# ...
places
end
def aggregate(pipeline)
Offer.collection.aggregate(pipeline)
end
end
I've omitted a lot of code from original project, just to present the way what I've been doing.
The most important thing here was the line:
places_hash.map { |attrs| Offer.new(attrs) { |o| o.new_record = false } }
Where both I'm creating an array of Offers, but additionally, manually I'm setting their new_record attribute to false, so they behave like any other documents get by simple Offer.where(...).
It's not beautiful, but it worked for me, and I could take the best of whole Aggregation Framework!
Hope that helps!
How could I append a new key/value pair to an existing JSON object in Ruby?
My output is:
{
"2d967df3-ee07-4e40-8f65-7bbff59bbb7e": {
"name": "Book1",
"author": "Author1"
}
}
I want to achieve something like this when I add a new key/value pair:
{
"2d967df3-ee07-4e40-8f65-7bbff59bbb7e": {
"name": "Book1",
"author": "Author1"
},
"c55a3632-9bed-4a41-ae40-c1abfe0f332a": {
"name": "Book2",
"author": "Author2"
}
}
This is my method to write to a JSON file:
def create_book(name, author)
tempHash = {
SecureRandom.uuid => {
"name" => name,
"author" => author
}
}
File.open("./books/book.json","w") do |f|
f.write(JSON.pretty_generate(tempHash))
end
end
To clarify, I need to add a second entry to the original file. I tried using append (<<), and that's where my code fails:
file = File.read("./books/book.json")
data_hash = JSON.parse(file)
newJson = data_hash << tempHash
How could I append a new key/value pair to existing JSON object in Ruby?
If you want to add it to an existing file then you should read the JSON first, extract data from it, then add a new hash to an array.
Maybe something like this will solve your problem:
def create_book(name, author)
tempHash = {
SecureRandom.uuid => {
"name" => name,
"author" => author
}
}
data_from_json = JSON[File.read("./books/book.json")]
data_from_json = [data_from_json] if data_from_json.class != Array
File.open("./books/book.json","w") do |f|
f.write(JSON.pretty_generate(data_from_json << tempHash))
end
end
There are also some other ways like manipulating the JSON as a common string but for safety you should extract the data and then create a new JSON file.
If you need the new key/value pair to be in the same JSON element as the previous data, instead of shoveling (<<) the hashes together, merge them.
Additionally this can allow you to put the new key/value pair in the start of the element or in the end, by flipping which hash you merge first.
So, take Maxim's solution from Apr 14 '15, but modify to merge the two hashes together.
data_from_json = JSON[http://File.read("./books/book.json")]
File.open("./books/book.json","w") do |f|
f.write(JSON.pretty_generate([data_from_json.merge(tempHash)])
end