ruby/logstash noob here using ELK stack.
I got a bunch of fields
[Message][Detail][Readout][Value1]
[Message][Detail][Readout][Value2]
[Message][Detail][Readout][Value3]
which I want to loop through using ruby in the logstash config.
Then I want to perform a simple operation on each, for example change them from hex to decimal e.g.
event.set('[currField]', event.get('[currField]').to_s.hex);
but I cant find the correct syntax using google.. any help appreciated.
I know the names of the fields, so worst case I'll have to hard code them, but I'd like to avoid that if possible.
EDIT: i have not tested my config yet, so i dont know if "Readout" will be a hash map; im using grok filter to add the values in the config
"(?<[Message][Detail][Readout][Value1]>(?<=0x.{8})([A-F0-9]{2}))",
"(?<[Message][Detail][Readout][Value2]>(?<=0x.{8})([A-F0-9]{2}))"
etc
Pseudo:
event.get('[Message][Detail][Readout]') each { |k, v|
event[k] = newValue;
}
You would use .each to iterate over the [Message][Detail][Readout] hash. Your pseudo-code would set the values at the top-level. To overwrite them use
ruby {
code => '
readout = event.get("[Message][Detail][Readout]")
if readout
readout.each { |k, v|
event.set("[Message][Detail][Readout][#{k}]", v.to_s.hex)
}
end
'
}
Related
I have this input repeated in 1850 files:
[
{
"id"=>66939,
"login"=>"XXX",
"url"=>"https://website.com/XX/users/XXX"
},
...
{}
]
And I wanted to make a list in a way that by looking for the login I can retrieve the ID using a syntax like:
users_list[XXX]
This is my desired output:
{"XXX"=>"66570", "XXX"=>"66570", "XXX"=>"66570", "XXX"=>"66570", ... }
My code is:
i2 = 1
while i2 != users_list_raw.parsed.count
temp_user = users_list_raw.parsed[i2]
temp_user_login = temp_user['login']
temp_user_id = temp_user['id']
user = {
temp_user_login => temp_user_id
}
users_list << user
i2 += 1
end
My output is:
[{"XXX":66570},{"XXX":66569},{"XXX":66568},{"XXX":66567},{"XXX":66566}, ... {}]
but this is not what I want.
What's wrong with my code?
hash[key] = value to add an entry in a hash. So I guess in your case users_list[temp_user_login] = temp_user_id
But I'm unsure why you'd want to do that. I think you could look up the id of a user by having the login with a statement like:
login = XXX
user = users_list.select {|user| user["login"] == login}.first
id = user["id"]
and maybe put that in a function get_id(login) which takes the login as its parameter?
Also, you might want to look into databases if you're going to manipulate large amounts of data like this. ORMs (Object Relational Mappers) are available in Ruby such as Data Mapper and Active Record (which comes bundled with Rails), they allow you to "model" the data and create Ruby objects from data stored in a database, without writing SQL queries manually.
If your goal is to lookup users_list[XXX] then a Hash would work well. We can construct that quite simply:
users_list = users_list_raw.parsed.each.with_object({}) do |user, list|
list[user['login']] = user['id']
end
Any time you find yourself writing a while loop in Ruby, there might be a more idiomatic solution.
If you want to keep track of a mapping from keys to values, the best data structure is a hash. Be aware that assignment via the array operator will replace existing values in the hash.
login_to_id = {}
Dir.glob("*.txt") { |filename| # Use Dir.glob to find all files that you want to process
data = eval(File.read(filename)) # Your data seems to be Ruby encoded hash/arrays. Eval is unsafe, I hope you know what you are doing.
data.each { |hash|
login_to_id[hash["login"]] = hash["id"]
}
}
puts login_to_id["XXX"] # => 66939
I have this as input, I need to parse all the keys and values and put in a separate event, can you please help me how to achieve, Actually I'm only one key and value when I using the below ruby code.
"entities": {
"PROCESS_GROUP_INSTANCE-EA653E6874E21338": "WebSphere AS nzapwa145Cell (nzapwa146 / ePIMS_pwa146)",
"PROCESS_GROUP_INSTANCE-64A2967FB7FC0C5B": "WebSphere AS ndmProd (nzapwa127 / ePIMS_Report_pwa127)",
"PROCESS_GROUP_INSTANCE-4738C9392BDBC296": "EPS Store - 900008014",
"PROCESS_GROUP_INSTANCE-BA196B9B53FF6323": "EPS Store - 900008040",
"PROCESS_GROUP_INSTANCE-D5D2DAB06C8FDAAF": "ws-server.jar prod-sj-userprofile-adapter-service- - -*",
"PROCESS_GROUP_INSTANCE-3EABAD79933CE911": "/apps/conf/httpd-p2_cdis.conf - KPATHS",
}
Code:
ruby {
code => '
event.get("[result][entities]").each { |key, value|
event.set("hostId", key)
event.set("serverName", value)
}
'
}
Output for above code:
{"serverName":"/apps/conf/httpd-p2_cdis.conf - KPATHS)","#timestamp":"2018-08-25T15:17:11.762Z","hostId":"PROCESS_GROUP_INSTANCE-3EABAD79933CE911"}
But desired output will be like this :
{"serverName":"WebSphere AS nzapwa145Cell (nzapwa146 / ePIMS_pwa146)","#timestamp":"2018-08-25T15:17:11.762Z","hostId":"PROCESS_GROUP_INSTANCE-EA653E6874E21338"}
{"serverName":"WebSphere AS ndmProd (nzapwa127 / ePIMS_Report_pwa127)","#timestamp":"2018-08-25T15:17:11.762Z","hostId":"PROCESS_GROUP_INSTANCE-64A2967FB7FC0C5B"}
{"serverName":"EPS Store - 900008014","#timestamp":"2018-08-25T15:17:11.762Z","hostId":"PROCESS_GROUP_INSTANCE-4738C9392BDBC296"}
{"serverName":"ws-server.jar prod-sj-userprofile-adapter-service- - -*","#timestamp":"2018-08-25T15:17:11.762Z","hostId":"PROCESS_GROUP_INSTANCE-BA196B9B53FF6323"}
{"serverName":"/apps/conf/httpd-p2_cdis.conf - KPATHS","#timestamp":"2018-08-25T15:17:11.762Z","hostId":"PROCESS_GROUP_INSTANCE-D5D2DAB06C8FDAAF"}
{"serverName":"/apps/conf/httpd-p2_cdis.conf - KPATHS)","#timestamp":"2018-08-25T15:17:11.762Z","hostId":"PROCESS_GROUP_INSTANCE-3EABAD79933CE911"}
Can you please help me to get the solution, I'm facing problem with ruby code.
I'm not a Logstash expert, but I think your Ruby filter overwrites (sets) the value hostId and serverName multiple times while iterating over all given key-value pairs. That leaves you with the values that were set in the last iteration. After all, there is only one event, so your keys need to be unique. To fix that, you need to extend your event's key field by either nesting it in tree structure or creating unique key strings.
Here, hostId is the unique key :
ruby {
code => '
event.get("[result][entities]").each do |key, value|
event.set(key, value)
end
'
}
Here an additional index is added to ensure unique keys:
ruby {
code => '
idx = 0
event.get("[result][entities]").each do |key, value|
event.set("[#{idx}][hostId]", key)
event.set("[#{idx}][serverName]", value)
idx += 1
end
'
}
I hope you find that helpful.
Im new in Scala. I need to know if is possible do something like this in Scala:
input2.lines.sort_by { |l| l.gsub(/.*?\+(.*?)\+(.*)\n/,"\\2\n").to_i }
Please help
It looks like you're trying to sort strings by a sub-section within each string. To do that you first need a regex with a capture group to select the region you're interested in.
val re = ".*\\+.*\\+(\\d+)".r
Now you can extract and modify what was captured and use the result as the sorting rule.
lines.sortBy{case re(n) => n.toInt}
Given this hash:
hash1= { node1: { node2: { node3: { node4: { node5: 1 } } } } }
We access inside nodes with square brackets like this:
hash1[:node1][:node2][:node3][:node4]
Now I have a hash that I know will always be nested as it is an XML response from a SOAP webservice, but neither the depth of the hash nor the names of the nodes stay the same. So it would be nice if I could ask the user of my application for the hash depth and store it in a variable. And then be able to do hash1[:hash_depth] and achieve the same result as above.
I have accomplished what I want by the following code:
str = 'node1,node2,node3,node4'
str_a = str.split(',')
hash_copy = hash1
str_a.each { |s| hash_copy = hash_copy.[](s.to_sym) }
hash_copy
=> {:node5=>1}
hash1[:node1][:node2][:node3][:node4]
=> {:node5=>1}
that is asking the user to enter the hash depth separated by commas, store it in a string, split it, make an array, clone the original hash, go down each level and modify the hash till I get to the desired node. Is there a way to do it with the square brackets notation and using a variable to store the depth without modifying the hash or needing to clone it?
Edit:
someone answered with the following (can't see his post anymore???)
hash_depth="[:node1][:node2][:node3][:node4]"
eval "hash1#{hash_depth}"
Although eval does everything you need, there is another approach, since you already have the working code for comma-separated list:
hash_depth="[:node1][:node2][:node3][:node4]"
csh = hash_depth.gsub(/\A\[:|\]\[:|\]\Z/, { '][:' => ',' })
#⇒ "node1,node2,node3,node4"
And now you are free to apply your existing function to csh.
If this is a webapp, I think you should prepare a list of short textareas, which starts with a single text item, and the user can keep adding a new item to the list by clicking on a button. The areas will be filled by the user, and will be sent.
Then, you will probably receive this through some serialized form. You decode this to get an array of strings:
str_a = ["node1", "node2", "node3", "node4"]
and you can reach the inner element by doing:
str_a.inject(hash1){|h, s| h[s.to_sym]} #=> {:node5 => 1}
I have a string which looks like the following:
string = " <SET-TOPIC>INITIATE</SET-TOPIC>
<SETPROFILE>
<PROFILE-KEY>predicates_live</PROFILE-KEY>
<PROFILE-VALUE>yes</PROFILE-VALUE>
</SETPROFILE>
<think>
<set><name>first_time_initiate</name>yes</set>
</think>
<SETPROFILE>
<PROFILE-KEY>first_time_initiate</PROFILE-KEY>
<PROFILE-VALUE>YES</PROFILE-VALUE>
</SETPROFILE>"
My objective is to be able to read out each top level that is in caps with the parse. I use a case statement to evaluate what is the top level key, such as <SETPROFILE> but there can be lots of different values, and then run a method that does different things with the contnts of the tag.
What this means is I need to be able to know very easily:
top_level_keys = ['SET-TOPIC', 'SET-PROFILE', 'SET-PROFILE']
when I pass in the key know the full value
parsed[0].value = {:PROFILE-KEY => predicates_live, :PROFILE-VALUE => yes}
parsed[0].key = ['SET-TOPIC']
I currently parse the whole string as follows:
doc = Nokogiri::XML::DocumentFragment.parse(string)
parsed = doc.search('*').each_with_object({}){ |n, h|
h[n.name] = n.text
}
As a result, I only parse and know of the second tag. The values from the first tag do not show up in the parsed variable.
I have control over what the tags are, if that helps.
But I need to be able to parse and know the contents of both tag as a result of the parse because I need to apply a method for each instance of the node.
Note: the string also contains just regular text, both before, in between, and after the XML-like tags.
It depends on what you are going to achieve. The problem is that you are overriding hash keys by new values. The easiest way to collect values is to store them in array:
parsed = doc.search('*').each_with_object({}) do |n, h|
# h[n.name] = n.text :: removed because it overrides values
(h[n.name] ||= []) << n.text
end