files started importing like this..through logstash..kindly suggest me how to remove the errors i use kv filter only in my conf program
{
"authserver" => "a_India RADIUS",
"proto" => "6",
"devname" => "FW_1",
"10:56:12\tdate" => "2020-06-22\tlocal7\tnotice\t\ttime=10:56:11",
"host" => "kali",
"dstintf" => "wan1",
"path" => "/root/Cybrotech-/log00",
"subtype" => "webfilter",
"srcintf" => "ssl.root",
"method" => "domain",
"eventtype" => "ftgd_allow",
"hostname" => "webmail.accessarellc.net",
"cat" => "33",
"srcintfrole" => "undefined",
"dstip" => "20.73.98.154",
"type" => "utm",
"sessionid" => "677535",
"dstintfrole" => "wan",
"srcport" => "6095",
"url" => "/",
"profile" => "monitor-all",
"srcip" => "10.212.134.190",
"logid" => "07013312",
"policyid" => "17",
"eventtime" => "12803571",
"direction" => "outgoing",
"level" => "notice",
"#version" => "1",
"reqtype" => "direct",
"catdesc" => ""Health",
"action" => "passthrough",
"vd" => "root",
"dstport" => "443",
"service" => "HTTPS",
"#timestamp" => 2020-07-13T09:10:47.811Z,
"sentbyte" => "192",
"devid" => "FG0TK19907000",
"group" => "SSLVPN_Group",
"msg" => "URL belongs to an allowed category in policy",
"user" => "\ASINGH",
"rcvdbyte" => "0"
}
after some time i got this error on screen..
"_type"=>"doc", "_id"=>"LzhxR3MBoH6QvDEw21Sy", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] in index [log00_210270] has been exceeded"}}}}
Related
I have a hash:
req = {
"count" => 50100,
"results" => [
{"listing_id" => 615929315, "state" => "active", "user_id" => 140604756, "category_id" => 69150367},
{"listing_id" => 615929311, "state" => "active", "user_id" => 152528025, "category_id" => 69150367}
]
}
I want to find and display the entire internal hash if a particular user_id exists. I can find it:
req["results"][0].select{|key, value| value == 152528025}
# => {"user_id" => 152528025}
How do I then display this entire (nested) hash?
{"listing_id" => 615929311, "state" => "active", "user_id" => 152528025, "category_id" => 69150367}
req["results"].select{|x| x["user_id"] == 152528025}
I would like to retrieve the Redis status using a Ruby script, I tried the following commands:
redis_status_command, stdeerr, status = Open3.capture3(`redis-cli -h 127.0.0.1 -p 6379 info|grep status`)
OR
redis_status_command, stdeerr, status = Open3.capture3(`systemctl status redis`)
Then when I try to print the variable redis_status_command it returns a blank space, but I know that the commands that are inside the Open3.capture3 part work in the command line. How can I retrieve the Redis status using Ruby? Thank you
With the "redis" gem, you can get all the info properties as follows:
require 'redis'
client = Redis.new(
url: "redis://your-host.your-domain.com",
port: 6379
)
client.info
The output will be a Hash of the info:
{
"redis_version" => "3.2.4",
"redis_git_sha1" => "0",
"redis_git_dirty" => "0",
"redis_build_id" => "0",
"redis_mode" => "standalone",
"os" => "Amazon ElastiCache",
"arch_bits" => "64",
"multiplexing_api" => "epoll",
"gcc_version" => "0.0.0",
"process_id" => "1",
"run_id" => "73f5f76133b7bfd66eb89850a2f9df43e838f567",
"tcp_port" => "6379",
"uptime_in_seconds" => "1754115",
"uptime_in_days" => "20",
"hz" => "10",
"lru_clock" => "5884700",
"executable" => "-",
"config_file" => "-",
"connected_clients" => "10",
"client_longest_output_list" => "0",
"client_biggest_input_buf" => "0",
"blocked_clients" => "0",
"used_memory" => "16110168",
"used_memory_human" => "15.36M",
"used_memory_rss" => "19808256",
"used_memory_rss_human" => "18.89M",
"used_memory_peak" => "19915496",
"used_memory_peak_human" => "18.99M",
"used_memory_lua" => "37888",
"used_memory_lua_human" => "37.00K",
"maxmemory" => "6501171200",
"maxmemory_human" => "6.05G",
"maxmemory_policy" => "volatile-lru",
"mem_fragmentation_ratio" => "1.23",
"mem_allocator" => "jemalloc-4.0.3",
"loading" => "0",
"rdb_changes_since_last_save" => "30264",
"rdb_bgsave_in_progress" => "0",
"rdb_last_save_time" => "1497302809",
"rdb_last_bgsave_status" => "ok",
"rdb_last_bgsave_time_sec" => "-1",
"rdb_current_bgsave_time_sec" => "-1",
"aof_enabled" => "0",
"aof_rewrite_in_progress" => "0",
"aof_rewrite_scheduled" => "0",
"aof_last_rewrite_time_sec" => "-1",
"aof_current_rewrite_time_sec" => "-1",
"aof_last_bgrewrite_status" => "ok",
"aof_last_write_status" => "ok",
"total_connections_received" => "29746",
"total_commands_processed" => "3809776",
"instantaneous_ops_per_sec" => "1",
"total_net_input_bytes" => "382270539",
"total_net_output_bytes" => "3134102675",
"instantaneous_input_kbps" => "0.05",
"instantaneous_output_kbps" => "0.03",
"rejected_connections" => "0",
"sync_full" => "1",
"sync_partial_ok" => "0",
"sync_partial_err" => "0",
"expired_keys" => "0",
"evicted_keys" => "0",
"keyspace_hits" => "674",
"keyspace_misses" => "318",
"pubsub_channels" => "0",
"pubsub_patterns" => "0",
"latest_fork_usec" => "162",
"migrate_cached_sockets" => "0",
"role" => "master",
"connected_slaves" => "1",
"slave0" => "ip=192.168.1.2,port=6379,state=online,offset=341359205,lag=0",
"master_repl_offset" => "341359205",
"repl_backlog_active" => "1",
"repl_backlog_size" => "1048576",
"repl_backlog_first_byte_offset" => "340310630",
"repl_backlog_histlen" => "1048576",
"used_cpu_sys" => "494.00",
"used_cpu_user" => "912.16",
"used_cpu_sys_children" => "0.00",
"used_cpu_user_children" => "0.00",
"cluster_enabled" => "0",
"db0" => "keys=3479,expires=0,avg_ttl=0"
}
I'm writing a logstash 2.4.0 configuration to go through HTTP logs.
We'd like to have the PORT that is passed in the Header field to be included in the Line fields below.
There is no specific end-event defined. Although I have tried adding an end event as well.
The input log file I'm currently using is:
HEADER 9200
LINE 1 2016-10-05 08:39:00 Some log data
LINE 2 2016-10-05 08:40:00 Some other log data
FOOTER
HEADER 9300
LINE 4 2016-11-05 08:39:00 Some log data in another log
LINE 5 2016-11-05 08:40:00 Some other log data in another log
FOOTER
I would like to have an output like this:
The Server_port fields are currently missing from the output
{"message" => "HEADER 9200",
"#version" => "1",
"#timestamp" => "2016-11-15T11:17:18.425Z",
"path" => "test.log",
"host" => "hostname",
"type" => "event",
"env" => "test",
"port" => 9200,
"tags" => [[0] "Header"] }
{"message" => "LINE 1 2016-10-05 08:39:00 Some log data",
"#version" => "1",
"#timestamp" => "2016-11-15T11:17:20.186Z",
"path" => "test.log",
"host" => "hostname",
"type" => "event",
"env" => "test",
"logMessage" => "1 2016-10-05 08:39:00 Some log data",
"Server_port" => 9200,
"tags" => [[0] "Line"]}
{"message" => "LINE 2 2016-10-05 08:40:00 Some other log data",
"#version" => "1",<
"#timestamp" => "2016-11-15T11:17:20.192Z",
"path" => "test.log",
"host" => "hostname",
"type" => "event",
"env" => "test",
"logMessage" => "2 2016-10-05 08:40:00 Some other log data",
"Server_port" => 9200,
"tags" => [[0] "Line"]}
{"message" => "FOOTER",
"#version" => "1",
"#timestamp" => "2016-11-15T11:17:20.195Z",
"path" => "test.log",
"host" => "hostname",
"type" => "event",
"env" => "test",
"tags" => [[0] "Footer"]}
After trying out different things, the configuration I'm currently using is as follows, with a hardcoded taskid='abcd' for testing:
input{ file{ path => "test.log"
start_position => "beginning"
sincedb_path => "/dev/null"
ignore_older => 0
type => "event"
add_field => { "env" => "test"} }
}
filter{
grok {
break_on_match => false
tag_on_failure => []
match => {"message" => ["^HEADER%{SPACE}%{INT:port:int}"]}
add_tag => ["Header"]
}
grok {
break_on_match => false
tag_on_failure => []
match => {"message" => "^LINE%{SPACE}%{GREEDYDATA:logMessage}"}
add_tag => ["Line"]
}
grok {
break_on_match => false
tag_on_failure => []
match => {"message" => "^FOOTER"}
add_tag => ["Footer"]
}
if "Header" in [tags]{
aggregate{
task_id => "abcd"
code => "map['server_port'] ||= 0; map['server_port']=event['port']"
push_map_as_event_on_timeout => true
push_previous_map_as_event => true
map_action => "create"
}
}
elseif "Line" in [tags]{
aggregate{
task_id => "abcd"
code => "event.set('server_port',map['server_port'])"
map_action => "update"
}
}
else if "Footer" in [tags]{
aggregate{
task_id => "abcd"
code => "event.set('server_port',map['server_port'])"
map_action => "update"
end_of_task => true
timeout => 120
}
}
}
output {
stdout { codec => rubydebug }
}
While this config runs without errors it's not creating the server_port fields.
Where am I going wrong?
After fiddling around some more I have a working test case.
I've changed the configuration as follows:
grok {
break_on_match => false
tag_on_failure => []
match => {
"message" => ["^HEADER%{SPACE}%{INT:taskid:int}%{SPACE}%{INT:port:int}"]
}
add_tag => ["Header"]
}
and
if "Header" in [tags]{
aggregate{
task_id => "%{taskid}"
code => "map['port']=event.get('port')"
map_action => "create"
}
}
elseif "Line" in [tags]{
aggregate{
task_id =>"%{taskid}"
code => "event.set('port',map['port'])"
map_action => "update"
}
}
else if "Footer" in [tags]{
aggregate{
task_id => "%{taskid}"
code => "event.set('port',map['port'])"
map_action => "update"
end_of_task => true
timeout => 120
}
}
And added a task id field to the logs:
HEADER 123 9200
LINE 123 2016-10-05 08:39:00 Some log data
Please I have a json Object which results from an xml input, it looks like this :
{
"#version" => "1",
"#timestamp" => "2016-04-11T15:35:07.372Z",
"host" => "YUSUF-PC",
"command" => "nana",
"doc" => {
"TotalResults" => "1892",
"Audit" => [
[0] {
"Id" => "2260167",
"Action" => "UPDATE",
"ParentId" => "30612",
"ParentType" => "defect",
"Time" => "2016-01-04 08:27:59",
"User" => "nana",
"Properties" => {
"Property" => [
[0] {
"Label" => "Statut",
"Name" => "status",
"NewValue" => [
[0] "En cours"
]
},
[1] {
"Label" => "Affecté à",
"Name" => "owner",
"NewValue" => [
[0] "nana"
]
},
[2] {
"Label" => "Priorité",
"Name" => "severity",
"NewValue" => [
[0] "nana"
]
}
]
}
},
[1] {
"Id" => "2260168",
"Action" => "UPDATE",
"ParentId" => "30612",
"ParentType" => "defect",
"Time" => "2016-01-04 09:45:33",
"User" => "nana",
"Properties" => {
"Property" => [
[0] {
"Label" => "Affecté à",
"Name" => "owner",
"NewValue" => [
[0] "nana"
],
"OldValue" => [
[0] "nana"
]
}
]
}
}
]
} }
I need to split this json to properties, ie to get each document containing one property, the problem is not the split operation, but when I insert this to elasticsearch, the "NewValue" field doesn't get into account... So I need to write a ruby filter to alter the value to value[0]. Anyone can help, I'm not good at ruby ?
I want to get a json like this one :
{
"#version" => "1",
"#timestamp" => "2016-04-11T15:35:07.372Z",
"host" => "YUSUF-PC",
"command" => "nana",
"doc" => {
"TotalResults" => "1892",
"Audit" => [
[0] {
"Id" => "2260167",
"Action" => "UPDATE",
"ParentId" => "30612",
"ParentType" => "defect",
"Time" => "2016-01-04 08:27:59",
"User" => "nana",
"Properties" => {
"Property" =>
{
"Label" => "Statut",
"Name" => "status",
"NewValue" => "En cours"
}
}
}
]
}
}
Thank you
I hope this helps.
old = {
"#version" => "1",
"#timestamp" => "2016-04-11T15:35:07.372Z",
"host" => "YUSUF-PC",
"command" => "nana",
"doc" => {
"TotalResults" => "1892",
"Audit" => [
{
"Id" => "2260167",
"Action" => "UPDATE",
"ParentId" => "30612",
"ParentType" => "defect",
"Time" => "2016-01-04 08:27:59",
"User" => "nana",
"Properties" => {
"Property" => [
{
"Label" => "Statut",
"Name" => "status",
"NewValue" => [
"En cours"
]
},
{
"Label" => "Affecté à",
"Name" => "owner",
"NewValue" => [
"nana"
]
},
{
"Label" => "Priorité",
"Name" => "severity",
"NewValue" => [
"nana"
]
}
]
}
},
{
"Id" => "2260168",
"Action" => "UPDATE",
"ParentId" => "30612",
"ParentType" => "defect",
"Time" => "2016-01-04 09:45:33",
"User" => "nana",
"Properties" => {
"Property" => [
{
"Label" => "Affecté à",
"Name" => "owner",
"NewValue" => [
"nana"
],
"OldValue" => [
"nana"
]
}
]
}
}
]
} }
##THIS IS THE LINE ACTUALLY DOING WORK.
old["doc"]["Audit"].map{|prop| prop["Properties"]["Property"].map{|value| value['NewValue']= value['NewValue'].first} }
old
=> {"#version"=>"1", "#timestamp"=>"2016-04-11T15:35:07.372Z", "host"=>"YUSUF-PC", "command"=>"nana", "doc"=>{"TotalResults"=>"1892", "Audit"=>[{"Id"=>"2260167", "Action"=>"UPDATE", "ParentId"=>"30612", "ParentType"=>"defect", "Time"=>"2016-01-04 08:27:59", "User"=>"nana", "Properties"=>{"Property"=>[{"Label"=>"Statut", "Name"=>"status", "NewValue"=>"En cours"}, {"Label"=>"Affecté à", "Name"=>"owner", "NewValue"=>"nana"}, {"Label"=>"Priorité", "Name"=>"severity", "NewValue"=>"nana"}]}}, {"Id"=>"2260168", "Action"=>"UPDATE", "ParentId"=>"30612", "ParentType"=>"defect", "Time"=>"2016-01-04 09:45:33", "User"=>"nana", "Properties"=>{"Property"=>[{"Label"=>"Affecté à", "Name"=>"owner", "NewValue"=>"nana", "OldValue"=>["nana"]}]}}]}}
I have the following logstash configuration:
input {
file{
path => ["C:/Users/MISHAL/Desktop/ELK_Files/rm/evsb.json"]
type => "json"
start_position => "beginning"
}
}
filter {
json {
source => "message"
}
mutate {
convert => [ "increasedFare", "float"]
convert => ["enq", "float"]
convert => ["bkd", "float"]
}
date{
match => [ "date" , "YYYY-MM-dd HH:mm:ss" ]
target => "#timestamp"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "localhost"
index => "zsx"
}
}
And this is the json data jt.json :
[{"id":1,"date":"2015-11-11 23:00:00","enq":"105","bkd":"9","increasedFare":"0"}, {"id":2,"date":"2015-11-15 23:00:00","eng":"55","bkd":"2","increasedFare":"0"}, {"id":3,"date":"2015-11-20 23:00:00","enq":"105","bkd":"9","increasedFare":"0"}, {"id":4,"date":"2015-11-25 23:00:00","eng":"55","bkd":"2","increasedFare":"0"}]
Tried running this in logstash however I am not able to parse the date or get the date in timestamp.
The following is the warning message im getting:
Failed parsing date from field {:field=>"[date]", :value=>"%{[date]}", :exception=>"Invalid format: \"%{[date]}\"", :config_parsers=>"YYYY-MM-dd HH:mm:ss", :config_locale=>"default=en_IN", :level=>:warn}
The following is the stdout
Logstash startup completed
{
"message" => "{\"id\":2,\"date\":\"2015-09-15 23:00:00\",\"enq\":\"34\",\"bkd\":\"2\",\"increasedFare\":\"0\"}\r",
"#version" => "1",
"#timestamp" => "2015-09-15T17:30:00.000Z",
"host" => "TCHWNG",
"path" => "C:/Users/MISHAL/Desktop/ELK_Files/jsonTest/jt.json",
"type" => "json",
"id" => 2,
"date" => "2015-09-15 23:00:00",
"enq" => 34.0,
"bkd" => 2.0,
"increasedFare" => 0.0
}
{
"message" => "{\"id\":3,\"date\":\"2015-09-20 23:00:00\",\"enq\":\"22\",\"bkd\":\"9\",\"increasedFare\":\"0\"}\r",
"#version" => "1",
"#timestamp" => "2015-09-20T17:30:00.000Z",
"host" => "TCHWNG",
"path" => "C:/Users/MISHAL/Desktop/ELK_Files/jsonTest/jt.json",
"type" => "json",
"id" => 3,
"date" => "2015-09-20 23:00:00",
"enq" => 22.0,
"bkd" => 9.0,
"increasedFare" => 0.0
}
{
"message" => "{\"id\":4,\"date\":\"2015-09-25 23:00:00\",\"enq\":\"66\",\"bkd\":\"2\",\"increasedFare\":\"0\"}\r",
"#version" => "1",
"#timestamp" => "2015-09-25T17:30:00.000Z",
"host" => "TCHWNG",
"path" => "C:/Users/MISHAL/Desktop/ELK_Files/jsonTest/jt.json",
"type" => "json",
"id" => 4,
"date" => "2015-09-25 23:00:00",
"enq" => 66.0,
"bkd" => 2.0,
"increasedFare" => 0.0
}
Been trying to solve this for two days and tried various things, But I am not able to solve this. Please tell what Im doing wrong here.