I need to parse my Date and it gives me an error.
input {
file {
path => "/home/osboxes/ELK/logstash/data/data.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["Date","Open","High","Low","Close","Volume","Adj Close"]
}
mutate {convert => ["High", "float"]}
mutate {convert => ["Open", "float"]}
mutate {convert => ["Low", "float"]}
mutate {convert => ["Close", "float"]}
mutate {convert => ["Volume", "float"]}
}
output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "stock"
workers => 1
}
stdout {}
}
The data.csv when I'm reading this is like this:
Date,Open,High,Low,Close,Volume,Adj Close
2015-04-02,125.03,125.56,124.19,125.32,32120700,125.32
2015-04-01,124.82,125.12,123.10,124.25,40359200,124.25
Where am I missing? Thanks in advance.
My logstash terminal only say this:
$ bin/logstash -f /home/osboxes/ELK/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
Add a date statement to the filter:
date {
match => [ "Date", "YYYY-MM-dd" ]
}
Related
I am trying to make the multiline field for csv file work in logstash.
But the multiline for a field is not working.
My log stash.conf content is:
input {
file {
type => "normal"
path => "/etc/logstash/*.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "."
negate => true
what => "previous"
}
}
}
filter {
if [type] == "normal" {
csv {
separator => ","
columns => ["make", "model", "doors"]
}
mutate {convert => ["doors","integer"] }
}
}
output {
if [type] == "normal" {
elasticsearch {
hosts => "<put_local_ip>"
user => "<put_user>"
password => "<put_password>"
index => "cars"
document_type => "sold_cars"
}
stdout {}
}
}
.csv with multiple line (in quotes) for a field make is:
make,model,doors
mazda,mazda6,4
"mitsubishi
4000k", galant,2
honda,civic,4
After I run "logstash -f /etc/logstash/logstash.conf"
I am getting parse failure, from the logs:
{
"tags" => [
[0] "_csvparsefailure"
],
"#timestamp" => 2020-07-13T19:13:11.339Z,
"type" => "normal",
"host" => "<host_ip_greyedout>",
"message" => "\"mitsubishi",
"#version" => "1",
"path" => "/etc/logstash/cars4.csv"
}
I'm attempting to take three columns and combine them into two new fields
Example:
Job_Date 6\5\2019
Job_Start_Time 0:00
Job_End_Time 0:00
Into New Fields:
timestamp_start 6/5/2019, 0:00
timestamp_end 6/5/2019, 0:00
The new fields are getting created but i'm getting the parse error below.
{
"#timestamp" => 2019-06-22T21:08:20.370Z,
"Warning" => 60,
"path" => "/Users/*******/Desktop/Logstash-Files/ax_batch_performance_test_new.csv",
"message" => "job",6/4/2019,13:45,13:45,6,120,60,15\r",
"tags" => [
[0] "_dateparsefailure"
],
"host" => "host",
"Job_Duration" => 6,
"timestamp_end" => "6/4/2019 13:45",
"Job_Start_Time" => "13:45",
"Critical" => 120,
"#version" => "1",
"Job_End_Time" => "13:45",
"Job_Date" => "6/4/2019",
"timestamp_start" => "6/4/2019 13:45",
"Target" => 15,
"Job_Name" => "job name"
I'm running logstash version 7.1.1. I have tried running the mutate command inside and outside of the date plugin.... If it matters I'm still learning.
I have successfully parsed a date format exactly like this before, but not by creating a new field and combining the data and time.
filter{
csv {
separator => ","
columns => ["Job_Name", "Job_Date", "Job_Start_Time", "Job_End_Time", "Job_Duration", "Critical", "Warning", "Target"]
}
mutate {convert => ["Job_Duration", "integer"]}
mutate {convert => ["Critical", "integer"]}
mutate {convert => ["Warning", "integer"]}
mutate {convert => ["Target", "integer"]}
mutate { add_field => {"timestamp_start" => "%{Job_Date} %{Job_Start_Time}"}}
mutate { add_field => {"timestamp_end" => "%{Job_Date} %{Job_End_Time}"}}
date {
match => ["timestamp_start", "M/d/yyyy, HH:MM"]
timezone => "UTC"
}
date {
match => ["timestamp_end", "M/d/yyyy, HH:MM"]
timezone => "UTC"
}
}
I'm expecting the date and time to be parsed and placed into #timestamp as a date.
This is my logstash.conf file:
input {
http {
host => "127.0.0.1"
port => 31311
}
}
filter {
mutate {
split => ["%{headers.request_path}", "/"]
add_field => { "index_id" => "%{headers.request_path[0]}" }
add_field => { "document_id" => "%{headers.request_path[1]}" }
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "%{index_id}"
document_id => "%{document_id}"
}
stdout {
codec => "rubydebug"
}
}
When I send a PUT request like
C:\Users\BolverkXR\Downloads\curl-7.64.1-win64-mingw\bin> .\curl.exe
-XPUT 'http://127.0.0.1:31311/twitter'
I want a new index to be created with the name twitter, instead of using the ElasticSearch default.
However, Logstash crashes immediately with the following (truncated) error message:
Exception in pipelineworker, the pipeline stopped processing new
events, please check your filter configuration and restart Logstash.
org.logstash.FieldReference$IllegalSyntaxException: Invalid
FieldReference: headers.request_path[0]
I am sure I have made a syntax error somewhere, but I can't see where it is. How can I fix this?
EDIT:
The same error occurs when I change the filter segment to the following:
filter {
mutate {
split => ["%{[headers][request_path]}", "/"]
add_field => { "index_id" => "%{[headers][request_path][0]}" }
add_field => { "document_id" => "%{[headers][request_path][1]}" }
}
}
To split the field the %{foo} syntax is not used. Also you should start at position [1] of the array, because in position [0] there will be an empty string("") due to the reason that there are no characters at the left of the first separator(/). Instead, your filter section should be something like this:
filter {
mutate {
split => ["[headers][request_path]", "/"]
add_field => { "index_id" => "%{[headers][request_path][1]}" }
add_field => { "document_id" => "%{[headers][request_path][2]}" }
}
}
You can now use the value in %{index_id} and %{document_id}. I tested this using logstash 6.5.3 version and used Postman to send the 'http://127.0.0.1:31311/twitter/1' HTTP request and the output in console was as follows:
{
"message" => "",
"index_id" => "twitter",
"document_id" => "1",
"#version" => "1",
"host" => "127.0.0.1",
"#timestamp" => 2019-04-09T12:15:47.098Z,
"headers" => {
"connection" => "keep-alive",
"http_version" => "HTTP/1.1",
"http_accept" => "*/*",
"cache_control" => "no-cache",
"content_length" => "0",
"postman_token" => "cb81754f-6d1c-4e31-ac94-fde50c0fdbf8",
"accept_encoding" => "gzip, deflate",
"request_path" => [
[0] "",
[1] "twitter",
[2] "1"
],
"http_host" => "127.0.0.1:31311",
"http_user_agent" => "PostmanRuntime/7.6.1",
"request_method" => "PUT"
}
}
The output section of your configuration does not change. So, your final logstash.conf file will be something like this:
input {
http {
host => "127.0.0.1"
port => 31311
}
}
filter {
mutate {
split => ["[headers][request_path]", "/"]
add_field => { "index_id" => "%{[headers][request_path][1]}" }
add_field => { "document_id" => "%{[headers][request_path][2]}" }
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "%{index_id}"
document_id => "%{document_id}"
}
stdout {
codec => "rubydebug"
}
}
I'm trying to push the data into elasticsearch which comes from kafka topic with Logstash but having this problem when I start my logstash
error code: A plugin had an unrecoverable error
how to fix this? The config file is below.
`input{
kafka{
bootstrap_servers =>"localhosts:9092"
topics => ["cars"]
}
}
filter{
csv {
separator =>","
columns => [ "maker", "model", "mileage", "manufacture_year", "engine_displacement", "engine_power", "body_type", "color_slug", "stk_year", "transmission", "door_count", "seat_count", "fuel_type", "date_created", "date_last_seen", "price_eur" ]
}
mutate {convert => ["mileage", "integer"] }
mutate {convert => ["price_eur", "float"] }
mutate {convert => ["engine_power", "integer"] }
mutate {convert => ["door_power", "integer"] }
mutate {convert => ["seat_count", "integer"] }
}
output{
elasticsearch {
hosts => ["localhost:9200"]
index => "cars1"
document_type=>"sold_cars"
}
stdout{}
}`
The convert mutate filter is a hash, not an array: https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-convert
Try it like this:
input {
kafka {
bootstrap_servers => "localhost:9092"
topics => ["cars"]
}
}
filter {
csv {
separator => ","
columns => ["maker", "model", "mileage", "manufacture_year", "engine_displacement", "engine_power", "body_type", "color_slug", "stk_year", "transmission", "door_count", "seat_count", "fuel_type", "date_created", "date_last_seen", "price_eur"]
}
mutate {
convert => {
"mileage" => "integer"
"price_eur" => "float"
"engine_power" => "integer"
"door_power" => "integer"
"seat_count" => "integer"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "cars1"
document_type => "sold_cars"
}
stdout {}
}
You can also use convert inside the csv filter itself, like so:
csv {
separator => ","
columns => ["maker", "model", "mileage", "manufacture_year", "engine_displacement", "engine_power", "body_type", "color_slug", "stk_year", "transmission", "door_count", "seat_count", "fuel_type", "date_created", "date_last_seen", "price_eur"]
convert => {
"mileage" => "integer"
"price_eur" => "float"
"engine_power" => "integer"
"door_power" => "integer"
"seat_count" => "integer"
}
}
i have the following json input that i want to dump to logstash (and eventually search/dashboard in elasticsearch/kibana).
{"vulnerabilities":[
{"ip":"10.1.1.1","dns":"z.acme.com","vid":"12345"},
{"ip":"10.1.1.2","dns":"y.acme.com","vid":"12345"},
{"ip":"10.1.1.3","dns":"x.acme.com","vid":"12345"}
]}
i'm using the following logstash configuration
input {
file {
path => "/tmp/logdump/*"
type => "assets"
codec => "json"
}
}
output {
stdout { codec => rubydebug }
elasticsearch { host => localhost }
}
output
{
"message" => "{\"vulnerabilities\":[\r",
"#version" => "1",
"#timestamp" => "2014-10-30T23:41:19.788Z",
"type" => "assets",
"host" => "av12612sn00-pn9",
"path" => "/tmp/logdump/stack3.json"
}
{
"message" => "{\"ip\":\"10.1.1.30\",\"dns\":\"z.acme.com\",\"vid\":\"12345\"},\r",
"#version" => "1",
"#timestamp" => "2014-10-30T23:41:19.838Z",
"type" => "assets",
"host" => "av12612sn00-pn9",
"path" => "/tmp/logdump/stack3.json"
}
{
"message" => "{\"ip\":\"10.1.1.31\",\"dns\":\"y.acme.com\",\"vid\":\"12345\"},\r",
"#version" => "1",
"#timestamp" => "2014-10-30T23:41:19.870Z",
"type" => "shellshock",
"host" => "av1261wag2sn00-pn9",
"path" => "/tmp/logdump/stack3.json"
}
{
"ip" => "10.1.1.32",
"dns" => "x.acme.com",
"vid" => "12345",
"#version" => "1",
"#timestamp" => "2014-10-30T23:41:19.884Z",
"type" => "assets",
"host" => "av12612sn00-pn9",
"path" => "/tmp/logdump/stack3.json"
}
obviously logstash is treating each line as an event and it thinks {"vulnerabilities":[ is an event and i'm guessing the trailing commas on the 2 subsequent nodes mess up the parsing, and the last node appears coorrect. how do i tell logstash to parse the events inside the vulnerabilities array and to ignore the commas at the end of the line?
Updated: 2014-11-05
Following Magnus' recommendations, I added the json filter and it's working perfectly. However, it would not parse the last line of the json correctly without specifying start_position => "beginning" in the file input block. Any ideas why not? I know it parses bottom up by default but would anticipate the mutate/gsub would handle this smoothly?
file {
path => "/tmp/logdump/*"
type => "assets"
start_position => "beginning"
}
}
filter {
if [message] =~ /^\[?{"ip":/ {
mutate {
gsub => [
"message", "^\[{", "{",
"message", "},?\]?$", "}"
]
}
json {
source => "message"
remove_field => ["message"]
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch { host => localhost }
}
You could skip the json codec and use a multiline filter to join the message into a single string that you can feed to the json filter.filter {
filter {
multiline {
pattern => '^{"vulnerabilities":\['
negate => true
what => "previous"
}
json {
source => "message"
}
}
However, this produces the following unwanted results:
{
"message" => "<omitted for brevity>",
"#version" => "1",
"#timestamp" => "2014-10-31T06:48:15.589Z",
"host" => "name-of-your-host",
"tags" => [
[0] "multiline"
],
"vulnerabilities" => [
[0] {
"ip" => "10.1.1.1",
"dns" => "z.acme.com",
"vid" => "12345"
},
[1] {
"ip" => "10.1.1.2",
"dns" => "y.acme.com",
"vid" => "12345"
},
[2] {
"ip" => "10.1.1.3",
"dns" => "x.acme.com",
"vid" => "12345"
}
]
}
Unless there's a fixed number of elements in the vulnerabilities array I don't think there's much we can do with this (without resorting to the ruby filter).
How about just applying the json filter to lines that look like what we want and drop the rest? Your question doesn't make it clear whether all of the log looks like this so this may not be so useful.
filter {
if [message] =~ /^\s+{"ip":/ {
# Remove trailing commas
mutate {
gsub => ["message", ",$", ""]
}
json {
source => "message"
remove_field => ["message"]
}
} else {
drop {}
}
}