So the data I want to store in Elasticsearch is an csv file. It contains the following:
1465309033156,84,http://test.com/purchase.php,200,OK,ThreadGroup 1-3,true,6,6,84,testPC,0
1465309033469,176,http://test.com/,200,OK,ThreadGroup 1-7,true,7,7,176,testPC,91
Note that the first line equals the time in UNIX_MS.
I'm trying to send the data with logstash. Here is my config file:
input {
file {
sincedb_path => "NUL"
ignore_older => 0
type => "csv"
path => ["C:/result/piet.jtl"]
start_position => "beginning"
}
}
filter {
csv {
columns => ["#jmeter_timestamp", ...]
separator => ","
}
date {
locale => "en"
match => ["#jmeter_timestamp", "UNIX_MS"]
remove_field => ["#jmeter_timestamp"]
target => "#timestamp"
timezone => "Europe/Amsterdam"
}
}
output {
elasticsearch {
template => "c:/result/piet.json"
template_name => "piet"
hosts => ["192.168.43.51:9200"]
index => "piet-%{+YYYY.MM.dd}"
}
}
Here is part of piet.json:
"mappings": {
"logs": {
"properties": {
"#timestamp": {
"type": "date"
},
Now i'm getting an error running the config file.
"error"=>{
"type"=>"mapper_parsing_exception",
"reason"=>"failed to parse [#timestamp]",
"caused_by"=>{
"type"=>"illegal_argument_exception",
"reason"=>"Invalid format: \"2016-06-07T14:17:34.611Z\" is malformed at \"-06-07T14:17:34.611Z\""
}}}}, :level=>:warn}
My stdout looks like this:
I'm just lost. How can I insert data from the csv file in Elasticsearch..
Solved it myself.
Elasticsearch is complaining about the date, because it doesn't recognize the date correctly.
So, I removed the mapping completely from Elasticsearch via the Kopf plugin.
I re-added the .json, including the following details. (Note the format)
"mappings": {
"logs": {
"properties": {
"#timestamp": {
"type": "date",
"format" : "strict_date_optional_time||epoch_millis"
It is working now. Huray
Related
I'm trying to index emails into elasticsearch with logstash
My conf file is like this :
sudo bin/logstash -e 'input
{ imap
{ host => "imap.googlemail.com"
password => "********"
user => "********#gmail.com"
port => 993
secure => "true"
check_interval => 10
folder => "Inbox"
verify_cert => "false" } }
output
{ stdout
{ codec => rubydebug }
elasticsearch
{ index => "emails"
document_type => "email"
hosts => "localhost:9200" } }'
The problem is that two fields of the outputs are parsed as String fields but they are supposed to be "date" fields
The format of the fields is as below :
"x-dbworld-deadline" => "31-Jul-2019"
"x-dbworld-start-date" => "18-Nov-2019"
How can I convert these two fields into date fields ?
Thanks!
How about create mapping of index on Elasticsearch.
It may look like this:
PUT date-test-191211
{
"mappings": {
"_doc": {
"properties": {
"x-dbworld-deadline": {
"type": "date",
"format": "dd-MMM-yyyy"
},
"x-dbworld-start-date": {
"type": "date",
"format": "dd-MMM-yyyy"
}
}
}
}
}
Then, those fields are recognized as Date format:
result:
[
This is the fourth time I attempt to do these mapping. But I do not hav my previous configuration files.
classes2.conf
input {
file {
path => "D:\Workspace.Elastic\FinalVersions\classes.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ["TITLE","PROFFESSOR","MAJOR","SEMESTER","student_count","unit","rating","submit_date","latitude","longitude"]
separator => ","
}
mutate {
convert => { "longitude" => "float" }
convert => { "latitude" => "float" }
rename => {
"longitude" => "[location][lon]"
"latitude" => "[location][lat]"
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "localhost:9200"
index => "geopointest"
}
}
classesRating_mapping2.json
{
"class": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
I get the following error:
[location] is defined as an object in mapping [doc] but this name is already used for a field in other types"
I have created geopointest index and added the json mapping like this:
curl -X PUT -H "Content-Type: application/json" http://localhost:9200/geopointest/class/_mapping --data-binary #classesRating_mapping2.json
What am I missing?
Many thanks.
The problem is that in your mapping you're using class which is a custom type name. That means you need to modify your elasticsearch output like this:
elasticsearch {
hosts => "localhost:9200"
index => "geopointest"
document_type => "class" <-- add this line
}
Without that line, Logstash uses the log type name and that was the reason why your mapping was off.
There are two conf files used to load data from 2 json files,testOrders and testItems, each containing only one document, into same index. I am trying to create parent child relationship between two documents.
Below is my conf for testorders
input{
file{
path => ["/path_data/testOrders.json"]
type => "json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
json {
source => "message"
target => "testorders_collection"
remove_field => [ "message" ]
}
ruby {
code => "
event.set('[my_join_field][name]', 'testorders')
"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "testorder"
document_id => "%{[testorders_collection][eId]}"
routing => "%{[testorders_collection][eId]}"
}
}
Below is the conf for testItems
input{
file{
path => ["/path_to_data/testItems.json"]
type => "json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
json {
source => "message"
target => "test_collection"
remove_field => [ "message" ]
}
}
filter {
ruby {
code => "
event.set('[my_join_field][name]', 'testItems')
event.set('[my_join_field][parent]', event.get('[test_collection][foreignKeyId]'))
"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "testorder"
document_id => "%{[test_collection][eId]}"
routing => "%{[test_collection][foreignKeyId]}"
}
}
As expected the logstash creates 1 record for testOrders but creates 2 records for testItems given 1 json document each for testOrders and testItems. One document is created properly with data but other is created as duplicate and there seems to be no data. The document that is created with data not parsed looks like as follows
{
"_index": "testorder",
"_type": "doc",
"_id": "%{[test_collection][eId]}",
"_score": 1,
"_routing": "%{[test_collection][foreignKeyId]}",
"_source": {
"type": "json",
"#timestamp": "2018-07-10T04:15:58.494Z",
"host": "<hidden>",
"test_collection": null,
"my_join_field": {
"name": "testItems",
"parent": null
},
"path": "/path_to_data/testItems.json",
"#version": "1"
}
Defining a mapping relationship in elastic search solved the issue. This is the way to define the relationship
PUT fulfillmentorder
{
"mappings": {
"doc": {
"properties": {
"my_join_field": {
"type": "join",
"relations": {
"fulfillmentorders": "orderlineitems"
}
}
}
}
}
}
I am learning ElasticSearch and have hit a block. I am trying to use logstash to load a simple CSV into ElasticSearch. This is the data, it is a postcode, longitude, latitude
ZE1 0BH,-1.136758103355,60.150855671143
ZE1 0NW,-1.15526666950369,60.1532197533966
I am using the following logstash conf file to filter the CSV to create a "location" field
input {
file {
path => "postcodes.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ["postcode", "lat", "lon"]
separator => ","
}
mutate { convert => {"lat" => "float"} }
mutate { convert => {"lon" => "float"} }
mutate { rename => {"lat" => "[location][lat]"} }
mutate { rename => {"lon" => "[location][lon]"} }
mutate { convert => { "[location]" => "float" } }
}
output {
elasticsearch {
action => "index"
hosts => "localhost"
index => "postcodes"
}
stdout { codec => rubydebug }
}
And I have added the mapping to ElasticSearch using the console in Kibana
PUT postcodes
{
"settings": {
"number_of_shards": 1
},
"mappings": {
"feature": {
"_all": { "enabled": true },
"properties": {
"postcode": {"type": "text"},
"location": {"type": "geo_point"}
}
}
}
}
I check the mappins for the index using
GET postcodes/_mapping
{
"postcodes": {
"mappings": {
"feature": {
"_all": {
"enabled": true
},
"properties": {
"location": {
"type": "geo_point"
},
"postcode": {
"type": "text"
}
}
}
}
}
}
So this all seems to be correct having looked at the documentation and the other questions posted.
However when i run
bin/logstash -f postcodes.conf
I get an error:
[location] is defined as an object in mapping [logs] but this name is already used for a field in other types
I have tried a number of alternative methods;
Deleted the index and the create a template.json and changed my conf file to have the extra settings:
manage_template => true
template => "postcode_template.json"
template_name =>"open_names"
template_overwrite => true
and this gets the same error.
I have managed to get the data loaded by not supplying a template however the data never gets loaded in as a geo_point so you cannot use the Kibana Tile Map to visualise the data
Can anyone explain why I am receiving that error and what method I should use?
Your problem is that you don't have a document_type => feature on your elasticsearch output. Without that, it's going to create the object on type logs which is why you are getting this conflict.
I have a csv file which I'm tryng to upload to ES using Logstash. My conf file is as follows:
input {
file {
path => ["filename"]
start_position => "beginning"
}
}
filter {
csv {
columns => ["name1", "name2", "name3", ...]
separator => ","
}
}
filter {
mutate {
remove_field => ["name31", "name32", "name33"]
}
}
output {
stdout{
codec => rubydebug
}
elasticsearch {
action => "index"
host => "localhost"
index => "newindex"
template_overwrite => true
document_type => "newdoc"
template => "template.json"
}
}
My template file looks like the following:
{
"mappings": {
"newdoc": {
"properties": {
"name1": {
"type": "integer"
},
"name2": {
"type": "float"
},
"name3": {
"format": "dateOptionalTime",
"type": "date"
},
"name4": {
"index": "not_analyzed",
"type": "string"
},
....
}
}
},
"settings": {
"number_of_replicas": 0,
"number_of_shards": 1
},
"template": "newindex"
}
When I try to overwrite the default mapping, I get an 400 error even when I only try to write one line:
failed action with response of 400, dropping action: ["index", + ...
What can be the problem? Everything works fine if I don't overwrite the mapping but that is not a solution for me. I'm using Logstash 1.5.1 and Elasticsearch 1.5.0 on Red Hat.
Thanks
You should POST your request 'mapping' to elasticsearch before loading data in elasticsearch
POST mapping
You don't need to create the index before running logstash , It does create the index if you haven't yet , but it's better to create your own mapping before runing your conf file with logstash . Gives you more control over your field types etc.. Here is a simple tutorial on how to import csv to elasticsearch using logstash : http://freefilesdl.com/how-to-connect-logstash-to-elasticsearch-output