I'm trying to index emails into elasticsearch with logstash
My conf file is like this :
sudo bin/logstash -e 'input
{ imap
{ host => "imap.googlemail.com"
password => "********"
user => "********#gmail.com"
port => 993
secure => "true"
check_interval => 10
folder => "Inbox"
verify_cert => "false" } }
output
{ stdout
{ codec => rubydebug }
elasticsearch
{ index => "emails"
document_type => "email"
hosts => "localhost:9200" } }'
The problem is that two fields of the outputs are parsed as String fields but they are supposed to be "date" fields
The format of the fields is as below :
"x-dbworld-deadline" => "31-Jul-2019"
"x-dbworld-start-date" => "18-Nov-2019"
How can I convert these two fields into date fields ?
Thanks!
How about create mapping of index on Elasticsearch.
It may look like this:
PUT date-test-191211
{
"mappings": {
"_doc": {
"properties": {
"x-dbworld-deadline": {
"type": "date",
"format": "dd-MMM-yyyy"
},
"x-dbworld-start-date": {
"type": "date",
"format": "dd-MMM-yyyy"
}
}
}
}
}
Then, those fields are recognized as Date format:
result:
[
Related
This is the fourth time I attempt to do these mapping. But I do not hav my previous configuration files.
classes2.conf
input {
file {
path => "D:\Workspace.Elastic\FinalVersions\classes.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ["TITLE","PROFFESSOR","MAJOR","SEMESTER","student_count","unit","rating","submit_date","latitude","longitude"]
separator => ","
}
mutate {
convert => { "longitude" => "float" }
convert => { "latitude" => "float" }
rename => {
"longitude" => "[location][lon]"
"latitude" => "[location][lat]"
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "localhost:9200"
index => "geopointest"
}
}
classesRating_mapping2.json
{
"class": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
I get the following error:
[location] is defined as an object in mapping [doc] but this name is already used for a field in other types"
I have created geopointest index and added the json mapping like this:
curl -X PUT -H "Content-Type: application/json" http://localhost:9200/geopointest/class/_mapping --data-binary #classesRating_mapping2.json
What am I missing?
Many thanks.
The problem is that in your mapping you're using class which is a custom type name. That means you need to modify your elasticsearch output like this:
elasticsearch {
hosts => "localhost:9200"
index => "geopointest"
document_type => "class" <-- add this line
}
Without that line, Logstash uses the log type name and that was the reason why your mapping was off.
I have a logstash config file below. Elastic is reading my data as a b where as i want it to read it as ab i found i need to use not_analyzed for my sscat filed and max_shingle_size , min_shingle_size for products to get the best result.
Should I use not_analyzed for products field as well? Will that give better result?
How should I fill my my_id_analyzer to actually use the analyzer on different fields?
How should I connect the template with logstash config file?
input{
file{
path => "path"
start_position =>"beginning"
}
}
filter{
csv{
separator => ","
columns => ["Index", "Category", "Scat", "Sscat", "Products", "Measure", "Price", "Description", "Gst"]
}
mutate{convert => ["Index", "float"] }
mutate{convert => ["Price", "float"] }
mutate{convert => ["Gst", "float"] }
}
output{
elasticsearch{
hosts => "host"
user => "elastic"
password => "pass"
index => "masterdb"
}
}
I also have a template that can do it for all the future files that i upload
curl user:pass host:"host" /_template/logstash-id -XPUT -d '{
"template": "logstash-*",
"settings" : {
"analysis": {
"analyzer": {
"my_id_analyzer"{
}
}
}
}
},
"mappings": {
"properties" : {
"id" : { "type" : "string", "analyzer" : "my_id_analyzer" }
}
}
}'
You can use "ignore_above:" to restrict to a max length along with "not_analyzed" while creating mapping so that text doesn't get analyzed.
Declaring type as keyword instead of text will be other alternative for you.
Regarding the connecting template with logstash, why you need this? Once you have template created on elasticsearch, you can create your index which will follow the created template definition and you can start indexing.
I am learning ElasticSearch and have hit a block. I am trying to use logstash to load a simple CSV into ElasticSearch. This is the data, it is a postcode, longitude, latitude
ZE1 0BH,-1.136758103355,60.150855671143
ZE1 0NW,-1.15526666950369,60.1532197533966
I am using the following logstash conf file to filter the CSV to create a "location" field
input {
file {
path => "postcodes.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ["postcode", "lat", "lon"]
separator => ","
}
mutate { convert => {"lat" => "float"} }
mutate { convert => {"lon" => "float"} }
mutate { rename => {"lat" => "[location][lat]"} }
mutate { rename => {"lon" => "[location][lon]"} }
mutate { convert => { "[location]" => "float" } }
}
output {
elasticsearch {
action => "index"
hosts => "localhost"
index => "postcodes"
}
stdout { codec => rubydebug }
}
And I have added the mapping to ElasticSearch using the console in Kibana
PUT postcodes
{
"settings": {
"number_of_shards": 1
},
"mappings": {
"feature": {
"_all": { "enabled": true },
"properties": {
"postcode": {"type": "text"},
"location": {"type": "geo_point"}
}
}
}
}
I check the mappins for the index using
GET postcodes/_mapping
{
"postcodes": {
"mappings": {
"feature": {
"_all": {
"enabled": true
},
"properties": {
"location": {
"type": "geo_point"
},
"postcode": {
"type": "text"
}
}
}
}
}
}
So this all seems to be correct having looked at the documentation and the other questions posted.
However when i run
bin/logstash -f postcodes.conf
I get an error:
[location] is defined as an object in mapping [logs] but this name is already used for a field in other types
I have tried a number of alternative methods;
Deleted the index and the create a template.json and changed my conf file to have the extra settings:
manage_template => true
template => "postcode_template.json"
template_name =>"open_names"
template_overwrite => true
and this gets the same error.
I have managed to get the data loaded by not supplying a template however the data never gets loaded in as a geo_point so you cannot use the Kibana Tile Map to visualise the data
Can anyone explain why I am receiving that error and what method I should use?
Your problem is that you don't have a document_type => feature on your elasticsearch output. Without that, it's going to create the object on type logs which is why you are getting this conflict.
So the data I want to store in Elasticsearch is an csv file. It contains the following:
1465309033156,84,http://test.com/purchase.php,200,OK,ThreadGroup 1-3,true,6,6,84,testPC,0
1465309033469,176,http://test.com/,200,OK,ThreadGroup 1-7,true,7,7,176,testPC,91
Note that the first line equals the time in UNIX_MS.
I'm trying to send the data with logstash. Here is my config file:
input {
file {
sincedb_path => "NUL"
ignore_older => 0
type => "csv"
path => ["C:/result/piet.jtl"]
start_position => "beginning"
}
}
filter {
csv {
columns => ["#jmeter_timestamp", ...]
separator => ","
}
date {
locale => "en"
match => ["#jmeter_timestamp", "UNIX_MS"]
remove_field => ["#jmeter_timestamp"]
target => "#timestamp"
timezone => "Europe/Amsterdam"
}
}
output {
elasticsearch {
template => "c:/result/piet.json"
template_name => "piet"
hosts => ["192.168.43.51:9200"]
index => "piet-%{+YYYY.MM.dd}"
}
}
Here is part of piet.json:
"mappings": {
"logs": {
"properties": {
"#timestamp": {
"type": "date"
},
Now i'm getting an error running the config file.
"error"=>{
"type"=>"mapper_parsing_exception",
"reason"=>"failed to parse [#timestamp]",
"caused_by"=>{
"type"=>"illegal_argument_exception",
"reason"=>"Invalid format: \"2016-06-07T14:17:34.611Z\" is malformed at \"-06-07T14:17:34.611Z\""
}}}}, :level=>:warn}
My stdout looks like this:
I'm just lost. How can I insert data from the csv file in Elasticsearch..
Solved it myself.
Elasticsearch is complaining about the date, because it doesn't recognize the date correctly.
So, I removed the mapping completely from Elasticsearch via the Kopf plugin.
I re-added the .json, including the following details. (Note the format)
"mappings": {
"logs": {
"properties": {
"#timestamp": {
"type": "date",
"format" : "strict_date_optional_time||epoch_millis"
It is working now. Huray
I have a csv file which I'm tryng to upload to ES using Logstash. My conf file is as follows:
input {
file {
path => ["filename"]
start_position => "beginning"
}
}
filter {
csv {
columns => ["name1", "name2", "name3", ...]
separator => ","
}
}
filter {
mutate {
remove_field => ["name31", "name32", "name33"]
}
}
output {
stdout{
codec => rubydebug
}
elasticsearch {
action => "index"
host => "localhost"
index => "newindex"
template_overwrite => true
document_type => "newdoc"
template => "template.json"
}
}
My template file looks like the following:
{
"mappings": {
"newdoc": {
"properties": {
"name1": {
"type": "integer"
},
"name2": {
"type": "float"
},
"name3": {
"format": "dateOptionalTime",
"type": "date"
},
"name4": {
"index": "not_analyzed",
"type": "string"
},
....
}
}
},
"settings": {
"number_of_replicas": 0,
"number_of_shards": 1
},
"template": "newindex"
}
When I try to overwrite the default mapping, I get an 400 error even when I only try to write one line:
failed action with response of 400, dropping action: ["index", + ...
What can be the problem? Everything works fine if I don't overwrite the mapping but that is not a solution for me. I'm using Logstash 1.5.1 and Elasticsearch 1.5.0 on Red Hat.
Thanks
You should POST your request 'mapping' to elasticsearch before loading data in elasticsearch
POST mapping
You don't need to create the index before running logstash , It does create the index if you haven't yet , but it's better to create your own mapping before runing your conf file with logstash . Gives you more control over your field types etc.. Here is a simple tutorial on how to import csv to elasticsearch using logstash : http://freefilesdl.com/how-to-connect-logstash-to-elasticsearch-output