I have some indexes in one of my machines. I need to copy them to another machine, how can i do that in elasticsearch.
I did get some good documentation here, but since im an newbie to elasticsearch ecosystem and since im toying with lesser data indices, I thought I would use some plugins or ways which would be less time consuming.
I would use Logstash with an elasticsearch input plugin and an elasticsearch output plugin.
After installing Logstash, you can create a configuration file copy.conf that looks like this:
input {
elasticsearch {
hosts => "localhost:9200" <--- source ES host
index => "source_index"
}
}
filter {
mutate {
remove_field => [ "#version", "#timestamp" ] <--- remove added junk
}
}
output {
elasticsearch {
host => "localhost" <--- target ES host
port => 9200
protocol => "http"
manage_template => false
index => "target_index"
document_id => "%{id}" <--- name of your ID field
workers => 1
}
}
And then after setting the correct values (source/target host + source/target index), you can run this with bin/logstash -f copy.conf
I can see 3 options here
Snapshot/Restore - You can move your data across geographical locations.
Logstash reindex - As pointed out by Val
Stream2ES - This is a more simpler solution
You can use Snapshot and restore feature as well, where you can take snapshot (backup) of one index and then can Restore to somewhere else.
Just have a look at
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
Related
I’m relatively new to Kibana and the ELK (Elasticsearch, Logstash and Kibana) stack and I’ve been doing pretty well setting one up, but I have run into what I see as an odd issue and need some help understanding what’s happening.
I’m using the ELK stack to crunch some Apache logs but I have my own custom type settings. So I need to explicitly specify field types and such instead of having Logstash (or is it Kibana?) guess what the data mapping would be.
From reading the Logstash documentation, it seems pretty clear that I can set the template value in the output.elasticsearch chunk of config shown here:
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-apache"
document_id => "%{[#metadata][fingerprint]}"
manage_template => false
template => "/path/to/logstash/logstash-apache.json"
template_name => "logstash-apache"
template_overwrite => true
}
stdout {
codec => rubydebug
}
}
100% sure I have the correct path set. But for some reason, if I use this, launch Logstash and let it do it’s things, the mappings I have specified in logstash-apache.json don’t show up. The index in Kibana is logstash-apache as well so this should work right?
So what I do now is preload the mappings template directly into Elasticsearch like this:
curl -ss -XPUT "http://localhost:9200/_template/logstash-apache/" -H 'Content-Type: application/json' -d #"/path/to/logstash/logstash-apache.json";
And it clearly works well and the data gets proper mapping… But doing something like this is fairly clunky. It would be cleaner to just have it all come from the logstash-apache.conf file I have setup.
So what am I doing wrong? What can I do to have my custom mappings template be used via that logstash-apache.conf without having to jump through the extra hoop of a curl command?
The problem is that you have set manage_template to false, which completely disables this template creation feature and requires you to create the template manually like you're doing right now.
So your output section should look like this instead and you should be good to go:
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-apache"
document_id => "%{[#metadata][fingerprint]}"
manage_template => true <-- change this line
template => "/path/to/logstash/logstash-apache.json"
template_name => "logstash-apache"
template_overwrite => true
}
Hi guy i have issue with updating multiple records in elastic using logstash.
My logstash configuration is bellow
output {
elasticsearch {
hosts => "******"
user => "xxxxx"
password => "yyyyyy"
index => "index_name"
document_type => "doc_type"
action => "update"
script_lang => "painless"
script_type => "inline"
document_id => "%{Id}"
script => 'ctx._source.Tags = params.event.get("Tags");'
}
}
My output to logstash dump folder looks like:
{"index_name":"feed_name","doc_type":"doc_type","Id":["b504d808-f82d-4eaa-b192-446ec0ba487f", "1bcbc54f-fa7a-4079-90e7-71da527f56a5"],"es_action":"update","Tags": ["tag1","tag2"]}
My biggest issue here is that I am not able to update those two recods at once but I need to create two records each with different ID.
Is there a why to solve this by writing query in my output configuration?
In sql that would look someting like this:
Update Table
SET Tags
WHERE ID in (guid1, guid2)
I know that in this case I can add two records in logstash and problem solved but I need to solve second issue where I need to replace all records that have one tag1 and give it newTag.
Have you considered to use the split filter in order to clone the event in events with one id each one? It seems the filter can help you.
I'm very new to the elasticsearch-kibana-logstash and can't seem to find solution for this one.
I'm trying to create index that I will see in kibana without having to use the POST command in Dev Tools section.
I have set test.conf-
input {
file {
path => "/home/user/logs/server.log"
type => "test-type"
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "new-index"
}
}
and then
bin/logstash -f test.conf from logstash directory
what i get is that I can't find the new-index in kibana (index patterns section), when I use elasticsearch - http://localhost:9200/new-index/ it presents an error and when I go to http://localhost:9600/ (the port it's showing) it doesn't seem to have any errors
Thanks a lot for the help!!
It's obvious that you won't be able to find the index which you've created using logstash in Kibana, unless you're manually creating it there within the Managemen section of Kibana.
Make sure, that you have the same name of the indice which you created using logstash. Have a look at the doc, which conveys:
When you define an index pattern, indices that match that pattern must
exist in Elasticsearch. Those indices must contain data.
which pretty much says that the indice should exist for you to create the index in Kibana. Hope it helps!
I have actually succeeded to create index even without first creating it in Kibana
I used the following config file -
input {
file {
path => "/home/hadar/tmp/logs/server.log"
type => "test-type"
id => "NEWTRY"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} - %{LOGLEVEL:level} - %{WORD:scriptName}.%{WORD:scriptEND} - " }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "new-index"
codec => line { format => "%{year}-%{month}-%{day} %{hour}:%{minute}:%{second} - %{level} - %{scriptName}.%{scriptEND} - \"%{message}\"" }
}
}
I made sure that the index wasn't already in Kibana (I tried with other indexes names too just to be sure...) and eventually I did see the index with the log's info in both Kibana (I added it in the index pattern section) and Elasticsearch when I went to http://localhost:9200/new-index
The only thing I should have done was to erase the .sincedb_XXX files which are created under data/plugins/inputs/file/ after every Logstash run
OR
the other solution (for tests environment only) is to add sincedb_path=>"/dev/null" to the input file plugin which indicates to not create the .sincedb_XXX file
You can create directly index in elastic search using https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
and these indices can be used in Kibana.
I have configured logstash to get input from one filebeat port.
Filebeat configured with two different paths. is it possible to display logs to two different index?
Logstash input part:
input{
beats
{
type => "stack"
port => 5044
}
Filebeat input part :
prospectors:
paths:
- E://stack/**/*.txt
- E://test/**/*.txt
Now i need to display "stack" in one index and "test" in other index.
How to configure logstash output part?
What you can do is to use the knowledge of the type property in order to decide in which index to store the log being processed.
So your elasticsearch output could simply look like this, i.e. depending on the type value, the selected index will be different.
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{type}"
}
}
I would like to create two separate indexes for two different systems that are sending data to the logstash server setup for udp - syslog. In Elasticsearch, I created an Index called CiscoASA01 and another Index called CiscoASA02. How can I configure Logstash to filter all events coming from the first device to go into the CiscoASA01 index and the events coming from the second device to go to the second index? Thank you.
You can use if to separate the logs. Assume your first device is CiscoASA01 & second is CiscoASA02.
Here is the output
output {
if [host] == "CiscoASA01"
{
elasticsearch {
host => "elasticsearch_server"
index => "CiscoASA01"
}
}
if [host] == "CiscoASA02"
{
elasticsearch {
host => "elasticsearch_server"
index => "CiscoASA02"
}
}
}
The [host] is the field in logstash event. You can use it to separate the log to different output.
Hope this can help you.