I'm using logstash (v2.3.3-1) to index about ~800k documents from S3 to AWS ElasticSearch, and some documents are being indexed 2 or 3 times, instead of only once.
The files are static (nothing is updating them or touching them) and they're very small (each is roughly 1.1KB).
The process takes a very long time to run on a t2.micro (~1day).
The config I'm using is:
input {
s3 {
bucket => "$BUCKETNAME"
codec => "json"
region => "$REGION"
access_key_id => '$KEY'
secret_access_key => '$KEY'
type => 's3'
}
}
filter {
if [type] == "s3" {
metrics {
meter => "events"
add_tag => "metric"
}
}
}
output {
if "metric" in [tags] {
stdout { codec => line {
format => "rate: %{[events][rate_1m]}"
}
}
} else {
amazon_es {
hosts => [$HOST]
region => "$REGION"
index => "$INDEXNAME"
aws_access_key_id => '$KEY'
aws_secret_access_key => '$KEY'
document_type => "$TYPE"
}
stdout { codec => rubydebug }
}
}
I've run this twice now with the same problem (into different ES indices) and the files that are being indexed >1x are different each time.
Any suggestions gratefully received!
The s3 input is very fragile. It records the time of the last file processed, so any files that share the same timestamp will not be processed and multiple logstash instances cannot read from the same bucket. As you've seen, it's also painfully slow to determine which files to process (a good portion of the blame goes to amazon here).
s3 only works for me when I use a single logstash to read the files, and then delete (or backup to another bucket/folder) the files to keep the original bucket as empty as possible, and then setting sincedb_path to /dev/null.
Related
I am working on an ELK stack setup I want to import data from a csv file from my PC to elasticsearch via logstash. Elasticsearch and Kibana is working properly.
Here is my logstash.conf file:
input {
file {
path => "C:/Users/aron/Desktop/es/archive/weapons.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => ["name", "type", "country"]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200/"]
index => "weapons"
document_type => "ww2_weapon"
}
stdout {}
}
And a sample row data from my .csv file looks like this:
Name
Type
Country
10.5 cm Kanone 17
Field Gun
Germany
German characters are all showing up.
I am running logstash via: logstash.bat -f path/to/logstash.conf
It starts working but it freezes and becomes unresponsive along the way, here is a screenshot of stdout
In kibana, it created the index and imported 2 documents but the data is all messed up. What am I doing wrong?
If your task is only to import that CSV you better use the file upload in Kibana.
Should be available under the following link (for Kibana > v8):
<your Kibana domain>/app/home#/tutorial_directory/fileDataViz
Logstash is used if you want to do this job on a regular basis with new files coming in over time.
You can try with below one. It is running perfectly on my machine.
input {
file {
path => "path/filename.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
filter {
csv {
separator => ","
columns => ["field1","field2",...]
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "https://localhost:9200"
user => "username" ------> if any
password => "password" ------> if any
index => "indexname"
document_type => "doc_type"
}
}
I am trying to send my Elasticsearch data to S3 using Logstash.
In the process, I have multiple ES indices which reside in the input filter of the Logstash.
I want to have different files for different indices with their name as file names.
Input Filter:-
input {
elasticsearch {
hosts => "localhost"
index => "index1,index2,index3,index4,index5,index6" #Multiple indices as input
query => '
{
"query": {
"match_all": {}
}
}
'
}
}
Now I want to save the output of all the indices in a file with index as the filename.
Like : index1 ---> index1.txt
Output filter:-
output {
s3 {
region => *********
bucket => ***********
size_file => ********
codec => "json"
encoding => "gzip"
prefix => "data/(indexName)/%{+YYYY}/%{+MM}/%{+dd}"
}
}
Use the docinfo => true setting in the Elasticsearch input to add #metadata fields about the source document. The setting is documented here. The field [#metadata][_index] will then be available for use in the output plugin.
An example based on your configuration:
input {
elasticsearch {
hosts => "localhost"
index => "index1,index2,index3,index4,index5,index6"
}
}
output {
s3 {
region => *********
bucket => ***********
size_file => ********
codec => "json"
encoding => "gzip"
prefix => "data/%{[#metadata][_index]}/%{+YYYY}/%{+MM}/%{+dd}"
}
}
The two important changes I've made in this config, compared to yours, are:
Enabled docinfo in the input plugin
Replaced (index) with %{[#metadata][_index]} in the output plugin
For readability I also pruned the match_all query.
I am relatively new to the whole of the ELK set up part, hence please bear along.
What I want to do is send the cloudtrail logs that are stored on S3 into a locally hosted (non-AWS I mean) ELK set up. I am not using Filebeat anywhere in the set up. I believe it isn't mandatory to use it. Logstash can directly deliver data to ES.
Am I right here ?
Once the data is in ES, I would simply want to visualize it in Kibana.
What I have tried so far, given that my ELK is up and running and that there is no Filebeat involved in the setup:
using the S3 logstash plugin
contents of /etc/logstash/conf.d/aws_ct_s3.conf
input {
s3 {
access_key_id => "access_key_id"
bucket => "bucket_name_here"
secret_access_key => "secret_access_key"
prefix => "AWSLogs/<account_number>/CloudTrail/ap-southeast-1/2019/01/09"
sincedb_path => "/tmp/s3ctlogs.sincedb"
region => "us-east-2"
codec => "json"
add_field => { source => gzfiles }
}
}
output {
stdout { codec => json }
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "attack-%{+YYYY.MM.dd}"
}
}
When logstash is started with the above conf, I can see all working fine. Using the head google chrome plugin, I can see that the documents are continuously getting added to the specified index.In fact when I browse it as well, I can see that there is the data I need. I am able to see the same on the Kibana side too.
The data that each of these gzip files have is of the format:
{
"Records": [
dictionary_D1,
dictionary_D2,
.
.
.
]
}
And I want to have each of these dictionaries from the list of dictionaries above to be a separate event in Kibana. With some Googling around I understand that I could use the split filter to achieve what I want to. And now my aws_ct_s3.conf looks something like :
input {
s3 {
access_key_id => "access_key_id"
bucket => "bucket_name_here"
secret_access_key => "secret_access_key"
prefix => "AWSLogs/<account_number>/CloudTrail/ap-southeast-1/2019/01/09"
sincedb_path => "/tmp/s3ctlogs.sincedb"
region => "us-east-2"
codec => "json"
add_field => { source => gzfiles }
}
}
filter {
split {
field => "Records"
}
}
output {
stdout { codec => json }
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "attack-%{+YYYY.MM.dd}"
}
}
And with this I am in fact getting the data as I need on Kibana.
Now the problem is
Without the filter in place, the number of documents that were being shipped by Logstash from S3 to Elasticsearch was in GBs, while after applying the filter it has stopped at roughly some 5000 documents alone.
I do not know what am I doing wrong here. Could someone please help ?
Current config:
java -XshowSettings:vm => Max Heap Size: 8.9 GB
elasticsearch jvm options => max and min heap size: 6GB
logstash jvm options => max and min heap size: 2GB
ES version - 6.6.0
LS version - 6.6.0
Kibana version - 6.6.0
This is how the current heap usage looks like:
I have issue with my logstash s3 iinput. The last messages I see in my kibana iinterface is from several days earlier In fact I have an AWS elb with logs enable. I've tested from command line and I can see that logsstash is continuously processing inputs, and never outputs. In the elb s3 bucket there is one folder per day/per month/per year and each folder contains several log files and with a total size of arround 60GB.
It was working fine at the begining, but as logs grow, it become slow, and now I'm seeing my logs in the outpiut size. Logstah is keeping doing input task, filter, and never output logs.
I created a dedicated configuration file for test with only s3 as input, and test in a dedicated machine from command line :
/opt/logstash/bin/logstash agent -f /tmp/s3.conf --debug 2>&1 | tee /tmp/logstash.log
```
the s3.conf file :
```
admin#ip-10-3-27-129:~$ cat /tmp/s3.conf
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# !!!!!!!!! This file is managed by SALT !!!!!!!!!
# !!!!!!!!! All changes will be lost !!!!!!!!!
# !!!!!!!!! DO NOT EDIT MANUALLY ! !!!!!!!!!
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
#--[ INPUT ]----------------------------------------------------------------
input
{
# Logs ELB API
s3
{
bucket => "s3.prod.elb.logs.eu-west-1.mydomain"
prefix => "rtb/smaato/AWSLogs/653589716289/elasticloadbalancing/"
interval => 30
region => "eu-west-1"
type => "elb_access_log"
}
}
#--[ FILTER ]---------------------------------------------------------------
filter
{
# Set the HTTP request time to #timestamp field
date {
match => [ "timestamp", "ISO8601" ]
remove_field => [ "timestamp" ]
}
# Parse the ELB access logs
if [type] == "elb_access_log" {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp:date} %{HOSTNAME:loadbalancer} %{IP:client_ip}:%{POSINT:client_port:int} (?:%{IP:backend_ip}:%{POSINT:backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:backend_status_code:int} %{INT:received_bytes:int} %{INT:sent_bytes:int} %{INT:sent_bytes_ack:int} \"%{WORD:http_method} %{URI:url_asked} HTTP/%{NUMBER:http_version}\" \"%{GREEDYDATA:user_agent}\" %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol}" ]
remove_field => [ "message" ]
}
kv {
field_split => "&?"
source => "url_asked"
}
date {
match => [ "timestamp", "ISO8601" ]
remove_field => [ "timestamp" ]
}
}
# Remove the filebeat input tag
mutate {
remove_tag => [ "beats_input_codec_plain_applied" ]
}
# Remove field tags if empty
if [tags] == [] {
mutate {
remove_field => [ "tags" ]
}
}
# Remove some unnecessary fields to make Kibana cleaner
mutate {
remove_field => [ "#version", "count", "fields", "input_type", "offset", "[beat][hostname]", "[beat][name]", "[beat]" ]
}
}
#--[ OUTPUT ]---------------------------------------------------------------
output
#{
# elasticsearch {
# hosts => ["10.3.16.75:9200"]
# }
#}
{
# file {
# path => "/tmp/logastash/elb/elb_logs.json"
# }
stdout { codec => rubydebug }
}
And I can see input processing, filter, and the message like "will start output worker....." but not output event received, never.
I created a new folder (named test_elb) on the bucket, and copy logs from one day folder (31/12/2016 for example) into it, and then set the new created as prefix in my input configuration like this :
s3
{
bucket => "s3.prod.elb.logs.eu-west-1.mydomain"
prefix => "rtb/smaato/AWSLogs/653589716289/test_elb/"
interval => 30
region => "eu-west-1"
type => "elb_access_log"
}
And with that s3 prefix, logstash is doing all the pipeline processing (input, filter, output) as expecting, and I see my logs outputs.
so for me it seems like the bucket is to large and losgstash-s3 plugin has difficult to process it.
Can someone here advise on that problematic please?
My logstash version : 2.2.4
Operating system: Debian Jessie
I've search and ask in the discuss.elastic forum, in the elasticseach IRC chan, and no real solution.
Do you thing it could be a bucket size matter
Thanks for the help.
Regards.
Configure the s3 input plugin to move files to a bucket/path not considered in the input once processed.
While there are many files in the input bucket/path you may need to run logstash on a subset of the data until it moves the files to the processing bucket/path.
This is what I'm doing to process about .5GiB (several hundred thousand) of files per day. Logstash will pull all of the object names prior to doing any inserts so it will appear that the process is stuck if you have a huge amount of files in your bucket.
bucket => "BUCKET_NAME"
prefix => "logs/2017/09/01"
backup_add_prefix => "sent-to-logstash-"
backup_to_bucket => "BUCKET_NAME"
interval => 120
delete => true
I'm not certain how durable the process is against data loss between the bucket moves, but for logs which aren't mission critical, this process is highly efficient considering the amount of files being moved.
This behaviour is controlled by the watch_for_new_files parameter. In the default, true, setting logstash will not process the existing files, and will wait for new files to arrive.
Example:
input {
s3
{
bucket => "the-bucket-name"
prefix => "the_path/ends_with_the_slash/"
interval => 30
region => "eu-west-1"
type => "elb_access_log"
watch_for_new_files => false
}
}
output {
stdout{}
}
I'm attempting to use Logstash to collect sales data for multiple clients from multiple vendors.
So far i've an S3 (Inbox) bucket that I can drop my files (currently CSVs) into and according the client code prefix on the file, the data gets pushed into an Elastic index for each client. This all works nicely.
The problem I have is that I have data from multiple vendors and need a way of identifying which file is from which vendor. Adding an extra column to the CSV is not an option, so my plan was to add this to the filename, so i'd end up with a filenaming convention something like clientcode_vendorcode_reportdate.csv.
I may be missing something, but it seems that on S3 I can't get access to the filename inside my config, which seems crazy given that the prefix is clearly being read at some point. I was intending to use a Grok or Ruby filter to simply split the filename on the underscore, giving me three key variables that I can use in my config, but all attempts so far have failed. I can't even seem to get the full source_path or filename as a variable.
My config so far looks something like this - i've removed failed attempts at using Grok/Ruby filters.
input {
s3 {
access_key_id =>"MYACCESSKEYID"
bucket => "sales-inbox"
backup_to_bucket => "sales-archive"
delete => "true"
secret_access_key => "MYSECRETACCESSKEY"
region => "eu-west-1"
codec => plain
prefix => "ABC"
type => "ABC"
}
s3 {
access_key_id =>"MYACCESSKEYID"
bucket => "sales-inbox"
backup_to_bucket => "sales-archive"
delete => "true"
secret_access_key => "MYSECRETACCESSKEY"
region => "eu-west-1"
codec => plain
prefix => "DEF"
type => "DEF"
}
}
filter {
if [type] == "ABC" {
csv {
columns => ["Date","Units","ProductID","Country"]
separator => ","
}
mutate {
add_field => { "client_code" => "abc" }
}
}
else if [type] == "DEF" {
csv {
columns => ["Date","Units","ProductID","Country"]
separator => ","
}
mutate {
add_field => { "client_code" => "def" }
}
}
mutate
{
remove_field => [ "message" ]
}
}
output {
elasticsearch {
codec => json
hosts => "myelasticcluster.com:9200"
index => "sales_%{client_code}"
document_type => "sale"
}
stdout { codec => rubydebug }
}
An guidance from those well versed in Logstash configs would be much appreciated!