This is the output i have in my .config file for Logstash:
output {
elasticsearch {
document_type => "apache"
}}
When output is set to stdout {} it's working perfectly.
I have Elasticsearch running locally with no issues.
When i run the command:
sudo ./logstash -f sample_1.conf --debug
Logstash just hangs.
This is what i got in the logs using --debug:
[DEBUG] 2018-03-04 03:28:56.245 [pool-2-thread-2] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2018-03-04 03:28:56.255 [pool-2-thread-2] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2018-03-04 03:28:58.448 [[main]<file] file - _globbed_files: /usr/share/logstash/bin/log_file.txt: glob is: ["/usr/share/logstash/bin/log_file.txt"]
[DEBUG] 2018-03-04 03:28:59.353 [Ruby-0-Thread-12: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:617] pipeline - Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x34284e61#/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[DEBUG] 2018-03-04 03:29:01.268 [pool-2-thread-2] jvm - collector name {:name=>"ParNew"}
[DEBUG] 2018-03-04 03:29:01.268 [pool-2-thread-2] jvm - collector name {:name=>"ConcurrentMarkSweep"}
[DEBUG] 2018-03-04 03:29:04.354 [Ruby-0-Thread-12: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:617] pipeline - Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x34284e61#/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
These lines just repeat themselves again and again indefinitely.
Both Logstash and Elasticsearch versions are 6.2.2
What could be the problem?
From version 6.0, ElasticSearch removed document types or mapping types. You can remove the document_type and try again.
Refer to https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html for more details.
Related
I have been trying for some time to send a simple log to Elasticsearch and after trying a very simple example, the logs are not been sent to Elasticsearch from logstash.
Services: In same server for this test
Operative Sytem: Centos 7
The logstash version is: 7.17.1
The Elasticsearch version is: 7.17.1
/etc/logstash/conf.d
input {
file {
path => "/var/log/Elasticsearch/Elasticsearch.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
}
output {
Elasticsearch {
hosts => ["localhost:9200"]
}
}
/var/log/logstash/logstash-plain.log
[2022-03-18T11:33:30,690][INFO ][org.reflections.Reflections] Reflections took 118 ms to scan 1 urls, producing 119 keys and 417 values
[2022-03-18T11:33:32,042][INFO ][logstash.outputs.Elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["//localhost:9200"]}
[2022-03-18T11:33:32,540][INFO ][logstash.outputs.Elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2022-03-18T11:33:32,889][WARN ][logstash.outputs.Elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2022-03-18T11:33:32,908][INFO ][logstash.outputs.Elasticsearch][main] Elasticsearch version determined (7.17.1) {:es_version=>7}
[2022-03-18T11:33:32,913][WARN ][logstash.outputs.Elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2022-03-18T11:33:33,037][INFO ][logstash.outputs.Elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2022-03-18T11:33:33,113][INFO ][logstash.outputs.Elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2022-03-18T11:33:33,311][INFO ][logstash.outputs.Elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-03-18T11:33:33,337][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x15acb961 run>"}
[2022-03-18T11:33:34,573][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.23}
[2022-03-18T11:33:34,664][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-03-18T11:33:34,761][INFO ][filewatch.observingtail ][main][2b6c69038f817ebf29690e5d479fe4c6e56f482b9d6cc052978d217447903269] START, creating Discoverer, Watch with file and sincedb collections
[2022-03-18T11:33:34,771][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
/var/log/Elasticsearch/Elasticsearch.log
[2022-03-18T01:30:00,079][INFO ][o.e.x.m.MlDailyMaintenanceService] [ip-.eu-west-2.compute.internal] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2022-03-18T10:23:44,010][INFO ][o.e.c.m.MetadataIndexTemplateService] [ip-.eu-west-2.compute.internal] adding template [logstash] for index patterns [logstash-]
[2022-03-18T10:23:44,189][INFO ][o.e.c.m.MetadataCreateIndexService] [ip-.eu-west-2.compute.internal] [logstash-2022.03.18-000001] creating index, cause [api], templates [logstash], shards [1]/[1]
[2022-03-18T10:23:44,522][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [ip-.eu-west-2.compute.internal] adding index lifecycle policy [logstash-policy]
[2022-03-18T10:23:44,603][INFO ][o.e.x.i.IndexLifecycleTransition] [ip-.eu-west-2.compute.internal] moving index [logstash-2022.03.18-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [logstash-policy]
[2022-03-18T10:23:44,671][INFO ][o.e.x.i.IndexLifecycleTransition] [ip-.eu-west-2.compute.internal] moving index [logstash-2022.03.18-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [logstash-policy]
[2022-03-18T10:23:44,726][INFO ][o.e.x.i.IndexLifecycleTransition] [ip-.eu-west-2.compute.internal] moving index [logstash-2022.03.18-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [logstash-policy]
[2022-03-18T10:23:55,371][INFO ][o.e.c.r.a.DiskThresholdMonitor] [ip-*.eu-west-2.compute.internal] low disk watermark [85%] exceeded on [r51WwHrKTE-VK6UCAaR4IA][ip-*8.eu-west-2.compute.internal][/var/lib/Elasticsearch/nodes/0] free: 1.1gb[14.1%], replicas will not be assigned to this node
Any help would be really appreciated : )
You need to specify the index in the output.
Pay attention, that you have free space problem in the Elasticsearch: low disk watermark [85%] exceeded.
I am trying to read logs to load them into elasticsearch using logstash im running it in a RHEL 7.9 integrated with elasticsearch and Kibana, but when I run it it stops at:
[INFO ] 2021-10-22 13:40:00.704 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
My file config is:
file {
path => [
"/home/logstash/connectors.log"
]
start_position => "beginning"
}
}
filter {
grok {
break_on_match => false
match => {
"message" => "%{TIMESTAMP_ISO8601:fecha} \[(?<threadname>[^\]]+)\] %{LOGLEVEL:loglevel}\s*\(%{JAVAFILE:file}:%{INT:line}\)\s*-\s*Dato\s*a\s*enviar:\s*\[%{GREEDYDATA:xml}\]"
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:8087"]
document_id => "%{[#metadata][fingerprint]}"
index => "wilobank-%{+YYYY.MM.dd}"
}
}
And the log of execution is:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-10-22 13:48:29.490 [main] runner - Starting Logstash {"logstash.version"=>"7.15.1", "jruby.version"=>"jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 11.0.12+7 on 11.0.12+7 +indy +jit [linux-x86_64]"}
[WARN ] 2021-10-22 13:48:29.818 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-10-22 13:48:30.990 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-10-22 13:48:31.589 [Converge PipelineAction::Create<main>] Reflections - Reflections took 73 ms to scan 1 urls, producing 120 keys and 417 values
[WARN ] 2021-10-22 13:48:32.589 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-10-22 13:48:32.648 [Converge PipelineAction::Create<main>] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-10-22 13:48:32.970 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-10-22 13:48:33.021 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-10-22 13:48:33.085 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:8087"]}
[INFO ] 2021-10-22 13:48:33.418 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:8087/]}}
[WARN ] 2021-10-22 13:48:33.566 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:8087/"}
[INFO ] 2021-10-22 13:48:33.616 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.15.1) {:es_version=>7}
[WARN ] 2021-10-22 13:48:33.618 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[WARN ] 2021-10-22 13:48:33.705 [[main]-pipeline-manager] grok - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-10-22 13:48:33.741 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2021-10-22 13:48:33.880 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/home/logstash/test.conf"], :thread=>"#<Thread:0x7a96f84a run>"}
[INFO ] 2021-10-22 13:48:34.785 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.9}
[INFO ] 2021-10-22 13:48:34.840 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_a171bd20c3269483fada27f50b68caf2", :path=>["/home/logstash/itecban-connectors.log"]}
[INFO ] 2021-10-22 13:48:34.860 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-10-22 13:48:34.897 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2021-10-22 13:48:34.916 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Can someone help me?
Thanks in advance
If you want to parse again a complete file, you need to :
delete sindedb files
OR only delete the corresponding line in sincedb file
Then, restart Logstash. Logstash will reparse the file.
For more info: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#sincedb_path
I am using terraform 0.13.0 and trying to the kubernetes-alpha provider (https://github.com/hashicorp/terraform-provider-kubernetes-alpha). I download the plugin for Mac and copied the plugin to ~/.terraform.d/plugins dir
when I run terraform init it doesn't find the local plugin, instead it is trying to find from hashicorp site
terraform init
2020/08/21 16:42:58 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
2020/08/21 16:42:58 [INFO] Terraform version: 0.13.0
2020/08/21 16:42:58 [INFO] Go runtime version: go1.14.2
2020/08/21 16:42:58 [INFO] CLI args: []string{"<$HOME>/bin/terraform", "init"}
2020/08/21 16:42:58 [DEBUG] Attempting to open CLI config file: <$HOME>/.terraformrc
2020/08/21 16:42:58 Loading CLI configuration from <$HOME>/.terraformrc
2020/08/21 16:42:58 [DEBUG] checking for credentials in "<$HOME>/.terraform.d/plugins"
2020/08/21 16:42:58 [DEBUG] checking for credentials in "<$HOME>/.terraform.d/plugins/darwin_amd64"
2020/08/21 16:42:58 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/08/21 16:42:58 [DEBUG] will search for provider plugins in <$HOME>/.terraform.d/plugins
2020/08/21 16:42:58 [DEBUG] ignoring non-existing provider search directory <$HOME>/Library/Application Support/io.terraform/plugins
2020/08/21 16:42:58 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2020/08/21 16:42:58 [INFO] CLI command args: []string{"init"}
2020/08/21 16:42:58 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
Initializing modules...
2020/08/21 16:42:58 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
2020/08/21 16:42:58 [DEBUG] Module installer: begin app
- app in app
2020/08/21 16:42:58 [DEBUG] Module installer: app installed at app
2020/08/21 16:42:58 [DEBUG] Module installer: begin gke
- gke in gke
2020/08/21 16:42:58 [DEBUG] Module installer: gke installed at gke
2020/08/21 16:42:58 [DEBUG] Module installer: begin iam
2020/08/21 16:42:58 [DEBUG] Module installer: iam installed at iam
2020/08/21 16:42:58 [DEBUG] Module installer: begin vpc
- iam in iam
2020/08/21 16:42:58 [DEBUG] Module installer: vpc installed at vpc
Initializing the backend...
2020/08/21 16:42:58 [DEBUG] New state was assigned lineage "7541d58f-fc27-1b61-d496-834e76d1fcdb"
2020/08/21 16:42:58 [DEBUG] checking for provisioner in "."
Initializing provider plugins...
- Finding latest version of hashicorp/kubernetes-alpha...
2020/08/21 16:42:58 [DEBUG] checking for provisioner in "<$HOME>/bin"
2020/08/21 16:42:58 [DEBUG] checking for provisioner in "<$HOME>/.terraform.d/plugins"
2020/08/21 16:42:58 [DEBUG] checking for provisioner in "<$HOME>/.terraform.d/plugins/darwin_amd64"
2020/08/21 16:42:58 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2020/08/21 16:42:58 [WARN] Failed to scan provider cache directory .terraform/plugins: cannot search .terraform/plugins: lstat .terraform/plugins: no such file or directory
2020/08/21 16:42:58 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/08/21 16:42:58 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
2020/08/21 16:42:58 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/kubernetes-alpha/versions
- Finding latest version of hashicorp/google...
- Installing hashicorp/google v3.35.0...
- Installed hashicorp/google v3.35.0 (unauthenticated)
2020/08/21 16:42:59 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
2020/08/21 16:42:59 [DEBUG] GET https://registry.terraform.io/v1/providers/-/kubernetes-alpha/versions
Error: Failed to install provider
Error while installing hashicorp/kubernetes-alpha: provider registry
registry.terraform.io does not have a provider named
registry.terraform.io/hashicorp/kubernetes-alpha
Next I tried to force the plugin by adding a requires
terraform {
required_providers {
kubernetes-alpha = {
source = "localdomain/provider/kubernetes-alpha"
version = "0.1.0"
}
}
}
and copied the plugin to
$HOME/.terraform.d/plugins/localdomain/provider/kubernetes-alpha/0.1.0/darwin_amd64
Initializing provider plugins...
2020/08/21 16:54:41 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
- Finding localdomain/provider/kubernetes-alpha versions matching "0.1.0"...
- Finding latest version of hashicorp/google...
- Finding latest version of hashicorp/kubernetes-alpha...
2020/08/21 16:54:42 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
2020/08/21 16:54:42 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/kubernetes-alpha/versions
- Installing hashicorp/google v3.35.0...
- Installed hashicorp/google v3.35.0 (unauthenticated)
- Installing localdomain/provider/kubernetes-alpha v0.1.0...
- Installed localdomain/provider/kubernetes-alpha v0.1.0 (unauthenticated)
2020/08/21 16:54:42 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
2020/08/21 16:54:42 [DEBUG] GET https://registry.terraform.io/v1/providers/-/kubernetes-alpha/versions
Error: Failed to install provider
Error while installing hashicorp/kubernetes-alpha: provider registry
registry.terraform.io does not have a provider named
registry.terraform.io/hashicorp/kubernetes-alpha
I can't figure out why it is trying to find the plugin on registry rather than using local.
I am new to terraform and wondering if I am missing something basic.
You'll have to run terraform state replace-provider 'registry.terraform.io/-/kubernetes-alpha' 'localdomain/provider/kubernetes-alpha' in order to fix any legacy / non-namespaced providers. See the 0.13 upgrade guide here for more details.
I have script that logs temperature + humidity from diffrent sensors and stores the data from each sensor to his directory and every day a new log is made in this format YYYY-MM-DD.log.
${data_root}/A/0/*.log
${data_root}/A/1/*.log
ETC..
the logs are in this format:
2018-03-02 03:48:14 25.00 27.10
(YYYY-MM-DD TIME Temperature Humidity)
I had trouble with understanding how to correctly config my logstash instance, I figured that my input should look something like this:
input {
file{ path => "/var/wlogs/a1/*.log" type=>"a1"}
file{ path => "/var/wlogs/a2/*.log" type=>"a2"}
etc..
}
and the filter should look something like this:
filter{
if [type] == "a1" {
grok {
match => { "message" => "(?<timestamp>%{YEAR}-%{MONTHNUM:month}-%{MONTHDAY:day} %{TIME}) %{NUMBER:temperature:float} %{NUMBER:humidity:float}" }
}
}
if [type] == "a2" {....}
Im trying to export the the data in the output section to ElasticSearch with no success.
output{
elasticsearch { hosts =>["ec2-xxxxxx.eu-west-2.compute.amazonaws.com:9200"] user=>"elastic" password=>"pass" index=>"{type}"}
stdout{ codec => rubydebug}
}
here is the console output when I try to run it:
ubuntu#ip-xxx-xxx:/usr/share/logstash$ sudo bin/logstash -f ~/logstash.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-03-02 13:43:34.633 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-03-02 13:43:34.647 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2018-03-02 13:43:35.063 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-03-02 13:43:35.209 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.2"}
[INFO ] 2018-03-02 13:43:35.430 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-03-02 13:43:36.145 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2018-03-02 13:43:36.318 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx#ec2-no.eu-west-2.compute.amazonaws.com:9200/]}}
[INFO ] 2018-03-02 13:43:36.327 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx#ec2-no.eu-west-2.compute.amazonaws.com:9200/, :path=>"/"}
[WARN ] 2018-03-02 13:43:36.447 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://elastic:xxxxxx#ec2-3no3.eu-west-2.compute.amazonaws.com:9200/"}
[INFO ] 2018-03-02 13:43:36.610 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>nil}
[WARN ] 2018-03-02 13:43:36.611 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2018-03-02 13:43:36.616 [[main]-pipeline-manager] elasticsearch - Using mapping template from {:path=>nil}
[INFO ] 2018-03-02 13:43:36.619 [[main]-pipeline-manager] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[INFO ] 2018-03-02 13:43:36.626 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//ec2-no.eu-west-2.compute.amazonaws.com:9200"]}
[INFO ] 2018-03-02 13:43:37.054 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x25b5f422#/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}
[INFO ] 2018-03-02 13:43:37.081 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
please help me figure out what I'm doing wrong and how to fix it :)
thanks in advance
P.S: Im using the latest versions of ElasticSearch, Kibana and Logstash
Don't see any error in the logs. Makes me think that the log files might have already been read in a previous attempt. Since the file offsets are maintained in the sincedb file in home directory, can you stop logstash, delete the file and try again?
For more details about the sincedb file, refer to https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
I'm currently working on a PoC ELK installation and I'd like to re-send every log line of a file which is registered in Filebeat for testing purposes.
This is what I do:
I stop Filebeat
I delete the index in Logstash through Kibana
I delete the Filebeat registry file
I start Filebeat
In Kibana I can see that twice as many events are there as log lines, and I can also see that every event is duplicated once.
Why is that?
Filebeat logs:
2017-05-05T14:25:16+02:00 INFO Setup Beat: filebeat; Version: 5.2.2
2017-05-05T14:25:16+02:00 INFO Max Retries set to: 3
2017-05-05T14:25:16+02:00 INFO Activated logstash as output plugin.
2017-05-05T14:25:16+02:00 INFO Publisher name: anonymized
2017-05-05T14:25:16+02:00 INFO Flush Interval set to: 1s
2017-05-05T14:25:16+02:00 INFO Max Bulk Size set to: 2048
2017-05-05T14:25:16+02:00 INFO filebeat start running.
2017-05-05T14:25:16+02:00 INFO No registry file found under: /var/lib/filebeat/registry. Creating a new registry file.
2017-05-05T14:25:16+02:00 INFO Loading registrar data from /var/lib/filebeat/registry
2017-05-05T14:25:16+02:00 INFO States Loaded from registrar: 0
2017-05-05T14:25:16+02:00 INFO Loading Prospectors: 1
2017-05-05T14:25:16+02:00 INFO Prospector with previous states loaded: 0
2017-05-05T14:25:16+02:00 INFO Loading Prospectors completed. Number of prospectors: 1
2017-05-05T14:25:16+02:00 INFO All prospectors are initialised and running with 0 states to persist
2017-05-05T14:25:16+02:00 INFO Starting Registrar
2017-05-05T14:25:16+02:00 INFO Start sending events to output
2017-05-05T14:25:16+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-05-05T14:25:16+02:00 INFO Starting prospector of type: log
2017-05-05T14:25:16+02:00 INFO Harvester started for file: /some/where/anonymized.log
2017-05-05T14:25:46+02:00 INFO Non-zero metrics in the last 30s: registrar.writes=2 libbeat.logstash.publish.read_bytes=54 libbeat.logstash.publish.write_bytes=32390 libbeat.logstash.published_and_acked_events=578 filebeat.harvester.running=1 registar.states.current=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.publisher.published_events=578 publish.events=579 filebeat.harvester.started=1 registrar.states.update=579 filebeat.harvester.open_files=1
2017-05-05T14:26:16+02:00 INFO No non-zero metrics in the last 30s
Deleting the registry file created the problem.
Filebeat management the state of a file and the ACK of the event with the prospector(in memory) and with the Registry File(persisted in disk).
Please read the documentation Here
You can management the _id field of each event by yourself, so that any event that is duplicated (for any reason, even in production environment) will not have two of them in elasticsearch, but will update the event.
Create the following configuration in your logstash pipeline config file.
#if your logs don't have a unique ID, use the following to generate one
fingerprint{
#with the message field or choose other(s) that can give you a uniqueID
source => ["message"]
target => "LogID"
key => "something"
method => "MD5"
concatenate_sources => true
}
#in your output section
elasticsearch{
hosts => ["localhost:9200"]
document_id => "%{LogID}"
index => "yourindex"
}