I have following k8s manifest:
apiVersion: v1
kind: Service
metadata:
name: springboot-k8s-svc
spec:
selector:
app: spring-boot-k8s
ports:
- protocol: "TCP"
port: 8080
targetPort: 8080
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: spring-boot-k8s
spec:
selector:
matchLabels:
app: spring-boot-k8s
replicas: 1
template:
metadata:
labels:
app: spring-boot-k8s
spec:
containers:
- name: spring-boot-k8s
image: springboot-k8s-example:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
Currently I have following things running in my minikube:
$ kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h52m
I start my dummy spring boot application:
$ kubectl apply -f deployment-n-svc.yaml
service/springboot-k8s-svc created
deployment.apps/spring-boot-k8s created
This app seem to start as desired appropriately:
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/spring-boot-k8s-bccc4c557-7wbrn 1/1 Running 0 5s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h53m
service/springboot-k8s-svc NodePort 10.99.136.27 <none> 8080:30931/TCP 5s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/spring-boot-k8s 1/1 1 1 5s
NAME DESIRED CURRENT READY AGE
replicaset.apps/spring-boot-k8s-bccc4c557 1 1 1 5s
When I try to hit REST end point, I get the desired output:
$ curl http://192.168.49.2:30931/message
OK!
Now I tried to autoscale the app:
$ kubectl autoscale deployment spring-boot-k8s --min=1 --max=5 --cpu-percent=10
horizontalpodautoscaler.autoscaling/spring-boot-k8s autoscaled
Started watching the hpa just started as shown below command. It seems to have started:
$ watch -n 1 kubectl get hpa
Every 1.0s: kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLIC
AS AGE
horizontalpodautoscaler.autoscaling/spring-boot-k8s Deployment/spring-boot-k8s <unknown>/10% 1 5 1
8m10s
Then I tried apache bench HTTP load test utility to create load on the spring boot server to check if k8s increases the number of pods:
$ab -n 1000000 -c 100 http://192.168.49.2:30931/message
However this did not increase number of pods. What I am missing?
PS:
When I kill ab command in with Ctrl+C, it gives following output (notice aprox 5s processing time per request):
$ ab -n 1000000 -c 100 http://192.168.49.2:32215/message
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.49.2 (be patient)
^C
Server Software:
Server Hostname: 192.168.49.2
Server Port: 32215
Document Path: /message
Document Length: 4 bytes
Concurrency Level: 100
Time taken for tests: 35.650 seconds
Complete requests: 601
Failed requests: 0
Total transferred: 81736 bytes
HTML transferred: 2404 bytes
Requests per second: 16.86 [#/sec] (mean)
Time per request: 5931.751 [ms] (mean)
Time per request: 59.318 [ms] (mean, across all concurrent requests)
Transfer rate: 2.24 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.5 0 7
Processing: 5001 5004 4.7 5003 5022
Waiting: 5000 5004 3.9 5002 5019
Total: 5001 5006 5.4 5003 5024
Percentage of the requests served within a certain time (ms)
50% 5003
66% 5005
75% 5007
80% 5009
90% 5013
95% 5020
98% 5023
99% 5023
100% 5024 (longest request)
Update
As asked in comments, here is output of some more commands:
$ kubectl describe hpa spring-boot-k8s
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name: spring-boot-k8s
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Fri, 03 Feb 2023 01:58:06 +0530
Reference: Deployment/spring-boot-k8s
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 10%
Min replicas: 1
Max replicas: 5
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 35m (x500 over 16h) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Notice what it says: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler.
Also it says: no metrics returned from resource metrics API. Thought my metric server is running:
$ kubectl get deployment -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
coredns 1/1 1 1 304d
metrics-server 1/1 1 1 40h
This seems to be the reason why its not working. But what could be the reason?
Apart from that the CPU utilization is not also increasing much. Till yesterday night, I used to see max 3% CPU utilization in the output of command watch kubectl top node:
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
minikube 141m 1% 1127Mi 7%
But now it shows following error:
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
I have installed Filebeat-oss 7.12.0 and opensearch-2.4.0 and opensearchDashboard-2.4.0 on Windows.
Every service is working fine.
But index is not getting created in Opensearch dashboard.
There is no error.
Logs are:
INFO log/harvester.go:302 Harvester started for file: D:\data\logs.txt
2022-12-08T18:28:17.584+0530 INFO [crawler] beater/crawler.go:141 Starting input (ID: 16780016071726099597)
2022-12-08T18:28:17.585+0530 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 2
2022-12-08T18:28:17.585+0530 INFO cfgfile/reload.go:164 Config reloader started
2022-12-08T18:28:17.584+0530 INFO [input.filestream] compat/compat.go:111 Input filestream starting
2022-12-08T18:28:17.585+0530 INFO cfgfile/reload.go:224 Loading of config files completed.
2022-12-08T18:28:20.428+0530 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2022-12-08T18:28:21.428+0530 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(http://localhost:9200))
2022-12-08T18:28:21.428+0530 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-12-08T18:28:21.428+0530 INFO [publisher] pipeline/retry.go:223 done
2022-12-08T18:28:21.433+0530 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 2.4.0
2022-12-08T18:28:21.537+0530 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 2.4.0
2022-12-08T18:28:21.620+0530 INFO template/load.go:117 Try loading template filebeat-7.12.0 to Elasticsearch
filebeat.yml is:
filebeat.inputs:
- type: log
paths:
- D:\data\*
- type: filestream
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- D:\data\*
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#============================== Kibana =====================================
setup.kibana:
host: "localhost:5601"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
I don't know what the problem is. No index is created in Opensearch dashboard with name filebeat-7.12.0.
#Android see my reply on this thread: https://stackoverflow.com/a/74984260/6101900.
You cannot forward events from filebeat to opensearch since its not elasticsearch.
I 'm trying to run filebeat on windows 10 and send to data to elasticsearch and kibana all on localhost. This is my config file filebeat.yml
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- D:\AppData\Elastic\filebeat\logs
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
### Multiline options
# Multiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
#multiline.negate: false
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#multiline.match: after
# filestream is an input for collecting log messages from files. It is going to replace log input in the future.
- type: filestream
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /var/log/*.log
- D:\AppData\Elastic\filebeat\logs
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#prospector.scanner.exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "localhost:5601"
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================
# Instrumentation support for the filebeat.
#instrumentation:
# Set to true to enable instrumentation of filebeat.
#enabled: false
# Environment in which filebeat is running on (eg: staging, production, etc.)
#environment: ""
# APM Server hosts to report instrumentation results to.
#hosts:
# - http://localhost:8200
# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:
# Secret token for the APM Server(s).
#secret_token:
# ================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
i 've run
./filebeat -c filebeat.yml -e
This is the result
2021-09-20T09:55:05.324+0700 INFO instance/beat.go:665 Home path: [D:\AppData\Elastic\filebeat] Config path: [D:\AppData\Elastic\filebeat] Data path: [D:\AppData\Elastic\filebeat\data] Logs path: [D:\AppData\Elastic\filebeat\logs]
2021-09-20T09:55:05.327+0700 INFO instance/beat.go:673 Beat ID: 3957662b-f353-4de0-a6a7-3260cb6481ba
2021-09-20T09:55:05.375+0700 INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "D:\\AppData\\Elastic\\filebeat", "data": "D:\\AppData\\Elastic\\filebeat\\data", "home": "D:\\AppData\\Elastic\\filebeat", "logs": "D:\\AppData\\Elastic\\filebeat\\logs"}, "type": "filebeat", "uuid": "3957662b-f353-4de0-a6a7-3260cb6481ba"}}}
2021-09-20T09:55:05.376+0700 INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "703d589a09cfdbfd7f84c1d990b50b6b7f62ac29", "libbeat": "7.14.1", "time": "2021-08-26T09:12:57.000Z", "version": "7.14.1"}}}
2021-09-20T09:55:05.376+0700 INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":12,"version":"go1.16.6"}}}
2021-09-20T09:55:05.403+0700 INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-09-20T08:27:03.12+07:00","name":"nnhai2","ip":["fe80::6036:7939:ebe1:1d3e/64","192.168.82.42/23","fe80::984a:b076:b82f:bedf/64","169.254.190.223/16","fe80::d148:87f2:9bc8:8452/64","169.254.132.82/16","fe80::c4c:5978:a65:9c2a/64","169.254.156.42/16","fe80::488c:c4a3:51de:f987/64","169.254.249.135/16","fe80::e1fb:7ed2:192d:c665/64","169.254.198.101/16","::1/128","127.0.0.1/8","fe80::a42f:ed21:3139:b3a7/64","172.28.96.1/20"],"kernel_version":"10.0.19041.1237 (WinBuild.160101.0800)","mac":["70:b5:e8:5a:d1:0a","ac:82:47:8d:80:2e","ac:82:47:8d:80:2f","ae:82:47:8d:80:2e","00:09:0f:fe:00:01","ac:82:47:8d:80:32","00:15:5d:d4:9f:62"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"19042.1237"},"timezone":"+07","timezone_offset_sec":25200,"id":"85952915-f150-4943-835a-55ae79b7bcb0"}}}
2021-09-20T09:55:05.404+0700 INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"cwd": "D:\\AppData\\Elastic\\filebeat", "exe": "D:\\AppData\\Elastic\\filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 15268, "ppid": 21388, "start_time": "2021-09-20T09:55:04.931+0700"}}}
2021-09-20T09:55:05.405+0700 INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.14.1
2021-09-20T09:55:05.405+0700 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.14.1' as ILM is enabled.
2021-09-20T09:55:05.405+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-09-20T09:55:05.406+0700 INFO [publisher] pipeline/module.go:113 Beat name: nnhai2
2021-09-20T09:55:05.410+0700 INFO instance/beat.go:473 filebeat start running.
2021-09-20T09:55:05.410+0700 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-09-20T09:55:05.418+0700 INFO memlog/store.go:119 Loading data file of 'D:\AppData\Elastic\filebeat\data\registry\filebeat' succeeded. Active transaction id=0
2021-09-20T09:55:05.418+0700 INFO memlog/store.go:124 Finished loading transaction log file for 'D:\AppData\Elastic\filebeat\data\registry\filebeat'. Active transaction id=0
2021-09-20T09:55:05.420+0700 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2021-09-20T09:55:05.420+0700 INFO [crawler] beater/crawler.go:71 Loading Inputs: 2
2021-09-20T09:55:05.420+0700 INFO [input] log/input.go:164 Configured paths: [D:\AppData\Elastic\filebeat\logs] {"input_id": "444806ec-503a-4a80-812f-a8c78e3f69a4"}
2021-09-20T09:55:05.421+0700 INFO [crawler] beater/crawler.go:141 Starting input (ID: 1263043090716372778)
2021-09-20T09:55:05.456+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_deprecation.log c:\ProgramData\Elastic\Elasticsearch\logs\*_deprecation.json] {"input_id": "aa771bd1-e31c-4061-bfe6-2897ff20dde4"}
2021-09-20T09:55:05.456+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\gc.log.* c:\ProgramData\Elastic\Elasticsearch\logs\gc.log] {"input_id": "5c90cdc0-ca6b-4d51-a33c-3ce661ff324b"}
2021-09-20T09:55:05.457+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*.log c:\ProgramData\Elastic\Elasticsearch\logs\*_server.json] {"input_id": "c7ba61e1-c4cb-42e6-8d9e-7acaa5c0d982"}
2021-09-20T09:55:05.457+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_index_search_slowlog.log c:\ProgramData\Elastic\Elasticsearch\logs\*_index_indexing_slowlog.log c:\ProgramData\Elastic\Elasticsearch\logs\*_index_search_slowlog.json c:\ProgramData\Elastic\Elasticsearch\logs\*_index_indexing_slowlog.json] {"input_id": "a83c170f-f55c-4de2-b6e3-969a3686c403"}
2021-09-20T09:55:05.458+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_access.log c:\ProgramData\Elastic\Elasticsearch\logs\*_audit.log c:\ProgramData\Elastic\Elasticsearch\logs\*_audit.json] {"input_id": "0696e6da-a676-4e5b-a550-0165db7b89af"}
2021-09-20T09:55:05.472+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\MySQL\MySQL Server*\error.log*] {"input_id": "cc0b720d-c796-47a2-87e7-7a3244fe8174"}
2021-09-20T09:55:05.472+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\MySQL\MySQL Server*\mysql-slow.log*] {"input_id": "2129affd-14ec-4468-8710-39eccfffb356"}
2021-09-20T09:55:05.487+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\nginx\logs\*access.log*] {"input_id": "44009138-8a88-4b07-8bd2-d04ab9d520bd"}
2021-09-20T09:55:05.487+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\nginx\logs\error.log*] {"input_id": "daa74eb0-538d-45fc-b657-13431186e186"}
2021-09-20T09:55:05.488+0700 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2021-09-20T09:55:05.488+0700 INFO cfgfile/reload.go:164 Config reloader started
2021-09-20T09:55:05.498+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*.log c:\ProgramData\Elastic\Elasticsearch\logs\*_server.json] {"input_id": "77a9d5c1-6f98-414d-b368-dec4f7163ed2"}
2021-09-20T09:55:05.499+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_index_search_slowlog.log c:\ProgramData\Elastic\Elasticsearch\logs\*_index_indexing_slowlog.log c:\ProgramData\Elastic\Elasticsearch\logs\*_index_search_slowlog.json c:\ProgramData\Elastic\Elasticsearch\logs\*_index_indexing_slowlog.json] {"input_id": "beaa87b1-8ce8-4374-bee5-7372846a1968"}
2021-09-20T09:55:05.500+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_access.log c:\ProgramData\Elastic\Elasticsearch\logs\*_audit.log c:\ProgramData\Elastic\Elasticsearch\logs\*_audit.json] {"input_id": "b6c46042-3acf-440e-b788-3dfdaf789c10"}
2021-09-20T09:55:05.500+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\*_deprecation.log c:\ProgramData\Elastic\Elasticsearch\logs\*_deprecation.json] {"input_id": "d673b872-e9b2-496d-96b5-1e9792169b76"}
2021-09-20T09:55:05.501+0700 INFO [input] log/input.go:164 Configured paths: [c:\ProgramData\Elastic\Elasticsearch\logs\gc.log.* c:\ProgramData\Elastic\Elasticsearch\logs\gc.log] {"input_id": "5e4465a8-9ffb-42c6-ab6f-c9d269419ed4"}
2021-09-20T09:55:05.501+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-09-20T09:55:08.389+0700 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2021-09-20T09:55:15.302+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
2021-09-20T09:55:15.454+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-gc-pipeline"}
2021-09-20T09:55:15.874+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-server-pipeline"}
2021-09-20T09:55:15.986+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-server-pipeline-plaintext"}
2021-09-20T09:55:16.108+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-server-pipeline-json"}
2021-09-20T09:55:16.227+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-slowlog-pipeline"}
2021-09-20T09:55:16.335+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-slowlog-pipeline-plaintext"}
2021-09-20T09:55:16.450+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-slowlog-pipeline-json"}
2021-09-20T09:55:16.558+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-audit-pipeline"}
2021-09-20T09:55:16.674+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-audit-pipeline-json"}
2021-09-20T09:55:16.789+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-audit-pipeline-plaintext"}
2021-09-20T09:55:16.898+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-deprecation-pipeline"}
2021-09-20T09:55:17.003+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-deprecation-pipeline-plaintext"}
2021-09-20T09:55:17.111+0700 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.14.1-elasticsearch-deprecation-pipeline-json"}
2021-09-20T09:55:17.116+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\MySQL\MySQL Server*\error.log*] {"input_id": "0043e24c-4f0f-487a-ab3e-2d2254e613ac"}
2021-09-20T09:55:17.117+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\MySQL\MySQL Server*\mysql-slow.log*] {"input_id": "eddd4b6e-3b7e-4cca-a0b0-e5422c6b7ccf"}
2021-09-20T09:55:17.117+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-09-20T09:55:17.120+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
2021-09-20T09:55:17.137+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\nginx\logs\*access.log*] {"input_id": "4af5fcde-53a7-4e67-9819-c153919b5f05"}
2021-09-20T09:55:17.137+0700 INFO [input] log/input.go:164 Configured paths: [c:\programdata\nginx\logs\error.log*] {"input_id": "f4dafaa1-89cf-48d8-927e-eb292c5b186f"}
2021-09-20T09:55:17.137+0700 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200
2021-09-20T09:55:17.141+0700 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
2021-09-20T09:55:17.153+0700 INFO cfgfile/reload.go:224 Loading of config files completed.
2021-09-20T09:55:35.468+0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":78,"time":{"ms":93}},"total":{"ticks":390,"time":{"ms":405},"value":390},"user":{"ticks":312,"time":{"ms":312}}},"handles":{"open":306},"info":{"ephemeral_id":"f9fdf685-5d19-459c-a646-fceea8d53c4e","uptime":{"ms":30152},"version":"7.14.1"},"memstats":{"gc_next":19839152,"memory_alloc":12307216,"memory_sys":34381016,"memory_total":62482208,"rss":62062592},"runtime":{"goroutines":78}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":3,"starts":3},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":10,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":12}}}}}
2021-09-20T09:56:05.478+0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":109,"time":{"ms":32}},"total":{"ticks":421,"time":{"ms":32},"value":421},"user":{"ticks":312}},"handles":{"open":307},"info":{"ephemeral_id":"f9fdf685-5d19-459c-a646-fceea8d53c4e","uptime":{"ms":60164},"version":"7.14.1"},"memstats":{"gc_next":19839152,"memory_alloc":12714808,"memory_total":62889800,"rss":62160896},"runtime":{"goroutines":78}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":10,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2021-09-20T09:56:35.470+0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":46}},"total":{"ticks":452,"time":{"ms":46},"value":452},"user":{"ticks":312}},"handles":{"open":304},"info":{"ephemeral_id":"f9fdf685-5d19-459c-a646-fceea8d53c4e","uptime":{"ms":90157},"version":"7.14.1"},"memstats":{"gc_next":19839152,"memory_alloc":13110648,"memory_total":63285640,"rss":62164992},"runtime":{"goroutines":72}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":10,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
but i have n't seen any data in kibana.
This is the result of GET _cat/indices?v
#! Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-minimal-setup.html to enable security.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases qeZJyRoKRlW7Xu7EE2ytcw 1 0 42 0 40.8mb 40.8mb
green open .apm-custom-link 5aba-WkIS5yl-Eui7vpfcQ 1 0 0 0 208b 208b
green open .kibana_task_manager_7.14.1_001 1zkRA6c6SXiMcqCtSyrSqQ 1 0 14 344 1.4mb 1.4mb
green open .apm-agent-configuration dppb9LlKQLWTzYBiLZjViA 1 0 0 0 208b 208b
yellow open filebeat-7.14.1-2021.09.19-000001 TpsVGELhRwC-_dxGH7nGKQ 1 1 0 0 208b 208b
green open .async-search KZgq-leNT_qt_dM8TOUQ6A 1 0 0 0 231b 231b
green open .kibana_7.14.1_001 D4UISLPMQlGmCjNgGIrOTw 1 0 2251 11 2.7mb 2.7mb
green open .kibana-event-log-7.14.1-000001 cspR3zh9T1emwvNA131noQ 1 0 3 0 16.4kb 16.4kb
green open .tasks 5dHd_BZpSVilDmVQy6kE7w 1 0 4 0 27.3kb 27.3kb
is it required specific structure log file or i can put any thing in there or where can i get sample log file to test the connection to put in my folder at D:\AppData\Elastic\filebeat\logs ?
Also, where can i find some best practice to config filebeat, i 've read the document at https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html. But it is too simple, many things were not explained like how to config and test modules (we have dozens modules pensando, postgresql, proofpoint, rabbitmq,....)
I am using filebeat and ELK stack.I am not getting the logs from filebeat to logstach. Can any one help.
Filebeaat version : 6.3.0
ELK version : 6.0.0
filebeat config :--
filebeat.prospectors:
- type: log
enabled: true
paths:
- '/var/lib/docker/containers/*/*.log'
ignore_older: 0
scan_frequency: 10s
json.message_key: log
json.keys_under_root: true
json.add_error_key: true
multiline.pattern: "^[[:space:]]+(at|\\.{3})\\b|^Caused by:"
multiline.negate: false
multiline.match: after
registry_file: usr/share/filebeat/data/registry
output.logstash:
hosts: ["172.31.34.173:5044"]
Filebeat logs :--
2018-07-23T08:29:34.701Z INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.0
2018-07-23T08:29:34.701Z INFO pipeline/module.go:81 Beat name: ff01ed6d5ae4
2018-07-23T08:29:34.702Z WARN [cfgwarn] beater/filebeat.go:61 DEPRECATED: prospectors are deprecated, Use `inputs` instead. Will be removed in version: 7.0.0
2018-07-23T08:29:34.702Z INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-07-23T08:29:34.702Z INFO instance/beat.go:315 filebeat start running.
2018-07-23T08:29:34.702Z INFO registrar/registrar.go:75 No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2018-07-23T08:29:34.704Z INFO registrar/registrar.go:112 Loading registrar data from /usr/share/filebeat/data/registry
2018-07-23T08:29:34.704Z INFO registrar/registrar.go:123 States Loaded from registrar: 0
2018-07-23T08:29:34.704Z WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-07-23T08:29:34.704Z INFO crawler/crawler.go:48 Loading Inputs: 1
2018-07-23T08:29:34.705Z INFO log/input.go:111 Configured paths: [/var/lib/docker/containers/*/*.log]
2018-07-23T08:29:34.705Z INFO input/input.go:87 Starting input of type: log; ID: 2696038032251986622
2018-07-23T08:29:34.705Z INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
2018-07-23T08:30:04.705Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":22}},"total":{"ticks":50,"time":{"ms":60},"value":50},"user":{"ticks":30,"time":{"ms":38}}},"info":{"ephemeral_id":"5193ce7d-8d09-4e9d-ab4e-e55a5972b4
Bit late to reply I know but I was having the same issue and after some searching, I found this layout to work for me.
filebeat.prospectors:
- paths:
- '<path to your log>'
multiline.pattern: '<whatever pattern is needed>'
multiline.negate: true
multiline.match: after
processors:
- decode_json_fields:
fields: ['<whatever field you need to decode']
target: json
Here's a link to a similar problem.
I am trying out the Kubernetes NFS volume claim in a replication controller example [1].
I have setup the NFS server, PV and PVC. And my replication controller looks like this
apiVersion: v1
kind: ReplicationController
metadata:
name: node-manager
labels:
name: node-manager
spec:
replicas: 1
selector:
name: node-manager
template:
metadata:
labels:
name: node-manager
spec:
containers:
-
name: node-manager
image: org/node-manager-1.0.0:1.0.0
ports:
-
containerPort: 9763
protocol: "TCP"
-
containerPort: 9443
protocol: "TCP"
volumeMounts:
- name: nfs
mountPath: "/mnt/data"
volumes:
- name: nfs
persistentVolumeClaim:
claimName: nfs
When I try to deploy the Replication Controller, the container is in the ContainerCreating status and I can see the following error in the journal of the minion
Feb 26 11:39:41 node-01 kubelet[1529]: Mounting arguments: 172.17.8.102:/ /var/lib/kubelet/pods/0e66affa-dc79-11e5-89b3-080027f84891/volumes/kubernetes.io~nfs/nfs nfs []
Feb 26 11:39:41 node-01 kubelet[1529]: Output: mount.nfs: requested NFS version or transport protocol is not supported
Feb 26 11:39:41 node-01 kubelet[1529]: E0226 11:39:41.908756 1529 kubelet.go:1383] Unable to mount volumes for pod "node-manager-eemi2_default": exit status 32; skipping pod
Feb 26 11:39:41 node-01 kubelet[1529]: E0226 11:39:41.923297 1529 pod_workers.go:112] Error syncing pod 0e66affa-dc79-11e5-89b3-080027f84891, skipping: exit status 32
Feb 26 11:39:51 node-01 kubelet[1529]: E0226 11:39:51.904931 1529 mount_linux.go:103] Mount failed: exit status 32
Used [2] Kubernetes-cluster-vagrant-cluster to setup my Kubernetes cluster.
my minion details:
core#node-01 ~ $ cat /etc/lsb-release
DISTRIB_ID=CoreOS
DISTRIB_RELEASE=969.0.0
DISTRIB_CODENAME="Coeur Rouge"
DISTRIB_DESCRIPTION="CoreOS 969.0.0 (Coeur Rouge)"
[1] - https://github.com/kubernetes/kubernetes/tree/master/examples/nfs
[2] - https://github.com/pires/kubernetes-vagrant-coreos-cluster
I had the same problem then realized that nfs-server.service status is disabled. After activating, the problem has been solved.
hence in order to resolve this nfs mount version issue by making the entry in /etc/nfsmount.conf in nfs server with Defaultvers=4 in the NFS server .The will resolved !!