Can packetbeat is used to monitor the tomcat server logs and windows logs?? or it will only monitor the database i.e., network monitoring?
Packetbeat only does network monitoring. But you can use it together with Logstash or Logstash-Forwarder to get visibility also into your logs.
It will do only network monitoring. you can use ELK for tomcat server logs.
#tsg is correct but now with the Beats 1.x release they are deprecating Logstash Forwarder in lieu of another Beat called Filebeat. Also they added Topbeat, which allows you to monitor server load and processes in your cluster.
See:
* https://www.elastic.co/blog/beats-1-0-0
You will likely want to install the package repo for your OS, then install each with:
{package manager cmd} install packetbeat
{package manager cmd} install topbeat
{package manager cmd} install filebeat
They each are installed in common directories. For example with Ubuntu (Linux) the config files are in /etc/<beat name>/<beat name>.yml where beat name is one of the 3 above. Each file are similar and you can disable the direct ES export and instead export to Logstash (comment ES and uncomment Logstash) and then add a beats import in your Logstash config. From thereon, Logstash listens for any beats over that port and can redistribute (or queue) using the [#metadata][beat] param to tell where it came from.
Libbeat also provides a framework to build your own so you can send any data you want to Logstash and it can queue and/or index. ;-)
Packetbeat is used mainly for network analysis . It currently supports following protocols :
ICMP (v4 and v6)
DNS
HTTP
Mysql
PostgreSQL
Redis
Thrift-RPC
MongoDB
Memcache
However , for visualizing tomcat logs you can configure them to use log4j and then configure logstash to take input from log4j and then using elasticsearch and kibana to visualise the logs.
To monitor windows logs you can use another beats platform Winlogbeat.
Related
When using logstash to retrieve airflow logs from a folder you have access to, would I still need to make any changes in the airflow.cfg file?
For instance, I have airflow and ELK deployed on same ec2 instance. The logstash .conf file has access to the airflow logs path since they are on the same instance. Do I need to turn on remote logging in airflow config?
In fact you have two options to push airflow logs to Elastic Search:
Using a log collector (logstash, fluentd, ...) to collect Airflow log then send it to Elastic Search server, in this case you don't need to change any Airflow config, you can just read the logs from the files or stdout and send it to ES.
Using Airflow remote logging feature, in this case Airflow will log directly to your remote logging server (ES in your case), and will store a local version of this log to show it when the remote server is unavailable.
So the answer to your question is no, if you have a logstash, you don't need Airflow remote logging config
I installed istio version 1.7.3 on windows 10 using Docker Desktop 3.2.2 and successfully run the sample bookinfo demo. I started prometheus to see http logs when sending requests to localhost/productpage. I want to be able to save those logs in my host machine (windows 10). Additionally I want to save istio metrics and logs components (mixer, citadel, envoy, etc.) to windows too, how can I achieve such a thing? and the path /dev/null specified in here it concerns the path for the pod itself?
EDIT:
the following documentation in istio v 1.1 uses fluentd, elasticsearch and kibana to collect logs, how can I apply this to istio v 1.7 ?
The Logging with Mixer and Fluentd sais that Istio 1.7 is the last one that support Mixer.
Mixer is deprecated. The functionality provided by Mixer is being
moved into the Envoy proxies. Use of Mixer with Istio will only be
supported through the 1.7 release of Istio.
So basically, if your Istio < 1.8, you still can use Istio Logging with Mixer and Fluentd and configure logs through Fluentd / Elasticsearch / Kibana stack.
Starting from version 1.8 Mixer is no longer supported in Istio
I am trying to build familiarity with SIEM systems in general and decided to set up an Elastic Stack via Digital Ocean. Everything was successful and my server as localhost is producing logs. It's been interesting to tinker with visualizations and that good stuff.
Obviously my interest isn't in logs from this remote server, though. I would like to configure some devices on my home network to send logs.
Current setup on server: filebeat > logstash > elasticsearch > kibana.
When I install filebeat onto, say, my laptop and configure the .yml file in a similar way to the server (comment out elastic output, uncomment logstash output) it is not able to connect. Basically I just set the hosts to serverip:logstash port and enabled filebeat on the system. Running the setup commands leads to a "couldn't connect to any configured elasticsearch hosts".
Instead of a direct answer, can someone explain for me generally what I need to be considering for this process? What is happening when connecting outside of the server LAN? and how do I handle authentication to the server, if needed?
Thank you, really. I know that the information is out there but I am deep in a rabbit hole and having a hard time finding what I need.
By default, the HTTP API is bound to only the host's local loopback interface,
ensuring that it is not accessible to the rest of the network. Because the API
includes neither authentication nor authorization and has not been hardened or
tested for use as a publicly-reachable API, binding to publicly accessible IPs
should be avoided where possible.
Even you set "http.host: 0.0.0.0" - you need to open port for your laptop (better if you already have public IP and open it only for your laptop)
For authentication - you have to investigate xpack - security features .
BR Alexey.
I have logstash installed in a server which will process logs and publish to elastic search. But, is it possible for logstash to pull logs from remote servers (linux) without installing filebeats in those servers.
Or if filebeats can be installed in the same server as logstash and can it fetch the logs? Please help me if there is any other option as well.
Thanks in advance
Neither Logstash nor Filebeat can pull/fetch log files from remote servers, you need to have some tool installed in the remote servers that will ship the logs elsewhere.
Logstash can consume logs from message queue systems like kafka, redis or rabbitmq, for example, but you need that your remote servers send the logs to those systems anyway, so you would need a log shipper on your remote servers.
How can I stream logs from multiple log4net in elasticsearch??
For example, if I have multiple Windows servers with .net apps and log4net running on each server, each server streaming logs on a separate UDP port. How do I setup logs on elasticsearch server to index logs from multiple ports ?
You can use logstash connectors for this purpose. Look at this :
Using logstash, ElasticSearch and log4net for centralized logging in Windows