If i want to monitor processes running on windows servers, could winlogbeats be configured used to pull this type of data and send to elasticsearch for indexing ?
Process infos like Image Name, CPU time, memory, User, PID ... as seen in task manager
WinlogBeat is used primarily for shipping windows event viewer logs into elastic. If you want to monitor critical events in your system event log over multiple machines, WinlogBeat is the way to go.
For metrics, you'd be better off using the perfmon options within MetricBeat. Any metrics you can pull from perfmon (which includes everything in task manager) you can ship to elastic via MetricBeat.
https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-windows-perfmon.html
Related
Our system consists of many python scripts that are run on "clean" machines, that is, they need to have as little additional software on them as possible. Is there a way we could use OpenTelemetry without having to run additional servers on those machines? Is there a push model for sending data instead of pull?
Considering your additional explanation I imagine you will eventually want to collect all telemetry from these systems. Using OTLP exporters you can send all three signals traces, metrics, logs to collector service (As of now only tracing is stable and metrics, logs work is experimental). You would not have to run any additional servers on these resource constrained servers for your use case. There are two deployments strategies recommended for opentelemetry collector.
As an agent - Runs along with the application on same host machine.
As a gateway - Runs on standalone server outside the application host machine.
Running collector agent on same application host machine offloads some of the work from language client libs and enhances the telemetry but can be resource incentive.
Read more about collector here https://opentelemetry.io/docs/collector/getting-started/
I have set up Flink UI for application running in Intellij IDEA. I would like to get some streaming metrics like - scheduling delay and processing time. However, I can not find the anywhere in UI. Should there be some specific setup for that or should I explicitly submit app jar?
Currently, Flink UI for the job looks like this:
All of the task metrics are exposed in the web UI, as Dominik mentioned, but for other metric scopes (e.g., job metrics) only some selected metrics are displayed. You can access all of the metrics via the REST API or by connecting a metrics reporter to send the metrics to an external metrics system.
I don't think any attempt has been to made to measure scheduling delay, but in the job metrics you will find things like restarting time and uptime.
In the UI you should have a tab Task Metrics when You select the currently running job. This tab allows You to choose a task and see all the metrics that are available. Although, I am not sure if the scheduling delay is one of currently available metrics.
Probably the better idea is to refer to expose the metrics for some collector of Your choice, You can find more info in the documentation: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html.
So I was running some tests across some machines and monitoring CPU/Memory usage by process.
To test it's accuracy I was also monitoring with VisualVM.
and the graphs were off slightly.
Also on Jmeter when I monitor CPU/memory but not per process name it gives the exact same results - so it's not seeing the process.
If i do it by process ID it works - but the process ID changes so don't want to go this route.
Beside process name there is occurence , anyone know what this is and whether it can be left blank or not.
I don't think the process is called kafka, my expectation is that both for Windows or for Linux/Unix you should be looking for java process, not Kafka
So try changing the process name to java and it should start working as expected:
If it doesn't - launch PerfMon Agent with --loglevel debug parameter - it should print a lot of useful information into stdout
Just in case check out How to Monitor Your Server Health & Performance During a JMeter Load Test article for more information.
I am working on a spark program that monitor each executors' performance such as mark down when one executor start to work and when it finishes its job. I am thinking two ways to do that:
First, develop programs so when the executor starts work, it mark down the current time to a file, when it finishes, mark down that time to the same file. In the ends, all "log" files will be spread the whole cluster networks except for the driver machine.
Second, since executors will report to driver periodically, each time the driver receives message from executors, if the message contains "start" and "finish" information, let the driver record everything.
Is that possible?
There are many ways to Monitor the executor performance as well as application performance
Best ways are to Monitor with the help of Spark Web UI and Other Monitoring tools available Open Source (Ganglia)
You Need to Monitor your application whether your cluster is under utilized or not how much resources are used by your application which you have created.
Monitoring can be done using various tools eg. Ganglia From Ganglia you can find CPU, Memory and Network Usage.Based on Observation about CPU and Memory Usage you can get a better idea what kind of tuning is needed for your application
Hope this Helps!!!....
I am trying to make Grafana display all my metrics (CPU, Memory, etc).
I have already configured Grafana on my server and have configured influxdb and of course I have configured Jmeter listener (Backend Listener) but still I cannot display all grpahas, any idea what should I do in order to make it work ?
It seems like that system metrics (CPU/Memory, etc.) are not in the scope of the JMeter Backend Listener implementation. Actually capturing those KPIs is a part of PerfMon plugin, which currently doesn't seem to support dumping the metrics to InfluxDB/Graphite (at least it doesn't seem to work for me). It might be a good idea to raise such a request at https://groups.google.com/forum/#!forum/jmeter-plugins. Until this gets done, I guess you also have the option of using some alternative metric-collection tools to feed data in InfluxDB/Graphite. Those would depend on the server OS you want to monitor (e.g. Graphite-PowerShell-Functions for Windows or collectd for everything else)
Are you sure that JMeter posts the data to InfluxDB? Did you see the default measurements created in influxDB?
I am able to send the data using backend listener to influxdb. I have given the steps in this site.
http://www.testautomationguru.com/jmeter-real-time-results-influxdb-grafana/