I have run a topology, and I used the Meter type in metric Reporting API v2. In the execute method I mark this metric. So it will mark an event whenever the execute method is called. But when I compare this value with the __execute-count, I see huge differences. Does anyone know why this happens?
These are the values from my log which are gathered at the same time:
9:v7 __execute-count {v0:v7=44500}
9:v7 tuple_inRate.count 664129
Update:
When I use the mark method on the Meter metric, I will get different results in comparison with the Counter metric. But still, I do not understand why the values from the counter metric (tuple counter) are not the same as the __execute-count.
As given in this answer, Storms Internal Metrics are just estimated by a percentage of the real data flow. Initially, it uses 5% of incoming tuples to make those estimations. This may lead to inaccuracies for extreme high or low throughputs.
EDIT: The documentation describes the following:
In general all of these tuple count metrics are randomly sub-sampled unless otherwise stated. This means that the counts you see both on the UI and from the built in metrics are not necessarily exact. In fact by default we sample only 5% of the events and estimate the total number of events from that. The sampling percentage is configurable per topology through the topology.stats.sample.rate config. Setting it to 1.0 will make the counts exact, but be aware that the more events we sample the slower your topology will run (as the metrics are counted in the same code path as tuples are processed). This is why we have a 5% sample rate as the default.
EDIT 2 In this post, there is more information about the estimation:
The way it works is that if you choose a sampling rate of 0.05, it will pick a random element of the next 20 events in which to increase the count by 20. So if you have 20 tasks for that bolt, your stats could be off by +-380.
By the way, execute_count is just an increasing number, while your tuple_inRate.count is a rate, isn`t it?
I'd like to measure avg request-response time for my webserver.
Apm has transaction.duration.us and it seems this could be the metric I'm looking for.
But I coulnd't find the documentation what it is.
Where can I find the meaning of the variable?
transaction.duration.us should indeed be what you're looking for. It's the duration in microseconds as an integer. Divide it by 1000 to get milliseconds, or by 1'000'000 to get seconds.
https://www.elastic.co/guide/en/apm/server/7.9/exported-fields-apm-transaction.html#_duration_2
We have cluster of instances whereas each instance has DropWizard metrics gatherer.
We're also trying to leverage AppDynamics custom metrics and that works so that custom script hits DropWizard exposed endpoint (/metrics) and sends metrics of interest to AppDynamics Controller.
AppDynamics has 2 cluster rollout strategies for how the metric is displayed in a whole application view (tier) - SUM and AVG.
While this works well for stuff like counts (sum is used) and average processing times (avg is used) - we for now don't have any idea of how to aggregate each instance percentiles exposed by DropWizard - neither sum nor avg looks correct.
Example:
instance1: p75=400
instance2: p75=600
instance3: p75=800
sum will give 1700 what of course isn't useful at all.
avg will give 600 - which isn't correct either - we're losing track of higher bound.
If AppDynamics had MAX Cluster rollout - that would be more or less fair - still not correct though. But AppDynamics doesn't have that.
We also understand that the only fully correct way of gathering cluster percentiles is to perform aggregation from all nodes at one place (e.g. logstash, etc..) and not on each instance. But for now that's what we have - just sending custom metrics periodically.
It would be great if anyone suggests something regarding that.
Thanks in advance,
After an update to Grafana v4.1.2 (commit: v4.1.2) which I am using together with the latest version of InfluxDB my dashboards changed their behaviour.
I am not using GROUP BY time($interval) in this particular dashboard, but give the possibility to define the group by value via template variable: GROUP BY time($group_time)
I know this is risky, but under our certain circumstances it needs to be this way.
Before the update if a very small value for group by variable was chosen together with a vast time interval obviously or it took a very long time to load or the Browser went in tilt. Now this behaviour changed. There seems to be a limit on the datapoints that grafana visualizes/ retrieves. To make this clearer see the example below:
Time interval with 20s GROUP BY
both curves entirely visible with this group by value
Same Time intervall with 10s GROUP BY
one curve entirely visible with this group by value, other curve cut after some points
Same Time intervall with 10s GROUP BY first curve turned to invisible
the second curve is entirely visible again
I suspect, that Grafana is now sending a LIMIT to InfluxDB. I found this bit of information for the combination Grafana & Graphite http://docs.grafana.org/installation/performance/ but nothing for Grafana & InfluxDB.
So where has this been changed and is there a way to set this limit manually? The limit that I am experiencing at the moment is most probably 10000, because I can limit my queries with values below 10000, but any higher value does not bring more points. Any kind of documentation of this now default limit would be very much appreciated.
I have been exploring the Graphite graphing tool for showing metrics from multiple servers, and it seems that the 'recommended' way is to send all metrics data to StatsD first. StatsD aggregates the data and sends it to graphite (or rather, Carbon).
In my case, I want to do simple aggregations like sum and average on metrics across servers and plot that in graphite. Graphite comes with a Carbon aggregator which can do this.
StatsD does not even provide aggregation of the kind I am talking about.
My question is - should I use statsd at all for my use case? Anything I am missing here?
StatsD operates over UDP, which removes the risk of carbon-aggregator.py being slow to respond and introducing latency in your application. In other words, loose coupling.
StatsD supports sampling of inbound metrics, which is useful when you don't want your aggregator to take 100% of all data points to compute descriptive statistics. For high-volume code sections, it is common to use 0.5%-1% sample rates so as to not overload StatsD.
StatsD has broad client-side support.
tldr: you will probably want statsd (or carbon-c-relay) if you ever want to look at the server-specific sums or averages.
carbon aggregator is designed to aggregate values from multiple metrics together into a single output metric, typically to increase graph rendering performance. statsd is designed to aggregate multiple data points in a single metric, because otherwise graphite only stores the last value reported in the minimum storage resolution.
statsd example:
assume that your graphite storage-schemas.conf file has a minimum retention of 10 seconds (the default) and your application is sending approximately 100 data points every 10 seconds to services.login.server1.count with a value of 1. without statsd, graphite would only store the last count received in each 10-second bucket. after the 100th message is received, the other 99 data points would have been thrown out. however, if you put statsd between your application and graphite, then it will sum all 100 datapoints together before sending the total to graphite. so, without statsd your graph only indicates if a login occurred in during the 10 second interval. with statsd, it indicates how many logins occurred in during that interval. (for example)
carbon aggregator example: assume you have 200 different servers reporting 200 separate metrics (services.login.server1.response.time, services.login.server2.response.time, etcetera). on your operations dashboard you show a graph of the average accross all servers using this graphite query: weightedAverage(services.login.server*.response.time, services.login.server*.response.count, 2). unfortunately, rendering this graph takes 10 seconds. to solve this problem, you can add a carbon aggregator rule to pre-calculate the average across all your servers and store the value in a new metric. now you can update your dashboard to simply pull a single metric (e.g. services.login.response.time). the new metric renders almost instantly.
side notes:
the aggregation rules in storage-aggregation.conf apply to all storage intervals in storage-schemas.conf except the first (smallest) retention period for each retention string. it is possible to use carbon-aggregator to aggregate data points within a metric for that first retention period. unfortunately, aggregation-rules.conf uses "blob" patterns rather than regex patterns. so you need to add a separate aggregation-rules.conf file entry for every path depth and aggregation type. the advantage of statsd is that the client sending the metric can specify the aggregation type rather than encoding it in the metric path. that gives you the flexibility to add a new metric on the fly regardless of metric path depth. if you wanted to configure carbon-aggregator to do statsd-like aggregation automatically when you add a new metric, your aggregation-rules.conf file would look something like this:
<n1>.avg (10)= avg <n1>.avg$
<n1>.count (10)= sum <n1>.count$
<n1>.<n2>.avg (10)= avg <n1>.<n2>.avg$
<n1>.<n2>.count (10)= sum <n1>.<n2>.count$
<n1>.<n2>.<n3>.avg (10)= avg <n1>.<n2>.<n3>.avg$
<n1>.<n2>.<n3>.count (10)= sum <n1>.<n2>.<n3>.count$
...
<n1>.<n2>.<n3> ... <n99>.count (10)= sum <n1>.<n2>.<n3> ... <n99>.count$
notes: the trailing "$" is not needed in graphite 0.10+ (currently pre-release) here is the relevant patch on github and here is the standard documentation on aggregation rules
the weightedAverage function is new in graphite 0.10, but generally the averageSeries function will give a very similar number as long as your load is evenly balanced. if you have some servers that are both slower and service fewer requests or you are just a stickler for precision, then you can still calculate a weighted average with graphite 0.9. you just need to build a more complex query like this:
divideSeries(sumSeries(multiplySeries(a.time,a.count), multiplySeries(b.time,b.count)),sumSeries(a.count, b.count))
if statsd is run on the client box this also reduces network load. although, in theory, you could run carbon-aggregator on the client side too. however, if you use one of the statsd client libraries, you can also use sampling to reduce the load on your application machine's cpu (e.g. creating loopback udp packets). furthermore, statsd can automatically perform multiple different aggregations on a single input metric (sum, mean, min, max, etcetera)
if you use statsd on each app server to aggregate response times, and then re-aggregate those values on the graphite server using carbon aggregator, you end up with an average response time weighted by app server rather than request. obviously, this only matters for aggregating using a mean or top_90 aggregation rule, and not min, max or sum. also, it only matters for mean if your load is unbalanced. as an example: assume you have a cluster of 100 servers, and suddenly 1 server is sent 99% of the traffic. consequentially, the response times quadruple on that 1 server, but remain steady on the other 99 servers. if you use client side aggregation, your overall metric would only go up about 3%. but if you do all your aggregation in a single server-side carbon aggregator, then your overall metric would go up by about 300%.
carbon-c-relay is essentially a drop-in replacement for carbon-aggregator written in c. it has improved performance and regex-based matching rules. the upshot being that you can do both statsd-style datapoint aggregation and carbon-relay style metric aggregation and other neat stuff like multi-layered aggregation all in the same simple regex-based config file.
if you use the cyanite back-end instead of carbon-cache, then cyanite will do the intra-metric averaging for you in memory (as of version 0.5.1) or at read time (in the version <0.1.3 architecture).
If the Carbon aggregator offers everything you need, there is no reason not to use it. It has two basic aggregation functions (sum and average), and indeed these are not covered by StatsD. (I'm not sure about the history, but maybe the Carbon aggregator already existed and the StatsD authors did not want to duplicate features?) Receiving data via UDP is also supported by Carbon, so the only thing you would miss would be the sampling, which does not matter if you aggregate by averaging.
StatsD supports different metric types by adding extra aggregate values (e.g. for timers: mean, lower, upper and upper Xth percentile, ...). I like them, but if you don't need them, the Carbon aggregator is a good way to go too.
I have been looking at the source code of the Carbon aggregator and StatsD (and Bucky, a StatsD implementation in Python), and they are all so simple, that I would not worry about resource usage or performance for either choice.
Looks like carbon aggregator and statsd support disjoint set of features:
statsd supports rate calculation and summation but does not support averaging values
carbon aggregator supports averaging but does not support rate calculation.
Because graphite has a minimum resolution, so you cannot save two different values for the same metric during defined interval. StatsD solves this problem by pre-aggregating them, and instead of saying "1 user registered now" and "1 user registered now" it says "2 users registered".
The other reason is performance because:
You send data to StatsD via UDP, which is a fire and forget protocol, stateless, much faster
StatsD etsy's implementation is in NodeJS which also increases the performance a lot.