I need to save data sets like (timestamp, event_name, event_value1, event_value2, event_value3, ...) with StatsD. I need this to track custom events in the web app I'm working on.
Official StatsD readme states that StatsD expects metrics to be sent in the format:
<metricname>:<value>|<type>
Is there any way to push multiple values, or any workaround to make this possible?
We're currently using Graphite as a backend service, but it can be changed for the sake of adding this feature.
You could use a naming convention to capture this information:
event_name-timestamp-event_value1_name:event_value1|<stat type>
event_name-timestamp-event_value2_name:event_value2|<stat type>
event_name-timestamp-event_value3_name:event_value3|<stat type>
etc.
Related
I already have micro-services running and would like to add event(Kafka).
For example, I have a customer service with 10000 customers in the db. I will be adding an event to the customer service so that whenever a new user is created, it publishes an event in which will be consumed by consumers (like recommendation-service, statistics-service, etc.)
I think the above is clear to me. However, I am not sure how to handle the already-registered customers (10000 customers) as the event will only be triggered when 'NEW' customer registers.
I can 'hack' the service to sync the data manually but what does most people do in this case?
Thank you
I tried to search the topic but couldn't find the ones that I am looking for.
There are basically two strategies that you can follow here. The first is a bulk load of fake "new customer" events into the Kafka topic, as you also suggested. The second approach would be to use the change data capture (CDC) pattern where there is an initial snapshot of all the observed data and then a constant streaming of new data change events, direclty from the database internal log (WAL).
To handle your entire use case, you could use a tool like Debezium Source Connector for Kafka Connect platform, but note that you will also need to map its change event into your message format. There are plugins to do that with a configuration-driven approach, but you can also create your custom logic using single message transformations (SMT).
run with flink-1.9.0 on yarn(2.6.0-cdh5.11.1), but the flink web ui metrics does'nt work, as shown below:
I guess you are looking at the wrong metrics. Due no data flows from one task to another (you can see only one box at the UI) there is nothing to show. The metrics you are looking at only show the data which flows from one flink task to another. At your example everything happens within this task.
Look at this example:
You can see two tasks sending data to the map-task which emits this data to another task. Therefore you see incoming and outgoing data.
But on the other hand a source task never has incoming data(I must admit that this is confusing at the first look):
The number of records recieved is 0 but it send a couple of records to the downstream task.
Back to your problem: What you can do is have a look at the operator metrics. If you look at the metrics tab (the one at the very right) you can select beside the task metrics also some operator metrics. These metrics have a name like 0.Map.numRecordsIn.
The name assembles like this <slot>.<operatorName>.<metricsname>. But be aware that this metrics are not recorded, you don't have any historic data and once you leave this tab or remove a metric the data collected until that point are gone. I would recommend to use a proper metrics backend like influx, prometheus or graphite. You can find a description at the flink docs.
Hope that helped.
I'm trying to ship Aerospike metrics to another node using some available methods, e.g., collectd.
For example, among the Aerospike monitoring metrics, given two fields: say X and Y, how can I define and send a derived metric like Z = X+Y or X/Y?
We could calculate it on the receiver side but it degrades the performance of our application overall. Will appreciate your guidance in advance.
Thanks.
It can't be done within the Aerospike collectd plugin, as the metrics are more or less shipped immediately once they are read. There's no variable that saves the metrics that have been shipped.
If you can use the Graphite plugin, it keeps track of all gathered metrics then sends once at the very end. You can add another stanza for your calculated metrics right before nmsg line. You'll have to search through the msg[] array for your source metrics.
The Nagios plugin is a very different method. It's a single metric pull, so a wrapper script would be needed to run the plugin for each operand, and run the calculation in the wrapper.
Or you can supplement existing plugins with your own script(s) just for derived metrics. All of our monitoring plugins utilize the Aerospike Info Protocol and you can use asinfo to gather metrics for your operands similar to the previous Nagios method.
Is it possible to retrieve data about A-B Tests using the Mixpanel API. For example fetching all AB-Tests including their tweaks? It seems like the java API only provides Events, People and aggregated data, but not the "master data" we configured in Mixpanel.
How did you set up the A/B testing data in Mixpanel? Usually these data is setup as property to an event e.g. "Variation Name: A" will be the super property that is attached to all relevant events.
You would then be able to pull any event and find that specific property through the API.
I am collecting system metrics using Collectd. I am collecting measures in small intervals to get accurate values. However i want to aggregate these values locally using Statsd. Statsd should aggregate the values and send them to librato in longer intervals. This will reduce costs.
I have completed the basic setup of Collectd and Statsd. How do i send data from Collectd to Statsd?
The Statsd plugin seems to be a replacement for Statsd itself and seems not to provide this functionality.
It doesn't seem like there is any established plugin to accomplish this. If you're already satisfied where/how collectd is sending the data, and just want to aggregate, you can use the aggregation plugin:
https://collectd.org/wiki/index.php/Plugin:Aggregation
If you really want to get the data into StatsD somehow, you might be able to use the collectd Network output plugin and have it point to StatsD's port (although you may have to manipulate the data somehow).
I think for the most part, though, these 2 exist in parallel, and if you needed both each daemon would send data to your librato separately, or you could consolidate by only using CollectD with the StatsD plugin.