I have kapacitor 1.3.1 and influxdb 1.2.4 running on my machine. Though i have enabled kapacitor to send its stats, i dont see _kapacitor database in influxdb.
What am i missing here?
kapacitor.config:
hostname = "localhost"
[stats]
# Emit internal statistics about Kapacitor.
# To consume these stats create a stream task
# that selects data from the configured database
# and retention policy.
#
# Example:
# stream|from().database('_kapacitor').retentionPolicy('autogen')...
#
enabled = true
stats-interval = "10s"
database = "_kapacitor"
retention-policy= "autogen"
[[influxdb]]
# Connect to an InfluxDB cluster
# Kapacitor can subscribe, query and write to this cluster.
# Using InfluxDB is not required and can be disabled.
enabled = true
default = true
name = "localhost"
urls = ["http://localhost:8086"]
username = ""
password = ""
timeout = 0
Q: What am i missing here?
A: You got the first step right by enabling the stats functionality in Kapacitor. The next thing you need to do here is to bounce the Kapacitor engine this is so that stats are getting written into its internal database periodically.
Now the catch is that you'll need to also define a TICK script to pull the stats out from Kapacitor's internal database, then you can choose what you want to do to it, manipulate the data and write it back to an InfluxDB or raise alerts.
Example:
var data = stream| from().database('_kapacitor').retentionPolicy('autogen')
data
|log()
.prefix('Kapacitor stat =>')
After you got your tick script going. You'll have to do the usual, like installing it into Kapacitor then enable it.
kapacitor define test -type stream -tick test.tick -dbrp _kapacitor.autogen
There is a catch here. You need to specify the retention policy that you have specified in the config or otherwise it won't know where to look for the data. In this case it is _kapacitor.autogen.
test stream disabled false ["_kapacitor"."autogen"]
Next you enable the stream task.
kapacitor enable test
Output:
[test:log2] 2017/07/26 00:49:21 I! Kapacitor stat =>
{"Name":"ingress","Database":"_kapacitor","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"cluster_id":"c80d02c0-8c51-4071-8904-1583164e90ec","database":"_internal","host":"kapacitor_stoh","measurement":"tsm1_cache","retention_policy":"monitor","server_id":"82a2d589-db45-4cc5-81b0-674cb80737ac","task_master":"main"},"Fields":{"points_received":4753},"Time":"2017-07-26T00:49:21.75615995Z"}
Related
As per an earlier discussion (Defining multiple outputs in Logstash whilst handling potential unavailability of an Elasticsearch instance) I'm now using pipelines in Logstash in order to send data input (from Beats on TCP 5044) to multiple Elasticsearch hosts. The relevant extract from pipelines.yml is shown below.
- pipeline.id: beats
queue.type: persisted
config.string: |
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => '/etc/logstash/config/certs/ca.crt'
ssl_key => '/etc/logstash/config/certs/forwarder-001.pkcs8.key'
ssl_certificate => '/etc/logstash/config/certs/forwarder-001.crt'
ssl_verify_mode => "force_peer"
}
}
output { pipeline { send_to => [es100, es101] } }
- pipeline.id: es100
path.config: "/etc/logstash/pipelines/es100.conf"
- pipeline.id: es101
path.config: "/etc/logstash/pipelines/es101.conf"
In each of the pipeline .conf files I have the related virtual address i.e. the file /etc/logstash/pipelines/es101.conf includes the following:
input {
pipeline {
address => es101
}
}
This configuration seems to work well i.e. data is received by each of the Elasticsearch hosts es100 and es101.
I need to ensure that if one of these hosts is unavailable, the other still receives data and thanks to a previous tip, I'm now using pipelines which I understand allow for this. However I'm obviously missing something key in this configuration as the data isn't received by a host if the other is unavailable. Any suggestions are gratefully welcomed.
Firstly, you should configure persistent queues on the downstream pipelines (es100, es101), and size them to contain all the data that arrives during an outage. But even with persistent queues logstash has an at-least-once delivery model. If the persistent queue fills up then back-pressure will cause the beats input to stop accepting data. As the documentation on the output isolator pattern says "If any of the persistent queues of the downstream pipelines ... become full, both outputs will stop". If you really want to make sure an output is never blocked because another output is unavailable then you will need to introduce some software with a different delivery model. For example, configure filebeat to write to kafka, then have two pipelines that read from kafka and write to elasticsearch. If kafka is configured with an at-most-once delivery model (the default) then it will lose data if it cannot deliver it.
I am performing this query with my Ruby mongo driver:
begin
User.collection.find({}).count()
rescue => e
Rails.logger.error e.to_s
end
I would like to catch all situations where this operation fails. The main reason it would fail is if the server was unavailable.
For example, one of the errors I see occasionally is:
Mongo::Error::NoServerAvailable (No server is available matching preference: #<Mongo::ServerSelector::Primary:0x70302744731080 tag_sets=[] max_staleness=nil> using server_selection_timeout=30 and local_threshold=0.015)
I want to catch errors after just 6 seconds.
From the docs I see that there are a few different timeout options (connect_timeout, server_selection_timeout, socket_timeout). But I am not sure which to pass, and how to pass them.
You're on the right track. server_selection_timeout is the correct option in this case -- it tells the driver how long to wait to find a suitable server before timing out. You can set that option on a new client in the Mongoid configuration file (config/mongoid.yml).
You want your configuration file to look something like this:
development: # (or production or whatever environment you're using)
clients:
default:
# your default client here
new_client: # the name of your new client with different options
database: mongoid
hosts:
- localhost:27017
options:
server_selection_timeout: 6
...
Read the Mongoid Configuration documentation to learn more about setting up config files.
Then, you want to use the new client you defined to perform your query, and rescue any Mongo::Error::NoServerAvailable errors.
begin
User.with(client: 'new_client').collection.find({}).count()
rescue Mongo::Error::NoServerAvailable => e
# perform error handling here
end
Note that this starts up a new Mongo::Client instance, which is an expensive operation if done repeatedly. I would recommend that you close the extra client once you are done using it, like so:
Mongoid::Clients.with_name('new_client').close
I am trying to invoke a simple lambda function (the lambda function prints hello world to console) using ruby . However when I run the code and look at the swf dashboard . I see the following error :
Reason: An Activity cannot send a response with data larger than 32768 characters. Please limit the size of the response. You can look at the Activity Worker logs to see the original response.
Could someone help me out to resolve this issue?
the code is as follows:
require 'aws/decider'
require 'aws-sdk'
class U_Act
extend AWS::Flow::Activities
activity :b_u do
{
version: "1.0"
}
end
def b_u(c_id)
lambda=Aws::Lambda::Client.new(
region: “xxxxxx”
access_key_id: “XxXXXXXXXXX”,
secret_access_key: “XXXXXXXXXX”
)
resp = lambda.invoke(
function_name: “s_u_1” # required
)
print "#{resp}"
end
Thanks
According to AWS documentation you cannot send input / result data set size larger than 32,000 characters. This limit affects activity or workflow execution result data, input data when scheduling activity tasks or workflow executions, and input sent with a workflow execution signal.
Workaround to resolve this issue are
Use AWS S3 to upload the message and send the path of the S3 message between the activities.
If you need high performance use Elasticache and store the values and pass the keys between the activities.
Stackers
I have a lot of messages in a RabbitMQ queue (running on localhost in my dev environment). The payload of the messages is a JSON string that I want to load directly into Elastic Search (also running on localhost for now). I wrote a quick ruby script to pull the messages from the queue and load them into ES, which is as follows :
#! /usr/bin/ruby
require 'bunny'
require 'json'
require 'elasticsearch'
# Connect to RabbitMQ to collect data
mq_conn = Bunny.new
mq_conn.start
mq_ch = mq_conn.create_channel
mq_q = mq_ch.queue("test.data")
# Connect to ElasticSearch to post the data
es = Elasticsearch::Client.new log: true
# Main loop - collect the message and stuff it into the db.
mq_q.subscribe do |delivery_info, metadata, payload|
begin
es.index index: "indexname",
type: "relationship",
body: payload
rescue
puts "Received #{payload} - #{delivery_info} - #{metadata}"
puts "Exception raised"
exit
end
end
mq_conn.close
There are around 4,000,000 messages in the queue.
When I run the script, I see a bunch of messages, say 30, being loaded into Elastic Search just fine. However, I see around 500 messages leaving the queue.
root#beep:~# rabbitmqctl list_queues
Listing queues ...
test.data 4333080
...done.
root#beep:~# rabbitmqctl list_queues
Listing queues ...
test.data 4332580
...done.
The script then silently exits without telling me an exception. The begin/rescue block never triggers an exception so I don't know why the script is finishing early or losing so many messages. Any clues how I should debug this next.
A
I've added a simple, working example here:
https://github.com/elasticsearch/elasticsearch-ruby/blob/master/examples/rabbitmq/consumer-publisher.rb
It's hard to debug your example when you don't provide examples of the test data.
The Elasticsearch "river" feature is deprecated, and will be removed, eventually. You should definitely invest time into writing your own custom feeder, if RabbitMQ and Elasticsearch are a central part of your infrastructure.
Answering my own question, I then learned that this is a crazy and stupid way to load a message queue of index instructions into Elastic. I created a river and can drain instructions much faster than I could with a ropey script. ;-)
I am using ganglia 3.6.0 for monitoring. I have an application that collect, aggregate some metrics for all hosts in the cluster. Then it sends them to gmond. The application runs on host1.
The problem here is, when setting spoof = false, ganglia eventually thinks this is metrics that only comes from host1. In fact, these metrics are generated by host1 but for all hosts in the cluster.
But when setting spoof=true, I expect gmond will accept the host name I specified. But it gmond doesn't accept the metrics at all. The metrics is event not shown on host1.
The code I use is copied from GangliaSink (from hadoop common) which applied Ganglia 3.1x format.
xdr_int(128); // metric_id = metadata_msg
xdr_string(getHostName()); // hostname
xdr_string(name); // metric name
xdr_int(1); // spoof = True
xdr_string(type); // metric type
xdr_string(name); // metric name
xdr_string(gConf.getUnits()); // units
xdr_int(gSlope.ordinal()); // slope
xdr_int(gConf.getTmax()); // tmax, the maximum time between metrics
xdr_int(gConf.getDmax()); // dmax, the maximum data value
xdr_int(1); /*Num of the entries in extra_value field for
Ganglia 3.1.x*/
xdr_string("GROUP"); /*Group attribute*/
xdr_string(groupName); /*Group value*/
// send the metric to Ganglia hosts
emitToGangliaHosts();
// Now we send out a message with the actual value.
// Technically, we only need to send out the metadata message once for
// each metric, but I don't want to have to record which metrics we did and
// did not send.
xdr_int(133); // we are sending a string value
xdr_string(getHostName()); // hostName
xdr_string(name); // metric name
xdr_int(1); // spoof = True
xdr_string("%s"); // format field
xdr_string(value); // metric value
// send the metric to Ganglia hosts
emitToGangliaHosts();
I do specified the host name for each metrics. But it seems not used/recognized by gmond.
Solved this... The hostName format issue. The format need to be like ip:hostname e.g 1.2.3.4:host0000001 or any string:string is fine :-)