I am using ganglia to monitor Hadoop. gmond and gmetad are running fine. When I telnet on gmond port (8649) and when I telnet gmetad on its xml answer port, I get no hadoop data. How can it be ?
cluster {
name = "my cluster"
owner = "Master"
latlong = "unspecified"
url = "unspecified"
}
host {
location = localhost
}
udp_send_channel {
#bind_hostname = yes
#mcast_join = 239.2.11.71
host = localhost
port = 8649
ttl = 1
}
udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
# buffer = 10485760
}
tcp_accept_channel {
port = 8649
# If you want to gzip XML output
gzip_output = no
}
I find out the issue. It was related to the hadoop metrics properties. I set up ganglia in the hadoop-metrics.properties but I had to set up hadoop-metrics.properties config file. Now ganglia throws correct metrics.
Related
I have a basic Gatling script on EC2 instance from which I want to push the results into an Influx database instance. I can successfully run a Gatling script and Influx is also running.
My Gatling configuration is the following:
data {
writers = [console, graphite] # The list of DataWriters to which Gatling write simulation data (currently supported : console, file, graphite)
console {
#light = false # When set to true, displays a light version without detailed request stats
#writePeriod = 5 # Write interval, in seconds
}
file {
#bufferSize = 8192 # FileDataWriter's internal data buffer size, in bytes
}
leak {
#noActivityTimeout = 30 # Period, in seconds, for which Gatling may have no activity before considering a leak may be happening
}
graphite {
light = false # only send the all* stats
host = "ec2-35-181-26-79.eu-west-3.compute.amazonaws.com" # The host where the Carbon server is located
port = 2003 # The port to which the Carbon server listens to (2003 is default for plaintext, 2004 is default for pickle)
protocol = "tcp" # The protocol used to send data to Carbon (currently supported : "tcp", "udp")
rootPathPrefix = "gatling" # The common prefix of all metrics sent to Graphite
bufferSize = 8192 # Internal data buffer size, in bytes
writePeriod = 1 # Write period, in seconds
}
And for Influx, I've setup a Telegraf with the following configuration
[[outputs.influxdb_v2]]
## The URLs of the InfluxDB cluster nodes.
##
## Multiple URLs can be specified for a single cluster, only ONE of the
## urls will be written to each interval.
## urls exp: http://127.0.0.1:8086
urls = ["http://ec2-35-181-26-79.eu-west-3.compute.amazonaws.com:8086"]
## Token for authentication.
token = "$INFLUX_TOKEN"
## Organization is the name of the organization you wish to write to; must exist.
organization = "Test"
## Destination bucket to write into.
bucket = "Test"
[[inputs.socket_listener]]
## URL to listen on
service_address = "tcp://:2003"
data_format = "graphite"
## Content encoding for message payloads, can be set to "gzip" to or
## "identity" to apply no encoding.
# content_encoding = "identity"
templates = [
"gatling.*.*.*.* measurement.simulation.request.status.field",
"gatling.*.users.*.* measurement.simulation.measurement.request.field"
]
With both Telegraf (with this configuration) and Influx running, I don't see any data pushed into the 'Test' bucket. Moreover I don't get any errors that could help me debugging.
Any help would be much appreciated. Thanks.
I have built an api web application, which is published on IIS Server, I am trying to configure Apache Flume to listen that web api and to save the response of http petitions in HDFS, this is the post method that I need to listen:
[HttpPost]
public IEnumerable<Data> obtenerValores(arguments arg)
{
Random rdm = new Random();
int ano = arg.ano;
int rdmInt;
decimal rdmDecimal;
int anoActual = DateTime.Now.Year;
int mesActual = DateTime.Now.Month;
List<Data> ano_mes_sales = new List<Data>();
while (ano <= anoActual)
{
int mes = 1;
while ((anoActual == ano && mes <= mesActual) || (ano < anoActual && mes <= 12))
{
rdmInt = rdm.Next();
rdmDecimal = (decimal)rdm.NextDouble();
Data anoMesSales = new Data(ano, mes,(rdmInt * rdmDecimal));
ano_mes_sales.Add(anoMesSales);
mes++;
}
ano++;
}
return ano_mes_sales;
}
Flume is running over a VMware Virtual Machine CentOs, this is my attempt to configure flume to listen that application:
# Sources, channels, and sinks are defined per # agent name, in this case 'tier1'.
a1.sources = source1
a1.channels = channel1
a1.sinks = sink1
a1.sources.source1.interceptors = i1 i2
a1.sources.source1.interceptors.i1.type = host
a1.sources.source1.interceptors.i1.preserveExisting = false
a1.sources.source1.interceptors.i1.hostHeader = host
a1.sources.source1.interceptors.i2.type = timestamp
# For each source, channel, and sink, set # standard properties.
a1.sources.source1.type = org.apache.flume.source.http.HTTPSource
a1.sources.source1.bind = transacciones.misionempresarial.com/CSharpFlume
a1.sources.source1.port = 80
# JSONHandler is the default for the httpsource #
a1.sources.source1.handler = org.apache.flume.source.http.JSONHandler
a1.sources.source1.channels = channel1
a1.channels.channel1.type = memory
a1.sinks.sink1.type = hdfs
a1.sinks.sink1.hdfs.path = /monthSales
a1.sinks.sink1.hdfs.filePrefix = event-file-prefix-
a1.sinks.sink1.hdfs.round = false
a1.sinks.sink1.channel = channel1
# Other properties are specific to each type of # source, channel, or sink. In this case, we # specify the capacity of the memory channel.
a1.channels.channel1.capacity = 1000
I am using curl to post, here is my attempt:
curl -X POST -H 'Content-Type: application/json; charset=UTF-8' -d '[{"ano":"2010"}]' http://transacciones.misionempresarial.com/CSharpFlume/api/SourceFlume/ObtenerValores
I only get this error:
{"Message":"Error."}
My question are, which is the right way to configure flume to listen http petitions to my web api, what I am missing?
The standard Flume 'HTTPSource', and its default JSONHandler, will only process an event in a specific, Flume-centric format.
That format is documented in the user manual, and also in the comments at the beginning of the JSONHandler source code.
In summary, it expects to receive a list of JSON objects, each one containing headers (key/value pairs, mapped to the Flume Event headers) and body (a simple string, mapped to the Flume Event body).
To take your example, if you send:
[{"headers": {}, "body": "{\"ano\":\"2010\"}"}]
I think you'd get what you were looking for.
If you don't have the flexibility to change what you send, then you may be able to use org.apache.flume.source.http.BLOBHandler, depending upon what processing you are trying to do (NB. there's no documentation in the manual for this, only for org.apache.flume.sink.solr.morphline.BlobHandler - they are not the same thing, but there are some notes in FLUME-2718), or you may need to provide your own implementation of Flume's HTTPSourceHandler interface instead.
Side note: the HTTP Source bind option requires a hostname or IP address. You may just be being lucky with your value being treated as the hostname, and the path being ignored.
Below is my flume config file. Even after the changing the rollInterval and rollSize only 10 events is getting written also the console shows rollCount=10 and events=10. Also I tried increasing the rollCount to 1000 but no change in output. Can anyone suggest to increase the file size being written in hdfs. Whats wrong with the below conf file?
#naming components
NetAgent.sources = NetCat_1 NetCat_2
NetAgent.sinks = HDFS
NetAgent.channels = MemChannel
NetAgent.sources.NetCat_1.type = netcat
NetAgent.sources.NetCat_1.bind = localhost
NetAgent.sources.NetCat_1.port = 8671
NetAgent.sources.NetCat_2.type = netcat
NetAgent.sources.NetCat_2.bind = localhost
NetAgent.sources.NetCat_2.port = 8672
NetAgent.sinks.HDFS.type = hdfs
NetAgent.sinks.HDFS.hdfs.path = file path here
NetAgent.sinks.HDFS.hdfs.filePrefix = test
NetAgent.sinks.HDFS.hdfs.rollSize = 67108864
NetAgent.sinks.HDFS.hdfs.rollInterval = 3600
NetAgent.sinks.HDFS.rollCount = 0
NetAgent.sinks.HDFS.hdfs.batchSize = 10000
NetAgent.sinks.HDFS.hdfs.writeFormat = Text
NetAgent.sinks.HDFS.hdfs.fileType = DataStream
NetAgent.channels.MemChannel.type = memory
NetAgent.channels.MemChannel.capacity = 20000
NetAgent.channels.MemChannel.transactionCapacity = 20000
NetAgent.sources.NetCat_1.channels = MemChannel
NetAgent.sources.NetCat_2.channels = MemChannel
NetAgent.sinks.HDFS.channel = MemChannel
The console logs as
(SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUg-org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java)]
rolling: rollCount: 10, events: 10
the image shows the files written in HDFS
You forgot to add hdfs to your rollCount configuration. It is using the default value of 10 because it doesn't see your configuration. Notice that your config for HDFS is:
NetAgent.sinks.HDFS.type = hdfs
NetAgent.sinks.HDFS.hdfs.rollSize = 67108864
NetAgent.sinks.HDFS.hdfs.rollInterval = 3600
NetAgent.sinks.HDFS.rollCount = 0
NetAgent.sinks.HDFS.hdfs.batchSize = 10000
NetAgent.sinks.HDFS.hdfs.writeFormat = Text
NetAgent.sinks.HDFS.hdfs.fileType = DataStream
In the rollCount line, it needs to be:
NetAgent.sinks.HDFS.hdfs.rollCount = 0
This will override the default rollCount and your Flume agent will behave how you want it to.
I want to monitor Hadoop (Hadoop version-0.20.2) multi node cluster using ganglia. My Hadoop is working properly.I have installed Ganglia after reading following blogs---
http://hakunamapdata.com/ganglia-configuration-for-a-small-hadoop-cluster-and-some-troubleshooting/
http://hokamblogs.blogspot.in/2013/06/ganglia-overview-and-installation-on.html
I have also studied Monitoring with Ganglia.pdf(APPENDIX B
Ganglia and Hadoop/HBase ).
I have modified only the following lines in **Hadoop-metrics.properties**(same on all Hadoop Nodes)==>
// Configuration of the "dfs" context for ganglia
dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=192.168.1.182:8649
// Configuration of the "mapred" context for ganglia
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=192.168.1.182:8649:8649
// Configuration of the "jvm" context for ganglia
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=192.168.1.182:8649
**gmetad.conf** (Only on Hadoop master Node )
data_source "Hadoop-slaves" 5 192.168.1.182:8649
RRAs "RRA:AVERAGE:0.5:1:302400" //Because i want to analyse one week data.
**gmond.conf** (on all the Hadoop Slave nodes and Hadoop Master)
globals {
daemonize = yes
setuid = yes
user = ganglia
debug_level = 0
max_udp_msg_len = 1472
mute = no
deaf = no
allow_extra_data = yes
host_dmax = 0 /*secs */
cleanup_threshold = 300 /*secs */
gexec = no
send_metadata_interval = 0
}
cluster {
name = "Hadoop-slaves"
owner = "Sandeep Priyank"
latlong = "unspecified"
url = "unspecified"
}
/* The host section describes attributes of the host, like the location */
host {
location = "CASL"
}
/* Feel free to specify as many udp_send_channels as you like. Gmond
used to only support having a single channel */
udp_send_channel {
host = 192.168.1.182
port = 8649
ttl = 1
}
/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
port = 8649
}
/* You can specify as many tcp_accept_channels as you like to share
an xml description of the state of the cluster */
tcp_accept_channel {
port = 8649
}
Now Ganglia is only giving system metrics(mem , disk etc.) for all the nodes. But it is not showing the Hadoop metrics( like jvm, mapred metrics
etc. ) on the web interface. how can i fix this problem ?
I do work Hadoop with Ganglia, and yes, I see on Ganglia a lot of metrics of Hadoop (Containers, map task, vmem). In fact, Hadoop specific report to Ganglio more of hundred metrics.
The hokamblogs Post was enough for this.
I edit hadoop-metrics2.properties on the master node and the content is:
namenode.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
namenode.sink.ganglia.period=10
namenode.sink.ganglia.servers=gmetad_hostname_or_ip:8649
resourcemanager.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
resourcemanager.sink.ganglia.period=10
resourcemanager.sink.ganglia.servers=gmetad_hostname_or_ip:8649
and I also edit the same files on the slaves:
datanode.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
datanode.sink.ganglia.period=10
datanode.sink.ganglia.servers=gmetad_hostname_or_ip:8649
nodemanager.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
nodemanager.sink.ganglia.period=10
nodemanager.sink.ganglia.servers=gmetad_hostname_or_ip:8649
Your remember restart Hadoop and Ganglia after change the files.
I hope this help you.
Thanks to everyone, If you are using older version of Hadoop then put following files( from new version of Hadoop) ==>
GangliaContext31.java
GangliaContext.java
In path ==> hadoop/src/core/org/apache/hadoop/metrics/ganglia
From the new version of Hadoop.
Compile your Hadoop using ant ( and set proper proxy while compiling).
If it gives error like function definition is missing then put that function definition( from new version) in proper java file and then compile Hadoop again. It will work.
I am new to Flume-Ng and need help to tail a file. I have a cluster running hadoop with flume running remotely. I communicate to this cluster by using putty. I want to tail a file on my PC and put it on the HDFS in the cluster. I am using the following code to this.
#flume.conf: http source, hdfs sink
# Name the components on this agent
tier1.sources = r1
tier1.sinks = k1
tier1.channels = c1
# Describe/configure the source
tier1.sources.r1.type = exec
tier1.sources.r1.command = tail -F /(Path to file on my PC)
# Describe the sink
tier1.sinks.k1.type = hdfs
tier1.sinks.k1.hdfs.path = /user/ntimbadi/flume/
tier1.sinks.k1.hdfs.filePrefix = events-
tier1.sinks.k1.hdfs.round = true
tier1.sinks.k1.hdfs.roundValue = 10
tier1.sinks.k1.hdfs.roundUnit = minute
# Use a channel which buffers events in memory
tier1.channels.c1.type = memory
tier1.channels.c1.capacity = 1000
tier1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
tier1.sources.r1.channels = c1
tier1.sinks.k1.channel = c1
I believe the mistake is in the source. This kind source does not take the host name or i.p to look for(in this case should be my PC). Could someone just give me a hint as to how to tail a file on my PC to upload it to the remotely located HDFS using flume.
The exec source in your configuration will run on the machine where you start the flume's tier1 agent. If you want to collect data from another machine, you'll need to start a flume agent on that machine too; to sum up you need:
an agent (remote1) running on the remote machine that has an avro source, which will listen for events from collector agents and will act like an aggregator.
an agent (local1) running on your machine (to act like a collector) that has an exec source and sends data to the remote agent via avro sink.
Or alternatively, you can have only one flume agent running on your local machine (having the same configuration you posted) and set the hdfs path as "hdfs://REMOTE_IP/hdfs/path" (though I'm not entirely sure this will work).
edit:
Below are the sample configurations for the 2-agents scenario (they may not work without some modification).
remote1.channels.mem-ch-1.type = memory
remote1.sources.avro-src-1.channels = mem-ch-1
remote1.sources.avro-src-1.type = avro
remote1.sources.avro-src-1.port = 10060
remote1.sources.avro-src-1.bind = 10.88.66.4 /* REPLACE WITH YOUR MACHINE'S EXTERNAL IP */
remote1.sinks.k1.channel = mem-ch-1
remote1.sinks.k1.type = hdfs
remote1.sinks.k1.hdfs.path = /user/ntimbadi/flume/
remote1.sinks.k1.hdfs.filePrefix = events-
remote1.sinks.k1.hdfs.round = true
remote1.sinks.k1.hdfs.roundValue = 10
remote1.sinks.k1.hdfs.roundUnit = minute
remote1.sources = avro-src-1
remote1.sinks = k1
remote1.channels = mem-ch-1
and
local1.channels.mem-ch-1.type = memory
local1.sources.exc-src-1.channels = mem-ch-1
local1.sources.exc-src-1.type = exec
local1.sources.exc-src-1.command = tail -F /(Path to file on my PC)
local1.sinks.avro-snk-1.channel = mem-ch-1
local1.sinks.avro-snk-1.type = avro
local1.sinks.avro-snk-1.hostname = 10.88.66.4 /* REPLACE WITH REMOTE IP */
local1.sinks.avro-snk-1.port = 10060
local1.sources = exc-src-1
local1.sinks = avro-snk-1
local1.channels = mem-ch-1