I am using RocketMQ and want to produce and consume data orderly.I have follwed the Order Example :
https://rocketmq.apache.org/docs/order-example/
But in consumer listener, message is not coming orderly.
Can any one tell how this Order messaging is working?
Did you set the "-o true" option when you created the topic?
For example as follows:
sh mqadmin updateTopic -c Default_Cluster -n 192.168.1.2:9876 -t TopicTest -o true -r 8 -w 8
Related
I am unable to use JMX exporter to expose kafka metrics. Can you look at my steps and correct me where needed.?
I am following steps here to enable kafka with JMX exporter.
Following are step by step instruction I followed
#get kafka
wget kafka_2.11-2.0.0
# Download Prometheus JMX exporter:
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.0/jmx_prometheus_javaagent-0.3.0.jar
sudo wget -P /opt/kafka/prometheus/ https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml
#Edit Prometheus JMX exporter config file; and append following lines
echo “- pattern : kafka.producer<type=producer-metrics, client-id=(.+)><>(.+):\w* name: kafka_producer_$2” >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo “— pattern : kafka.consumer<type=consumer-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_$2” >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo “— pattern : kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_$2” >> /opt/kafka/prometheus/kafka-0-8-2.yml
#start zookeeper in terminal 0
/opt/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties
#start kafka broker in terminal 1
KAFKA_HEAP_OPTS=’”-Xmx1000M -Xms1000M”’
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7071
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
#start kafka consumer in terminal 2
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7072:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7072
/opt/kafka/bin/kafka-console-consumer.sh — bootstrap-server 0.0.0.0:9092 — topic test — from-beginning
#start kafka producer in terminal 3
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7073:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7073
/opt/kafka/bin/kafka-console-producer.sh — broker-list 0.0.0.0:9092 — topic test
After above steps zookeeper and kafka is running fine.
I can type in producer terminal a message and it is received on consumer console. However no kafka metrics is visible on Prometheus. To debug this I checked ports 7071/2/3 by
netstat -tlnp | grep 7071
netstat -tlnp | grep 7072
netstat -tlnp | grep 7073
which resulted in blank response; this means no service is using above ports. I feel like JMX exporter is not enabled correctly.
Can you help me with above issues?
From the looks of your question, you put the variables on their own lines, while the blog has them on the same line...
e.g. This is how to start the Kafka server
KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"' KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml' JMX_PORT=7081 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Otherwise, you need to export the variables so the sub-process will pick them up like you did in your previous question, which seemed to be working fine for exposing the metrics
export KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"'
export KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml'
export JMX_PORT=7081
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Note: The blog you linked to doesn't use JMX_PORT, but those ports cannot be the same as the exporter.
I would also suggest at least downloading a version newer than 0.3 - https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/
and using the configs for Kafka 2.0 - https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml
Sidenote: netstat -tlnp | grep 707 would show you all them at once
thank you cricket-007 for your help.
I am listing steps i followed here for simplicity
wget -q -O /tmp/kafka.tgz https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
tar -xf /tmp/kafka.tgz --directory /opt/kafka --strip 1
rm -f /tmp/kafka.tgz
wget -q -O /tmp/kafka.tgz https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
mkdir /opt/kafka
tar -xf /tmp/kafka.tgz --directory /opt/kafka --strip 1
rm -f /tmp/kafka.tgz
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-2_0_0.yml
cd kafka
export KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.12.0.jar=7071:/opt/kafka/prometheus/kafka-2_0_0.yml"
export KAFKA_HEAP_OPTS="-Xmx1000M -Xms1000M"
mv ../kafka-2_0_0.yml prometheus/
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
netstat -tlnpu |grep 70
tcp6 0 0 :::7071 :::* LISTEN 209455/java
udp6 0 0 :::40705 :::*
curl -s localhost_or_IP:7071 | grep -i kafka
long list of metrics will be dumped on stdout -
I am trying to find a command or a sample shell snippet where I can wait until the list of Kubernetes pods is successful. I have checked the answer but it was not giving any output. Can someone guide me or suggest an approach, I am completely new to kubernetes.
kubectl -n test-ns get jobs -w
NAME DESIRED SUCCESSFUL AGE
test-1 1 1 2d
test-2 1 1 2d
test-3 1 1 2d
test-4 1 1 2d
until kubectl get jobs -n test-ns -o jsonpath='{.status.conditions[?(#.type=="Complete")].status}' | grep True ; do sleep 1 ; done
This is not giving any output
To wait until your pods are running, check for "condition=ready" and filter by app label, for example:
$ kubectl wait --for=condition=ready pod -l app=netshoot
pod/netshoot-58785d5fc7-xt6fg condition met
you need to use this command
kubectl rollout status
If you want to use kubectl as described here where it gets all the jobs, you need to use .items[*]... in your JSONpath (That answer is for just one specific job). For example:
kubectl -n test-ns get jobs \
-o jsonpath='{.items[*].status.conditions[?(#.type=="Complete")].status}' \
| grep True
I am writing a shell script where i want to ssh to a server and get the cpu and memory details data of that displayed as a result. I’m using the help of top command here.
Script line:
ssh -q user#host -n “cd; top -n 1 | egrep ‘Cpu|Mem|Swap’”
But the result is
TERM environment variable is not set.
I had checked the same in the server by entering set | grep TERM and got result as TERM=xterm
Please someone help me on this. Many thanks.
Try using the top -b flag:
ssh -q user#host -n "cd; top -bn 1 | egrep 'Cpu|Mem|Swap'"
This tells top to run non-interactively, and is intended for this sort of use.
top need an environment. You have to add the parameter -t to get the result:
ssh -t user#host -n "top -n 1 | egrep 'Cpu|Mem|Swap'"
Got it..!! Need to make a small modification for the below script line.
ssh -t user#host -n "top -n 1 | egrep 'Cpu|Mem|Swap'"
Instead of -t we need to give -tt. It worked for me.
To execute command top after ssh’ing. It requires a tty to run. Using -tt it will enable a force pseudo-tty allocation.
Thanks stony for providing me a close enough answer!! :)
I want to execute cql queries from bash command.
[cqlsh 3.1.8 | Cassandra 1.2.19 | CQL spec 3.0.5 | Thrift protocol 19.36.2]
[root#hostname ~]# /opt/apache-cassandra-1.2.19/bin/cqlsh -k "some_keyspace" -e "SELECT column FROM Users where key=value"
I got:
cqlsh: error: no such option: -e
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-C, --color Always use color output
--no-color Never use color output
-u USERNAME, --username=USERNAME
Authenticate as user.
-p PASSWORD, --password=PASSWORD
Authenticate using password.
-k KEYSPACE, --keyspace=KEYSPACE
Authenticate to the given keyspace.
-f FILE, --file=FILE Execute commands from FILE, then exit
-t TRANSPORT_FACTORY, --transport-factory=TRANSPORT_FACTORY
Use the provided Thrift transport factory function.
--debug Show additional debugging information
--cqlversion=CQLVERSION
Specify a particular CQL version (default: 3.0.5).
Examples: "2", "3.0.0-beta1"
-2, --cql2 Shortcut notation for --cqlversion=2
-3, --cql3 Shortcut notation for --cqlversion=3
Any suggestions ?
First of all, you should seriously consider upgrading. You are missing out on a lot of new features and bug fixes.
Secondly, with cqlsh in Cassandra 1.2 you can use the -f flag to specify a file containing cql statements:
$ echo "use system_auth; SELECT role,is_superuser FROM roles WHERE role='cassandra';" > userQuery.cql
$ bin/cqlsh -u aploetz -p reindeerFlotilla -f userQuery.cql
role | is_superuser
-----------+--------------
cassandra | True
(1 rows)
You can use -f to execute from a file or SOURCE once you start CQLSH. I don't think -e is a valid option with that version.
It's bit dirty and unstable, but here is the answer:
/opt/apache-cassandra-1.2.19/bin/cqlsh -k "keyspace" -f /path/to/file.cql > /path/to/output.txt
tail -2 /path/to/output.txt | head -1 > /path/to/output-value.txt
I want to use nc as a simple TCP/IP server. So far I run it using:
$ nc -k -l 3000 > temp.tmp
This works, but writes all the received data of all connections into one single file. But, I would like to have individual files.
Basically I get this if I skip the -k switch, but then nc shuts down as soon as the first connection is gone.
How can I keep nc running, but use an individual file for each incoming request? (Or, if this is not possible, is there an alternative *nix tool that is able to do this?)
Please note that I do not want to restart nc, but I want it all with a single running instance.
PS: I know that SO doesn't allow questions on finding tools, but for me this is only the fallback in case nc isn't able to do that by itself. So, sorry for the part in parentheses…
On the receiver:
$ nc -p 3000 -l | tar -x
On the sender:
$ tar -c * | nc <ip_address> 3000
Omit the -k and run it in a loop:
n=0
while nc -l 3000 > "$n".txt ; do
n=$((n+1))
done
I think Cronolog can help in this case
-p option you can determine the period option
Also the filewatcher utility "incron" can be used which will check logs to one file & can split to some other files