I have more than five hundred topic in one cluster with six nodes, I create the topic like this :
./mqadmin updateTopic -c MyCluster -n 192.168.1.100:9876 -r 16 -w 16 -t topic1;
./mqadmin updateTopic -c MyCluster -n 192.168.1.100:9876 -r 16 -w 16 -t topic500;
I can send the message without delay ,but I find sometime that the client can not consume the message because the broker do not have consume client.How to fix this bug?
do following check
more than 1 consumer instance started in one machine.
if so , make every consumer instance client id or client name different
consumer instances in this c_fy_core consumer group sub different topic and tags.
if so , make sure all consumer instances in one consumer group sub same topic-and-tags
Related
I have a container creating celery tasks, and a container running a worker.
I have removed the worker container, so I expected that tasks would accumulate in the redis list of tasks.
But I can't see any tasks in redis.
This is with django. I need to isolate the worker and queue, hence the settings
A typical queue name is 'test-dear', that is, SHORT_HOSTNAME='test-dear'
CELERY_DATABASE_NUMBER = 0
CELERY_BROKER_URL = f"redis://{REDIS_HOST}:6379/{CELERY_DATABASE_NUMBER}"
CELERY_RESULT_BACKEND = f"redis://{REDIS_HOST}:6379/{CELERY_DATABASE_NUMBER}"
CELERY_BROKER_TRANSPORT_OPTIONS = {'global_keyprefix': SHORT_HOSTNAME }
CELERY_TASK_DEFAULT_QUEUE = SHORT_HOSTNAME
CELERY_TASK_ACKS_LATE = True
After starting everything, and stopping the worker, I add tasks.
For example, on the producer container after python manage.py shell
>>> from cached_dear import tasks
>>> t1 = tasks.purge_deleted_masterdata_fast.delay()
<AsyncResult: 9c9a564a-d270-444c-bc71-ff710a42049e>
t1.get() does not return.
then in redis:
127.0.0.1:6379> llen test-dear
(integer) 0
I was not expecting 0 entries.
What I am doing wrong or not understanding?
I did this from the redis container
redis-cli monitor | grep test-dear
and sent a task.
The list is test-deartest-dear and
llen test-deartest-dear
works to show the number of tasks which have not yet been sent to a worker.
The queue name is f"{global_keyprefix}{queue_name}
I read the consumer group example in here(provided by Shopify/sarama),
and I wonder that how would I know how many consumers are in a consumer group, are there any ways I could control the number of consumers?
Thanks a lot
To list the consumers currently assigned to a group, you can use the kafka-consumer-groups .sh shell script. Assuming that $KAFKA_HOME is the root directory of your Kafka installation, you can do:
$KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --group YOURGROUP --zookeeper ZOOKEEPER_HOST:PORT
If you want to list current groups, run the following command:
$KAFKA_HOME/bin/kafka-consumer-groups.sh --list --zookeeper ZOOKEEPER_HOST:PORT
You can use ClusterAdmin interface in Sarama library.
https://pkg.go.dev/github.com/Shopify/sarama#ClusterAdmin
A utility bash function to clear a kafka topic by manipulating the retention interval is as follows:
clearKafka() {
tname=$1 ;
kafka-topics.sh --zookeeper localhost:2181 --alter --topic $tname --config retention.ms=1000
sleep 25s;
kafka-console-consumer.sh --from-beginning --bootstrap-server localhost:9092 --property print.key=true --property print.value=false --property print.partition --topic $tname --timeout-ms 300 | tail -n 10|grep "Processed a total of"
kafka-topics.sh --zookeeper localhost:2181 --alter --topic $tname --config retention.ms=600000;
sleep 25s;
kafka-console-consumer.sh --from-beginning --bootstrap-server localhost:9092 --property print.key=true --property print.value=false --property print.partition --topic $tname --timeout-ms 300 | tail -n 10|grep "Processed a total of"
}
This is a little messy (WARNING's..) but does work on linux - notice the last information that found 0 messages.
$ clearKafka airsmall
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
Going forward, please use kafka-configs.sh for this functionality
Updated config for topic airsmall.
[2019-05-13 03:11:01,552] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 2000 messages
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
Going forward, please use kafka-configs.sh for this functionality
Updated config for topic airsmall.
[2019-05-13 03:11:01,552] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 0 messages
The same utility on mac ends up with showing 2000 messages even at the very end.
Processed a total of 2000 messages
And in fact still shows 2000 messages with that command even several minutes later . So what is the deal on mac for this ? How to clear a topic on mac?
Turns out the command does work on Mac .. it just takes much longer: after another few minutes - maybe 5 or 10 minutes total - the topic is showing up as cleared
Processed a total of 0 messages
Not sure why linux gets it done in a few tens of seconds but mac requires minutes..
I was executing two mqsc commands on IBM MQ v8 to find out the number of connections made by App1 on queue manager QMGR and found a discrepancy between the output values on one of the attribute called CONNAME
echo 'dis chs('APP1.SVRCONN.CHL')'| runmqsc QMGR | grep CONNAME
This time I got 14 IP's as CONNAME
echo 'dis qs(APP1.QUEUE) type(handle) All where(CHANNEL eq 'APP1.SVRCONN.CHL')'| runmqsc QMGR | grep CONNAME
When I checked the number of handles on the only App1 queue connected by App1 channel I got only 7 IP's as CONNAME
So there are 7 handles on the App1 queue but there are 14 connections by the App1 channel on queue manager. How can I relate these two values? Is this multi-threaded connections on each queue handles and if so how can I find a relationship between them?
The DIS CHS command will show you how many channel instances are running which is really how many TCP sessions there are established between your client and the queue manager.
MQ v7.0.1 and higher supports shared conversations within each channel. Each time the app makes a connection to the queue manager this can be another conversation in the same channel instance. The client and the server can set a maximum number of shared connections. On the queue manager's SVRCONN the attribute is SHARECNV, on the client it could be in a CCDT or programatically. When the channel starts up the client and server will negotiate to the lowest value between them. On a DIS CHS you can add MAXSHCNV to see what the running channel instances negotiated the maximum to and CURSHCNV to see how many conversations are running in each channel instance.
The DIS QS command with TYPE(HANDLE) will only show you connections where the application has a handle open to that queue at the point in time you ran the command.
An application can connect to the queue manager without having a queue open, or it may open and close the queue, if you don't catch it with the queue open you won't see a handle.
A more accurate command to show what the channel is doing is DIS CONN which will show all connections, each connection represents a conversation, so the number of connections should match the number total of CURSHCNV from all channel instances.
The following command will show you all connections running through a channel with the name specified.
echo "DIS CONN(*) TYPE(ALL) WHERE(CHANNEL EQ APP1.SVRCONN.CHL)"|runmqsc QMGR
The output may look like this:
AMQ8276: Display Connection details.
CONN(ABABABABABABABAB)
EXTCONN(ABABABABABABABABABABABABABABABAB)
TYPE(*)
PID(99998) TID(998)
APPLDESC(WebSphere MQ Channel) APPLTAG(App1)
APPLTYPE(USER) ASTATE(NONE)
CHANNEL(APP1.SVRCONN.CHL) CLIENTID( )
CONNAME(10.10.10.10)
CONNOPTS(MQCNO_HANDLE_SHARE_BLOCK,MQCNO_SHARED_BINDING)
USERID(appuser) UOWLOG( )
UOWSTDA( ) UOWSTTI( )
UOWLOGDA( ) UOWLOGTI( )
URTYPE(QMGR)
EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
QMURID(0.0) UOWSTATE(NONE)
AMQ8276: Display Connection details.
CONN(BABABABABABABABA)
EXTCONN(BABABABABABABABABABABABABABABABA)
TYPE(*)
PID(99999) TID(999)
APPLDESC(WebSphere MQ Channel) APPLTAG(App1)
APPLTYPE(USER) ASTATE(STARTED)
CHANNEL(APP1.SVRCONN.CHL) CLIENTID( )
CONNAME(10.10.10.10)
CONNOPTS(MQCNO_HANDLE_SHARE_BLOCK,MQCNO_SHARED_BINDING)
USERID(appuser) UOWLOG( )
UOWSTDA(2018-05-21) UOWSTTI(10.11.27)
UOWLOGDA( ) UOWLOGTI( )
URTYPE(QMGR)
EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
QMURID(0.99999) UOWSTATE(ACTIVE)
OBJNAME(APP1.QUEUE) OBJTYPE(QUEUE)
ASTATE(ACTIVE) HSTATE(INACTIVE)
OPENOPTS(MQOO_INPUT_SHARED,MQOO_BROWSE,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
In the above output CONN(ABABABABABABABAB) only has a connection to the queue manager and no objects open while CONN(BABABABABABABABA) had a connection to the queue manager and has the queue APP1.QUEUE open.
If you were to add to the above command |grep 'APP1.QUEUE' the count should match the number of handles from the DIS QS command.
You may have a app that browses the queue on one connection to look for messages and then dispatches to other threads to actually process the messages. In your case 7 channel instances may have the queue open for BROWSE and the other 7 would be waiting to open the queue for INPUT to read a message off.
Below is a command I use on Linux that will spit out a CSV format of the DIS CONN command along with each object that is open.
echo "DIS CONN(*) TYPE(ALL) WHERE(CHANNEL EQ CHL_NAME)"|runmqsc QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["APPLTAG"], p["USERID"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'
The fields in the CSV output are:
"CONN","CHANNEL","APPLTAG","USERID","CONNAME","OBJNAME","OBJTYPE","OPENOPTS"
Objects can be the QMGR itself, a QUEUE, TOPIC, etc. The CONN id will be repeated in lines of the output if the same CONN has multiple objects open. If the CONN has no objects open you will see a single entry with no objects listed at the end of the line, if at least one object is open there will not be a line representing no objects:
I am trying to transfer a 60mb file to queue, but Websphere MQ fte stall the transfer and keep recovering. I am running WebSphere MQ FTE on default configuration.
I have tested following scenario with different results according to configuration changes I made.
These commands were issued to create monitor:
fteCreateTransfer -sa AGENT1 -sm TQM.FTE -da AGENT2 -dm QM.FTE -dq FTE.TEST.Q -p QM.FTE -de overwrite -sd delete -gt /var/IBM/WMQFTE/config/TQM.FTE/TEST_TRANSFER.xml D:\\rvs\\tstusrdat\\ALZtoSIP\\INC\\*.zip
fteCreateMonitor -ma AGENT1 -mn TEST_MONITOR -md D:\\rvs\\tstusrdat\\ALZtoSIP\\INC -mt /var/IBM/WMQFTE/config/TQM.FTE/TEST_TRANSFER.xml -tr match,*.zip
Test was performed on files: 53MB and 30MB
Default configuration (just enableQueueInputOutput=true added to AGENT2.properties)
1) all default
no success, transfer status: "recovering"
for both files
2) added maxInputOutputMessageLength=60000000, destination queue max message length changed to 103809024
result transfer status: "failed" with following exception PM71138: BFGIO0178E: A QUEUE WRITE FAILED DUE TO A WMQAPIEXCEPTION WITH MESSAGE TEXT CC=2 RC=2142 MQRC_HEADER_ERROR
for both files
After reading this: http://pic.dhe.ibm.com/infocenter/wmqfte/v7r0/topic/com.ibm.wmqfte.doc/message_length.htm I came with working settings:
3) maxInputOutputMessageLength=34603008 (its maximum value), destination queue max message length still to 103809024
result for file with size 30MB: succcess
result for file with size 53MB: "failed" with following exception PM71138: BFGIO0178E: A QUEUE WRITE FAILED DUE TO A WMQAPIEXCEPTION WITH MESSAGE TEXT CC=2 RC=2142 MQRC_HEADER_ERROR
So according to this I am afraid one can't transfer larger then 34603008 bytes.
If you are transferring file to queue you definitely can't use the default settings. You have to add "enableQueueInputOutput=true" to agent.properties for agent thet uses queue as source or destination.