I am trying to run multiple instances of my kafka streams application on different machines but getting following error:
Could not lock global state directory. This could happen if multiple KafkaStreams instances are running on the same host using the same state directory.
kstream-test-prod-6f9bc47d9c-bmq2z: org.apache.kafka.streams.errors.LockException: Failed to lock the global state directory: /tmp/kafka-streams/test-app/global
This is predictable in case i launch more than 1 instance on same machine, but here i am launching them on different machines, still getting the same issue.
You can run multiple instance of single kafka stream with same application/client id in different host. Kafka Stream it self will load balance between different host. There may be chance this lock may happend when you tried mutiple instance in same host. Try clean up the directory and try again
Related
Im running Kafka connect in distributed mode having two different connectors with one task each. Each connector is running in different instance which is exactly I want.
Is it always ensure the same behaviour that Kafka connect cluster share the load properly ?
Connectors in Kafka Connect run with one, or more tasks. The number of tasks depends on how you have configured the connector, and whether the connector itself can run multiple tasks. An example would be the JDBC Source connector, which if ingesting more than one table from a database will run (if configured to do so) one task per table.
When you run Kafka Connect in distributed mode, tasks from all the connectors are executed across the available workers. Each task will only be executing on one worker at one time.
If a worker fails (or is shut down) then Kafka Connect will rebalance the tasks across the remaining worker(s).
Therefore, you may see one connector running across different workers (instances), but only if it has more than one task.
If you think you are seeing the same connector's task executing more than once then it suggests a misconfiguration of the Kafka Connect cluster, and I would suggest reviewing https://rmoff.net/2019/11/22/common-mistakes-made-when-configuring-multiple-kafka-connect-workers/.
In my application I'm running several integration tests in parallel, spread among different maven profiles.
before each intgeration-test step, I'm using fabric8 to start Couchbase container, for which I use the reserve-network-port task (of builder-helper-maven-plugin) in order to allocate avaliable ports to be used by the Couchbase container.
The issue is that, multiple Jenkins jobs (different BB branches) of by (BB) repository are running in parallel on the same Jenkins machine.
from time to time, integration tests init is failed (for one or more profile), on 'address is already binded' error - meaning that 1 of the allocated tcp ports, been reserverd by the reserve-network-port task (of builder-helper-maven-plugin) has been already been allocated by another job.
Is there a way to avoid this race condition? (e.g. using some kind of resource locking when the reserve-network-port task is been execute? other solution?)
I have a Spring batch partitioned job. The job is always started with a unique set of parameters so always a new job.
My remoting fabric is JMS with request/response queues configured for communication between the masters and slaves.
One instance of this partitioned job processes files in a given folder. Master step gets the file names from the folder and submits the file names to the slaves; each slave instance processes one of the files.
Job works fine.
Recently, I started to execute multiple instances (completely separate JVMs) of this job to process files from multiple folders. So I essentially have multiple master steps running but the same set of slaves.
Randomly; I notice the following behavior sometimes - the slaves will finish their work but the master keeps spinning thinking the slaves are still doing something. The step status will show successful in the job repo but at the job level the status is STARTING with an exit code of UNKNOWN.
All masters share the set of request/response queues; one queue for requests and one for responses.
Is this a supported configuration? Can you have multiple master steps sharing the same set of queues running concurrently? Because of the behavior above I'm thinking the responses back from the workers are going to the incorrect master.
i have a jmeter distributed system with 1 master and 4 slaves.
the test is configured to run for 60 minutes.
somehow suddenly a random slave finish the test and the load is distributed between the other 3.
all the slaves configured the same way.
the instances are aws ec2 instances on the same subnet
is there any explanation for this behaviour?
It might be the case you configured JMeter to stop thread when the error occurs yourself:
if you have marked settings under Thread Group it might be the case the Threads (virtual users) are being stopped or the whole test gets stopped on error
If unexpected error occurs there should be a corresponding entry in jmeter.log file, make sure to execute JMeter slave process providing log file location via -j command-line argument like:
./jmeter -s -j jmeter-slave.log .....
It might be the case your JMeter instance runs out of memory and the whole JVM gets terminated so make sure to properly tune it for high loads
Check operating system log of your Amazon instance
There could be multiple reason for it:
Possibly load balancing was not happening properly, more sets of request are getting drived toward one instance. That can cause the VM to crash
OR It could be the crashed AWS instance. The disk space got full.
I suggest you check the disk usage of crashed vm.
I'd like to have multiple instances of rabbitmq running on my developement machine. It seems like I can only install one rabbitmq service, so I'm wondering how can I spawn multiple rabbitmq processes ?
My objective is to launch and kill "n" rabbitmq node locally via c# in my automated integration test.
If you want to kill the nodes you could create multiple RabbitMQ nodes on your local machine which run under the same service.
You can find more information here. Although this is under the 'Clustering' documentation, you should be able to create multiple nodes and just skip the 'clustering' bit.