after searching a while, I couldn't find how to submit a SpringXD app to Yarn using a specifiq queue.
Right now I have everything working, but when I submit the job it fails, presumably because the user doesn't have permission to submit jobs to the default queue (Any help getting the actual cause of failure will be greatly appreciated too).
I need the containers to run on higher queues, so my question is, how can I submit the Yarn application to the desired Yarn Queue?
Thanks.
Related
I don't want to use message expiry as it has dependency on sending application and don't want to use pub/sub as well because if the applications don't take the messages it will fill up the filesystem etc. I don't want the messages to be piled up in the queue because application is down.
This setup is required so that there wont be any outage because of this queue and the application consuming it. Any advice?
CAPEXPRY allows the administrator to set message expiry without application changes. See https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.ref.dev.doc/q097495_.htm
I have a Kafka cluster running on AWS MSK with Kafka producer and consumer go clients running in kubernetes. The producer is responsible for sending the stream of data to Kafka. I need help solving the following problems:
Let's say, there is some code change in producer code and have to redeploy it in kubernetes. How can I do that? Since the data is continuously generated, I cannot just simply stop the already running producer and deploy the updated one. In this case, I will lose the data between the update process.
Sometimes due to a panic(golang) in the code, the client crashes, but since it is running as a pod, kubernetes restarts it. I am not able to understand as to whether it's a good thing or bad.
Thanks
For your first question, I would suggest having rolling update of your deployment in the cluster.
For second, that is the general behavior of deployments in kubernetes. I could think of an external monitoring solution that de-deploys your application or stops handling requests in case of a panic.
It would be better if you could explain why exactly you need such kind of behavior.!
We have a microservice which is developed using spring boot. couple of the functionalities it implements is
1) A scheduler that triggers, at a specified time, a file download using webhdfs and process it and once the data is processed, it will send an email to users with the data process summary.
2) Read messages from kafka and once the data is read, send an email to users.
We are now planning to make this application high available either in Active-Active or Active-passive set up. The problem we are facing now is if both the instances of the application are running then both of them will try to download the file/read the data from kafka, process it and send emails. How can this be avoided? I mean to ensure that only one instance triggers the download and process it ?
Please let me know if there is known solution for this kind of scenarios as this seems to be a common scenario in most of the projects? Is master-slave/leader election approach a correct solution?
Thanks
Let the service download that file, extract the information and publish them via kafka.
Check beforehand if the information was already processed by querying kafka or a local DB.
You also could publish an DataProcessed-Event that triggers the EmailService, that sends the corresponding E-Mail.
I have following problem to solve:
Multiple Users can submit Jobs to a Queue via a Web Interface.
This Jobs are then stored in the Database via the database Queue Driver.
Now my problem is: I want the Queue to run all Jobs until inside a job I say something like $queue->pause() because for the next Job to run I need some confirmation from the User.
How would I do something like this?
run jobs
inside one job the job determines that it needs some confirmation from the user
halt the queue and keep that job which needs the confirmation in the queue
any user on the website can press a button which then would delete this confirmation job and start the queue again.
My current "solution" which didn't work was this:
create 2 different job types:
ImageProcessingJob
UserNotificaitonJob
The queue worked all ImageProcessingJobs until it hit a UserNotificationJob.
Inside the UserNotificationJob->handle() I called Artisan::call("queue:restart"); which stopped the Queue.
The problem with this solution is: The UserNotificationJob also got deleted. So if I would start the Queue again the Queue would immediately start with the remainig ImageProcessingJobs without waiting for the actual confirmation.
I'm also open to other architectural solutions without a Queue system.
One approach which avoids pausing the queue, is to have the UserNotificationJob wait on a SyncEvent (the SyncEvent is set when then confirmation comes back from the user). You can have this wait timing-out if you like, but then you need to repost the job to the queue. If you decide to timeout and repost, you can use job chaining to setup dependencies between jobs so that nothing can be run until the UserNotificationJob competes.
Another approach might be to simply avoid posting the remaining jobs until the confirmation is sent from the user.
I am looking for a clean way to implement a java event system that hooks into Hadoop v2. I know there is a notification url and I have used that in the past. What I want to do is to hook into the JobStatus and have events posted to a queue service for propagating events to client. I tried extending job and assigning the status using reflection to my custom JobStatus class but this is not working. I have also cursory looked into Yarn's event system to add a hook that will allow me to listen for yarn events and propagate those. I really need an expert opinion on how to accomplish this kind of task. I want to get log messages, status change events in real time over to a web client.
Thanks for any assistance in advance.