I have a certain processing task that I want to solve with kubernetes. The basic concept is that there is a certain number of items in a work queue that I want to process. Items can be added to the queue and are deleted as soon as a pod finished processing one.
The preferred work flow would be:
Defining a maximum number of pods(e.g.40)
push items to queue(e.g. 20)
number of pods is created according to number of items in queue(=>20)
while the pods are still processing on the 20 items another 40 items are pushed to the queue resulting in 20 more pods that are created(maximum number is reached), and as soon as the first ones finish additional pods are created until the end of the queue is reached.
Is there any build in solution using kubectl? Using the job pattern I can define the number of parallel Pods but those are running all the time until success and not scaled up upon other criteria.
Thanks for your help!
Use a Horizontal Pod Autoscaler. You may have to define Custom Metrics for getting the number of items in the Queue and use it in HPA.
Related
In Apache NiFi, dockerized version 1.15, a cluster of 3 NiFi nodes is created. When load balancing is used via default port 6342, flow files get stuck in some of the queues, in the queue in which load balancing is enabled. But, when "List queue" is tried, the message "The queue has no FlowFiles." is issued:
The part of the NiFi processor group where the issue happens:
Configuration of NiFi queue in which flow files seem to be stuck:
Another problem, maybe not related, is that after this happens, some of the flow files reach the subsequent NiFi processors, but get stuck before the MergeContent processors. This time, the queues can be listed:
The part of code when the second issue occurs:
The part of code when the second issue occurs
The configuration of the queue:
The listing of the FlowFiles in the queue:
The MergeContent processor configuration. The parameter "max_num_for_merge_smxs" is set to 100:
Load balancing is used because data are gathered from the SFTP server, and that processor runs only on the Primary node.
If you need more information, please let me know.
Thank you in advance!
Edited:
I put the load-balancing queues between the ConsumeMQTT (working on the Primary node only) and UpdataAttribute processors, but Flow files are seemingly staying in the load-balancing queue, but when the listing is done, the message is "The queue has no FlowFiles.". Please check:
Changed position of the load-balancing queue:
The message that there are no flow files in the queues:
Take notice that the processors before and after the queue are stopped while doing "List queue".
Edit 2:
I changed the configuration in the nifi.properties to the following:
nifi.cluster.load.balance.connections.per.node=20
nifi.cluster.load.balance.max.thread.count=60
nifi.cluster.load.balance.comms.timeout=30 sec
I also restarted the NiFi containers, so I will monitor the behaviour. For now, there are no stuck Flow files in the load-balancing queues, they go to the processor that follows the queue.
"The queue has no FlowFiles" is normal behaviour of a queue that is feeding into a Merge - the flowfiles are pending to be merged.
The most likely cause of them being "stuck" before a Merge is that you have Round Robin distributed the FlowFiles across many nodes, and then you are setting a Minimum count on the Merge. This minimum is per node and there are not enough FlowFiles on each node to hit the Minimum, so they are stuck waiting for more FlowFiles to trigger the Merge.
-- Edit
"The queue has no FlowFiles" is also expected on a queue that is active - in your flow, the load balancing queue is drained immediately into the output queue of your merge PGs Input port - so there are no FFs sitting around in the load balancing queue. If you were to STOP the Input ports inside the merge PG, you should be able to list them on the LB queue.
It sounds like you are doing GetSFTP (Primary) and then distributing the files. The better approach would be to use ListSFTP (Primary) -> Load Balance -> FetchSFTP - this would avoid shuffling large files, and would instead load balance the file names between all nodes, with each node then fetching a subset of the files.
Secondly, I would review your Merge config - you have a parameter #{max_num_for_merge_xmsx} defined, but this set in the Minimum Number of Entries for the Merge - so you are telling Merge to only ever merge when at least #{max_num_for_merge_xmsx} amount of FlowFiles is reached.
I want to use a queue for file uploads. Users can upload files. Each file will have around 500 rows. Now I want to implement this logic:
Maximum of 5 files can be processed at the same time. The remaining files should be in the queue.
Each file should have 5 processes, so 5 rows will be inserted into databases at the same time. Shortly, there are will be a maximum of 25
processes (5 processes in every 5 files).
Now I am adding all files to one queue. Files processing one by one. Shortly first-come, first out. 2nd file needs to wait to finish 1st file.
How can I implement this? Or do you have any other suggestions?
What exactly is the difference between processing a file, and inserting rows into the DB?
If you want to run multiple workers for the same queue, you can simply start more workers using php artisan queue:work and additionally use flags to specify the queues --queue=process-files for example. See the documentation.
In a production environment, consider to configure a supervisor to run a specific amount of workers on a queue using numprocs directive.
Do I understand correctly you want to run 25 queue workers per user? That does not seem right. Instead, you should consider creating queues for fast/slow jobs.
In my Laravel 5.1 project I want to start my second job when first will finished.
Here is my logic.
\Queue::push(new MyJob())
and when this job finish I want to start this job
\Queue::push(new ClearJob())
How can i realize this?
If you want this, you just should define 1 Queue.
A queue is just a list/line of things waiting to be handled in order,
starting from the beginning. When I say things, I mean jobs. - https://toniperic.com/2015/12/01/laravel-queues-demystified
To get the opposite of what you want: async executed Jobs, you should define a new Queue for every Job.
Multiple Queues and Workers
You can have different queues/lists for
storing the jobs. You can name them however you want, such as “images”
for pushing image processing tasks, or “emails” for queue that holds
jobs specific to sending emails. You can also have multiple workers,
each working on a different queue if you want. You can even have
multiple workers per queue, thus having more than one job being worked
on simultaneously. Bear in mind having multiple workers comes with a
CPU and memory cost. Look it up in the official docs, it’s pretty
straightforward.
I want to run only one pod of my kubernetes app at a time(relaunch in case of failure), I am using job controller.
But as per documentations, kubernetes may launch more than one pods and will eventually achieve specified replicas. Is there any way to achieve exactly one pod at a time or any recommended design pattern for such use cases.
My app is reading data from HDFS and writing it to a message queue. It exits after processing all the files. I want to minimize possibility of writing duplicate records.
I suggest you use replicasets for this. Set the number of replica to 1. More here https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/#when-to-use-a-replicaset
In principle, in the case of Jobs with no paralellism implied, there shouldn't be this kind of "race condition" ("should be 1" according to the documentation [1]). The job would be rescheduled only if an attempt fails. Did you come across the situation where 2 pods from the same job were being executed at the same time?
In any case, if you want to be completely sure, you may want to implement an extra coordination method or external solution.
[1] https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
If I understand your question correctly I think you are looking for: .spec.strategy.rollingUpdate.maxSurge
If you set this to 0 then the existing pods will be killed before starting an new one.
I have a single agent and many builds. There are frequently several builds in the queue that take an hour a piece to execute. I want to trigger daily at a specific time a build which takes less than five seconds but needs to run immediately (next in the queue). Is there any way to do this?
Build priorities are suggested in various places but they do not help. I set the priority to the max value of 100 and it was placed at 15 out of 17 in the queue.
You can use Teamcity REST to trigger the build and put on top of the queue. You can make use of triggering option queueAtTop="true"
I ended up working around the problem and moving this build to another practically dedicated teamcity agent which means this executes promptly. This is not a good solution and I would prefer to accept an actual answer if anyone is able to offer one.