I have a .Net Core application with Hangfire implementation.
There is a recurring job per minute as below:-
RecurringJob.AddOrUpdate<IS2SScheduledJobs>(x => x.ProcessInput(), Cron.MinuteInterval(1));
var hangfireOptions = new BackgroundJobServerOptions
{
WorkerCount = 20,
};
_server = new BackgroundJobServer(hangfireOptions);
The ProcessInput() internally checks the BlockingCollection() of some Ids to process, it keeps on continuously processing.
There is a time when the first ten ProcessInput() jobs keeps on processing with 10 workers where as the other new ProcessInput() jobs gets enqueued.
For this purpose, I wanted to increase the workers count, say around 50, so that there would be 50 ProcessInput() jobs being processed parallely.
Please suggest.
Thanks.
In the .NET Core version you can set the worker count when adding the Hangfire Server:
services.AddHangfireServer(options => options.WorkerCount = 50);
you can try increasing the worker process using this method. You have to know the number of processors running on your servers. To get 100 workers, you can try
var options = new BackgroundJobServerOptions { WorkerCount = Environment.ProcessorCount * 25 };
app.UseHangfireServer(options);
Source
Related
I am currently dispatching queued jobs to send API Events instantly, in busy times these queued jobs need to be held until overnight when the API is less busy, how can I hold these queued jobs or schedule them to only run from 01:00am the following day.
the Queued Job call currently looks like:
EliQueueIdentity::dispatch($EliIdentity->id)->onQueue('eli');
there are other jobs on the same queue, all of which will need to be held in busy times
Use delay to run job at a certain time.
EliQueueIdentity::dispatch($EliIdentity->id)
->onQueue('eli')
->delay($this->scheduleDate());
Helper for calculating the time, handling a edge case between 00:00 to 01:00, where it would delay it a whole day. While not specified how to handle busy, made an pseudo example you can implement.
private function scheduleDate()
{
$now = Carbon::now();
if (! $this->busy()) {
return $now;
}
// check for edge case of 00:00:00 to 01
if ($now->hour <= 1) {
$now->setTime(1, 0, 0);
return $now;
}
return Carbon::tomorrow()->addHour();
}
You can use delayed dispatching (see https://laravel.com/docs/6.x/queues#delayed-dispatching):
// Run it 10 minutes later:
EliQueueIdentity::dispatch($EliIdentity->id)->onQueue('eli')->delay(
now()->addMinutes(10)
);
Or pass another carbon instance like:
// Run it at the end of the current week (i believe this is sunday 23:59, havent checked).
->delay(Carbon::now()->endOfWeek());
// Or run it at february second 2020 at 00:00.
->delay(Carbon::createFromFormat('Y-m-d', '2020-02-02'));
You get the picture.
We are receiving data in spark streaming from Kafka. Once execution has been started in Spark Streaming, it executes only one batch and the remaining batches starting queuing up in Kafka.
Our data is independent and can be processes in Parallel.
We tried multiple configurations with multiple executor, cores, back pressure and other configurations but nothing worked so far. There are a lot messages queued and only one micro batch has been processed at a time and rest are remained in queue.
We want to achieve parallelism at maximum, so that not any micro batch is queued, as we have enough resources available. So how we can reduce time by maximum utilization of resources.
// Start reading messages from Kafka and get DStream
final JavaInputDStream<ConsumerRecord<String, byte[]>> consumerStream = KafkaUtils.createDirectStream(
getJavaStreamingContext(), LocationStrategies.PreferConsistent(),
ConsumerStrategies.<String, byte[]>Subscribe("TOPIC_NAME",
sparkServiceConf.getKafkaConsumeParams()));
ThreadContext.put(Constants.CommonLiterals.LOGGER_UID_VAR, CommonUtils.loggerUniqueId());
JavaDStream<byte[]> messagesStream = consumerStream.map(new Function<ConsumerRecord<String, byte[]>, byte[]>() {
private static final long serialVersionUID = 1L;
#Override
public byte[] call(ConsumerRecord<String, byte[]> kafkaRecord) throws Exception {
return kafkaRecord.value();
}
});
// Decode each binary message and generate JSON array
JavaDStream<String> decodedStream = messagesStream.map(new Function<byte[], String>() {
private static final long serialVersionUID = 1L;
#Override
public String call(byte[] asn1Data) throws Exception {
if(asn1Data.length > 0) {
try (InputStream inputStream = new ByteArrayInputStream(asn1Data);
Writer writer = new StringWriter(); ) {
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(asn1Data);
GZIPInputStream gzipInputStream = new GZIPInputStream(byteArrayInputStream);
byte[] buffer = new byte[1024];
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
int len;
while((len = gzipInputStream.read(buffer)) != -1) {
byteArrayOutputStream.write(buffer, 0, len);
}
return new String(byteArrayOutputStream.toByteArray());
} catch (Exception e) {
//
producer.flush();
throw e;
}
}
return null;
}
});
// publish generated json gzip to kafka
cache.foreachRDD(new VoidFunction<JavaRDD<String>>() {
private static final long serialVersionUID = 1L;
#Override
public void call(JavaRDD<String> jsonRdd4DF) throws Exception {
//Dataset<Row> json = sparkSession.read().json(jsonRdd4DF);
if(!jsonRdd4DF.isEmpty()) {
//JavaRDD<String> jsonRddDF = getJavaSparkContext().parallelize(jsonRdd4DF.collect());
Dataset<Row> json = sparkSession.read().json(jsonRdd4DF);
SparkAIRMainJsonProcessor airMainJsonProcessor = new SparkAIRMainJsonProcessor();
airMainJsonProcessor.processAIRData(json, sparkSession);
}
}
});
getJavaStreamingContext().start();
getJavaStreamingContext().awaitTermination();
getJavaStreamingContext().stop();
Technology that we are using:
HDFS 2.7.1.2.5
YARN + MapReduce2 2.7.1.2.5
ZooKeeper 3.4.6.2.5
Ambari Infra 0.1.0
Ambari Metrics 0.1.0
Kafka 0.10.0.2.5
Knox 0.9.0.2.5
Ranger 0.6.0.2.5
Ranger KMS 0.6.0.2.5
SmartSense 1.3.0.0-1
Spark2 2.0.x.2.5
Statistics that we got from difference experimentations:
Experiment 1
num_executors=6
executor_memory=8g
executor_cores=12
100 Files processing time 48 Minutes
Experiment 2
spark.default.parallelism=12
num_executors=6
executor_memory=8g
executor_cores=12
100 Files processing time 8 Minutes
Experiment 3
spark.default.parallelism=12
num_executors=6
executor_memory=8g
executor_cores=12
100 Files processing time 7 Minutes
Experiment 4
spark.default.parallelism=16
num_executors=6
executor_memory=8g
executor_cores=12
100 Files processing time 10 Minutes
Please advise, how we can process maximum so no queued.
I was facing same issue and I tried a few things in trying to resolve the issue and came to following findings:
First of all. Intuition says that one batch must be processed per executor but on the contrary, only one batch is processed at a time but jobs and tasks are processed in parallel.
Multiple batch processing can be achieved by using spark.streaming.concurrentjobs, but it's not documented and still needs a few fixes. One of problems is with saving Kafka offsets. Suppose we set this parameter to 4 and 4 batches are processed in parallel, what if 3rd batch finishes before 4th one, which Kafka offsets would be committed. This parameter is quite useful if batches are independent.
spark.default.parallelism because of its name is sometimes considered to make things parallel. But its true benefit is in distributed shuffle operations. Try different numbers and find an optimum number for this. You will get a considerable difference in processing time. It depends upon shuffle operations in your jobs. Setting it too high would decrease the performance. It's apparent from you experiments results too.
Another option is to use foreachPartitionAsync in place of foreach on RDD. But I think foreachPartition is better as foreachPartitionAsync would queue up the jobs whereas batches would appear to be processed but their jobs would still be in the queue or in processing. May be I didn't get its usage right. But it behaved same in my 3 services.
FAIR spark.scheduler.mode must be used for jobs with lots of tasks as round-robin assignment of tasks to jobs, gives opportunity to smaller tasks to start receiving resources while bigger tasks are processing.
Try to tune your batch duration+input size and always keep it below processing duration otherwise you're gonna see a long backlog of batches.
These are my findings and suggestions, however, there are so many configurations and methods to do streaming and often one set of operation doesn't work for others. Spark Streaming is all about learning, putting your experience and anticipation together to get to a set of optimum configuration.
Hope it helps. It would be a great relief if someone could tell specifically how we can legitimately process batches in parallel.
We want to achieve parallelism at maximum, so that not any micro batch is queued
That's the thing about stream processing: you process the data in the order it was received. If you process your data at the rate slower than it arrives it will be queued. Also, don't expect that processing of one record will suddenly be parallelized across multiple nodes.
From your screenshot, it seems your batch time is 10 seconds and your producer published 100 records over 90 seconds.
It took 36s to process 2 records and 70s to process 17 records. Clearly, there is some per-batch overhead. If this dependency is linear, it would take only 4:18 to process all 100 records in a single mini-batch thus beating your record holder.
Since your code is not complete, it's hard to tell what exactly takes so much time. Transformations in the code look fine but probably the action (or subsequent transformations) are the real bottlenecks. Also, what's with producer.flush() which wasn't mentioned anywhere in your code?
I was facing the same issue and I solved it using Scala Futures.
Here are some link that show how to use it:
https://alvinalexander.com/scala/how-use-multiple-scala-futures-in-for-comprehension-loop
https://www.beyondthelines.net/computing/scala-future-and-execution-context/
Also, this is piece of my code when I used Scala Futures:
messages.foreachRDD{ rdd =>
val f = Future {
// sleep(100)
val newRDD = rdd.map{message =>
val req_message = message.value()
(message.value())
}
println("Request messages: " + newRDD.count())
var resultrows = newRDD.collect()//.collectAsList()
processMessage(resultrows, mlFeatures: MLFeatures, conf)
println("Inside scala future")
1
}
f.onComplete {
case Success(messages) => println("yay!")
case Failure(exception) => println("On no!")
}
}
It's hard to tell without having all the details, but general advice to tackle issues like that -- start with very simple application, "Hello world" kind. Just read from input stream and print data into log file. Once this works you prove that problem was in application and you gradually add your functionality back until you find what was culprit. If even simplest app doesn't work - you know that problem in configuration or Spark cluster itself. Hope this helps.
I have created an application using IBM XMS.NET. All is good and I am able to read the mssages from queue.
I want to read only those messages which are older that 2 mins from now.
How to use selector in this case. Below is code I have created.
var time = 120000; // 2 mins in miliseconds
var currentTime = (DateTime.Now - DateTime.MinValue).TotalMilliseconds; // current time in milliseconds
long finaltime = Convert.ToInt64(currentTime - time); // Time in milliseconds after substracting 2 minutes
var dtt = Convert.ToInt64(((new DateTime(1970, 01, 01, 01, 00, 00)) - DateTime.MinValue).TotalMilliseconds); // Time in miliseconds till 1970
finaltime = finaltime - dtt; // substracting milliseconds till 1976 as JMSTimestamp shows time after 1970
string selector = "JMSTimestamp <=" + finaltime.ToString();
Here selector is getting set as fixed value for example 1454322340382.
How I can set selector to choose latest DateTime.Now and then look for message older that DateTime.Now - 2 minutes.
Selecting on those messages that are older than 2 minutes is probably the most inefficient way to look at those messages. You don't say why you want to do this. If you simply want to discard them, then I suggest you have the producer of the message add an expiry time on those messages. If you cannot get the producer of the message to change their application to do this, then consider using the CAPEXPRY administrative over-ride.
If you want to look at them, then browsing through all the messages and only operating on those which are the right age would be more efficient that selecting them I'm sure.
Because the selector is passed as parameter at the time of creating consumer, it can be changed without closing and recreating the consumer.
MessageConsumer receiver;
receiver = session.createConsumer(stockQueue, selector);
Update:
Evaluation of selector expression happens during creation of consumer. The
DateTime.Now - 2 expression evaluates to a fixed value and does not change. For example "JMSTimestamp <= 1454322340382". So when a consumer is created with that selection string, consumer will get only those messages that match the above condition.
While the above is fine. But when the consumer is getting messages, new messages can come into the queue. Those message may become older than 2 minutes by the time consumer attempts to get them. Consumer will not get those messages even though they are older than two minutes because their JMSTimestamp is higher, for example 1454666666666. To remove such messages you will have to close the consumer and create it again with updated selector condition.
Hope I am clear.
For your use case, I would go for MQ Base .NET API instead of XMS .NET. First browse messages and if a message is older than 2 minutes remove it.
queueBrowse = queueManager.AccessQueue(strQueueName, MQC.MQOO_BROWSE + MQC.MQOO_INPUT_AS_Q_DEF + MQC.MQOO_FAIL_IF_QUIESCING);
try
{
MQMessage msgBrowse = new MQMessage();
MQGetMessageOptions mqgmoBrowse = new MQGetMessageOptions();
mqgmoBrowse.Options = MQC.MQGMO_BROWSE_NEXT;
queueBrowse.Get(msgBrowse, mqgmoBrowse);
TimeSpan ts = DateTime.Now.ToUniversalTime().Subtract(msgBrowse.PutDateTime);
if (ts.TotalMinutes > 2)
{
MQMessage msgDelete = new MQMessage();
msgDelete.MessageId = msgBrowse.MessageId;
MQGetMessageOptions mqgmo = new MQGetMessageOptions();
mqgmo.MatchOptions = MQC.MQMO_MATCH_MSG_ID;
queueBrowse.Get(msgDelete, mqgmo);
Console.WriteLine("Message older than 2 minutes deleted");
}
else
{
Console.WriteLine("Message not older than 2 minutes");
}
}
catch (MQException ex)
{
}
I've got a system that uses Akka 2.2.4 which creates a bunch of local actors and sets them as the routees of a Broadcast Router. Each worker handles some segment of the total work, according to some hash range we pass it. It works great.
Now, I've got to cluster this application for failover. Based on the requirement that only one worker per hash range exist/be triggered on the cluster, it seems to me that setting up each one as a ClusterSingletonManager would make sense..however I'm having trouble getting it working. The actor system starts up, it creates the ClusterSingletonManager, it adds the path in the code cited below to a Broadcast Router, but it never instantiates my actual worker actor to handle my messages for some reason. All I get is a log message: "unhandled event ${my message} in state Start". What am I doing wrong? Is there something else I need to do to start up this single instance cluster? Am I sending the wrong actor a message?
here's my akka config(I use the default config as a fallback):
akka{
cluster{
roles=["workerSystem"]
min-nr-of-members = 1
role {
workerSystem.min-nr-of-members = 1
}
}
daemonic = true
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = "127.0.0.1"
port = ${akkaPort}
}
}
actor{
provider = akka.cluster.ClusterActorRefProvider
single-message-bound-mailbox {
# FQCN of the MailboxType. The Class of the FQCN must have a public
# constructor with
# (akka.actor.ActorSystem.Settings, com.typesafe.config.Config) parameters.
mailbox-type = "akka.dispatch.BoundedMailbox"
# If the mailbox is bounded then it uses this setting to determine its
# capacity. The provided value must be positive.
# NOTICE:
# Up to version 2.1 the mailbox type was determined based on this setting;
# this is no longer the case, the type must explicitly be a bounded mailbox.
mailbox-capacity = 1
# If the mailbox is bounded then this is the timeout for enqueueing
# in case the mailbox is full. Negative values signify infinite
# timeout, which should be avoided as it bears the risk of dead-lock.
mailbox-push-timeout-time = 1
}
worker-dispatcher{
type = PinnedDispatcher
executor = "thread-pool-executor"
# Throughput defines the number of messages that are processed in a batch
# before the thread is returned to the pool. Set to 1 for as fair as possible.
throughput = 500
thread-pool-executor {
# Keep alive time for threads
keep-alive-time = 60s
# Min number of threads to cap factor-based core number to
core-pool-size-min = ${workerCount}
# The core pool size factor is used to determine thread pool core size
# using the following formula: ceil(available processors * factor).
# Resulting size is then bounded by the core-pool-size-min and
# core-pool-size-max values.
core-pool-size-factor = 3.0
# Max number of threads to cap factor-based number to
core-pool-size-max = 64
# Minimum number of threads to cap factor-based max number to
# (if using a bounded task queue)
max-pool-size-min = ${workerCount}
# Max no of threads (if using a bounded task queue) is determined by
# calculating: ceil(available processors * factor)
max-pool-size-factor = 3.0
# Max number of threads to cap factor-based max number to
# (if using a bounded task queue)
max-pool-size-max = 64
# Specifies the bounded capacity of the task queue (< 1 == unbounded)
task-queue-size = -1
# Specifies which type of task queue will be used, can be "array" or
# "linked" (default)
task-queue-type = "linked"
# Allow core threads to time out
allow-core-timeout = on
}
fork-join-executor {
# Min number of threads to cap factor-based parallelism number to
parallelism-min = 1
# The parallelism factor is used to determine thread pool size using the
# following formula: ceil(available processors * factor). Resulting size
# is then bounded by the parallelism-min and parallelism-max values.
parallelism-factor = 3.0
# Max number of threads to cap factor-based parallelism number to
parallelism-max = 1
}
}
}
}
Here's where I create my Actors(its' written in Groovy):
Props clusteredProps = ClusterSingletonManager.defaultProps("worker".toString(), PoisonPill.getInstance(), "workerSystem",
new ClusterSingletonPropsFactory(){
#Override
Props create(Object handOverData) {
log.info("called in ClusterSingetonManager")
Props.create(WorkerActorCreator.create(applicationContext, it.start, it.end)).withDispatcher("akka.actor.worker-dispatcher").withMailbox("akka.actor.single-message-bound-mailbox")
}
} )
ActorRef manager = system.actorOf(clusteredProps, "worker-${it.start}-${it.end}".toString())
String path = manager.path().child("worker").toString()
path
when I try to send a message to the actual worker actor, should the path above resolve? Currently it does not.
What am I doing wrong? Also, these actors live within a Spring application, and the worker actors are set up with some #Autowired dependencies. While this Spring integration worked well in a non-clustered environment, are there any gotchyas in a clustered environment I should be looking out for?
thank you
FYI:I've also posted this in the akka-user google group. Here's the link.
The path in your code is to the ClusterSingletonManager actor that you start on each node with role "workerSystem". It will create a child actor (WorkerActor) with name "worker-${it.start}-${it.end}" on the oldest node in the cluster, i.e. singleton within the cluster.
You should also define the name of the ClusterSingletonManager, e.g. system.actorOf(clusteredProps, "workerSingletonManager").
You can't send the messages to the ClusterSingletonManager. You must send them to the path of the active worker, i.e. including the address of the oldest node. That is illustrated by the ConsumerProxy in the documentation.
I'm not sure you should use a singleton at all for this. All workers will be running on the same node, the oldest. I would prefer to discuss alternative solutions to your problem at the akka-user google group.
I wrote a little test for a simple scenario:
One publisher and one subscriber
Publisher send 1000000 messages
Subscriber receive the 1000000 messages
First test with RabbitMQ, fanout Exchange, RabbitMq node type Ram : 320 seconds
Second test with Redis, basic pub/Sub : 24 seconds
Am i missing something? Why a such difference ? Is this a configuration problem or something?
First scenario: one node.js process for the subscriber, one for the publisher, each one, one connection to rabbitmq with amqp node module.
Second scénario: one node.js process for the subscriber, one for the publisher, each one got one connection to redis.
Any help is welcom to understand... I can share the code if needed.
i'm pretty new to all of this.
What i need, is a high performances pub / sub messaging system. I'd like to have clustering capabilities.
To run my test, i just launch the rabbitMq server (default configuration) and i use the following
Publisher.js
var sys = require('sys');
var amqp = require('amqp');
var nb_messages = process.argv[2];
var connection = amqp.createConnection({url: 'amqp://guest:guest#localhost:5672'});
connection.addListener('ready', function () {
exchangeName = 'myexchange';
var start = end = null;
var exchange = connection.exchange(exchangeName, {type: 'fanout'}, function(exchange){
start = (new Date()).getTime();
for(i=1; i <= nb_messages; i++){
if (i%1000 == 0){
console.log("x");
}
exchange.publish("", "hello");
}
end = (new Date()).getTime();
console.log("Publishing duration: "+((end-start)/1000)+" sec");
process.exit(0);
});
});
Subscriber.js
var sys = require('sys');
var amqp = require('amqp');
var nb_messages = process.argv[2];
var connection = amqp.createConnection({url: 'amqp://guest:guest#localhost:5672'});
connection.addListener('ready', function () {
exchangeName = 'myexchange';
queueName = 'myqueue'+Math.random();
var queue = connection.queue(queueName, function (queue) {
queue.bind(exchangeName, "");
queue.start = false;
queue.nb_messages = 0;
queue.subscribe(function (message) {
if (!queue.start){
queue.start = (new Date()).getTime();
}
queue.nb_messages++;
if (queue.nb_messages % 1000 == 0){
console.log('+');
}
if (queue.nb_messages >= nb_messages){
queue.end = (new Date()).getTime();
console.log("Ending at "+queue.end);
console.log("Receive duration: "+((queue.end - queue.start)/1000));
process.exit(0);
}
});
});
});
Check to ensure that:
Your RabbitMQ queue is not configured as persistent (since that would require disk writes for each message)
Your prefetch count on the subscriber side is 0
You are not using transactions or publisher confirms
There are other things which could be tuned, but without knowing the details of your test it's hard to guess. I would just make sure that you are comparing "apples to apples".
Most messaging products can be made to go as fast as humanly possible at the expense of various guarantees (like delivery assurance, etc) so make sure you understand your application's requirements first. If your only requirement is for data to get shoveled from point A to point B and you can tolerate the loss of some messages, pretty much every messaging system out there can do that, and do it well. The harder part is figuring out what you need beyond raw speed, and tuning to meet those requirements as well.