I am running an Apache Storm benchmark on a local machine.
However, I am seeing a weird behavior. One of the benchmarks i.e., SOL (Speed of light) test, uses a RandomMessageSpout to generate random tuples as source. Here is the nextTuple() code for that spout:
public void nextTuple() {
final String message = messages[rand.nextInt(messages.length)];
if(ackEnabled) {
collector.emit(new Values(message), messageCount);
messageCount++;
} else {
collector.emit(new Values(message));
}
}
When I run this benchmark and profile it using a Java profiler (Yourkit in my case). The spout thread shows sleep intervals in accordance with the SleepSpoutWaitStrategy.emptyEmit(). As my understanding goes, this function is called when nextTuple() has no tuples to emit and thus the spout thread goes to sleep for a configurable amount of time, as shown in the screenshot.
I do not understand why this function would be called given this particular nextTuple() implementation that will always return a tuple. What I am misunderstanding here?
Empty emit is also called in following situations
If the number of unacked messages reach the max spout pending.
If the executor send queue as well as the overflow buffer of spout is full.
Related
I have the following code:
class ARouter {
public static ActorRef getRouter(actorContext, param1, param2, routerName) {
ActorRef router;
try {
RoundRobinPool roundRobinPool = new RoundRobinPool(1);
Props props = Props.create(MyActor.class, param1, param2, param3);
router = actorContext.actorOf(roundRobinPool.props(props), routerName);
} catch (Exception e) {
router = null;
}
return router;
}
}
and somewhere in my code I do this
ActorRef router = ARouter.getRouter(actorContext, param1, param2, routerName);
anObject.getAListOfItems().forEach(listItem -> router.tell(listItem, getSelf()));
I would expect to to see one thread because although I send the messages to the router to dispatch them to the actors, the router was created with only one routee (If I understand it correctly).
I tried with different number of instances but I always get 8 threads. The only think that worked (and of course "crashed") was setting new RoundRobinPool(0) which worked and the application protested that no actors were available.
No custom configuration file is used. Is there something in the logic of routers that I don't understand?
It's not completely clear what you're asking (your code nowhere refers to threads), but in Akka, a dispatcher schedules an actor's message processing to run on a thread when that actor has a message to process. The standard implementation leverages a thread pool (in 2.6, the default pool has a size equal to the number of cores (counting a hyperthread as a core), 2.5 by default uses a larger pool to guard against inadvertent blocking starving system components): an actor's message processing in that implementation can happen in any thread in the pool.
So if your actors are logging which thread they're running on, for instance, you may see that the actor is running on multiple threads. This is generally desirable: the actor's one-message-at-a-time processing model still ensures safety, and not being pinned to a particular thread in turn means that with n threads in the pool, any combination of n actors can be processing at the same time.
There are alternative dispatcher implementations which will pin an actor to a thread: if actors A and B are both pinned to thread T, then B cannot process a message if A is processing a message. In some scenarios, this reduces context-switch overhead and improves throughput at some cost to latency.
In general, an actor shouldn't care which particular thread it's running on.
I have a task that is potentially long running (hours). The task is performed by multiple workers (AWS ECS instances in my case) that read from a message queue (AWS SQS in my case). I have multiple users adding messages to the queue. The problem is that if Bob adds 5000 messages to the queue, enough to keep the workers busy for 3 days, then Alice comes along and wants to process 5 tasks, Alice will need to wait 3 days before any of Alice's tasks even start.
I would like to feed messages to the workers from Alice and Bob at an equal rate as soon as Alice submits tasks.
I have solved this problem in another context by creating multiple queues (subqueues) for each user (or even each batch a user submits) and alternating between all subqueues when a consumer asks for the next message.
This seems, at least in my world, to be a common problem, and I'm wondering if anyone knows of an established way of solving it.
I don't see any solution with ActiveMQ. I've looked a little at Kafka with it's ability to round-robin partitions in a topic, and that may work. Right now, I'm implementing something using Redis.
I would recommend Cadence Workflow instead of queues as it supports long running operations and state management out of the box.
In your case I would create a workflow instance per user. Every new task would be sent to the user workflow via signal API. Then the workflow instance would queue up the received tasks and execute them one by one.
Here is a outline of the implementation:
public interface SerializedExecutionWorkflow {
#WorkflowMethod
void execute();
#SignalMethod
void addTask(Task t);
}
public interface TaskProcessorActivity {
#ActivityMethod
void process(Task poll);
}
public class SerializedExecutionWorkflowImpl implements SerializedExecutionWorkflow {
private final Queue<Task> taskQueue = new ArrayDeque<>();
private final TaskProcesorActivity processor = Workflow.newActivityStub(TaskProcesorActivity.class);
#Override
public void execute() {
while(!taskQueue.isEmpty()) {
processor.process(taskQueue.poll());
}
}
#Override
public void addTask(Task t) {
taskQueue.add(t);
}
}
And then the code that enqueues that task to the workflow through signal method:
private void addTask(WorkflowClient cadenceClient, Task task) {
// Set workflowId to userId
WorkflowOptions options = new WorkflowOptions.Builder().setWorkflowId(task.getUserId()).build();
// Use workflow interface stub to start/signal workflow instance
SerializedExecutionWorkflow workflow = cadenceClient.newWorkflowStub(SerializedExecutionWorkflow.class, options);
BatchRequest request = cadenceClient.newSignalWithStartRequest();
request.add(workflow::execute);
request.add(workflow::addTask, task);
cadenceClient.signalWithStart(request);
}
Cadence offers a lot of other advantages over using queues for task processing.
Built it exponential retries with unlimited expiration interval
Failure handling. For example it allows to execute a task that notifies another service if both updates couldn't succeed during a configured interval.
Support for long running heartbeating operations
Ability to implement complex task dependencies. For example to implement chaining of calls or compensation logic in case of unrecoverble failures (SAGA)
Gives complete visibility into current state of the update. For example when using queues all you know if there are some messages in a queue and you need additional DB to track the overall progress. With Cadence every event is recorded.
Ability to cancel an update in flight.
See the presentation that goes over Cadence programming model.
I have a requirement of add sleep statement if i am unable to consume message and want to retry after 5s. To do this do i need to set any configuration properties?
rdd.foreachPartition(new VoidFunction<Iterator<ConsumerRecord<String, Object>>>() {
#Override
public void call(Iterator<ConsumerRecord<String, Object>> record)
throws Exception {
while (record.hasNext()) {
ConsumerRecord<String, Object> consumerRecord = record.next();
boolean flag=false;
while(flag){
flag= processmessage(record.value())
if(!flag)
Thread.sleep(1000)
}
}
}
});
Currently, i am unable to run my job
you can use sleep in spark streaming application.
But wait,
Spark streaming job runs micro batches and we define the stream interval time which is usually in few seconds ( could be 1s, 2s,... etc ). If you use sleep in your spark streaming code, it will take additional time to finish running each micro batch. This might impact the performance if data is coming very frequently.
It totally depends on application requirement whether sleep will make any performance issue or delay or it might just not impact if data is coming after long intervals.
Hope this helps.
Let me give a brief context first:
I have a scenario where the RSUs will broadcast a fixed message 'RSUmessage' about every TRSU seconds. I have implemented the following code for RSU broadcast (also, these fixed messages have Psid = -100 to differentiate them from others):
void TraCIDemoRSU11p::handleSelfMsg(cMessage* msg) {
if (WaveShortMessage* wsm = dynamic_cast<WaveShortMessage*>(msg)) {
if(wsm->getPsid()==-100){
sendDown(RSUmessage->dup());
scheduleAt(simTime() + trsu +uniform(0.02, 0.05), RSUmessage);
}
}
else {
BaseWaveApplLayer::handleSelfMsg(wsm);
}
}
A car can receive these messages from other cars as well as RSUs. RSUs discard the messages received from cars. The cars will receive multiple such messages, do some comparison stuff and periodically broadcast a similar type of message : 'aggregatedMessage' per interval Tcar. aggregatedMessage also have Psid=-100 ,so that the message can be differentiated from other messages easily.
I am scheduling the car events using self messages. (Though it could have been done inside handlePositionUpdate I believe). The handleSelfMsg of a car is following:
void TraCIDemo11p::handleSelfMsg(cMessage* msg) {
if (WaveShortMessage* wsm = dynamic_cast<WaveShortMessage*>(msg)) {
wsm->setSerial(wsm->getSerial() +1);
if (wsm->getPsid() == -100) {
sendDown(aggregatedMessage->dup());
//sendDelayedDown(aggregatedMessage->dup(), simTime()+uniform(0.1,0.5));
scheduleAt(simTime()+tcar+uniform(0.01, 0.05), aggregatedMessage);
}
//send this message on the service channel until the counter is 3 or higher.
//this code only runs when channel switching is enabled
else if (wsm->getSerial() >= 3) {
//stop service advertisements
stopService();
delete(wsm);
}
else {
scheduleAt(simTime()+1, wsm);
}
}
else {
BaseWaveApplLayer::handleSelfMsg(msg);
}
}
PROBLEM: With this setting, the simulation is very very slow. I get about 50 simulation seconds in 5-6 hours or more in Express mode in OMNET GUI. (No. of RSU: 64, Number of Vehicle: 40, around 1kmx1km map)
Also, I am referring to this post. The OP says that he got faster speed by removing the sending of message after each RSU received a message. In my case I cannot remove that, because I need to send out the broadcast messages after each interval.
Question: I think that this slowness is because every node is trying to sendDown messages at the beginning of each simulated second. Is it the case that when all vehicles and nodes schedules and sends message at the same time OMNET slows down? (Makes sense to slow down , but by what degree) But, there are only around 100 nodes overall in the simulation. Surely it cannot be this slow.
What I tried : I tried using sendDelayedDown(wsm->dup(), simTime()+uniform(0.1,0.5)); to spread the sending of the messages through out 1st half of each simulated second. This seems to stop messages piling up at the beginning of each simulation seconds and sped things up a bit, but not so much overall.
Can anybody please let me know if this is normal behavior or whether I am doing something wrong.
Also I apologize for the long post. I could not explain my problem without giving the context.
It seems you are flooding your network with messages: Every message from an RSU gets duplicated and transmitted again by every Car which has received this message. Hence, the computational time increases quadratically with the number of nodes (sender of messages) in your network (every sent message has to be handled by every node which is in range to receive it). The limit of 3 transmissions per message does not seem to help much and, as the comment in the code indicates, is not used at all, if there is no channel switching.
Therefore, if you can not improve/change your code to simply send less messages, you have to live with that. Your little tweak to send the messages in a delayed manner only distributes the messages over one second but does not solve the problem of flooding.
However, there are still some hints you can follow to improve the performance of your simulation:
Compile in release mode: make MODE=Release
Run your simulation in the terminal environment Cmdenv: ./run -u Cmdenv ...
If you need to use the graphical environment by all means, you should at least speed up the animations by using the slider in the upper part of the interface.
Removing the simtime-resolution parameter from the omnetpp.ini file solves the problem.
It seems the simulation kernel has an issue when the channel delay does not match the simulation-time resolution.
You can verify the solution by cloning the following repository. Note that you need a functional installation of the OMNeT++ framework. Specifically, I test this fix in OMNeT++ 5.6.2.
https://github.com/Ryuuba/flooding
Now I am learning Storm's Guaranteeing Message Processing and am confused by some concepts in this part.
To guarantee a message emitted by a spout is fully processed, Storm uses acker to achieve this. Each time a spout emits a tuple, acker will assign "ack val" initialized as 0 to store the status of the tuple tree. Each time the downstream bolts of this tuple emit new tuple or ack an "old" tuple, the tuple ID will be XOR with "ack val". The acker only needs to check whether "ack val" is 0 or not to know the tuple has been fully processed. Let's see the code below:
public class WordReader implements IRichSpout {
... ...
while((str = reader.readLine()) != null){
this.collector.emit(new Values(str), str);
... ...
}
The code piece above is a spout in word count program from "Getting Started with Storm" tutorial. In the emit method, the 2nd parameter "str" is the messageId. I am confused by this parameter:
1) As I understand, each time a tuple (i.e., a message) is emitted no matter in spouts or in bolts, it should be Storm's responsibility to assign a 64-bit messageId to that message. Is that correct? Or here "str" is just a human-readable alias to this message?
2) No matter what's answer to 1), here "str" would be the same word in two different messages because in a text file there should be many duplicate words. If this is true, then how does Storm differentiate different messages? And what's the meaning of this parameter?
3) In some code piece, I see some spouts use the following code to set the message Id in Spout emit method:
public class RandomIntegerSpout extends BaseRichSpout {
private long msgId = 0;
collector.emit(new Values(..., ++msgId), msgId);
}
This is much closer to what I think it should be: the message ID should be totally different across different messages. But for this code piece, another confusion is: what will happen to private field "msgId" across different executors? Because each executor has its own msgId initialized as 0, then messages in different executors will be named from 0, 1, 2, and so on. Then how does Storm differentiate these messages?
I am novice to Storm, so maybe these problems are naive. Hope someone could help me to figure out. Thanks!
About message ID is general: internally it might be a 64bit value, but this 64bit value is computed as a hash from the msgID object provided in emit() within Spout. So you can hand any object as message ID (the probability that two objects hash to the same value is close to zero).
About using str: I think in this example, str contains a line (and not a word) and it is very unlikely that document contains the exact same line twice (if there are no empty lines which might be many).
About the counter as message id: you are absolutely right about you observation -- if multiple spouts are running in parallel, this would give message ID conflict and would break fault tolerance.
If you want to "fix" the counter approach, each counter should be initialized differently (best, from 1...#SpoutTasks). You can use the taskID for this (which is unique and can be accessed via TopologyContext provided in Spout.open()). Basically, you get all taskIDs for all parallel spout tasks, sort them, and assign each spout task its ordering number. Furthermore, you need to increment by "number of parallel spouts" instead of 1.