Crystal lang. How to realize pattern producer-consumer with a few producers? - fifo

A few days ago I asked Does it have a crystal-lang Queue?, and #Johannes Müller answered me for using Channel::Buffered(T). Now, I have a littlebit another question. What if I have more than one producer? And all of them using the same Channel?
For example:
channel = Channel(String).new
# first producer
spawn do
5.times do |i|
puts "first producer #{i}"
channel.send "first #{i}"
sleep 0.1
end
end
# second producer, that send data to the same with first producer channel
spawn do
5.times do |i|
puts "second producer #{i}"
channel.send "second #{i}"
sleep 0.1
end
end
# consumer
spawn do
loop do
data = channel.receive
puts "receive: #{data}"
sleep 0.5
end
end
sleep 6
Output will be:
$ crystal ./test.cr
first producer 0 # ok. first produced 0
second producer 0 # ok. second produced 0
receive: first 0 # ok. received from the first producer
first producer 1 # o_O. Where received data from the second producer?
receive: first 1
first producer 2
receive: first 2
first producer 3
receive: first 3
first producer 4
receive: first 4
receive: second 0 # aa. It's here... Why it's happend only, when the first producer was produced all?
second producer 1
receive: second 1
second producer 2
receive: second 2
second producer 3
receive: second 3
second producer 4
receive: second 4
As you can see, the "second producer" send first package almost at the same time with the first, but it's was ignored untill the first producer finished the job. And doesn't matter, channel buffered or not.
Why did not they work consistently? If the first producer, works in an eternal cycle, data from the second producer never been received.
Probably this behavior should be, and send data to one place from several - obviously bad practice?

Read the docs!
Not sure if i properly understood what you meant, but is this the expected output?:
First producer sent: 0
Consumer received "First: 0"!
Second producer sent: 0
Consumer received "Second: 0"!
First producer sent: 1
Consumer received "First: 1"!
Second producer sent: 1
Consumer received "Second: 1"!
First producer sent: 2
Consumer received "First: 2"!
Second producer sent: 2
Consumer received "Second: 2"!
First producer sent: 3
Consumer received "First: 3"!
Second producer sent: 3
Consumer received "Second: 3"!
First producer sent: 4
Consumer received "First: 4"!
Second producer sent: 4
Consumer received "Second: 4"!
If so; this is the code:
class CrystalIsAwesome
##products = 0
##channel = Channel(String).new
def self.produce(which, times, produce_time = 0)
##products += times
spawn do
times.times do |i|
puts "#{which} producer sent: #{i}"
##channel.send "#{which}: #{i}"
sleep produce_time.seconds
end
end
end
# first producer
produce "First", 5
# second producer
produce "Second", 5
##products.times do |_|
puts %(Consumer received "#{##channel.receive}"!)
end
end
Try adding a third producer (produce "Third", 2, 1) and see how it plays out.
Again; see the Concurrency docs for why this works like this.

Related

RabbitMQ Consumer-Increment-Count Configuration In Spring Boot

I have these configurations:
container.setMaxConcurrentConsumers(100);
container.setConcurrentConsumers(1);
container.setPrefetchCount(1);
container.setAutoStartup(true);
container.setConsecutiveActiveTrigger(1);
And this works like: starts with 1 consumer and goes on 1 + 1 + 1 + 1....100(max-consumer) with each active consecutive trigger. Is there a way to increase it like: starts with 1 consumer and goes on 1 + 5 + 5 + 5 ... 100(max-consumer) with each active consecutive trigger?
So it increases the consumer count 1 by 1. But I want to change it like 5 by 5 or 10 by 10.
No; only one consumer is added for each trigger.
I suggest you open a new feature request: https://github.com/spring-projects/spring-amqp/issues

How to make ruby loops wait for 15 min after each 300 element?

I want to make ruby script that will print all followers for any account, but twitters API will gife me an error (too many requests) after 300 printed follower, how can i make loop to print the frist 300 then wait for 15 min then to start where its done to another 300?
you can do it like:
some_variable = 0
loop do
#**your code that puts element **
some_variable += 1
sleep(15*60) if (some_variable % 300).zero?
end

Errors with inter-group communications

I have to implement a scenario where I have 2 groups. Group 1 contains only process 0 and 1. Group 2 contains all the processes. Now the processes in Group 2 read some data and send it to Group 1(process 0 or 1, equal division). It should look like this:
Grp1 Grp2
Rank
(grp)
0 0 0
1 1 1
2 2
3 3
4 4 and so on..
Now to implement this, I have used the following code (All the variables and other things are taken care)
program mpi
use mpi
integer ierr,new_rank1,new_grp1,new_rank2,new_grp2,new_comm1,new_comm2
integer numtasks,rank,orig_grp,tag,sendbuf,recvbuf
integer ranks1(2),ranks2(8),stat(MPI_STATUS_SIZE)
data ranks1 /0,1/,ranks2 /0,1,2,3,4,5,6,7/
call mpi_init(ierr)
call mpi_comm_rank(mpi_comm_world,rank,ierr)
call mpi_comm_size(mpi_comm_world,numtasks,ierr)
call MPI_COMM_GROUP(MPI_COMM_WORLD,orig_grp,ierr)
tag = 342
call MPI_GROUP_INCL(orig_grp,8,ranks2,new_grp2,ierr)
call mpi_comm_create_group(mpi_comm_world,new_grp2,112,new_comm2,ierr)
call mpi_comm_rank(new_comm2,new_rank2,ierr)
if(new_rank2 == 4) then
call mpi_send(123,1,MPI_INT,0,tag,new_comm2,ierr) !send a msg from rank 4 of group 2 to rank 0 of group 1
end if
if(rank <2) then
call MPI_GROUP_INCL(orig_grp,2,ranks1,new_grp1,ierr)
call mpi_comm_create_group(mpi_comm_world,new_grp1,111,new_comm1,ierr)
call mpi_comm_rank(new_comm1,new_rank1,ierr)
if(new_rank1 == 0) then
call mpi_recv(recvbuf,1,MPI_INT,4,tag,new_comm2,stat,ierr)
print*,recvbuf
end if
end if
call mpi_finalize(ierr)
end
Facing errors in sending and receiving data. The other things work fine. And also if the whole concept is wrong, I am open to suggestions.

IO bound threads in ruby

In a ruby application I have a bunch of tasks which share no state and I want to launch them off many at a time. Crucially, I don't care about the order they are started in, nor their return values (as they will each incur database transactions before they complete). I'm aware that depending on my ruby implementation the GIL may prevent these tasks from actually running at the same time, but that's OK because I'm not actually interested in true concurrency: these worker threads will be IO bound over network requests anyways.
What I've got so far is this:
def asyncDispatcher(numConcurrent, stateQueue, &workerBlock)
workerThreads = []
while not stateQueue.empty?
while workerThreads.length < numConcurrent
nextState = stateQueue.pop
nextWorker =
Thread.new(nextState) do |st|
workerBlock.call(st)
end
workerThreads.push(nextWorker)
end # inner while
workerThreads.delete_if{|th| not th.alive?} # clean up dead threads
end # outer while
workerThreads.each{|th| th.join} # join any remaining workers
end # asyncDispatcher
And I invoke it like this:
asyncDispatcher(2, (1..10).to_a ) {|x| x + 1}
Are there any lurking bugs or concurrency pitfalls here? Or perhaps something in the runtime which would simplify this task?
Use a Queue:
require 'thread'
def asyncDispatcher(numWorkers, stateArray, &processor)
q = Queue.new
threads = []
(1..numWorkers).each do |worker_id|
threads << Thread.new(processor, worker_id) do |processor, worker_id|
while true
next_state = q.shift #shift() blocks if q is empty, which is the case now
break if next_state == q #Some sentinel that won't appear in your data
processor.call(next_state, worker_id)
end
end
end
stateArray.each {|state| q.push state}
stateArray.each {q.push q} #Some sentinel that won't appear in your data
threads.each(&:join)
end
asyncDispatcher(2, (1..10).to_a) do |state, worker_id|
time = sleep(Random.rand 10) #How long it took to process state
puts "#{state} is finished being processed: worker ##{worker_id} took #{time} secs."
end
--output:--
2 is finished being processed: worker #1 took 4 secs.
3 is finished being processed: worker #1 took 1 secs.
1 is finished being processed: worker #2 took 7 secs.
5 is finished being processed: worker #2 took 1 secs.
6 is finished being processed: worker #2 took 4 secs.
7 is finished being processed: worker #2 took 1 secs.
4 is finished being processed: worker #1 took 8 secs.
8 is finished being processed: worker #2 took 1 secs.
10 is finished being processed: worker #2 took 3 secs.
9 is finished being processed: worker #1 took 9 secs.
Okay, okay, someone is going look at that output and cry out,
Hey, #2 took a total of 13 seconds to do four jobs in a row, while #1
took only 8 secs. for a job, so #1's output for the 8 sec. job should
have come earlier. There's no thread switching in Ruby! Ruby is
broken!".
Well, while #1 was sleeping for its first two jobs for a total of 5 seconds, #2 was sleeping at the same time, so #2 only had 2 more seconds left to sleep when #1 finished it's first two jobs. So replace #2's 7 secs by 2 secs, and you'll see that after number #1 finished its first two jobs, #2 took a total of 8 seconds for its run of four jobs in a row, which tied #1 for it's 8 second job.

Perform a loop for a certain time interval or while condition is met

I am trying to have a check fire off every second for 30 seconds. I haven't found a clear way to do this with Ruby yet. Trying something like this currently:
until counter == 30
sleep 1
if condition
do something
break
else
counter +=1
end
Problem with something like that is it has to use sleep, which stops the thread in its tracks for a full second. Is there another way to achieve something similar to the above without the use of sleep? Is there a way to have something cycle though on a time based interval?
You can approximate what you're looking for with something along these lines:
now = Time.now
counter = 1
loop do
if Time.now < now + counter
next
else
puts "counting another second ..."
end
counter += 1
break if counter > 30
end
You could do something simple like..
max_runtime = 10.seconds.from_now
puts 'whatever' until Time.now > max_runtime
you can try this it allows for interval controls
counter == 30
interval = 5 # Check every 5 seconds
interval_timer = 1 # must start at 1
now = Time.now
while Time.now - now < counter
if interval_timer % interval == 0 #Every 5 attempts the activity will process
if condition
stuff
end
end
process_timer = process_timer + 1
end
This will happen under a guaranteed 30 seconds the interval can be set to any value 1 or greater. Some things process via milliseconds this will give you an option that will save you cycles on processing. Works well in graphics processing.

Resources