I have a parallel map in Elixir using Task.async. I am using it with System.cmd to parallel ruby rbenv install a list of ruby version strings.
The script runs and installs the ruby versions. However it does not exit the Task once the ruby versions have been installed and errors with a Task.await timeout error.
I have tried passing a simple IO.puts function into the parallel map and behaves correctly, exiting when the work is done and not raising an error. What am I missing in my System.cmd to ensure that each process is ended once each rbenv process is finished.
# My parallel map
def pmap(collection, func) do
collection
|> Enum.map(&(Task.async(fn -> func.(&1) end)))
|> Enum.map(&Task.await/1)
end
# System.cmd is being passed into the map like this
def parallel_install(ruby_versions) do
pmap(ruby_versions, &(System.cmd("rbenv", ["install", &1])))
end
Output with Error:
rbenv: /Users/lewis.jones/.rbenv/versions/2.4.4 already exists
rbenv: /Users/lewis.jones/.rbenv/versions/2.5.1 already exists
rbenv: /Users/lewis.jones/.rbenv/versions/2.1.10 already exists
rbenv: /Users/lewis.jones/.rbenv/versions/2.3.7 already exists
rbenv: /Users/lewis.jones/.rbenv/versions/2.2.10 already exists
** (exit) exited in: Task.await(%Task{owner: #PID<0.73.0>, pid: #PID<0.82.0>,
ref: #Reference<0.100168651.1374158856.61975>}, 5000)
** (EXIT) time out
(elixir) lib/task.ex:491: Task.await/2
(elixir) lib/enum.ex:1270: Enum."-map/2-lists^map/1-0-"/2
(elixir) lib/code.ex:376: Code.require_file/2
Task.await defaults to a timeout of 5 seconds. I'm guessing rbenv install takes longer than that. You can either increase the timeout or set it to infinity.
300 seconds:
|> Enum.map(&Task.await(&1, 300_000))
or infinity:
|> Enum.map(&Task.await(&1, :infinity))
Related
I'm learning Erlang and I've written a simple module to test "spawn" function:
-module(concurrent).
-export([go/0, loop/0]).
go() ->
Pid2 = spawn(echo, loop, []).
loop() -> 2.
but when I run concurrent:go(). I get this error message:
=ERROR REPORT==== 10-Feb-2023::14:41:34.586000 === Error in process
<0.84.0> with exit value: {undef,[{echo,loop,[],[]}]}
I don't understand what I'm doing wrong.
You try to spawn a process running function loop from module echo. But you export function loop from module named concurrent and not echo.
I have installed the ElixirLS Extension. I need help with opening the IEx session from the directory containing the Elixir module. Kindly help in setting the path
checkout.ex
defmodule Checkout do
def total_cost(price, tax_rate) do
price * (tax_rate + 1)
end
end
MacBook-Pro:elixir-intro sudha$ elixirc checkout.ex
MacBook-Pro:elixir-intro sudha$ iex
Erlang/OTP 25 [erts-13.1.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit] [dtrace]
Interactive Elixir (1.14.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Checkout.total_cost(100,0.2)
this is what I got -
** (UndefinedFunctionError) function Checkout.total_cost/2 is undefined (module Checkout is not available)
Checkout.total_cost(100, 0.2)
iex:1: (file)
iex(1)>
You need to load the file in the iex session. Try either:
iex -r checkout.ex or inside of iex try c "checkout.ex"
Using Ruby (tested with versions 2.6.9, 2.7.5, 3.0.3, 3.1.1) and forking processes to handle socket communication there seems to be a huge difference between MacOS OSX and a Debian Linux.
While running on Debian, the forked processes get called in a balanced manner - that mean: if having 10 tcp server forks and running 100 client calls, each fork will get 10 calls. The order of the pid call stack is also always the same even not ordered by pid (caused by load when instantiating the forks).
Doing the same on a MacOS OSX (Catalina) the forked processes will not get called balanced - that mean: "pid A" might get called 23 or whatever times while e.g. "pid G" was never used.
Sample code (originally from: https://relaxdiego.com/2017/02/load-balancing-sockets.html)
#!/usr/bin/env ruby
# server.rb
require 'socket'
# Open a socket
socket = TCPServer.open('0.0.0.0', 9999)
puts "Server started ..."
# For keeping track of children pids
wpids = []
# Forward any relevant signals to the child processes.
[:INT, :QUIT].each do |signal|
Signal.trap(signal) {
wpids.each { |wpid| Process.kill(:KILL, wpid) }
}
end
5.times {
wpids << fork do
loop {
connection = socket.accept
connection.puts "Hello from #{ Process.pid }"
connection.close
}
end
}
Process.waitall
Run some netcat to the server on a second terminal:
for i in {1..20}; do nc -d localhost 9999; done
As said: if running on Linux each forked process will get 4 calls - doing same on MacOS OSX its a random usage per forked process.
Any solution or correction to make it work on MacOS OSX in a balanced manner also?
The problem is that the default socket backlog size is 5 on MacOS and 128 on Linux. You can change the backlog size by passing it as the second argument to TCPServer#listen:
socket.listen(128)
Or you can use the backlog size from the environment variable SOMAXCONN:
socket.listen(ENV.fetch('SOMAXCONN', 128).to_i)
An example output from capistrano:
INFO [94db8027] Running /usr/bin/env uptime on leehambley#example.com:22
DEBUG [94db8027] Command: /usr/bin/env uptime
DEBUG [94db8027] 17:11:17 up 50 days, 22:31, 1 user, load average: 0.02, 0.02, 0.05
INFO [94db8027] Finished in 0.435 seconds command successful.
As you can see, each line starts with "{type} {hash}". I assume the hash is some unique identifier for either the server or the running thread, as I've noticed if I run capistrano over several servers, each one has it's own distinct hash.
My question is, how do I get this value? I want to manually output some message during execution, and I want to be able to match my output, with the server that triggered it.
Something like: puts "DEBUG ["+????+"] Something happened!"
What do I put in the ???? there? Or is there another, built in way to output messages like this?
For reference, I am using Capistrano Version: 3.2.1 (Rake Version: 10.3.2)
This hash is a command uuid. It is tied not to the server but to a specific command that is currently run.
If all you want is to distinguish between servers you may try the following
task :some_task do
on roles(:app) do |host|
debug "[#{host.hostname}:#{host.port}] something happened"
end
end
If there is more than one way, please list them. I only know of one, but I'm wondering if there is a cleaner, in-Ruby way.
The difference between the Process.getpgid and Process::kill approaches seems to be what happens when the pid exists but is owned by another user. Process.getpgid will return an answer, Process::kill will throw an exception (Errno::EPERM).
Based on that, I recommend Process.getpgid, if just for the reason that it saves you from having to catch two different exceptions.
Here's the code I use:
begin
Process.getpgid( pid )
true
rescue Errno::ESRCH
false
end
If it's a process you expect to "own" (e.g. you're using this to validate a pid for a process you control), you can just send sig 0 to it.
>> Process.kill 0, 370
=> 1
>> Process.kill 0, 2
Errno::ESRCH: No such process
from (irb):5:in `kill'
from (irb):5
>>
#John T, #Dustin: Actually, guys, I perused the Process rdocs, and it looks like
Process.getpgid( pid )
is a less violent means of applying the same technique.
For child processes, other solutions like sending a signal won't behave as expected: they will indicate that the process is still running when it actually exited.
You can use Process.waitpid if you want to check on a process that you spawned yourself. The call won't block if you're using the Process::WNOHANG flag and nil is going to be returned as long as the child process didn't exit.
Example:
pid = Process.spawn('sleep 5')
Process.waitpid(pid, Process::WNOHANG) # => nil
sleep 5
Process.waitpid(pid, Process::WNOHANG) # => pid
If the pid doesn't belong to a child process, an exception will be thrown (Errno::ECHILD: No child processes).
The same applies to Process.waitpid2.
This is how I've been doing it:
def alive?(pid)
!!Process.kill(0, pid) rescue false
end
You can try using
Process::kill 0, pid
where pid is the pid number, if the pid is running it should return 1.
Under Linux you can obtain a lot of attributes of running programm using proc filesystem:
File.read("/proc/#{pid}/cmdline")
File.read("/proc/#{pid}/comm")
A *nix-only approach would be to shell-out to ps and check if a \n (new line) delimiter exists in the returned string.
Example IRB Output
1.9.3p448 :067 > `ps -p 56718`
" PID TTY TIME CMD\n56718 ttys007 0:03.38 zeus slave: default_bundle \n"
Packaged as a Method
def process?(pid)
!!`ps -p #{pid.to_i}`["\n"]
end
I've dealt with this problem before and yesterday I compiled it into the "process_exists" gem.
It sends the null signal (0) to the process with the given pid to check if it exists. It works even if the current user does not have permissions to send the signal to the receiving process.
Usage:
require 'process_exists'
pid = 12
pid_exists = Process.exists?(pid)