Ruby, strange forked processes behaviour on MacOS vs Debian - ruby

Using Ruby (tested with versions 2.6.9, 2.7.5, 3.0.3, 3.1.1) and forking processes to handle socket communication there seems to be a huge difference between MacOS OSX and a Debian Linux.
While running on Debian, the forked processes get called in a balanced manner - that mean: if having 10 tcp server forks and running 100 client calls, each fork will get 10 calls. The order of the pid call stack is also always the same even not ordered by pid (caused by load when instantiating the forks).
Doing the same on a MacOS OSX (Catalina) the forked processes will not get called balanced - that mean: "pid A" might get called 23 or whatever times while e.g. "pid G" was never used.
Sample code (originally from: https://relaxdiego.com/2017/02/load-balancing-sockets.html)
#!/usr/bin/env ruby
# server.rb
require 'socket'
# Open a socket
socket = TCPServer.open('0.0.0.0', 9999)
puts "Server started ..."
# For keeping track of children pids
wpids = []
# Forward any relevant signals to the child processes.
[:INT, :QUIT].each do |signal|
Signal.trap(signal) {
wpids.each { |wpid| Process.kill(:KILL, wpid) }
}
end
5.times {
wpids << fork do
loop {
connection = socket.accept
connection.puts "Hello from #{ Process.pid }"
connection.close
}
end
}
Process.waitall
Run some netcat to the server on a second terminal:
for i in {1..20}; do nc -d localhost 9999; done
As said: if running on Linux each forked process will get 4 calls - doing same on MacOS OSX its a random usage per forked process.
Any solution or correction to make it work on MacOS OSX in a balanced manner also?

The problem is that the default socket backlog size is 5 on MacOS and 128 on Linux. You can change the backlog size by passing it as the second argument to TCPServer#listen:
socket.listen(128)
Or you can use the backlog size from the environment variable SOMAXCONN:
socket.listen(ENV.fetch('SOMAXCONN', 128).to_i)

Related

Running awesome-client from a script executing as root

Running Awesome on Debian (11) testing
awesome v4.3 (Too long)
• Compiled against Lua 5.3.3 (running with Lua 5.3)
• D-Bus support: ✔
• execinfo support: ✔
• xcb-randr version: 1.6
• LGI version: 0.9.2
I'm trying to signal to Awesome when systemd triggers suspend. After fiddling with D-Bus directly for awhile and getting nowhere, I wrote a couple of functions that somewhat duplicate the functionality of signals.
I tested it by running the following command in a shell, inside of my Awesome session:
$ awesome-client 'require("lib.syskit").signal("awesome-client", "Hello world!")'
This runs just fine. A notification posts to the desktop "Hello world!" as expected. I added the path to my lib.syskit code to the $LUA_PATH in my ~/.xsessionrc. Given the error described below, I doubt this is an issue.
Now for the more difficult part. I put the following in a script located at /lib/systemd/system-sleep/pre-suspend.sh
#!/bin/bash
if [ "${1}" == "pre" ]; then
ERR=$(export DISPLAY=":0"; sudo -u naddan awesome-client 'require("lib.syskit").signal("awesome-client", "pre-suspend")' 2>&1)
echo "suspending at `date`, ${ERR}" > /tmp/systemd_suspend_test
elif [ "${1}" == "post" ]; then
ERR=$(export DISPLAY=":0"; sudo -u naddan awesome-client 'require("lib.syskit").signal("awesome-client", "post-suspend")' 2>&1)
echo "resuming at `date`, ${ERR}" >> /tmp/systemd_suspend_test
fi
Here's the output written to /tmp/systemd_suspend_test
suspending at Thu 22 Jul 2021 10:58:01 PM MDT, Failed to open connection to "session" message bus: /usr/bin/dbus-launch terminated abnormally without any error message
E: dbus-send failed.
resuming at Thu 22 Jul 2021 10:58:05 PM MDT, Failed to open connection to "session" message bus: /usr/bin/dbus-launch terminated abnormally without any error message
E: dbus-send failed.
Given that I'm already telling it the $DISPLAY that Awesome is running under (this is a laptop), and that I'm running awesome-client as my user, not root, what else am I missing that's keeping this from working?
Is there a better way that I could achieve telling Awesome when the system suspends?
awesome-client is a shell script. It is a thin wrapper around dbus-send. Thus, since you write "After fiddling with D-Bus directly for awhile and getting nowhere", I guess the same reasoning applies.
Given that I'm already telling it the $DISPLAY that Awesome is running under (this is a laptop), and that I'm running awesome-client as my user, not root, what else am I missing that's keeping this from working?
You are missing the address of the dbus session bus. For me, it is:
$ env | grep DBUS
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
Is there a better way that I could achieve telling Awesome when the system suspends?
Instead of sending a message directly to awesome via some script, you could use the existing mechanism for this: Dbus signals. These are broadcasts that interested parties can listen to.
Google suggests that there is already a PrepareForSleep signal:
https://serverfault.com/questions/573379/system-suspend-dbus-upower-signals-are-not-seen
Based on this, Google then gave me the following AwesomeWM lua code that listens for logind's PrepareForSleep signal (written by yours truely - thanks Google for finding that!):
https://github.com/awesomeWM/awesome/issues/344#issuecomment-328354719
local lgi = require("lgi")
local Gio = lgi.require("Gio")
local function listen_to_signals()
local bus = lgi.Gio.bus_get_sync(Gio.BusType.SYSTEM)
local sender = "org.freedesktop.login1"
local interface = "org.freedesktop.login1.Manager"
local object = "/org/freedesktop/login1"
local member = "PrepareForSleep"
bus:signal_subscribe(sender, interface, member, object, nil, Gio.DBusSignalFlags.NONE,
function(bus, sender, object, interface, signal, params)
-- "signals are sent right before (with the argument True) and
-- after (with the argument False) the system goes down for
-- reboot/poweroff, resp. suspend/hibernate."
if not params[1] then
-- This code is run before suspend. You can replace the following with something else.
require("gears.timer").start_new(2, function()
mytextclock:force_update()
end)
end
end)
end
listen_to_signals()

warning using Parallel::ForkManager but only in Windows

I sometimes get this warning when using Parallel::ForkManager but only in Windows, not on a Unix based system. What does it mean and should I worry about it?
child process '-17108' disappeared. A call to waitpid outside of
Parallel::ForkManager might have reaped it.
Here is the sample code from the docs that my code is based on:
use LWP::Simple;
use Parallel::ForkManager;
my #links=(
["http://www.foo.bar/rulez.data","rulez_data.txt"],
["http://new.host/more_data.doc","more_data.doc"],
);
# Max 30 processes for parallel download
my $pm = Parallel::ForkManager->new(30);
LINKS:
foreach my $linkarray (#links) {
$pm->start and next LINKS; # do the fork
my ($link, $fn) = #$linkarray;
warn "Cannot get $fn from $link"
if getstore($link, $fn) != RC_OK;
$pm->finish; # do the exit in the child process
}
$pm->wait_all_children;
I had the similar issue and placing a sleep 1 before "$pm->start and next LINKS;"
fixed the issue. I guess its due to continues forking, where Perl lost track of the fork processes. I may be wrong!

Thor Start Jekyll then Open Page in Browser

Hi I want Thor to start a server - Jekyll / Python / PHP etc then open the browser
However the starting is a blocking task.
Is there a way to create a child process in Thor; or spawn a new terminal window - couldnt see and google gave me no reasonable answers.
My Code
##
# Project Thor File
#
# #use thor list
##
class IanWarner < Thor
##
# Open Jekyll Server
#
# #use thor ian_warner:openServer
##
desc "openServer", "Start the Jekyll Server"
def openServer
system("clear")
say("\n\t")
say("Start Server\n\t")
system("jekyll --server 4000 --auto")
say("Open Site\n\t")
system("open http://localhost:4000")
say("\n")
end
end
It looks like you are messing things up. Thor is in general a powerful CLI wrapper. CLI itself is in general singlethreaded.
You have two options: either to create different Thor descendents and run them as different threads/processes, forcing open thread/process to wait until jekyll start is running (preferred,) or to hack with system("jekyll --server 4000 --auto &") (note an ampersand at the end.)
The latter will work, but you still are to control the server is started (it may take a significant amount of time.) The second ugly hack to achieve this is to rely on sleep:
say("Start Server\n\t")
system("jekyll --server 4000 --auto &")
say("Wait for Server\n\t")
system("sleep 3")
say("Open Site\n\t")
system("open http://localhost:4000")
Upd: it’s hard to imagine what do you want to yield. If you want to leave your jekyll server running after your script is finished:
desc "openServer", "Start the Jekyll Server"
def openServer
system "clear"
say "\n\t"
say "Starting Server…\n\t"
r, w = IO.pipe
# Jekyll will print it’s running status to STDERR
pid = Process.spawn("jekyll --server 4000 --auto", :err=>w)
w.close
say "Spawned with pid=#{pid}"
rr = ''
while (rr += r.sysread(1024)) do
break if rr.include?('WEBrick::HTTPServer#start')
end
Process.detach(pid) # !!! Leave the jekyll running
say "Open Site\n\t"
system "open http://localhost:4000"
end
If you want to shutdown the jekyll after the page is opened, you are to spawn the call to open as well and Process.waitpid for it.

How to shutdown Ruby XMLRPC server?

I'm testing Ruby XMLRPC support right now. It all works fine, except XMLRPC::Server#shutdown.
If I run the following Ruby 1.9.3 test code, it fails to shut down the server on both Windows 7 and OSX 10.7:
# server.rb
require "xmlrpc/server"
require 'thread'
Thread.new { sleep 10; $server.shutdown() }
$server = XMLRPC::Server.new( 1234 )
$server.add_handler( "test" ) { true }
$server.serve()
# client.rb
require "xmlrpc/client"
server = XMLRPC::Client.new( "localhost", "/", 1234 )
loop { server.call( "test" ); sleep 0.1 }
After ten seconds, the server writes "INFO going to shutdown ..." to stdout, but won't actually shut down and continues to handle incoming requests. What am I doing wrong?
Have you noticed that without incoming requests it shutdowns properly? Also, after you end the client, it will shut down as it should, returning :Stop symbol. It waits for the client to stop pumping data before shutting down.
I have examined XMLRPC::Server source code. It seems a bug/feature that prevents shutdown if client uses connection with keep-alive HTTP flag.
The workaround is to use call_async instead of call.

How can I check from Ruby whether a process with a certain pid is running?

If there is more than one way, please list them. I only know of one, but I'm wondering if there is a cleaner, in-Ruby way.
The difference between the Process.getpgid and Process::kill approaches seems to be what happens when the pid exists but is owned by another user. Process.getpgid will return an answer, Process::kill will throw an exception (Errno::EPERM).
Based on that, I recommend Process.getpgid, if just for the reason that it saves you from having to catch two different exceptions.
Here's the code I use:
begin
Process.getpgid( pid )
true
rescue Errno::ESRCH
false
end
If it's a process you expect to "own" (e.g. you're using this to validate a pid for a process you control), you can just send sig 0 to it.
>> Process.kill 0, 370
=> 1
>> Process.kill 0, 2
Errno::ESRCH: No such process
from (irb):5:in `kill'
from (irb):5
>>
#John T, #Dustin: Actually, guys, I perused the Process rdocs, and it looks like
Process.getpgid( pid )
is a less violent means of applying the same technique.
For child processes, other solutions like sending a signal won't behave as expected: they will indicate that the process is still running when it actually exited.
You can use Process.waitpid if you want to check on a process that you spawned yourself. The call won't block if you're using the Process::WNOHANG flag and nil is going to be returned as long as the child process didn't exit.
Example:
pid = Process.spawn('sleep 5')
Process.waitpid(pid, Process::WNOHANG) # => nil
sleep 5
Process.waitpid(pid, Process::WNOHANG) # => pid
If the pid doesn't belong to a child process, an exception will be thrown (Errno::ECHILD: No child processes).
The same applies to Process.waitpid2.
This is how I've been doing it:
def alive?(pid)
!!Process.kill(0, pid) rescue false
end
You can try using
Process::kill 0, pid
where pid is the pid number, if the pid is running it should return 1.
Under Linux you can obtain a lot of attributes of running programm using proc filesystem:
File.read("/proc/#{pid}/cmdline")
File.read("/proc/#{pid}/comm")
A *nix-only approach would be to shell-out to ps and check if a \n (new line) delimiter exists in the returned string.
Example IRB Output
1.9.3p448 :067 > `ps -p 56718`
" PID TTY TIME CMD\n56718 ttys007 0:03.38 zeus slave: default_bundle \n"
Packaged as a Method
def process?(pid)
!!`ps -p #{pid.to_i}`["\n"]
end
I've dealt with this problem before and yesterday I compiled it into the "process_exists" gem.
It sends the null signal (0) to the process with the given pid to check if it exists. It works even if the current user does not have permissions to send the signal to the receiving process.
Usage:
require 'process_exists'
pid = 12
pid_exists = Process.exists?(pid)

Resources