Ruby script with parrallel threads and logging bash output - ruby

Question

I have a ruby script I'm making to upload a directory of roles to chef server. Doing this 1 at a time with a .each loop is slow. So I added parallelism by running each command in separate threads. Now I'm trying to figure out how to store the output of the commands so I can read them back in order of the threads that were created. The roles array is already in alphabetical order. We also use bash_profile aliases for running the knife command with different configuration files for dev and prod.
I've tried many different ways to run the bash command and trying to store the output in an array or to a file etc... Currently this displays the output from each thread as it runs or finishes so out put is hard to read or tell if everything finished correctly and the files that the bash command output is supposed to be redirecting to get created but have empty content.
Sorry if this script isn't the easiest to read. I've only been doing ruby for a little over a year now and self taught myself it when we started to get into chef. I didn't have a programming background before that.
#!/opt/chefdk/embedded/bin/ruby
def print_usage_and_exit
puts 'Need to specify 1 or more role.json files or no arguments to upload all roles'
puts "ruby #{__FILE__} or ruby #{__FILE__} [role1.json] [role2.json] [...]"
exit(1)
end
def fetch_roles
roles = []
current_dir = File.dirname(__FILE__)
Dir.foreach("#{current_dir}/roles") do |role|
next if role == '.' || role == '..' || role == 'README.md'
roles.push(role)
end
roles
end
upload = []
i = 0
roles = (ARGV.empty? ? fetch_roles : ARGV[0..-1])
# Probably redundant, but a cheap check to make sure we're only looking at json files
roles.keep_if { |b| b.end_with?('.json') }
print_usage_and_exit if roles.empty?
print "\nSpecify new knife command if you have seperate knife command for dev and prod created with .bash_profile function."
print "\nLeave blank to use default 'knife' command"
print "\nWhich knife command to use: "
knife = ($stdin.gets.chomp('') ? 'knife' : $stdin.gets.chomp)
print "\n**** Starting upload of roles to chef server ****\n"
roles.each do |role|
upload[i] = Thread.new{
system("bash", "-cl", "#{knife} role from file #{role} > /tmp/#{role}.log")
}
i += 1
end
upload.each {|t| t.join}
roles.each do |role|
logfile = "/tmp/#{role}.log"
logmsg = open(logfile)
print "\n#{logmsg.read}\n"
#FileUtils.rm("/tmp/#{role}.log")
end
print "\n**** Finished uploading roles to chef server ****\n"

Answer 1

The right way to do this is knife upload roles/. That doesn't actually answer your question per se, but I think you'll find it a lot simpler.

Answer 2

I prefer to use Open3's caputure3 function to execute subprocesses, as it makes it easy to handle all the various details ( stdin, stdout, stderr, environment variables, etc ).
Pair that with the use of thread-local data, a built in feature of ruby threads, and you have a pretty easy method of running subprocesses. I'm a big fan of using threads for this kind of concurrency. The GIL prevents ruby from running all the threads concurrently, but the capture3 subprocesses run concurrently anyway, so it doesn't really matter.
require 'open3'
commands = [
'true',
'echo "a more complex command from `pwd`" 1>&2 && echo "and stdout"',
]
threads = []
commands.each_with_index do |cmd, i|
threads[i] = Thread.new do
stdout, stderr, status = Open3.capture3("bash", stdin_data: cmd)
Thread.current['stdout'] = stdout
Thread.current['stderr'] = stderr
Thread.current['status'] = status
end
end
threads.each_with_index do |th,i|
th.join
puts "Thread # #{i}:"
%w( stdout stderr status ).each do |s|
puts "\t#{s}: #{th[s]}"
end
puts
end
The results are exactly what you'd expect:
$ ruby ./t.rb
Thread # 0:
stdout:
stderr:
status: pid 34244 exit 0
Thread # 1:
stdout: and stdout
stderr: a more complex command from /Users/dfarrell/t
status: pid 34243 exit 0
You can use the exit status to give a final summary of how many commands failed or succeeded.