I have two Ruby scripts: loop.rb and event.rb. When I run event.rb while running a loop in loop.rb, the former must catch the latter and change its behaviour.
There are many hacks that I can use; for example, loop.rb watches some env-var, and event.rb changes it. But I feel it a bit dirty.
What is the right way to send message from one Ruby script to another in Linux?
I think best option would be to use ZeroMQ:
https://github.com/zeromq/rbzmq
require "zmq"
context = ZMQ::Context.new(1)
puts "Opening connection for READ"
inbound = context.socket(ZMQ::UPSTREAM)
inbound.bind("tcp://127.0.0.1:9000")
outbound = context.socket(ZMQ::DOWNSTREAM)
outbound.connect("tcp://127.0.0.1:9000")
p outbound.send("Hello World!")
p outbound.send("QUIT")
loop do
data = inbound.recv
p data
break if data == "QUIT"
end
Related
Lets say I have 4 folders with 25 folders in each. In each of those 25 folders there is 20 folders each with 1 very long text document. The method i'm using now seems to have room to improve and in every scenario in which I implement ruby's threads, the result is slower than before. I have an array of the 54names of the folders. I iterate through each and use a foreach method to get the deeply nested files. In the foreach loop I do 3 things. I get the contents of today's file, I get the contents of yesterday's file, and I use my diff algorithm to find what has changed from yesterday to today. How would you do this faster with threads.
def backup_differ_loop device_name
device_name.strip!
Dir.foreach("X:/Backups/#{device_name}/#{#today}").each do |backup|
if backup != "." and backup != ".."
#today_filename = "X:/Backups/#{device_name}/#{#today}/#{backup}"
#yesterday_filename = "X:/Backups/#{device_name}/#{#yesterday}/#{backup.gsub(#today, #yesterday)}"
if File.exists?(#yesterday_filename)
today_backup_content = File.open(#today_filename, "r").read
yesterday_backup_content = File.open(#yesterday_filename, "r").read
begin
Diffy::Diff.new(yesterday_backup_content, today_backup_content, :include_plus_and_minus_in_html => true, :context => 1).to_s(:html)
rescue
#do nothing just continue
end
end
else
#file not found
end
end
end
The first part of your logic is finding all files in a specific folder. Instead of doing Dir.foreach and then checking against "." and ".." you can do this in one line:
files = Dir.glob("X:/Backups/#{device_name}/#{#today}/*").select { |item| File.file?(item)}
Notice the /* at the end? This will search 1 level deep (inside the #today folder). If you want to search inside sub-folders too, replace it with /**/* so you'll get array of all files inside all sub-folders of #today.
So I'd first have a method which would give me a double array containing a bunch of arrays of matching files:
def get_matching_files
matching_files = []
Dir.glob("X:/Backups/#{device_name}/#{#today}/*").select { |item| File.file?(item)}.each do |backup|
today_filename = File.absolute_path(backup) # should get you X:/Backups...converts to an absolute path
yesterday_filename = "X:/Backups/#{device_name}/#{#yesterday}/#{backup.gsub(#today, #yesterday)}"
if File.exists?(yesterday_filename)
matching_files << [today_filename, yesterday_filename]
end
end
return matching_files
end
and call it:
matching_files = get_matching_files
NOW we can start the multi-threading which is where things probably slow down. I'd first get all the files from the array matching_files into a queue, then start 5 threads which will go until the queue is empty:
queue = Queue.new
matching_files.each { |file| queue << file }
# 5 being the number of threads
5.times.map do
Thread.new do
until queue.empty?
begin
today_file_content, yesterday_file_content = queue.pop
Diffy::Diff.new(yesterday_backup_content, today_backup_content, :include_plus_and_minus_in_html => true, :context => 1).to_s(:html)
rescue
#do nothing just continue
end
end
end
end.each(&:join)
I can't guarantee my code will work because I don't have the entire context of your program. I hope I've given you some ideas.
And the MOST important thing: The standard implementation of Ruby can run only 1 thread at a time. This means even if you implement the code above, you won't get a significant performance difference. So get Rubinius or JRuby which allow more than 1 threads to be running at a time. Or if you prefer to use the standard MRI Ruby, then you'll need to re-structure your code (you can keep your original version) and start multiple processes. You'll just need something like a shared database where you can store the matching_files (as a single row, for example) and every time a process will 'take' something from that database, it will mark that row as 'used'. SQLite is a good db for this I think because it's thread safe by default.
Most Ruby implementations don't have "true" multicore threading i.e. threads won't gain you any performance improvement since the interpreter can only run one thread at a time. For applications like yours with lots of disk IO this is especially true. In fact, even with real multithreading your applications might be IO-bound and still not see much of an improvement.
You are more likely to get results by finding some inefficient algorithm in your code and improving it.
I'm using EventMachine to process incoming emails which could at times be very high volume. The code that I have so far definitely works for emails that come in separated by at least about 5 seconds, but somewhere below that, only one email will be processed out of however many arrive. I've tried adding EM.defer statements in a few different places which I thought would help, but to no avail. I should also note, if it makes any difference, that I'm using the em-imap gem in this example as well.
The relevant section of the code is here:
EM.run do
client = EM::IMAP.new('imap.gmail.com', 993, true)
client.connect.bind! do
client.login('me#email.com', 'password123')
end.bind! do
client.select('INBOX')
end.bind! do
client.wait_for_new_emails do |response|
client.fetch(response.data).callback do |fetched|
currentSubjectLine = fetched.first.attr.values[1].subject
desiredCommand = parseSubjectLine(currentSubjectLine)
if desiredCommand == 0
if fetched.first.attr.values[0].parts.length == 2
if fetched.first.attr.values[0].parts[1].subtype.downcase != "pdf"
puts 'Error: Missing attachment, or attachment of the wrong type.'
else
file_name = fetched.first.attr.values[0].parts[1].param.values[0]
client.fetch(response.data, "BODY[2]").callback do |attachments|
attachment = attachments[0].attr["BODY[2]"]
File.new(file_name,'wb+').write(Base64.decode64(attachment))
end
end...
Am I somehow blocking the reactor in this code segment? Is it possible that some library that I'm using isn't appropriate here? Could GMail's IMAP server have something to do with it? Do you need any more information about what happens in some given situation before you can answer with confidence? As always, any help is greatly appreciated. Thank you!
Update with Minimized Code
Just in case anything in my organization has anything to do with it, I'm including everything that I think might possibly be relevant.
module Processing
def self.run
EM.run do
client = EM::IMAP.new('imap.gmail.com', 993, true)
client.connect.bind! do
client.login('me#email.com', 'password123')
end.bind! do
client.select('INBOX')
end.bind! do
client.wait_for_new_emails do |response|
client.fetch(response.data).callback do |fetched|
puts fetched[0].attr.values[1].subject
end
end
end.errback do |error|
puts "Something failed: #{error}"
end
end...
Processing.run
Don't hate me for saying this, but refactor that pyramid of doom spaggheti thingy that makes Demeter twitch into something readable and the error will reveal itself :)
If it doesn't reveal itself you will be able to boil it down to the simplest possible code that reproduces the problem and submit it as an issue to https://github.com/eventmachine/eventmachine
However, EM isn't really supported any more, the devs went a bit awol so think about moving to https://github.com/celluloid/celluloid and https://github.com/celluloid/celluloid-io
PS
just saw this
File.new(file_name,'wb+').write(Base64.decode64(attachment))
is a blocking call afaik, try playing with this and you might be able to reproduce the issue. See https://github.com/martinkozak/em-files and http://eventmachine.rubyforge.org/EventMachine.html#defer-class_method on possible ways to go around this
I've been looking for this answer in the internet for a while and have found other people asking the same thing, even here. So this post will be a presentation of my case and a response to the "solutions" that I have found.
I am such new in Ruby, but for learning purposes I decided to create a gem, here.
I am trying to implement a keyboard navigation to this program, that will allow the user use short-cuts to select what kind of request he want to see. And in the future, arrow navigations, etc.
My problem: I can't find a consistent way to get the keyboard events from the user's console with Ruby.
Solutions that I have tried:
Highline gem: Seems do not support this feature anymore. Anyway it uses the STDIN, keep reading.
STDIN.getch: I need to run it in a parallel loop, because at the same time that the user can use a short-cut, more data can be created and the program needs to show it. And well, I display formated text in the console, (Rails log). When this loop is running, my text lost the all the format.
Curses: Cool but I need to set position(x,y) to display my text every time? It will get confusing.
Here is where I am trying to do it.
You may note that I am using "stty -raw echo" (turns raw off) before show my text and "stty raw -echo" (turns raw on) after. That keeps my text formated.
But my key listener loop is not working. I mean, It works in sometimes but is not consistent. If a press a key twice it don't work anymore and sometimes it stops alone too.
Let me put one part of the code here:
def run
# Two loops run in parallel using Threads.
# stream_log loops like a normal stream in the file, but it also parser the text.
# break it into requests and store in #requests_queue.
# stream_parsed_log stream inside the #requests_queue and shows it in the screen.
#requests_queue = Queue.new
#all_requests = Array.new
# It's not working yet.
Thread.new { listen_keyboard }
Thread.new { stream_log }
stream_parsed_log
end
def listen_keyboard
# not finished
loop do
char = STDIN.getch
case char
when 'q'
puts "Exiting."
exit
when 'a'
#types_to_show = ['GET', 'POST', 'PUT', 'DELETE', 'ASSET']
requests_to_show = filter_to_show(#all_requests)
command = true
when 'p'
#types_to_show = ['POST']
requests_to_show = filter_to_show(#all_requests)
command = true
end
clear_screen if command
#requests_queue += requests_to_show if command
command = false
end
end
I need a light in my path, what should I do?
That one was my mistake.
It's just a logic error in another part of code that was running in another thread so the ruby don't shows the error by default. I used ruby -d and realized what was wrong. This mistake was messing my keyboard input.
So now it's fixed and I am using STDIN.getch with no problem.
I just turn the raw mode off before show any string. And everything is ok.
You can check here, or in the gem itself.
That's it.
I am writing a web based debugger for Ruby, but in order to do this I need to be able to call the Ruby debugger from within a Ruby program on the server side. Has this ever been done? Is this even possible?
The end product being built will allow Ruby code to be edited, executed and stepped through using just a web browser. The ruby code that is to be debugged will be "eval"ed on the server side.
I have since been pointed in the right direction by one of the stackoverflow users who has suggested using popen or expect. I have tried both of these now but have encountered the following problems:
popen: When waiting for the console you have to use a timeout block to signal the end of the debug console's output (The command line terminal can detect this, so why can't ruby).
expect: In the program below the debugger inputs get out of sync with the debugger. Why is that?
require 'pty'
require 'expect'
$expect_verbose = true
PTY.spawn("rdebug deb.rb") do |from_debugger, to_debugger, pid|
a=nil
while ( a != "end" ) do
from_debugger.expect(/\(rdb:1\)/ ) do |input|
a = gets
to_debugger.puts( a + "\n" )
end
from_debugger.flush
end
end
I think you could make use of ruby-debug. I would imagine this as opening the same script on the server-side, and sending keystrokes to it. I think this is how I'd do it.
Think about it this way: the client pastes his code in your form. Optionally, he may click on some button that places a breakpoint on some line. Then he submits the whole thing. On server side, you process that form, and create a .rb file with the contents of the form. At the line where he put the breakpoint, you insert a debugger call. Then you run :
ruby client_script.rb
and this will stop the process once it reaches the debugger line. I would suggest you do all of this with Expect for Ruby or whatever. So, when the client presses step over on the page or inspect whatever variable, you send keystrokes to the process you spawned.
For example, if he wants to inspect the value of the variable x, you'd send this to the process:
p x
and you'd have to capture that output and send it back to the client's browser. Expect is kind of like popen , but a bit better.
I finally figured out how to do this using expect:
require 'pty'
require 'expect'
PTY.spawn("rdebug deb.rb") do |output,input,pid|
while true do
begin
buffer = ""
puts "-------"
puts output.readpartial(1024,buffer) until buffer =~ /\(rdb:1\)/
a = gets
input.puts a
rescue
break
end
end
end
Is it possible from within shoooes to spawn a separate thread which will read from a named pipe and then print whatever is written in that name pipe into a text box? Would anyone have an example of how to set that up?
It's pretty easy to manipulate text across threads. Try this code, for example:
Shoes.app do
#text = para 'Do you like ponies?'
Thread.new do
sleep(4)
#text.text += " Of course I do!"
end
timer(2) {#text.text += "\nWhat a silly question."}
end
As to reading from a named pipe, they can be treated like any other file, with the caveat that they will block until the other side of the pipe is set up. So, either make them non-blocking or just set the other side up before you open the pipe.