I would like to upload data I generated at runtime in Ruby, something like feeding the upload from a block.
All examples I found only show how to stream a file that must be on disk prior to the request but I do not want to buffer the file.
What is the best solution besides rolling my own socket connection?
This a pseudocode example:
post_stream('127.0.0.1', '/stream/') do |body|
generate_xml do |segment|
body << segment
end
end
Code that works.
require 'thread'
require 'net/http'
require 'base64'
require 'openssl'
class Producer
def initialize
#mutex = Mutex.new
#body = ''
#eof = false
end
def eof!()
#eof = true
end
def eof?()
#eof
end
def read(size)
#mutex.synchronize {
#body.slice!(0,size)
}
end
def produce(str)
if #body.empty? && #eof
nil
else
#mutex.synchronize { #body.slice!(0,size) }
end
end
end
data = "--60079\r\nContent-Disposition: form-data; name=\"file\"; filename=\"test.file\"\r\nContent-Type: application/x-ruby\r\n\r\nthis is just a test\r\n--60079--\r\n"
req = Net::HTTP::Post.new('/')
producer = Producer.new
req.body_stream = producer
req.content_length = data.length
req.content_type = "multipart/form-data; boundary=60079"
t1 = Thread.new do
producer.produce(data)
producer.eof!
end
res = Net::HTTP.new('127.0.0.1', 9000).start {|http| http.request(req) }
puts res
There's a Net::HTTPGenericRequest#body_stream=( obj.should respond_to?(:read) )
You use it more or less like this:
class Producer
def initialize
#mutex = Mutex.new
#body = ''
end
def read(size)
#mutex.synchronize {
#body.slice!(0,size)
}
end
def produce(str)
#mutex.synchronize {
#body << str
}
end
end
# Create a producer thread
req = Net::HTTP::Post.new(url.path)
req.body_stream = producer
res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req) }
Related
I have a plain ruby class Espresso::MyExampleClass.
module Espresso
class MyExampleClass
def my_first_function(value)
puts "my_first_function"
end
def my_function_to_run_before
puts "Running before"
end
end
end
With some of the methods in the class, I want to perform a before or after callback similar to ActiveSupport callbacks before_action or before_filter. I'd like to put something like this in my class, which will run my_function_to_run_before before my_first_function:
before_method :my_function_to_run_before, only: :my_first_function
The result should be something like:
klass = Espresso::MyExampleClass.new
klass.my_first_function("yes")
> "Running before"
> "my_first_function"
How do I use call backs in a plain ruby class like in Rails to run a method before each specified method?
Edit2:
Thanks #tadman for recommending XY problem. The real issue we have is with an API client that has a token expiration. Before each call to the API, we need to check to see if the token is expired. If we have a ton of function for the API, it would be cumbersome to check if the token was expired each time.
Here is the example class:
require "rubygems"
require "bundler/setup"
require 'active_support/all'
require 'httparty'
require 'json'
module Espresso
class Client
include HTTParty
include ActiveSupport::Callbacks
def initialize
login("admin#example.com", "password")
end
def login(username, password)
puts "logging in"
uri = URI.parse("localhost:3000" + '/login')
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new(uri.request_uri)
request.set_form_data(username: username, password: password)
response = http.request(request)
body = JSON.parse(response.body)
#access_token = body['access_token']
#expires_in = body['expires_in']
#expires = #expires_in.seconds.from_now
#options = {
headers: {
Authorization: "Bearer #{#access_token}"
}
}
end
def is_token_expired?
#if Time.now > #expires.
if 1.hour.ago > #expires
puts "Going to expire"
else
puts "not going to expire"
end
1.hour.ago > #expires ? false : true
end
# Gets posts
def get_posts
#Check if the token is expired, if is login again and get a new token
if is_token_expired?
login("admin#example.com", "password")
end
self.class.get('/posts', #options)
end
# Gets comments
def get_comments
#Check if the token is expired, if is login again and get a new token
if is_token_expired?
login("admin#example.com", "password")
end
self.class.get('/comments', #options)
end
end
end
klass = Espresso::Client.new
klass.get_posts
klass.get_comments
A naive implementation would be;
module Callbacks
def self.extended(base)
base.send(:include, InstanceMethods)
end
def overridden_methods
#overridden_methods ||= []
end
def callbacks
#callbacks ||= Hash.new { |hash, key| hash[key] = [] }
end
def method_added(method_name)
return if should_override?(method_name)
overridden_methods << method_name
original_method_name = "original_#{method_name}"
alias_method(original_method_name, method_name)
define_method(method_name) do |*args|
run_callbacks_for(method_name)
send(original_method_name, *args)
end
end
def should_override?(method_name)
overridden_methods.include?(method_name) || method_name =~ /original_/
end
def before_run(method_name, callback)
callbacks[method_name] << callback
end
module InstanceMethods
def run_callbacks_for(method_name)
self.class.callbacks[method_name].to_a.each do |callback|
send(callback)
end
end
end
end
class Foo
extend Callbacks
before_run :bar, :zoo
def bar
puts 'bar'
end
def zoo
puts 'This runs everytime you call `bar`'
end
end
Foo.new.bar #=> This runs everytime you call `bar`
#=> bar
The tricky point in this implementation is, method_added. Whenever a method gets bind, method_added method gets called by ruby with the name of the method. Inside of this method, what I am doing is just name mangling and overriding the original method with the new one which first runs the callbacks then calls the original method.
Note that, this implementation neither supports block callbacks nor callbacks for super class methods. Both of them could be implemented easily though.
So witness and observe the following code, my questions is why do i never make it to the on_connect after starting the cool.io loop in send_to_server, the l.run should fire off the request as per the documented example on the github, and how the code handles incoming connections in module Server #socket.attach(l)
l.run
which does work and accepts the incoming data and sends it to my parser, which does work and fires off all the way up until the aforementioned send_to_server. So what is going on here?
require 'cool.io'
require 'http/parser'
require 'uri'
class Hash
def downcase_key
keys.each do |k|
store(k.downcase, Array === (v = delete(k)) ? v.map(&:downcase_key) : v)
end
self
end
end
module ShadyProxy
extend self
module ClientParserCallbacks
extend self
def on_message_complete(conn)
lambda do
puts "on_message_complete"
PluginHooks.before_request_to_server(conn)
end
end
def on_headers_complete(conn)
lambda do |headers|
conn.headers = headers
end
end
def on_body(conn)
lambda do |chunk|
conn.body << chunk
end
end
end
module PluginHooks
extend self
def before_request_to_server(conn)
# modify request here
conn.parser.headers.delete "Proxy-Connection"
conn.parser.headers.downcase_key
send_to_server(conn)
end
def send_to_server(conn)
parser = conn.parser
uri = URI::parse(parser.request_url)
l = Coolio::Loop.default
puts uri.scheme + "://" + uri.host
c = ShadyHttpClient.connect(uri.scheme + "://" + uri.host,uri.port).attach(l)
c.connection_reference = conn
c.request(parser.http_method,uri.request_uri)
l.run
end
def before_reply_to_client(conn)
end
end
class ShadyHttpClient < Coolio::HttpClient
def connection_reference=(conn)
puts "haz conneciton ref"
#connection_reference = conn
end
def connection_reference
#connection_reference
end
def on_connect
super
#never gets here
#headers = nil
#body = ''
#buffer = ''
end
def on_connect_failed
super
# never gets here either
end
def on_response_header(header)
#headers = header
end
def on_body_data(data)
puts "on data?"
#body << data
STDOUT.write data
end
def on_request_complete
puts "Headers"
puts #headers
puts "Body"
puts #body
end
def on_error(reason)
STDERR.puts "Error: #{reason}"
end
end
class ShadyProxyConnection < Cool.io::TCPSocket
attr_accessor :headers, :body, :buffer, :parser
def on_connect
#headers = nil
#body = ''
#buffer = ''
#parser = Http::Parser.new
#parser.on_message_complete = ClientParserCallbacks.on_message_complete(self)
#parser.on_headers_complete = ClientParserCallbacks.on_headers_complete(self)
#parser.on_body = ClientParserCallbacks.on_body(self)
end
def on_close
puts "huh?"
end
def on_read(data)
#buffer << data
#parser << data
end
end
module Server
def run(opts)
begin
# Start our server to handle connections (will raise things on errors)
l = Coolio::Loop.new
#socket = Cool.io::TCPServer.new(opts[:host],opts[:port], ShadyProxy::ShadyProxyConnection)
#socket.attach(l)
l.run
# Handle every request in another thread
loop do
Thread.new s = #socket.accept
end
# CTRL-C
rescue Interrupt
puts 'Got Interrupt..'
# Ensure that we release the socket on errors
ensure
if #socket
#socket.close
puts 'Socked closed..'
end
puts 'Quitting.'
end
end
module_function :run
end
end
ShadyProxy::Server.run(:host => '0.0.0.0',:port => 1234)
I would like to create a Pipe class to emulate Unix commands in Ruby in a two step fashion. First step is to compile a pipeline by adding a number of commands, and the second step is to run that pipeline. Here is a mockup:
#!/usr/bin/env ruby
p = Pipe.new
p.add(:cat, input: "table.txt")
p.add(:cut, field: 2)
p.add(:grep, pattern: "foo")
p.add(:puts, output: "result.txt")
p.run
The question is how to code this using lazy evaluation, so that the pipe is processed record by record when run() is called without loading all of the data into memory at any one time?
Take a look at the http://ruby-doc.org/core-2.0.0/Enumerator.html class. The Pipe class will stitch together an Enumerator, e.g. add(:cat, input: 'foo.txt') will create an enumerator which yields lines of foo.txt. add(:grep) will filter it according to regexp etc.
Here's the lazy file reader
require 'benchmark'
def lazy_cat(filename)
e = Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
e.lazy
end
def cat(filename)
Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
end
lazy = Benchmark.realtime { puts lazy_cat("log.txt").map{|s| s.upcase}.take(1).to_a }
puts "Lazy: #{lazy}"
eager = Benchmark.realtime { puts cat("log.txt").map{|s| s.upcase}.take(1).to_a }
puts "Eager: #{eager}"
Eager version takes 7 seconds for 10 million line file, lazy version takes pretty much no time.
For what I understood you can simply read one line at a time and move this single line thought the pipeline, then write it to the output. Some code:
output = File.new("output.txt")
File.new("input.txt").each do |line|
record = read_record(line)
newrecord = run_pipeline_on_one_record(record)
output.write(dump_record(newrecord))
end
Another much heavier option would be create actual IO blocking pipes and use one thread for each task in the pipeline. This somewhat reassembles what Unix does.
Sample usage with OP's syntax:
class Pipe
def initialize
#actions = []
end
def add(&block)
#actions << block
end
def run(infile, outfile)
output = File.open(outfile, "w")
File.open(infile).each do |line|
line.chomp!
#actions.each {|act| line = act[line] }
output.write(line+"\n")
end
end
end
p = Pipe.new
p.add {|line| line.size.to_s }
p.add {|line| "number of chars: #{line}" }
p.run("in.txt", "out.txt")
Sample in.txt:
aaa
12345
h
Generated out.txt:
number of chars: 3
number of chars: 5
number of chars: 1
This seems to work:
#!/usr/bin/env ruby
require 'pp'
class Pipe
def initialize
#commands = []
end
def add(command, options = {})
#commands << [command, options]
self
end
def run
enum = nil
#commands.each do |command, options|
enum = method(command).call enum, options
end
enum.each {}
enum
end
def to_s
cmd_string = "Pipe.new"
#commands.each do |command, options|
opt_list = []
options.each do |key, value|
if value.is_a? String
opt_list << "#{key}: \"#{value}\""
else
opt_list << "#{key}: #{value}"
end
end
cmd_string << ".add(:#{command}, #{opt_list.join(", ")})"
end
cmd_string << ".run"
end
private
def cat(enum, options)
Enumerator.new do |yielder|
enum.map { |line| yielder << line } if enum
File.open(options[:input]) do |ios|
ios.each { |line| yielder << line }
end
end.lazy
end
def cut(enum, options)
Enumerator.new do |yielder|
enum.each do |line|
fields = line.chomp.split(%r{#{options[:delimiter]}})
yielder << fields[options[:field]]
end
end.lazy
end
def grep(enum, options)
Enumerator.new do |yielder|
enum.each do |line|
yielder << line if line.match(options[:pattern])
end
end.lazy
end
def save(enum, options)
Enumerator.new do |yielder|
File.open(options[:output], 'w') do |ios|
enum.each do |line|
ios.puts line
yielder << line
end
end
end.lazy
end
end
p = Pipe.new
p.add(:cat, input: "table.txt")
p.add(:cut, field: 2, delimiter: ',\s*')
p.add(:grep, pattern: "4")
p.add(:save, output: "result.txt")
p.run
puts p
https://stackoverflow.com/a/20049201/3183101
require 'benchmark'
def lazy_cat(filename)
e = Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
e.lazy
end
def cat(filename)
Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
end
lazy = Benchmark.realtime { puts lazy_cat("log.txt").map{|s| s.upcase}.take(1).to_a }
puts "Lazy: #{lazy}"
eager = Benchmark.realtime { puts cat("log.txt").map{|s| s.upcase}.take(1).to_a }
puts "Eager: #{eager}"
This could have been simplified to the following, which I think makes the diff between the two methods easier to see.
require 'benchmark'
def cat(filename, evaluation_strategy: :eager)
e = Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
e.lazy if evaluation_strategy == :lazy
end
lazy = Benchmark.realtime { puts cat("log.txt", evaluation_strategy: :lazy).map{ |s|
s.upcase}.take(1).to_a
}
puts "Lazy: #{lazy}"
eager = Benchmark.realtime { puts cat("log.txt", evaluation_strategy: :eager).map{ |s|
s.upcase}.take(1).to_a
}
puts "Eager: #{eager}"
I would have just put this in a comment, but I'm too 'green' here to be permitted to do so. Anyway, the ability to post all of the code I think makes it clearer.
This builds on previous answers, and serves as a warning about a gotcha regarding enumerators. An enumerator that hasn't been exhausted (i.e. raised StopIteration) will not run ensure blocks. That means a construct like File.open { } won't clean up after itself.
Example:
def lazy_cat(filename)
f = nil # visible to the define_singleton_method block
e = Enumerator.new do |yielder|
# Also stored in #f for demonstration purposes only, so we examine it later
#f = f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
e.lazy.tap do |enum|
# Provide a finish method to close the File
# We can't use def enum.finish because it can't see 'f'
enum.define_singleton_method(:finish) do
f.close
end
end
end
def get_first_line(path)
enum = lazy_cat(path)
enum.take(1).to_a
end
def get_first_line_with_finish(path)
enum = lazy_cat(path)
enum.take(1).to_a
ensure
enum.finish
end
# foo.txt contains:
# abc
# def
# ghi
puts "Without finish"
p get_first_line('foo.txt')
if #f.closed?
puts "OK: handle was closed"
else
puts "FAIL: handle not closed!"
#f.close
end
puts
puts "With finish"
p get_first_line_with_finish('foo.txt')
if #f.closed?
puts "OK: handle was closed"
else
puts "FAIL: handle not closed!"
#f.close
end
Running this produces:
Without finish
["abc\n"]
FAIL: handle not closed!
With finish
["abc\n"]
OK: handle was closed
Note that if you don't provide the finish method, the stream won't be closed, and you'll leak file descriptors. It's possible that GC will close it, but you shouldn't depend on that.
I am attempting to write a chat server with EventMachine. How do I pass a message from one EventMachine connection, to another, in a thread-safe manner?
I see a messaging protocol (Stomp) being supported but I can't figure out how to use it. Any help is appreciated.
Stomp in EventMachine - http://eventmachine.rubyforge.org/EventMachine/Protocols/Stomp.html
See http://eventmachine.rubyforge.org/EventMachine/Channel.html
you can try something in these lines:
require 'eventmachine'
class Chat < EventMachine::Connection
def initialize channel
#channel = channel
end
def post_init
send_data 'Hello'
#sid = #channel.subscribe do |msg|
send_data msg
end
end
def receive_data(msg)
#channel.push msg
end
def unbind
#channel.unsubscribe #sid
end
end
EM.run do
#channel = EventMachine::Channel.new
EventMachine.start_server '127.0.0.1', 8081, Chat, #channel
end
EDIT: also check out https://github.com/eventmachine/eventmachine/tree/master/examples/guides/getting_started - there is a nice chatroom example
Try starting out with an in memory message dispatcher.
require 'thread'
class Room
def initialize
#users = []
end
def join(user)
#users << user
end
def leave(user)
#user.delete(user)
end
def broadcast(message)
#users.each do |user|
user.enqueue(message)
end
end
end
class User
def initialize
#mutex = Mutex.new
#queued_messages = []
end
def enqueue(message)
#mutex.synchronize do
#queued_message << message
end
end
def get_new_messages
#mutex.synchronize do
output = #queued_messages
#queued_messages = []
end
return output
end
end
UPDATE
ROOM = Room.new
class Connection
def user_logged_in
# #user = ...
ROOM.join(#user)
end
def received_message(message)
ROOM.broadcast(message)
end
def receive_send_more_messages_request(req)
messages = #user.get_new_messages
# write messages
end
def connection_closed
ROOM.leave(#user)
end
end
Is there a way to scope variables to the thread without having to pass everything around, given a class with the following methods:
def initialize
#server = TCPServer.new('localhost',456)
end
def start
threads = []
while (upload = #server.accept)
threads << Thread.new(upload) do |connection|
some_method_here(connection)
end
end
threads.each {|t| t.join }
end
def some_method_here(connection)
variable = "abc"
another_method(connection,variable)
end
def another_method(connection,variable)
puts variable.inspect
connection.close
end
if I get you right you want to use thread local variables (see the ruby rdoc for Thread#[])
From the rdoc:
a = Thread.new { Thread.current["name"] = "A"; Thread.stop }
b = Thread.new { Thread.current[:name] = "B"; Thread.stop }
c = Thread.new { Thread.current["name"] = "C"; Thread.stop }
Thread.list.each {|x| puts "#{x.inspect}: #{x[:name]}" }
produces:
#<Thread:0x401b3b3c sleep>: C
#<Thread:0x401b3bc8 sleep>: B
#<Thread:0x401b3c68 sleep>: A
#<Thread:0x401bdf4c run>:
So your example would use
Thread.current[:variable] = "abc"
Thread.current[:variable] # => "abc"
wherever you were using just variable before
Thread.current[:variable_name] = ... ?
You can try an around_filter in ApplicationController
around_filter :apply_scope
def apply_scope
Document.where(:user_id => current_user.id).scoping do
yield
end