I have a very simple Sinatra app that allows me to receive SMS messages through my Twilio number and will print them to the same terminal session that the app is running on. I would like to save these messages to a local .csv file. Adding CSV.open() to the app throws some errors.
require 'sinatra'
require 'twilio-ruby'
require 'csv'
post '/receive_sms' do
#body = params["Body"].to_s
#sid = params["MessageSid"].to_s
#sender = params["From"].delete('+').to_i
content_type 'text/xml'
puts #body
puts #sender
puts #sid
CSV.open("/home/ubuntu/Twilio_SMS/smsLog.csv", "a") do |csv|
csv << [#sender, #body, #sid]
end
end
This gives me the following errors:
ERROR IOError: closed stream
/home/ubuntu/.rvm/gems/ruby-2.2.1/gems/rack-1.6.4/lib/rack/body_proxy.rb:16:in `close'
/home/ubuntu/.rvm/gems/ruby-2.2.1/gems/rack-1.6.4/lib/rack/handler/webrick.rb:117:in `ensure in service'
/home/ubuntu/.rvm/gems/ruby-2.2.1/gems/rack-1.6.4/lib/rack/handler/webrick.rb:117:in `service'
/home/ubuntu/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/webrick/httpserver.rb:138:in `service'
/home/ubuntu/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/webrick/httpserver.rb:94:in `run'
/home/ubuntu/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/webrick/server.rb:294:in `block in start_thread'
I have tried moving the CSV call outside of the post method, but this only writes , , to the file every time I start the applicaiton.
What is the proper way to save this informaiton to the CSV file and make sure that every message is added even if they are recieved in rapid succession?
Try adding a valid return value to the method.
require 'sinatra'
require 'twilio-ruby'
require 'csv'
post '/receive_sms' do
#body = params["Body"].to_s
#sid = params["MessageSid"].to_s
#sender = params["From"].delete('+').to_i
content_type 'text/xml'
puts #body
puts #sender
puts #sid
CSV.open("/home/ubuntu/Twilio_SMS/smsLog.csv", "a") do |csv|
csv << [#sender, #body, #sid]
end
'done'
end
Because 'CSV.open' was the last method you ran, Sinatra tried to read from it to generate an HTTP reply - and invoked an IOError from trying to read from a closed stream.
Related
I have been trying to use the ticketmaster api to obtain some data. However, it throws an error when I try to run the code. I do not know why the error is popping up. I do not know if it is because of the url that I gave. Open method accept the url as string so I converted to string and this url is working in postman if I try. The exact error is
C:/RailsInstaller/Ruby2.3.0/lib/ruby/2.3.0/delegate.rb:87:in `method_missing': u
ndefined method `string' for #<Tempfile:0x28c2d50> (NoMethodError)
from test.rb:20:in `connection'
from test.rb:47:in `<main>'
and my code is ;
require 'open-uri'
require 'openssl'
require 'json'
class Test
#silence_warnings do
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE #unless Rails.env.production?
#end
def initialize()
#url="https://app.ticketmaster.com/discovery/v2/events.json?apikey=XXXXXXXXXXXXXXXXXT&city=Binghamton"
end
def connection
puts "url", #url
result=open(#url)
response_status=result.status
#putstus
if(response_status[0]=="200")
#body=JSON.parse(result.string)
puts("Status is "+ response_status[0],"Server Message is "+ response_status[1])
return #body
else
connection_error
puts("Status is ", response_status[0])
end
end
def connection_error
puts "Connection error with the api"
end
def silence_warnings
with_warnings(nil) { yield }
end
def get_content
raise notImplementedMethod
end
end
a=Test.new
a.connection
I have a simple ruby script meant to send all received messages as sms messages. However, somehow for some reason it does not execute.
Here is the sample code;
/etc/aliases
motor: "|/home/motorcare/sms_script.rb"
sms_script.rb
#!/usr/bin/env ruby
require "json"
require "httparty"
require 'net/http'
require 'uri'
require "cgi"
require "mail"
# Reading files
mail = Mail.read(ARGV[0])
destination = mail.subject
message = mail.body.decoded
#first_line = lines[0].strip
if destination =~ /^(256)/
send(destination, message)
else
destination = "256#{destination.gsub(/^0+/,"")}"
send(destination, message)
end
# Sending message
def send(destination, message)
url = "http://xxxxxxxxxx.com/messages?token=c19ae2574be1875f0fa09df13b0dde0b&to=#{phone_number}&from=xxxxxx&message=#{CGI.escape(message)}"
5.times do |i|
response = HTTParty.get(url)
body = JSON.parse(response.body)
if body["status"] == "Success"
break
end
end
end
Anyone with a similar script to assist with this one?
You have 2 errors.
1st error is that send is already defined in Ruby. See this SO post What does send() do in Ruby?
see this code
$ cat send.rb
#!/usr/bin/env ruby
puts defined? send
puts send :class
$ ./send.rb
method
Object
2nd error is that you call the method before it's even defined. See this sample code (calling welcome before def welcome)
$ cat welcome.rb
#!/usr/bin/env ruby
welcome('hello from welcome')
def welcome(msg)
puts msg
end
$ ./welcome.rb
./welcome.rb:3:in `<main>': undefined method `welcome' for main:Object (NoMethodError)
Change the method name from send to something else, e.g. send_sms, and put the definition before calling the method
So this should be sth like:
#!/usr/bin/env ruby
require "json"
require "httparty"
require 'net/http'
require 'uri'
require "cgi"
require "mail"
# Sending message
def send_sms(destination, message)
url = "http://xxxxxxxxxx.com/messages?token=c19ae2574be1875f0fa09df13b0dde0b&to=#{phone_number}&from=xxxxxx&message=#{CGI.escape(message)}"
5.times do |i|
response = HTTParty.get(url)
body = JSON.parse(response.body)
if body["status"] == "Success"
break
end
end
end
# Reading files
mail = Mail.read(ARGV[0])
destination = mail.subject
message = mail.body.decoded
#first_line = lines[0].strip
if destination =~ /^(256)/
send_sms(destination, message)
else
destination = "256#{destination.gsub(/^0+/,"")}"
send_sms(destination, message)
end
And also adding logging to the script would give you info about what's going in inside when it's run and pipped. So you can easily debug the beaviour. Logging is the easies approach to DEBUG.
I built a simple API with one endpoint. It scrapes files and currently has around 30,000 records. I would ideally like to be able to fetch all those records in JSON with one http call.
Here is my Sinatra view code:
require 'sinatra'
require 'json'
require 'mongoid'
Mongoid.identity_map_enabled = false
get '/' do
content_type :json
Book.all
end
I've tried the following:
using multi_json with
require './require.rb'
require 'sinatra'
require 'multi_json'
MultiJson.engine = :yajl
Mongoid.identity_map_enabled = false
get '/' do
content_type :json
MultiJson.encode(Book.all)
end
The problem with this approach is I get Error R14 (Memory quota exceeded). I get the same error when I try to use the 'oj' gem.
I would just concatinate everything one long Redis string, but Heroku's redis service is $30 per month for the instance size I would need (> 10mb).
My current solution is to use background task that creates objects and stuffs them full of jsonified objects at near the Mongoid object size limit (16mb). The problems with this approach: It still takes nearly 30 seconds to render, and I have to run post-processing on the receiving app to properly extract the json from the objects.
Does anyone have any better idea for how I can render json for 30k records in one call without switching away from Heroku?
Sounds like you want to stream the JSON directly to the client instead of building it all up in memory. It's probably the best way to cut down memory usage. You could for example use yajl to encode JSON directly to a stream.
Edit: I rewrote the entire code for yajl, because its API is much more compelling and allows for much cleaner code. I also included an example for reading the response in chunks. Here's the streamed JSON array helper I wrote:
require 'yajl'
module JsonArray
class StreamWriter
def initialize(out)
super()
#out = out
#encoder = Yajl::Encoder.new
#first = true
end
def <<(object)
#out << ',' unless #first
#out << #encoder.encode(object)
#out << "\n"
#first = false
end
end
def self.write_stream(app, &block)
app.stream do |out|
out << '['
block.call StreamWriter.new(out)
out << ']'
end
end
end
Usage:
require 'sinatra'
require 'mongoid'
Mongoid.identity_map_enabled = false
# use a server that supports streaming
set :server, :thin
get '/' do
content_type :json
JsonArray.write_stream(self) do |json|
Book.all.each do |book|
json << book.attributes
end
end
end
To decode on the client side you can read and parse the response in chunks, for example with em-http. Note that this solution requires the clients memory to be large enough to store the entire objects array. Here's the corresponding streamed parser helper:
require 'yajl'
module JsonArray
class StreamParser
def initialize(&callback)
#parser = Yajl::Parser.new
#parser.on_parse_complete = callback
end
def <<(str)
#parser << str
end
end
def self.parse_stream(&callback)
StreamParser.new(&callback)
end
end
Usage:
require 'em-http'
parser = JsonArray.parse_stream do |object|
# block is called when we are done parsing the
# entire array; now we can handle the data
p object
end
EventMachine.run do
http = EventMachine::HttpRequest.new('http://localhost:4567').get
http.stream do |chunk|
parser << chunk
end
http.callback do
EventMachine.stop
end
end
Alternative solution
You could actually simplify the whole thing a lot when you give up the need for generating a "proper" JSON array. What the above solution generates is JSON in this form:
[{ ... book_1 ... }
,{ ... book_2 ... }
,{ ... book_3 ... }
...
,{ ... book_n ... }
]
We could however stream each book as a separate JSON and thus reduce the format to the following:
{ ... book_1 ... }
{ ... book_2 ... }
{ ... book_3 ... }
...
{ ... book_n ... }
The code on the server would then be much simpler:
require 'sinatra'
require 'mongoid'
require 'yajl'
Mongoid.identity_map_enabled = false
set :server, :thin
get '/' do
content_type :json
encoder = Yajl::Encoder.new
stream do |out|
Book.all.each do |book|
out << encoder.encode(book.attributes) << "\n"
end
end
end
As well as the client:
require 'em-http'
require 'yajl'
parser = Yajl::Parser.new
parser.on_parse_complete = Proc.new do |book|
# this will now be called separately for every book
p book
end
EventMachine.run do
http = EventMachine::HttpRequest.new('http://localhost:4567').get
http.stream do |chunk|
parser << chunk
end
http.callback do
EventMachine.stop
end
end
The great thing is that now the client does not have to wait for the entire response, but instead parses every book separately. However, this will not work if one of your clients expects one single big JSON array.
I am trying to build a unit test for a REST API I built using Sinatra. For right now I just want to test that my echo function works right. Echo uses POST and will return the exact same payload from the post. I am still new with ruby, so forgive me if I don't use the proper lingo.
Here is the code I want to test:
post '/echo' do
request.body.read
end
This is the unit test I am trying to make:
ENV['RACK_ENV'] = 'test'
require './rest_server'
require 'test/unit'
require 'rack/test'
require 'json'
class RestServer < Test::Unit::TestCase
def app
Sinatra::Application
end
def test_check_methods
data = '{"dataIn": "hello"}'
response = post '/echo', JSON.parse(data)
assert.last_response.ok?
assert(response.body == data)
end
end
With the above code, here is the error:
NoMethodError: undefined method `dataIn' for Sinatra::Application:Class
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1285:in `block in compile!'
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1285:in `each_pair'
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1285:in `compile!'
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1267:in `route'
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1256:in `post'
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1688:in `block (2 levels) in delegate'
/Users/barrywilliams/RubymineProjects/project/rest_server_test.rb:20:in `test_check_methods'
If I try doing it without the JSON.parse, I get
NoMethodError: undefined method `key?' for "{\"dataIn\": \"hello\"}":String
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1265:in `route'
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1256:in `post'
/Users/barrywilliams/.rvm/gems/ruby-1.9.3-p448/gems/sinatra-1.3.4/lib/sinatra/base.rb:1688:in `block (2 levels) in delegate'
/Users/barrywilliams/RubymineProjects/project/rest_server_test.rb:20:in `test_check_methods'
If I try doing it where data = 'hello', then I get the same undefined method 'key?' error
I've tried this suggestion, with no success:
http://softwareblog.morlok.net/2010/12/18/testing-post-with-racktest/
I get an error saying that post only takes 2 arguments, not 3.
So, in summary, I need to be able to make a call, have the code I'm testing receive the call and return a response, then I need to be able to read that response and verify it was the original data. Right now it looks like it's getting stuck at just making the call.
I did a thing a little similar, it might help you :
The application post definition :
post '/' do
data = JSON.parse request.body.read.to_s
"Hello !\n#{data.to_s}"
end
The .to_s is necessary, else the conversions will not be exactly the same :-/
Then on the test file :
class RootPostTest < Test::Unit::TestCase
include Rack::Test::Methods
def app
Sinatra::Application
end
def test_return_the_parameters
data = {
'reqID' => 1,
'signedReqID' => "plop",
'cert' => "mycert"
}
post '/', data.to_json, "CONTENT_TYPE" => "application/json"
assert last_response.ok?
body_espected = "Hello !\n#{JSON.parse(data.to_json).to_s}"
assert_equal last_response.body, body_espected
end
end
Hope it helped you.
Rack Test will give you back the response body in last_response.body, no need to save it to a variable. You're also not echoing back what you've sent - data in the code you've given is JSON, but you converted it to a hash and posted that, so it's not going to match what comes back. Either send JSON, or convert it to JSON in the Sinatra route if you want to do that (see https://stackoverflow.com/a/12138793/335847 for more).
In the Sinatra app:
require 'json'
post '/echo' do
# Don't use request.body.read as you're not posting JSON
params.to_json
end
and in the test file:
def test_check_methods
data = '{"dataIn": "hello"}'
post '/echo', JSON.parse(data)
assert.last_response.ok?
assert(last_response.body == data)
end
If you do end up wanting to post JSON (which I think is usually not a good idea if it's easy to convert or already have the data as a hash) then use :provides => "json" as a condition to the route, and consider using Rack::Test::Accepts to make life easier writing the test for that (note: that's a shameless plug for a gem I wrote;)
Maybe this is the wrong approach, but I'm trying to parallelize em-hiredis puts and lookups in Goliath with EM::Synchrony::Multi or EM::Synchrony::FiberIterator. However, I can't seem to access basic values initialized in the config. I keep getting method_missing errors.
Here's the basic watered down version of what I'm trying to do:
/lib/config/try.rb
config['redisUri'] = 'redis://localhost:6379/0'
config['redis_db'] ||= EM::Hiredis.connect
config['user_agent'] = "MyCrawler Mozilla/5.0 Compat etc."
Here's the basic Goliath Setup
/try.rb
require "goliath"
require "em-hiredis"
require "em-synchrony/fiber_iterator"
require "em-synchrony/em-hiredis"
require "em-synchrony/em-multi"
class Try < Goliath::API
use Goliath::Rack::Params
use Goliath::Rack::DefaultMimeType
def response(env)
case env['REQUEST_PATH']
when "/start" then
start_crawl()
body = "STARTING"
[200, {}, body]
end
end
def start_crawl
urls = ["http://www.example.com/",
"http://www.example.com/photos/",
"http://www.example.com/video/",
]
EM::Synchrony::FiberIterator.new(urls, 3).each do |url|
p "#{user_agent}"
redis_db.sadd 'test_queue', url
end
# multi = EM::Synchrony::Multi.new
# urls.each_with_index do |url, index|
# p "#{user_agent}"
# multi.add index, redis_db.sadd('test_queue', url)
# end
end
end
However, I keep getting errors where Goliath doesn't know what user_agent is or redis_db which were initialized in the config.
[936:INFO] 2012-09-21 23:47:10 :: Starting server on 0.0.0.0:9000 in development mode. Watch out for stones.
/Users/ewu/.rvm/gems/ruby-1.9.3-p194#crawler/gems/goliath-1.0.0/lib/goliath/api.rb:143:in `method_missing': undefined local variable or method `user_agent' for #<Try:0x007ff5a431c4e0 #opts={}> (NameError)
from ./lib/try.rb:27:in `block in start_crawl'
from /Users/ewu/.rvm/gems/ruby-1.9.3-p194#crawler/gems/em-synchrony-1.0.2/lib/em-synchrony/fiber_iterator.rb:10:in `call'
from /Users/ewu/.rvm/gems/ruby-1.9.3-p194#crawler/gems/em-synchrony-1.0.2/lib/em-synchrony/fiber_iterator.rb:10:in `block (2 levels) in each'
...
...
...
Ideally I'd be able to get FiberIterator working, because I have additional conditionals to check for:
EM::Synchrony::FiberIterator.new(urls, 3).each do |new_url}
is_member = redis_db.sismember('crawled_urls', new_url)
is_member += redis_db.sismember('queued_urls', new_url)
if is_member == 0
redis_db.lpush 'crawl_queue', new_url
redis_db.sadd 'queued_urls', new_url
end
end
I don't think your config file is getting loaded. The name of try.rb needs to match the name of the robojin.rb file in the config directory.