Summary
Within a Sinatra web app, how can I make a virtual request to the application and get the response body back as text? For example, these routes...
get('/foo'){ "foo" }
get('/bar'){ "#{spoof_request '/foo'} - bar" }
...should result in the response "foo - bar" when requesting "/bar" with the web browser.
Motivation
My application has a page representing an bug entry, with lots of details about that bug entry: what version was the bug experienced in, how important is it, what tags are associated with it, to whom is the bug assigned, etc.
The user may edit individual pieces of data on this page interactively. Using my AJAXFetch jQuery plugin, JavaScript uses AJAX to swap out a read-only section of the page (e.g. the name of the person that this bug is assigned to) with an HTML partial form for editing just that section. The user submits the form, and AJAX makes a new request for the static version of that field.
In order to be DRY, I want the Haml view that creates the page to use the exact same request that AJAX makes when creating the individual static pieces. For example:
#notifications.section
%h2 Email me if someone...
.section-body= spoof_request "/partial/notifications/#{#bug.id}"
Not-Quite-Working Code
The following helper defining spoof_request worked under Sinatra 1.1.2:
PATH_VARS = %w[ REQUEST_PATH PATH_INFO REQUEST_URI ]
def spoof_request( uri, headers=nil )
new_env = env.dup
PATH_VARS.each{ |k| new_env[k] = uri.to_s }
new_env.merge!(headers) if headers
call( new_env ).last.join
end
Under Sinatra 1.2.3, however, this no longer works. Despite setting each of the PATH_VARS to the desired URI, the call( new_env ) still causes Sinatra to process the route for the current request, not for the specified path. (This results in infinite recursion until the stack level finally bottoms out.)
This question differs from Calling Sinatra from within Sinatra because the accepted answer to that (old) question does not maintain the session of the user.
The code I was using is more complex than the answer in the Sinatra README, but relied on the same mechanism. Neither my code nor the answer from the README worked under 1.2.3 due to a bug in that version. Both now work under 1.2.6.
Here's a test case of a simple helper that works:
require 'sinatra'
helpers do
def local_get(url)
call(env.merge("PATH_INFO" => url)).last.join
end
end
get("/foo"){ "foo - #{local_get '/bar'}" }
get("/bar"){ "bar" }
In action:
phrogz$ curl http://localhost:4567/foo
foo - bar
The following appears to work as needed under Sinatra 1.2.3:
ENV_COPY = %w[ REQUEST_METHOD HTTP_COOKIE rack.request.cookie_string
rack.session rack.session.options rack.input]
# Returns the response body after simulating a request to a particular URL
# Maintains the session of the current user.
# Pass custom headers if you want to set or change them, e.g.
#
# # Spoof a GET request, even if we happen to be inside a POST
# html = spoof_request "/partial/assignedto/#{#bug.id}", 'REQUEST_METHOD'=>'GET'
def spoof_request( uri, headers=nil )
new_env = env.slice(*ENV_COPY).merge({
"PATH_INFO" => uri.to_s,
"HTTP_REFERER" => env["REQUEST_URI"]
})
new_env.merge!(headers) if headers
call( new_env ).last.join
end
where Hash#slice is defined as:
class Hash
def slice(*keys)
{}.tap{ |h| keys.each{ |k| h[k] = self[k] } }
end
end
It feels like I'm missing what you are trying to do but why not just call the defined method?
get('/foo'){ "foo" }
get('/bar'){ "#{self.send("GET /foo")} - bar" }
That is one funky method name by the way. Don't ask me why it's even allowed.
PS. This only works pre version 1.2.3. DS.
Related
I can't find a basic explanation anywhere about how I can test, with Rack::Test, that a Ruby/Sinatra post method successfully saves data to a YAML store/file. (This explains testing get, which I can do(!), but not post; other mentions of testing post methods with rack/test seem irrelevant.) For self-study, I'm building a "to do" app in Ruby/Sinatra and I'm trying to use TDD everything and unit test like a good little boy. A requirement I have is: When a user posts a new task, it is saved in the YML store.
I was thinking of testing this either by seeing if a "Task saved" was shown in the response to the user (which of course isn't directly testing the thing itself...but is something I'd also like to test):
assert last_response.body.include?("Task saved")
or by somehow testing that a test task's description is now in the YML file. I guess I could open up the YML file and look, and then delete it from the YML file, but I'm pretty sure that's not what I'm supposed to do.
I've confirmed post does correctly save to a YML file:
get('/') do |*user_message|
# prepare erb messages
#user_message = session[:message] if session[:message]
#overlong_description = session[:overlong_description] if
session[:overlong_description]
session[:message] = nil # clear message after being used
session[:overlong_description] = nil # ditto
#tasks = store.all
erb :index #, user_message => {:user_message => params[:user_message]}
end
post('/newtask') do
#task = Task.new(store, params)
# decide whether to save & prepare user messages
if #task.complete == true # task is complete!
#task.message << " " + "Task saved!"
session[:message] = #task.message # use session[:message] for user messages
#task.message = ""
store.save(#task)
else
#task.message << " " + "Not saved." # task incomplete
session[:message] = #task.message # use session[:message] for user messages
session[:overlong_description] = #task.overlong_description if
#task.overlong_description
#task.message = ""
#task.overlong_description = nil
end
redirect '/'
end
As you can see, it ends in a redirect...one response I want to test is actually on the slash route, not on the /newtask route.
So of course the test doesn't work:
def test_post_newtask
post('/newtask', params = {"description"=>"Test task 123"})
# Test that "saved" message for user is in returned page
assert last_response.body.include?("Task saved") # boooo
end
Github source here
If you can give me advice on a book (chapter, website, blog, etc.) that goes over this in a way accessible to a relative beginner, I'd be most grateful.
Be gentle...I'm very new to testing (and programming).
Nobody answered my question and, since I have figured out what the answer is, I thought I would share it here.
First of all, I gather that it shouldn't be necessary to check if the data is actually saved to the YAML store; the main thing is to see if the web page returns the correct result (we assume the database is groovy if so).
The test method I wrote above was correct; it was simply missing the single line follow_redirect!. Apparently I didn't realize that I needed to instruct rake/test to follow the redirect.
Part of the problem was that I simply hadn't found the right documentation. This page does give the correct syntax, but doesn't give much detail. This page helped a lot, and this bit covers redirects.
Here's the updated test method:
def test_post_newtask
post "/newtask", params = {"description" => "Write about quick brown foxes",
"categories" => "writing823"}
follow_redirect!
assert last_response.body.include?("Task saved")
assert last_response.body.include?("Write about quick brown foxes")
end
(With thanks to the Columbus Ruby Brigade.)
I have been tasked with creating a Ruby API that retrieves youtube URL's. However, I am not sure of the proper way to create an 'API'... I did the following code below as a Sinatra server that serves up JSON, but what exactly would be the definition of an API and would this qualify as one? If this is not an API, how can I make in an API? Thanks in advance.
require 'open-uri'
require 'json'
require 'sinatra'
# get user input
puts "Please enter a search (seperate words by commas):"
search_input = gets.chomp
puts
puts "Performing search on YOUTUBE ... go to '/videos' API endpoint to see the results and use the output"
puts
# define query parameters
api_key = 'my_key_here'
search_url = 'https://www.googleapis.com/youtube/v3/search'
params = {
part: 'snippet',
q: search_input,
type: 'video',
videoCaption: 'closedCaption',
key: api_key
}
# use search_url and query parameters to construct a url, then open and parse the result
uri = URI.parse(search_url)
uri.query = URI.encode_www_form(params)
result = JSON.parse(open(uri).read)
# class to define attributes of each video and format into eventual json
class Video
attr_accessor :title, :description, :url
def initialize
#title = nil
#description = nil
#url = nil
end
def to_hash
{
'title' => #title,
'description' => #description,
'url' => #url
}
end
def to_json
self.to_hash.to_json
end
end
# create an array with top 3 search results
results_array = []
result["items"].take(3).each do |video|
#video = Video.new
#video.title = video["snippet"]["title"]
#video.description = video["snippet"]["description"]
#video.url = video["snippet"]["thumbnails"]["default"]["url"]
results_array << #video.to_json.gsub!(/\"/, '\'')
end
# define the API endpoint
get '/videos' do
results_array.to_json
end
An "API = Application Program Interface" is, simply, something that another program can reliably use to get a job done, without having to busy its little head about exactly how the job is done.
Perhaps the simplest thing to do now, if possible, is to go back to the person who "tasked" you with this task, and to ask him/her, "well, what do you have in mind?" The best API that you can design, in this case, will be the one that is most convenient for the people (who are writing the programs which ...) will actually have to use it. "Don't guess. Ask!"
A very common strategy for an API, in a language like Ruby, is to define a class which represents "this application's connection to this service." Anyone who wants to use the API does so by calling some function which will return a new instance of this class. Thereafter, the program uses this object to issue and handle requests.
The requests, also, are objects. To issue a request, you first ask the API-connection object to give you a new request-object. You then fill-out the request with whatever particulars, then tell the request object to "go!" At some point in the future, and by some appropriate means (such as a callback ...) the request-object informs you that it succeeded or that it failed.
"A whole lot of voodoo-magic might have taken place," between the request object and the connection object which spawned it, but the client does not have to care. And that, most of all, is the objective of any API. "It Just Works.™"
I think they want you to create a third-party library. Imagine you are schizophrenic for a while.
Joe wants to build a Sinatra application to list some YouTube videos, but he is lazy and he does not want to do the dirty work, he just wants to drop something in, give it some credentials, ask for urls and use them, finito.
Joe asks Bob to implement it for him and he gives him his requirements: "Bob, I need YouTube library. I need it to do:"
# Please note that I don't know how YouTube API works, just guessing.
client = YouTube.new(api_key: 'hola')
video_urls = client.videos # => ['https://...', 'https://...', ...]
And Bob says "OK." end spends a day in his interactive console.
So first, you should figure out how you are going to use your not-yet-existing lib, if you can – sometimes you just don't know yet.
Next, build that library based on the requirements, then drop it in your Sinatra app and you're done. Does that help?
Say I have some HTML documents stored on S3 likes this:
http://alan.aws-s3-bla-bla.com/posts/1.html
http://alan.aws-s3-bla-bla.com/posts/2.html
http://alan.aws-s3-bla-bla.com/posts/3.html
http://alan.aws-s3-bla-bla.com/posts/1/comments/1.html
http://alan.aws-s3-bla-bla.com/posts/1/comments/2.html
http://alan.aws-s3-bla-bla.com/posts/1/comments/3.html
etc, etc
I'd like to serve these with a Rack (preferably Sinatra) application, mapping the following routes:
get "/posts/:id" do
render "http://alan.aws-s3-bla-bla.com/posts/#{params[:id]}.html"
end
get "/posts/:posts_id/comments/:comments_id" do
render "http://alan.aws-s3-bla-bla.com/posts/#{params[:posts_id]}/comments/#{params[:comments_id}.html"
end
Is this a good idea? How would I do it?
There would obviously be a wait while you grabbed the file, so you could cache it or set etags etc to help with that. I suppose it depends on how long you want to wait and how often it is accessed, its size etc as to whether it's worth storing the HTML locally or remotely. Only you can work that bit out.
If the last expression in the block is a string that will automatically be rendered, so there's no need to call render as long as you've opened the file as a string.
Here's how to grab an external file and put it into a tempfile:
require 'faraday'
require 'faraday_middleware'
#require 'faraday/adapter/typhoeus' # see https://github.com/typhoeus/typhoeus/issues/226#issuecomment-9919517 if you get a problem with the requiring
require 'typhoeus/adapters/faraday'
configure do
Faraday.default_connection = Faraday::Connection.new(
:headers => { :accept => 'text/plain', # maybe this is wrong
:user_agent => "Sinatra via Faraday"}
) do |conn|
conn.use Faraday::Adapter::Typhoeus
end
end
helpers do
def grab_external_html( url )
response = Faraday.get url # you'll need to supply this variable somehow, your choice
filename = url # perhaps change this a bit
tempfile = Tempfile.open(filename, 'wb') { |fp| fp.write(response.body) }
end
end
get "/posts/:whatever/" do
tempfile = grab_external_html whatever # surely you'd do a bit more here…
tempfile.read
end
This might work. You may also want to think about closing that tempfile, but the garbage collector and the OS should take care of it.
I found some solutions using post_connect_hook and pre_connect_hook, but it seems like they don't work. I'm using the latest Mechanize version (2.1). There are no [:response] fields in the new version, and I don't know where to get them in the new version.
https://gist.github.com/search?q=pre_connect_hooks
https://gist.github.com/search?q=post_connect_hooks
Is it possible to make Mechanize return a UTF8 encoded version, instead of having to convert it manually using iconv?
Since Mechanize 2.0, arguments of pre_connect_hooks() and post_connect_hooks() were changed.
See the Mechanize documentation:
pre_connect_hooks()
A list of hooks to call before retrieving a response. Hooks are called with the agent, the URI, the response, and the response body.
post_connect_hooks()
A list of hooks to call after retrieving a response. Hooks are called with the agent, the URI, the response, and the response body.
Now you can't change the internal response-body value because an argument is not array. So, the next best way is to replace an internal parser with your own:
class MyParser
def self.parse(thing, url = nil, encoding = nil, options = Nokogiri::XML::ParseOptions::DEFAULT_HTML, &block)
# insert your conversion code here. For example:
# thing = NKF.nkf("-wm0X", thing).sub(/Shift_JIS/,"utf-8") # you need to rewrite content charset if it exists.
Nokogiri::HTML::Document.parse(thing, url, encoding, options, &block)
end
end
agent = Mechanize.new
agent.html_parser = MyParser
page = agent.get('http://somewhere.com/')
...
I found a solution that works pretty well:
class HtmlParser
def self.parse(body, url, encoding)
body.encode!('UTF-8', encoding, invalid: :replace, undef: :replace, replace: '')
Nokogiri::HTML::Document.parse(body, url, 'UTF-8')
end
end
Mechanize.new.tap do |web|
web.html_parser = HtmlParser
end
No issues were found yet.
In your script, just enter: page.encoding = 'utf-8'
However, depending on your scenario, you may alternatively need to enter the reverse (the encoding of the website Mechanize is working with) instead. For that, open Firefox, open the website you want Mechanize to work with, select Tools in the menubar, and then open Page Info. Determine what the page is encoded in from there.
Using that info, you would instead enter what the page is encoded in (such as page.encoding = 'windows-1252').
How about something like this:
class Mechanize
alias_method :original_get, :get
def get *args
doc = original_get *args
doc.encoding = 'utf-8'
doc
end
end
I'm using webrick (the built-in ruby webserver) to serve .rhtml
files (html with ruby code embedded --like jsp).
It works fine, but I can't figure out how to access parameters
(e.g. http://localhost/mypage.rhtml?foo=bar)
from within the ruby code in the .rhtml file.
(Note that I'm not using the rails framework, only webrick + .rhtml files)
Thanks
According to the source code of erbhandler it runs the rhtml files this way:
Module.new.module_eval{
meta_vars = servlet_request.meta_vars
query = servlet_request.query
erb.result(binding)
}
So the binding should contain a query (which contains a hash of the query string) and a meta_vars variable (which contains a hash of the environment, like SERVER_NAME) that you can access inside the rhtml files (and the servlet_request and servlet_response might be available too, but I'm not sure about them).
If that is not the case you can also try querying the CGI parameter ENV["QUERY_STRING"] and parse it, but this should only be as a last resort (and it might only work with CGI files).
This is the solution:
(suppose the request is http://your.server.com/mypage.rhtml?foo=bar)
<html>
<body>
This is my page (mypage.rhtml, served by webrick)
<%
# embedded ruby code
servlet_request.query ["foo"] # this simply prints bar on console
%>
</body>
</html>
You don't give much details, but I imagine you have a servlet to serve the files you will process with erb, and by default the web server serves any static file in a public directory.
require 'webrick'
include WEBrick
require 'erb'
s = HTTPServer.new( :Port => 8080,:DocumentRoot => Dir::pwd + "/public" )
class MyServlet < HTTPServlet::AbstractServlet
def do_GET(req, response)
File.open('public/my.rhtml','r') do |f|
#template = ERB.new(f.read)
end
response.body = #template.result(binding)
response['Content-Type'] = "text/html"
end
end
s.mount("/my", MyServlet)
trap("INT"){
s.shutdown
}
s.start
This example is limited, when you go to /my always the same file is processed. Here you should construct the file path based on the request path. Here I said a important word: "request", everything you need is there.
To get the HTTP header parameters, use req[header_name]. To get the parameters in the query string, use req.query[param_name]. req is the HTTPRequest object passed to the servlet.
Once you have the parameter you need, you have to bind it to the template. In the example we pass the binding object from self (binding is defined in Kernel, and it represents the context where code is executing), so every local variable defined in the do_GET method would be available in the template. However, you can create your own binding for example passing a Proc object and pass it to the ERB processor when calling 'result'.
Everything together, your solution would look like:
def do_GET(req, response)
File.open('public/my.rhtml','r') do |f|
#template = ERB.new(f.read)
end
foo = req.query["foo"]
response.body = #template.result(binding)
response['Content-Type'] = "text/html"
end
Browsing the documentation, it looks like you should have an HTTPRequest from which you can get the query string. You can then use parse_query to get a name/value hash.
Alternatively, it's possible that just calling query() will give you the hash directly... my Ruby-fu isn't quite up to it, but you might want to at least give it a try.