I need to post using three parameters and a body which consists of 512 bytes. I can get the body right but I can't seem to get the parameters to take:
require 'net/http'
#ip_address = Array['cueserver.dnsalias.com']
#cueserver = 0
#playback = 'p1'
def send_cuescript(data)
params = {'id' => '1', 'type' => "20",'dst' => 'RES' }
begin
url = URI.parse('http://'+ #ip_address[#cueserver] + '/set.cgi')
http = Net::HTTP.new(url.host, url.port)
response, body = http.post(url.path, params, data)
rescue Timeout::Error, Errno::EINVAL, Errno::ECONNRESET, EOFError,
Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError => e
end
response_array = []
puts 'got this value: ' + response.to_s
response.body.each_byte { |e| response_array.push(e.to_s(16))}
end
data_array = Array.new(512, "\x80")
send_cuescript(data_array.join)
I am getting an error from the initialize_http_header. I know there must be a way to set the parameters and the body separately but I can't seem to find any reference to this.
Why do you have to send part of the params in the url and part of it in the body?
If you have to do this, try
url = URI.parse('http://'+ #ip_address[#cueserver] + '/set.cgi?' + params.to_param)
PS: to_param is from active support. You need to write your own if you are not using active support.
Related
I'm using faraday with net-http-persistent adapter to make HTTP requests.
I want to optimise my requests by making them execute asynchronously but as I need a persistent connection I keep getting errors such as too many connections reset which I assume is due to the fact that I have multiple threads creating new connections.
I tried changing my adapter to typhoeus but as the connection is not persistent the final result of executing all request is not as expected.
My goal is to add items to a basket by making this HTTP requests. Without the persistent connection items are not added to the basket.
So, my question is:
Is it possible to make persistent HTTP requests reusing the connection between threads? If so, how can this be achieved?
Here is a piece of my code:
Create the connection:
Faraday.new do |c|
c.use :cookie_jar, jar: cookie_jar
c.options.open_timeout = 5
c.options.timeout = 10
c.request :url_encoded
c.response :logger, logger
c.adapter :net_http_persistent do |http| # yields Net::HTTP::Persistent
http.idle_timeout = 2
end
end
Creating threads and getting the result of each one of them
result = []
threads = []
total_items = items.size
items.each_slice(5) do |sliced_items|
# Create a thread for a batch of 5 items and store its result
threads << Thread.new do
Thread.current[:output] = browser.add_all_items(sliced_items)
end
end
# Wait for all threads to finish their work and store their output into result
threads.each do |t|
t.join
result << t[:output]
end
add_all_items and add_to_cart methods:
# Add a batch of items by the key passed (id, gtin, url)
def add_all_items(items_info, key)
results = []
items_info.each do |item|
begin
add_to_cart(item[key], item[:quantity])
item[:message] = nil
rescue => e
item[:message] = e.message
puts "---------- BACKTRACE -------------- \n #{e.backtrace}"
end
puts "\n--------- MESSAGE = #{item[:message]} --------- \n"
results << item
puts "-------- RESULTS #{results}"
end
results
end
def add_to_cart(url, count = 1)
response = connection.get(url) do |req|
req.headers["User-Agent"] = #user_agent
end
doc = Nokogiri::HTML(response.body)
stoken = doc.search('form.js-oxProductForm input[name=stoken]').attr('value').value
empty_json = '""'
product_id = get_item_id(url)
data = { #removed payload for security reasons }
# Using example.com for question purposes
response = connection.post('https://www.example.com/index.php?') do |req|
req.headers["Origin"] = "https://www.example.com"
req.headers["Content-Type"] = "application/x-www-form-urlencoded; charset=UTF-8"
req.headers["Accept"] = "application/json, text/javascript, */*; q=0.01"
req.headers["Referer"] = url
req.headers["Pragma"] = "no-cache"
req.headers["Accept-Language"] = "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
req.headers["User-Agent"] = #user_agent
req.headers["Cache-Control"] = "no-cache"
req.headers["Connection"] = "keep-alive"
req.headers["DNT"] ="1"
req.headers["Content-Length"] = data.size.to_s
req.headers["Accept"] = "*/*"
req.headers["X-Requested-With"] = "XMLHttpRequest"
req.headers["Connection"] = "keep-alive"
req.body = data
end
begin
json = JSON.parse(response.body)
raise "Could not add item: #{json['message']}" if json['success'] != 1 || json['item'] != product_id
rescue JSON::ParserError => e
puts "JSON Error"
end
end
def get_item_id(url)
response = connection.get(url) do |req|
req.headers["User-Agent"] = #user_agent
end
doc = Nokogiri::HTML(response.body)
doc.search('.js-oxProductForm input[name=aid]').attr('value').value
end
Thanks in advance.
With Ruby, my app:
checks if the page status is 200
Parses the PDF files if so
sends via email the result of scraping
Having tested all the parts of the code, everything works fine, except one thing, the mail that is sent doesn't contain the result of my scrpaing;
What is the issue, is it related to the variable #monscrape that may be not recongnised in the final party of the code ?
My code:
require 'open-uri'
require "net/http"
require 'rubygems'
require 'pdf/reader'
require 'mail'
options = { :address => "smtp.gmail.com",
:port => 587,
:domain => 'gmail.com',
:user_name => 'mail#gmail.com',
:password => 'pwd',
:authentication => 'plain',
:enable_starttls_auto => true
}
lien= "http://www.example.com"
url = URI.parse(lien)
req = Net::HTTP.new(url.host, url.port)
res = req.request_head(url.path)
if res.code == "200"
io = open('http://www.example.com')
reader = PDF::Reader.new(io)
reader.pages.each do |page|
res = page.text
#monscrape = res.scan(/text[\s\S]*text/)
end
Mail.defaults do
delivery_method :smtp, options
end
Mail.deliver do
to 'mail#hotmail.com'
from 'Author <mail#gmail.com>'
subject 'testing sendmail'
html_part do
content_type 'text/html; charset=UTF-8'
body '<h1>Please find below the scrape <%= #monscrape %></h1>'
end
end
else
puts "the link doenst work"
end
The problem is the Mail.deliver block is evaluated using instance_eval. Therefore no local instance #variables will be visible to the Mail block.
So #monscrape will always be nil inside the Mail.deliver block.
One solution is to use a local (non-instance) variable instead:
monscrape = "test"
Mail.deliver do
...
body "<h1>Please find below the scrape #{monscrape}</h1>"
...
end
Also note that Mail does not support ERB(!) therefore you cannot use something like <%= monscrape %> in the body. You have to treat it like a normal string using string expansion with double quotes " and not single quotes '.
See further discussion and options here:
Why can't the Mail block see my variable?
You can't use
res = req.request_head(url.path)
when url.path returns "". request_head expects a path of at least "/". That implies you need to fix up the URL being passed so it at least has the root path "/".
url = URI.parse('http://www.example.com')
url.path # => ""
req.request_head(url.path)
*** ArgumentError Exception: HTTP request path is empty
vs.
url = URI.parse('http://www.example.com/')
url.path # => "/"
req.request_head(url.path)
#<Net::HTTPOK 200 OK readbody=true>
The second problem is you're trying to read something as PDF that isn't a PDF file. Example.com returns HTML, which is text. You can't use:
io = open('http://www.example.com')
reader = PDF::Reader.new(io)
Trying to returns "PDF does not contain EOF marker".
It's really important that you understand what types of objects/resources are being returned by a site when you request a URL. You can't declare them willy-nilly and expect code to accept it without errors.
I'm having trouble getting parameters passed in an HTTP Put call, using ruby. Take a look at the "put_data" variable.
When I leave it as a hash, ruby says:
undefined method `bytesize' for #<Hash:0x007fbf41a109e8>
if I convert to a string, I get:
can't convert Net::HTTPUnauthorized into String
I've also tried doing just - '?token=wBsB16NSrfVDpZPoEpM'
def process_activation
uri = URI("http://localhost:3000/api/v1/activation/" + self.member_card_num)
Net::HTTP.start(uri.host, uri.port) do |http|
headers = {'Content-Type' => 'text/plain; charset=utf-8'}
put_data = {:token => "wBsB16NSrfVDpZPoEpM"}
response = http.send_request('PUT', uri.request_uri, put_data, headers)
result = JSON.parse(response)
end
if result['card']['state']['state'] == "active"
return true
else
return false
end
end
I've searched all around, including rubydocs, but can't find an example of how to encode parameters. Any help would be appreciated.
Don't waste your time with NET::HTTP. I used 'rest-client' and had this thing done in minutes...
def process_activation
response = RestClient.put 'http://localhost:3000/api/v1/card_activation/'+ self.member_card_num, :token => "wBsB1pjJNNfiK6NSrfVDpZPoEpM"
result = JSON.parse(response)
return result['card']['state']['state'] == "active"
end
I have a little sinatra api that I'm trying to beautify. Most of my routes are simple db operations but a few involve calling an external service before doing db operations. In all cases most of the code is the same except for how I respond to the service response. Is there any slick functional programming approach?
Here's an example of one of these routes:
get '/update_x' do
validateParams(params,:x)
xid = params[:x]
xName = getNameFromId(xid)
if xName
# Make request to proxy service
rid = generateRandomHexNumber(16) # generate requestId
params['m'] = 'set'
params['rid'] = rid
json = "{}"
begin
response = #resource["/"+"?rid=#{rid}&id=#{xid}&json=#{json}"].get
status = response.code
body = response.body
parsed_json = JSON(body)
if parsed_json['response'] and parsed_json['response']['success'] and parsed_json['response']['success']=='false'
msg = {:success => "false", :response => "unknown error"}
if parsed_json['response']['response']
msg = {:success => "false", :response => parsed_json['response']['response']}
end
content_type :json
msg.to_json
else
#### Here is stuff specific to this api call
updateDBHelper(xid,buildUpdateOptions(params))
params['ss_status'] = status
content_type :json
params.to_json
#### End specific to api call
end
rescue Exception=>e
params['ss_status'] = status
params['exception'] = e
content_type :json
params.to_json
end
else
msg = {:success => "false", :response => "Not found"}
content_type :json
msg.to_json
end
end
In general if you have a general pattern with some arbitrary code that changes every time then the simplest thing would be accepting a block with those customizations.
def make_api_request(some, params)
# do what you need to do
yield(variables, that, your_custom_code, needs)
# do some more, maybe cleanup
end
get '/some_route' do
make_api_request do |variables, that, your_custom_code, needs|
# do custom stuff here
end
end
How can I send HTTP GET request with parameters via ruby?
I have tried a lot of examples but all of those failed.
I know this post is old but for the sake of those brought here by google, there is an easier way to encode your parameters in a URL safe manner. I'm not sure why I haven't seen this elsewhere as the method is documented on the Net::HTTP page. I have seen the method described by Arsen7 as the accepted answer on several other questions also.
Mentioned in the Net::HTTP documentation is URI.encode_www_form(params):
# Lets say we have a path and params that look like this:
path = "/search"
params = {q: => "answer"}
# Example 1: Replacing the #path_with_params method from Arsen7
def path_with_params(path, params)
encoded_params = URI.encode_www_form(params)
[path, encoded_params].join("?")
end
# Example 2: A shortcut for the entire example by Arsen7
uri = URI.parse("http://localhost.com" + path)
uri.query = URI.encode_www_form(params)
response = Net::HTTP.get_response(uri)
Which example you choose is very much dependent on your use case. In my current project I am using a method similar to the one recommended by Arsen7 along with the simpler #path_with_params method and without the block format.
# Simplified example implementation without response
# decoding or error handling.
require "net/http"
require "uri"
class Connection
VERB_MAP = {
:get => Net::HTTP::Get,
:post => Net::HTTP::Post,
:put => Net::HTTP::Put,
:delete => Net::HTTP::Delete
}
API_ENDPOINT = "http://dev.random.com"
attr_reader :http
def initialize(endpoint = API_ENDPOINT)
uri = URI.parse(endpoint)
#http = Net::HTTP.new(uri.host, uri.port)
end
def request(method, path, params)
case method
when :get
full_path = path_with_params(path, params)
request = VERB_MAP[method].new(full_path)
else
request = VERB_MAP[method].new(path)
request.set_form_data(params)
end
http.request(request)
end
private
def path_with_params(path, params)
encoded_params = URI.encode_www_form(params)
[path, encoded_params].join("?")
end
end
con = Connection.new
con.request(:post, "/account", {:email => "test#test.com"})
=> #<Net::HTTPCreated 201 Created readbody=true>
I assume that you understand the examples on the Net::HTTP documentation page but you do not know how to pass parameters to the GET request.
You just append the parameters to the requested address, in exactly the same way you type such address in the browser:
require 'net/http'
res = Net::HTTP.start('localhost', 3000) do |http|
http.get('/users?id=1')
end
puts res.body
If you need some generic way to build the parameters string from a hash, you may create a helper like this:
require 'cgi'
def path_with_params(page, params)
return page if params.empty?
page + "?" + params.map {|k,v| CGI.escape(k.to_s)+'='+CGI.escape(v.to_s) }.join("&")
end
path_with_params("/users", :id => 1, :name => "John&Sons")
# => "/users?name=John%26Sons&id=1"