Ruby Eventmachine & HTTP requests via proxy - ruby

I'm coding on a high scalable web harvester running on top of Eventmachine. All works fine and fast. Recently I'm trying to fire the requests through a bunch of proxies which also works fine, e.g.:
EventMachine.run do
connect_opts = { :proxy => { :host => '11.12.13.14', :port => 3128 } }
request_opts = { :proxy => { :authorization => ['jdoe', 'mysecretpass'] } }
req = EventMachine::HttpRequest.new('http://www.example.com/', connect_opts).get request_opts
req.callback { }
end
I'm iterating over hundreds of proxies and firing several hundred requests per second. What I need now is to know which proxy was used for which request to store this metadata in a db. How would you fetch this information from the req object?
I hoped there's is some method like req.proxy_foo (equivalent to req.response, req.response_header etc.) to get this information but didn't find any appropriate way yet.

Related

Simple POST request failing with em-http-request

The following query works with requestmaker:
URI:
http://www.cleverbot.com/webservicemin/
Query:
start=y&icognoid=wsf&fno=0&sub=Say&islearning=1&cleanslate=false&stimulus=!!!%20there%20was%20an%20error%20!!!&icognocheck=af71393ce00d9126a247df2f53948e79
But it does not work with em-http-request:
require 'eventmachine'
require 'em-http-request'
uri = 'http://www.cleverbot.com/webservicemin/'
query = 'start=y&icognoid=wsf&fno=0&sub=Say&islearning=1&cleanslate=false&stimulus=!!!%20there%20was%20an%20error%20!!!&icognocheck=af71393ce00d9126a247df2f53948e79'
EM.run do
http = EM::HttpRequest.new(uri).post(query: query)
http.callback { puts http.response; EM.stop }
http.errback { puts 'There was an error'; EM.stop }
end
which prints There was an error. I feel stumped because this simple example works with any other method of sending a request and I've checked around to see if my usage was wrong but it doesn't seem to be.
Edit: Just for reference, this is not the correct way to use cleverbot. I made a second mistake by sending the data under :query. If you use http.post(body: query) it will work
Looks like a badly implemented server: it aborts the TCP connection without returning a proper HTTP status code, which is why you see "connection closed by server" when you query http.error.
If you change the default user agent to curl's UA string, you get a response:
http = EM::HttpRequest.new(uri).post({
:query => query,
:head => {'User-Agent' => 'curl/7.30.0'}
})

How can I upload files to Redmine via ActiveResource / REST API?

I am trying to batch-upload images to Redmine and link them each to a certain wiki pages.
The docs (Rest_api, Using the REST API with Ruby) mention some aspects, but the examples fail in various ways. I also tried to derive ideas from the source - without success.
Can anyone provide a short example that shows how to upload and link an image from within Ruby?
This is a bit tricky as both attachments and wiki APIs are relatively new, but I have done something similar in the past. Here is a minimal working example using rest-client:
require 'rest_client'
require 'json'
key = '5daf2e447336bad7ed3993a6ebde8310ffa263bf'
upload_url = "http://localhost:3000/uploads.json?key=#{key}"
wiki_url = "http://localhost:3000/projects/some_project/wiki/some_wiki.json?key=#{key}"
img = File.new('/some/image.png')
# First we upload the image to get attachment token
response = RestClient.post(upload_url, img, {
:multipart => true,
:content_type => 'application/octet-stream'
})
token = JSON.parse(response)['upload']['token']
# Redmine will throw validation errors if you do not
# send a wiki content when attaching the image. So
# we just get the current content and send that
wiki_text = JSON.parse(RestClient.get(wiki_url))['wiki_page']['text']
response = RestClient.put(wiki_url, {
:attachments => {
:attachment1 => { # the hash key gets thrown away - name doesn't matter
:token => token,
:filename => 'image.png',
:description => 'Awesome!' # optional
}
},
:wiki_page => {
:text => wiki_text # original wiki text
}
})

Post request via Net::HTTP doesn't get a response and times out

I want to use the Microsoft Translate API and I am already stuck obtaining the access token.
This is how my code looks like:
authUri = "https://datamarket.accesscontrol.windows.net/v2/OAuth2-13/"
paramHash = {
"client_id" => "test",
"client_secret" => "*****",
"scope" => "http://api.microsofttranslator.com",
"grant_type" => "client_credentials"
}
postData = Net::HTTP.post_form(URI.parse(authUri), paramHash)
puts postData.body
But this won't terminate and after a while I get a Timeout Error:
`rescue in rbuf_fill': Timeout::Error (Timeout::Error)
I already tried the a Rest-Client gem and curl. Both worked perfectly fine, but I don't like incorporating another gem for such a simple task.
When I send the post request to localhost and listen with netcat I see the exact same requests (except for User-Agent, but even changing that didn't solve the problem)

Rails pagination in API using Her, Faraday

I've been trying to figure this out all day, and it's driving me crazy.
I have two rails apps, ServerApp and ClientApp. ClientApp gets data from ServerApp through an API, using the Her gem. Everything was great until I needed pagination information.
This is the method I am using to get the orders (this uses kamainari for pagination and ransack for search):
# ServerApp
def search
#search = Order.includes(:documents, :client).order('id desc').search(params[:q])
#orders = #search.result(distinct: true).page(params[:page]).per(params[:per])
respond_with #orders.as_json(include: :documents)
end
It returns an array of hashes in json, which Her uses as a collection of orders. That works fine.
# Response
[
{
"client_id": 239,
"created_at": "2013-05-15T15:37:03-07:00",
"id": 2422,
"ordered_at": "2013-05-15T15:37:03-07:00",
"origin": "online",
"updated_at": "2013-05-15T15:37:03-07:00",
"documents": [
{ ... }
]
},
...
]
But I needed pagination information. It looked like I needed to send it as metadata with my json. So I change my response to this:
respond_to do |format|
format.json do
render json: { orders: #orders.as_json(include: :documents), metadata: 'sent' }
end
end
This does indeed send over metadata, so in my ClientApp I can write #orders.metadata and get 'sent'. But now my orders are nested in an array inside of 'orders', so I need to use #orders.orders, and then it treats it like an array instead of a Her collection.
After doing some reading, it seemed sending pagination info through headers was the way a lot of other people did this (I was able to get the headers set up in an after_filter using this guide). But I am even more lost on how to get those response headers in my ClientApp - I believe I need a Faraday Middleware but I just am having no luck getting this to work.
If anyone knows how I can just get this done, I would be very grateful. I can't take another day of banging my head against the wall on this, but I feel like I am just one vital piece of info away from solving this!
I encountered the same issue and solved it by adding my own middleware and rewriting the "parse" and "on_complete" methods without that much hassle and avoiding the use of global variables.
Here's the code:
class CustomParserMiddleware < Her::Middleware::DefaultParseJSON
def parse(env)
json = parse_json(env[:body])
pagination = parse_json(env[:response_headers][:pagination_key]) || {}
errors = json.delete(:errors) || {}
metadata = json.delete(:metadata) || {}
{
:data => json,
:errors => errors,
:metadata => {
:pagination => pagination,
:additional_metadata => metadata
},
end
def on_complete(env)
env[:body] = case env[:status]
when 204
parse('{}')
else
parse(env)
end
end
end
then, you can access the pagination as follows:
model = Model.all
model.metadata[:pagination]
I finally got this working. The trick was to use a global variable in the faraday on_complete - I tried to find a better solution but this was the best I could do. Once again, I got the header code from here. Here's the full guide to how to get pagination working with Her:
First, on my server side, I have the Kaminari gem, and I pass page and per as params to the server from the client. (This is also using ransack for searching)
def search
#search = Order.order('id desc').search(params[:q])
#orders = #search.result(distinct: true).page(params[:page]).per(params[:per])
respond_with #orders.as_json(include: :items)
end
My client makes the request like so:
#orders = Order.search(q: { client_id_eq: #current_user.id }, page: params[:page], per: 3)`
Back on the server, I have this in my ApiController (app controller for api):
protected
def self.set_pagination_headers(name, options = {})
after_filter(options) do |controller|
results = instance_variable_get("##{name}")
headers["X-Pagination"] = {
total_count: results.total_count,
offset_value: results.offset_value
}.to_json
end
end
In the server orders_controller.rb, I set the pagination headers for the search method:
class OrdersController < ApiController
set_pagination_headers :orders, only: [:search]
...
end
Now to receive the headers we need a Faraday middleware in Her on the client.
# config/initializers/her.rb
Her::API.setup url: Constants.api.url do |c|
c.use TokenAuthentication
c.use HeaderParser # <= This is my middleware for headers
c.use Faraday::Request::UrlEncoded
c.use Her::Middleware::DefaultParseJSON
c.use Faraday::Adapter::NetHttp
c.use Faraday::Response::RaiseError
end
# lib/header_parser.rb
# don't forget to load this file in application.rb with something like:
# config.autoload_paths += Dir[File.join(Rails.root, "lib", "*.rb")].each { |l| require l }
class HeaderParser < Faraday::Response::Middleware
def on_complete(env)
unless env[:response_headers]['x-pagination'].nil?
# Set the global var for pagination
$pagination = JSON.parse(env[:response_headers]['x-pagination'], symbolize_names: true)
end
end
end
Now back in your client controller, you have a global variable of hash called $pagination; mine looks like this:
$pagintation = { total_count: 0, offset_value: 0 }`
Finally, I added Kaminari gem to my client app to paginate the array and get those easy pagination links:
#orders = Kaminari.paginate_array(#orders, total_count: $pagination[:total_count]).page(params[:page]).per(params[:per_page])`
I hope this can help someone else, and if anyone knows a better way to do this, let me know!
You can pass header options to Faraday when setting up the connection, see the docs at http://rubydoc.info/gems/faraday/0.8.7/Faraday/Connection:initialize
Sometimes it helps to do a curl request first, esp. use -vv option for verbose output where you will see all headers. (Maybe you can attach some log outputs from the Server too)
You can use e.g. clogger (http://clogger.rubyforge.org/) do monitor header information on the Rails server side

Parse WSDL file with SOAP4R

Is there any example of WSDL Parser using SOAP4R? I'm trying to list all operations of WSDL file but I can't figure it out :( Can you post me some tutorial?
Thx
Maybe that isn't answer you want, but I recommend you switch to Savon. For example, your task looks like this snippet (this example taken from github's savon page):
require "savon"
# create a client for your SOAP service
client = Savon::Client.new("http://service.example.com?wsdl")
client.wsdl.soap_actions
# => [:create_user, :get_user, :get_all_users]
# execute a SOAP request to call the "getUser" action
response = client.request(:get_user) do
soap.body = { :id => 1 }
end
response.body
# => { :get_user_response => { :first_name => "The", :last_name => "Hoff" } }

Resources