error!': 301 "Moved Permanently" (Net::HTTPRetriableError) - ruby

I am trying to use a code that I used to use with another Mac.
now when I run it with a new Mac (2018) I get the following error
This is the code
require 'net/http'
base = 'www.uniprot.org'
tool = 'uploadlists'
params = {
'from' => 'ACC', 'to' => 'P_REFSEQ_AC', 'format' => 'tab',
'query' => 'P13368 P20806 Q9UM73 P97793 Q17192'
}
http = Net::HTTP.new base
$stderr.puts "Submitting...\n";
response = http.request_post '/' + tool + '/',
params.keys.map {|key| key + '=' + params[key]}.join('&')
loc = nil
while response.code == '302'
loc = response['Location']
response = http.request_get loc
end
while loc
wait = response['Retry-After'] or break
$stderr.puts "Waiting (#{wait})...\n";
sleep wait.to_i
response = http.request_get loc
end
response.value # raises http error if not 2xx
puts response.body
and this is the error I get
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/net/http/response.rb:120:in `error!': 301 "Moved Permanently" (Net::HTTPRetriableError)
from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/net/http/response.rb:129:in `value'
from conver.rb:28:in `<main>'

You're receiving an HTTP 301 Moved Permanently response code. When you look closer you can see it's pointing you to https://www.uniprot.org:443/uploadlists/ for the new location; this is what typically happens when a resource that was previously reachable via HTTP is now only reachable via HTTPS. It doesn't have anything to do with using a new computer; it's just coincidental that it happened around the same time.
If you change the URL to HTTPS it should work as you expect. That said, I never encourage the use of Net::HTTP directly because it's clunky. Take a look at how awkward it is just to make a connection using HTTPS! It's not worth the headache.
I prefer to use HTTParty because it's straightforward and easy to use, as well as being very popular in the Ruby community. Here's an example of how to accomplish your task with HTTParty in fewer lines of code:
require 'httparty'
params = {
'from' => 'ACC', 'to' => 'P_REFSEQ_AC', 'format' => 'tab',
'query' => 'P13368 P20806 Q9UM73 P97793 Q17192'
}
response = HTTParty.post(
'https://www.uniprot.org:443/uploadlists/',
{
body: params.keys.map { |key| key + '=' + params[key] }.join('&'),
headers: { 'Content-Type' => 'application/x-www-form-urlencoded' }
}
)
Then you can inspect the response body:
puts response.body
From To
P13368 NP_511114.2
Q9UM73 NP_004295.2
P97793 NP_031465.2
Q17192 XP_004934106.1
Additionally, I can tell from the stack trace that you posted that you're using the system version of Ruby that came with macOS. My advice is: Don't use system Ruby.
Instead, you should install a Ruby manager like RVM:
Install RVM with \curl -sSL https://get.rvm.io | bash -s stable
Reload your shell
Install Ruby with rvm install 2.5.3
Reinstall your gems (gem install httparty)
Then you can re-run your application.

Related

enforcing Faraday adapter :typhoeus to use HTTP/2 for requests

How to enforce Faraday adapter typhoeus to use HTTP/2 for requests to servers which supported HTTP/2?
I have tested this over service https://http2.pro/doc/api and result was like this:
body="{\"http2\":1,\"protocol\":\"HTTP\\/2.0\",\"push\":0,\"user_agent\":\"Faraday v0.12.2\"}",
\"http2\":1, what means that HTTP/2 not used for request!
There are two things at play here. The first is that the remote API is lying to you in the response body. Their documentation says:
http2: Possible values are 0 (HTTP/2 was used) and 1 (HTTP/2 was not used).
Even though the response body shows 'http2': 1 indicating that HTTP2 was not used, it is being used. You can most easily confirm this using Chrome's dev tools:
So once we know that the API is lying in the response body, how can we independently confirm that Typhoeus is using HTTP2?
(this answer assumes you are using pry as your REPL, not IRB)
First let's confirm that Typhoeus alone will use HTTP2:
require 'typhoeus'
response = Typhoeus.get("https://http2.pro/api/v1", http_version: :httpv2_0)
response.class
=> Typhoeus::Response < Object
response.body
=> "{\"http2\":1,\"protocol\":\"HTTP\\/2.0\",\"push\":0,\"user_agent\":\"Typhoeus - https:\\/\\/github.com\\/typhoeus\\/typhoeus\"}" # this is the lying API response
response.http_version
=> "2" # this is what Typhoeus tells us was actually used
Now let's test it in Faraday:
require 'faraday'
require 'typhoeus'
require 'typhoeus/adapters/faraday'
conn = Faraday.new do |faraday|
faraday.adapter :typhoeus, http_version: :httpv2_0
end
response = conn.get("https://http2.pro/api/v1")
response.body
=> "{\"http2\":1,\"protocol\":\"HTTP\\/2.0\",\"push\":0,\"user_agent\":\"Faraday v0.17.0\"}" # again we get the lying API response
But how can we confirm it was HTTP2? This doesn't work:
response.http_version
NoMethodError: undefined method `http_version' for #<Faraday::Response:0x00007f99935519a8>
Because response isn't a Typhoeus::Response object, it's a Faraday object:
response.class
=> Faraday::Response < Object
So we need to get into the gem itself to figure out where it's creating the Typhoeus::Response object so we can call .http_version on it manually and confirm it's using the protocol we expect. As it turns out, that's right here.
Let's take the easy route and stick binding.pry into our local copy of the gem (you'll need to restart pry to pick up the changes to the gem):
def typhoeus_request(env)
opts = {
:method => env[:method],
:body => env[:body],
:headers => env[:request_headers]
}.merge(#adapter_options)
binding.pry
::Typhoeus::Request.new(env[:url].to_s, opts)
end
Then re-run the request:
require 'faraday'
require 'typhoeus'
require 'typhoeus/adapters/faraday'
conn = Faraday.new do |faraday|
faraday.adapter :typhoeus, http_version: :httpv2_0
end
response = conn.get("https://http2.pro/api/v1")
And you'll see:
Frame number: 0/3
From: /Users/foo/.rvm/gems/ruby-2.6.3/gems/typhoeus-1.3.1/lib/typhoeus/adapters/faraday.rb # line 127 Faraday::Adapter::Typhoeus#typhoeus_request:
120: def typhoeus_request(env)
121: opts = {
122: :method => env[:method],
123: :body => env[:body],
124: :headers => env[:request_headers]
125: }.merge(#adapter_options)
126: binding.pry
=> 127: ::Typhoeus::Request.new(env[:url].to_s, opts)
128: end
Now enter:
response = ::Typhoeus::Request.new(env[:url].to_s, opts).run
And confirm it's a Typhoeus::Response object:
response.class
=> Typhoeus::Response < Object
And confirm it's using HTTP2:
response.http_version
=> "2"
And confirm the API response body is a dirty liar:
response.body
=> "{\"http2\":1,\"protocol\":\"HTTP\\/2.0\",\"push\":0,\"user_agent\":\"Faraday v0.17.0\"}"
And that's how you use Typhoeus as a Faraday adapter to make an HTTP2 request.

Open-Uri Alternative - Getting a response from a website

The following code works in Ruby 1.9.3p-551
require "open-uri"
res = open("http://example.com/version").read
p res => {"buildNumber": 2496, "buildDate": "2015-09-29 11:18:02 +0200", "timestamp": 1443639212 }
In any Ruby version higher than 1.9.3 I get the following error;
from /Users/imac/.rbenv/versions/2.1.0/lib/ruby/2.1.0/net/http/response.rb:357:in `finish': incorrect header check (Zlib::DataError)
I need to use a higher version as this will be used in a Rails 4 app.
Any ideas for alternatives?
Turns out the gzip encoding is not accepted by default. Or at least that's what I'm guessing.
The following works.
res = open("http://someurl.com/version", "Accept-Encoding" => "plain").read
Interesting how this changed from Ruby 2.0.0+
Another neat solution for this.
require 'rest-client'
url = "http://example.com/version"
def get_response(url)
begin
return RestClient.get(url, {:accept => :json})
rescue RestClient::GatewayTimeout
"GatewayTimeout"
rescue RestClient::RequestTimeout
"RequestTimeout"
rescue SocketError
"SocketError"
end
end
p get_response(url)
# => "{\"buildNumber\": 2535, \"buildDate\": \"2015-09-30 17:41:42 +0200\", \"timestamp\": 1444085042 }"

Why Am I getting a NoMethodError when using the google API to run a query against google Big Query?

I'm trying to run a Query against Google Big Query, using the Ruby API.
This is my first project with Ruby and I'm still learning the language.
This is also my first project using the Google API.
ENVIORNMENT:
Windows 7
Ruby 1.9
Faraday 0.90
Goolge API - Service Account Authentication
My Code runs without giving any warnings or error messages through:
#client.authorization.fetch_access_token!
doc = File.read('bigQueryAPI.json')
#bigQuery = #client.register_discovery_document('bigquery', 'v2', doc)
NOTE: #bigQuery is loaded from a file because when I try to load #bigquery with
#bigquery = #client.discovered_api('bigquery', 'v2')
I get Google::APIClient::ClientError: Not Found and inspect only prints
#<Google::APIClient::API:0x17c94cc ID:bigquery:v2>
However If I save the Big Query API as a text file from
https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest
and then load it as a text file with
doc = File.read('bigQueryAPI.json')
#bigQuery = #client.register_discovery_document('bigquery', 'v2', doc)
then #bigQuery.inspect actually returns something useful.
#bigQuery.inspect output.
However, When I try to actually run a query, like so:
result = #client.execute!(
:api_method => #bigQuery.batch_path.query,
:body_object => { "query" => "SELECT count(DISTINCT repository_name) as repository_total, " +
"count(payload_commit) as commits_total, " +
"count(DISTINCT repository_name) / count(payload_commit) as average, " +
"FROM [githubarchive:github.timeline]" }, #,
:parameters => { "projectId" => #project_id })
I get the following error:
NoMethodError: undefined method `query_values' for nil:NilClass
Here's the Full Stacktrace of the error:
1) Error:
test_averages(Test_GitHub_Archive):
NoMethodError: undefined method `query_values' for nil:NilClass
C:/Ruby193/lib/ruby/gems/1.9.1/gems/google-api-client-0.7.1/lib/google/api_client/request.rb:145:in `uri='
C:/Ruby193/lib/ruby/gems/1.9.1/gems/google-api-client-0.7.1/lib/google/api_client/request.rb:101:in `initialize'
C:/Ruby193/lib/ruby/gems/1.9.1/gems/google-api-client-0.7.1/lib/google/api_client.rb:518:in `new'
C:/Ruby193/lib/ruby/gems/1.9.1/gems/google-api-client-0.7.1/lib/google/api_client.rb:518:in `generate_request'
C:/Ruby193/lib/ruby/gems/1.9.1/gems/google-api-client-0.7.1/lib/google/api_client.rb:583:in `execute!'
C:/Users/tfburton/Documents/private/ProjectSuggestor/RubyStats/GitHub_Archive.rb:39:in `get_averages'
C:/Users/tfburton/Documents/private/ProjectSuggestor/RubyStats/TestSpec/test_GitHub_Archive.rb:26:in `test_averages'
Here is the results for #client.inspect
NOTE: I would have pasted here, but my post ended over the length limit.
After doing some digging. It looks like I'm not passing the proper #bigQuery prameter to get the query function.
Looking at the dump for #bigQuery.inspect I need to pass the method at line 751.
However I can't seem to figure out how to pass that method.
If you strip out the rest of the inspect output the "path" looks like this:
{ "resources => { "jobs" => { "methods" => { "query"
I've tried #bigQuery.Jobs.query and that results in an error stating that #bigQuery.Jobs doesn't exist.
So am I creating #bigQuery correctly?
Why doesn't #bigQuery.Jobs.query work?
Here's how I got it to work with the bigquery.jobs.query method, which is probably what you need.
I had to set OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE because otherwise the authorization process would fail miserably, but that might be specific to min Win7/MgitSys environment. In any case, this specific line is not safe in prod.
require 'google/api_client'
require 'google/api_client/client_secrets'
require 'google/api_client/auth/installed_app'
require 'google/api_client/auth/file_storage'
require 'openssl'
require 'json'
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
# Initialize the client.
client = Google::APIClient.new(
:application_name => 'Example Ruby application',
:application_version => '1.0.0'
)
CREDENTIAL_STORE_FILE = "#{$0}-oauth2.json"
file_storage = Google::APIClient::FileStorage.new(CREDENTIAL_STORE_FILE)
# Initialize Google+ API. Note this will make a request to the
# discovery service every time, so be sure to use serialization
# in your production code. Check the samples for more details.
#bigQuery = client.discovered_api('bigquery', 'v2')
# Load client secrets from your client_secrets.json.
client_secrets = Google::APIClient::ClientSecrets.load
file_storage = Google::APIClient::FileStorage.new(CREDENTIAL_STORE_FILE)
if file_storage.authorization.nil?
client_secrets = Google::APIClient::ClientSecrets.load
# The InstalledAppFlow is a helper class to handle the OAuth 2.0 installed
# application flow, which ties in with FileStorage to store credentials
# between runs.
flow = Google::APIClient::InstalledAppFlow.new(
:client_id => client_secrets.client_id,
:client_secret => client_secrets.client_secret,
:scope => ['https://www.googleapis.com/auth/cloud-platform','https://www.googleapis.com/auth/bigquery']
)
client.authorization = flow.authorize(file_storage)
else
client.authorization = file_storage.authorization
end
puts "authorized, requesting"
# Make an API call.
result = client.execute!(
:api_method => #bigQuery.jobs.query,
:body_object => { "query" => "SELECT count(DISTINCT repository_name) as repository_total, " +
"count(payload_commit) as commits_total, " +
"count(DISTINCT repository_name) / count(payload_commit) as average, " +
"FROM [githubarchive:github.timeline]" }, #,
:parameters => { "projectId" => "845227657643" })
puts JSON.dump result.data

Ruby POST request with cookies?

I have a Ruby script that sends a POST request with a cookie using:
curl.exe -H "Cookie: SomeCookie=#{cookie}" -d "SomaData=#{data}" http://somesite.com/post
I tried to rewrite this into native Ruby using Net::HTTP, but this code doesn't work:
Net::HTTP.post_form(URI('http://somesite.com/post'),
{'SomeData' => '#{data}',
'Cookie' => 'SomeCookie=#{cookie}'} )
How do I solve this problem?
I'am using MRI Ruby 1.9.3 on Windows 7.
Why not look into using Curb? It's a Ruby interface to libcurl, and has an interface that's closer to cURL than Net::HTTP.
This is from the documentation:
http = Curl.get("http://www.google.com/")
puts http.body_str
http = Curl.post("http://www.google.com/", {:foo => "bar"})
puts http.body_str
http = Curl.get("http://www.google.com/") do|http|
http.headers['Cookie'] = 'foo=1;bar=2'
end
puts http.body_str

NET:HTTP headers ruby gem

I am trying to use the NET:HTTP gem to add an api-key to http header of a client, but it just doesn't seem to be working for some reason when I try and test it out.Basically the server requires the http header of the client or anything to have http_x_api header in order to serve the request.
Server code
require 'sinatra'
before do
halt 400 if (env['API_KEY']) != 'wow'
end
get '/' do
"boo"
end
Client code
require 'net/http'
require 'uri'
port = ENV['PORT'] || '7474'
res = Net::HTTP.start('localhost', port ) { |h| h.get('/')}
res.add_field('api-key', 'wow')
res.each_header do |key, value|
p "#{key} => #{value}"
end
puts (res.code == '200' && res.body == 'boo') ? 'OK' : 'FAIL'
this the response i get back :=>
"x-frame-options => sameorigin"
"x-xss-protection => 1; mode=block"
"content-type => text/html;charset=utf-8"
"content-length => 0"
"connection => keep-alive"
"server => thin 1.5.0 codename Knife"
"api-key => wow"
FAIL
On the server, the HTTP header variables in env are prefixed with HTTP_, so you need to check env['HTTP_API_KEY']. From the documentation:
HTTP_ Variables: Variables corresponding to the client-supplied HTTP request headers (i.e., variables whose names begin with HTTP_). The presence or absence of these variables should correspond with the presence or absence of the appropriate HTTP header in the request.

Resources