When sending an HTTParty.get() request to a URL that redirects to another (one, or possibly more) URL(s), I get this error:
Zlib::DataError:
invalid code lengths set
# ./file/file_name.rb:63:in `block (3 levels) in <top (required)>'
How do I get around this, and why is it happening?
Set the Accept-Encoding param in the headers option to blank:
resp = HTTParty.get(article.old_url, :verify => false, :headers => {'Accept-Encoding' => ''})
HTTParty was having errors decompressing the response (specifically gzip in my case), possibly because something strange was happening with the received Content-Type header as a result of the redirect(s). Setting the Accept-Encoding header to blank returned something that HTTParty knew how to parse.
Related
When using Octokit to do post request to the graphql endpoint, how do you set custom Accept: headers in order to turn on a "preview" API?
I need to do a GraphQL query that is part of an API preview, so it requires me to set the Accept header to application/vnd.github.starfox-preview+json (see GitHub API documentation).
I am using Octokit (4.15.0) in a Rails project, and am successfully doing normal queries (not preview mode) using both the v3 and v4 (GraphQL) APIs.
For normal queries (not API preview), my code looks like this, and works flawlessly. Here, the method graphql_query_string forms the correct query string, and github_machine_user is an instance of an Octokit Client object.
Code that works (when custom Accept: header not needed)
def perform_graphql_query(repo_name, org_name, after="")
graphql_query_string = graphql_query(repo_name, org_name, after)
options = { query: graphql_query_string }.to_json
github_machine_user.post '/graphql', options
end
This results in the default Accept header being used, which is, apparently "application/vnd.github.v3+json" (more on how I know this below.)
I tried several ways of adding a custom Accept: header, but none of them work. Here's what I tried:
Unsuccessful Attempt #1:
def perform_graphql_query(repo_name, org_name, after="")
graphql_query_string = graphql_query(repo_name, org_name, after)
options = {:query => graphql_query_string,
:headers => {:accept => "application/vnd.github.starfox-preview+json"}
}.to_json
github_machine_user.post('/graphql', options)
end
Unsuccessful Attempt #2:
def perform_graphql_query(repo_name, org_name, after="")
graphql_query_string = graphql_query(repo_name, org_name, after)
options = {
:query => graphql_query_string,
:accept => "application/vnd.github.starfox-preview+json"
}.to_json
github_machine_user.post('/graphql', options)
end
Neither of theses sets the proper header. From the source code for the post method and the underlying request method, I would have expected this to work. Here's what those look like.
# Make a HTTP POST request
#
# #param url [String] The path, relative to {#api_endpoint}
# #param options [Hash] Body and header params for request
# #return [Sawyer::Resource]
def post(url, options = {})
request :post, url, options
end
def request(method, path, data, options = {})
if data.is_a?(Hash)
options[:query] = data.delete(:query) || {}
options[:headers] = data.delete(:headers) || {}
if accept = data.delete(:accept)
options[:headers][:accept] = accept
end
end
Is it me, or a bug in Octokit?
I can tell the header isn't being set not just because I get results indicating Field 'project' doesn't exist on type 'AddedToProjectEvent', because I also followed the advice in the Octokit documentation about turning on debugging info and I can see the headers on my request.
Here's how they appear (other than the redacted credential)
I, [2020-07-24T12:26:37.989030 #64350] INFO -- request: POST https://api.github.com/graphql
I, [2020-07-24T12:26:37.989109 #64350] INFO -- request: Accept: "application/vnd.github.v3+json"
User-Agent: "Octokit Ruby Gem 4.15.0"
Content-Type: "application/json"
Authorization: "token REDACTED_FOR_STACK_OVERFLOW_POST"
So my requests to set the accept header are not being respected.
I did verify that using curl, I was able to get the query to work properly by passing the correct Accept header. So the graphQl query is properly formed, and when the Accept header is properly put into the query, everything is fine. But I just can't seem to figure out how to get Octokit to respect my wishes in setting the Accept header. I've even looked into the source, and it seems like what I'm doing should work.
Can anyone help?
Update: Posted as an issue on Octokit's GitHub Repo too.
Update #2: I tried removing the to_json. Unfortunately, that gives a stack trace as shown below.
If I remove the to_json and then only pass the {:accept => "application/vnd.github.starfox-preview+json"} then it does set the correct header. But if try including :query in that hash, it results in the error below, unless I have the to_json on the hash. I just can't seem to win.
NoMethodError undefined method `each' for #<String:0x00007fb05be10b00>
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/utils/params_hash.rb:28:in `update'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/sawyer-0.8.2/lib/sawyer/agent.rb:99:in `block in call'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/connection.rb:489:in `block in run_request'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/connection.rb:506:in `block in build_request'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/request.rb:55:in `block in create'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/request.rb:54:in `tap'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/request.rb:54:in `create'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/connection.rb:502:in `build_request'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/connection.rb:484:in `run_request'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/faraday-1.0.0/lib/faraday/connection.rb:279:in `post'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/sawyer-0.8.2/lib/sawyer/agent.rb:94:in `call'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/octokit-4.18.0/lib/octokit/connection.rb:156:in `request'
/Users/pconrad/.rvm/gems/ruby-2.6.5/gems/octokit-4.18.0/lib/octokit/connection.rb:28:in `post'
/Users/pconrad/github/project-anacapa/anacapa-github-linker/app/jobs/course/course_github_repo_get_sdlc_events.rb:105:in `perform_graphql_query'
Update #3: Added my own answer below but it seems hacky. I had to call a private method, which one is not supposed to do, and definitely not supposed to "have to do". So, still hoping for a better solution.
This is what finally worked. I had to get "dirty" and invoke a private method using .send which is not ideal.
graphql_query_string = graphql_query(repo_name, org_name, after).gsub("\n","")
data = {
:query => graphql_query_string,
}.to_json
options = {
:headers => {
:accept => Octokit::Preview::PREVIEW_TYPES[:project_card_events]
}
}
github_machine_user.send :request, :post, '/graphql', data,options
If there is a better way, I'd welcome the suggestion. I agree with the other posters that said it doesn't make any sense to pass to_json on the data part, but without it, I get the stack trace about each not being defined on a string.
Remove calling #to_json on options. You're passing a String into #post as options. options gets put into #request as data. #request will do its thing on data if it was a Hash but since it's a String, it won't populate the new options that is passed to the client in #request.
I am trying to create a widget on Dashing that will extract data from Icescrum and update a specific widget.
The thing is I found info on how to program jira, so i've been trying to mess around with it to make it work with icescrum, but with no luck.
I tried this one:
require "uri"
require "net/http"
require "json"
SCHEDULER.every '2s', :first_in => 0 do |job|
uri = URI.parse("http://website.ca:8080/icescrum")
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Get.new("uri.request_uri")
req.basic_auth username, password
response = http.request(req)
issuesinProgress = JSON.parse(response.body)["total"]
send_event('synergy', value: issuesinProgress)
end
but I got this error:
scheduler caught exception:
undefined local variable or method `username' for main:Object
/home/administrator/dashboard/jobs/sample.rb:27:in `block in <top (required)>'
/var/lib/gems/1.9.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:230:in `call'
/var/lib/gems/1.9.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:230:in `trigger_block'
/var/lib/gems/1.9.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/jobs.rb:204:in `block in trigger'
/var/lib/gems/1.9.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/scheduler.rb:430:in `call'
/var/lib/gems/1.9.1/gems/rufus-scheduler-2.0.24/lib/rufus/sc/scheduler.rb:430:in `block in trigger_job'
Your code has a line:
req.basic_auth username, password
It's expecting two variables, "username" and "password", but neither look to be defined in your code, here. If they aren't defined you'll get an error that looks an awful lot like what you're getting.
So finally I just used a cURL function to get the information. I'll leave it here if anyone else needs it. It also sends the results to an xml file (if you don't want that then just remove the last part)
.rb file:
%x{curl -v -H "Content-Type: application/xml" -X POST --data "#filename.xml" -u username:password http://website.ca:8080/icescrum/ --output result.xml}
Remember to change these dummy names with your links:
filename.xml
username
password
website.ca:8080/icescrum/
result.xml
a complete set up of the widget is also available here :
https://github.com/Flash-Dash/Icescrum-widget
I'm trying to pass in certain values and headers via Ruby but not sure how to do it. What I have so far:
uri = URI.parse('http://jira.test.local/rest/zapi/latest/execution')
req = Net::HTTP::Get.new(uri)
req.basic_auth 'userid', 'password'
res = Net::HTTP.start(uri.hostname, uri.port) {|http| http.request(req)}
puts res.body
values = "{\n \"issueId\": 32640,\n \"versionId\": \"11163\",\n \"cycleId\": \"5\",\n \"projectId\": 10460\n,\n \"status\": \"1\"}"
headers = {:content_type => "application/json"}
Net::HTTP.start(uri.hostname, uri.port) do | http |
response = http.post(req, values)
puts response.body
end
I'm getting this error right now:
{"status":{"1":{"id":1,"color":"#75B000","description":"Test was executed and passed successfully.","name":"PASS"},"2":{"id":2,"color":"#CC3300","description":"Test was executed and failed.","name":"FAIL"},"3":{"id":3,"color":"#F2B000","description":"Test execution is a work-in-progress.","name":"WIP"},"4":{"id":4,"color":"#6693B0","description":"The test execution of this test was blocked for some reason.","name":"BLOCKED"},"-1":{"id":-1,"color":"#A0A0A0","description":"The test has not yet been executed.","name":"UNEXECUTED"}},"executions":[],"currentlySelectedExecutionId":""}
undefined method `empty?' for #<Net::HTTP::Get GET> (NoMethodError)
/Users/fyousuf/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http/generic_request.rb:27:in `initialize'
/Users/fyousuf/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http/request.rb:14:in `initialize'
/Users/fyousuf/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http.rb:1390:in `new'
/Users/fyousuf/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http.rb:1390:in `send_entity'
/Users/fyousuf/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http.rb:1179:in `post'
./features/step_definitions/zapi_farooq.rb:32:in `block (2 levels) in <top (required)>'
/Users/fyousuf/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http.rb:852:in `start'
/Users/fyousuf/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http.rb:582:in `start'
./features/step_definitions/zapi_farooq.rb:31:in `/^I test zapi update$/'
features/zapi_farooq.feature:4:in `* I test zapi update'
The first line of the output is good, it's giving me the proper outpul from the res.body, but after that is the error.
Summary: I want to go to the uri, authenticate with creds and post with the values provided and with the proper headers (all using Ruby).
I'm trying to create a new execution as per this API: http://docs.getzephyr.apiary.io/#executionresourceapis (Create a new Execution)
Playing around I got it to work with:
values = "{\n \"status\": 2\n}"
headers = {:content_type => "application/json"}
response = RestClient::Resource.new 'http://jira.test.local/rest/zapi/latest/execution/343/execute', 'username', 'password'
response.put values, headers
puts response
puts response.body
Gives output of:
http://jira.test.local/rest/zapi/latest/execution/343/execute
<html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>
Also it changes the value of the issue in Jira (intended goal).
I just used RestClient b/c the original docs (http://docs.getzephyr.apiary.io/#executionresourceapis) were using it and got some help here too: https://github.com/rest-client/rest-client
I recently switched from Ruby's Net:HTTP class to rest-client 1.6.7.
I find it a lot easier to form requests, but unlike Net:HTTP request, when rest-client gets anything other than a 200, the request dies. I've tried putting a breakpoint directly after the RestClient.get, and it never gets hit - so I'm doing something wrong.
def get_member_using_card
resource = "#{#settings_app_uri}api/v1/card/#{self.member_card_num}?token=#{#settings.api_key}"
response = RestClient.get resource
if response.code == 200
card = JSON.parse(response.body)
self.customer_id = card['card']['customer_id']
else
return 0
end
end
Which results in this stacktrace:
RestClient::ResourceNotFound - 404 Resource Not Found:
/Users/tim/.rvm/gems/ruby-1.9.2-p290/gems/rest-client-1.6.7/lib/restclient/abstr
act_response.rb:48:in `return!'
/Users/tim/.rvm/gems/ruby-1.9.2-p290/gems/rest-client-1.6.7/lib/restclient/reque
st.rb:230:in `process_result'
/Users/tim/.rvm/gems/ruby-1.9.2-p290/gems/rest-client-1.6.7/lib/restclient/reque
st.rb:178:in `block in transmit'
/Users/tim/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:627:in `start'
/Users/tim/.rvm/gems/ruby-1.9.2-p290/gems/rest-client-1.6.7/lib/restclient/reque
st.rb:172:in `transmit'
/Users/tim/.rvm/gems/ruby-1.9.2-p290/gems/rest-client-1.6.7/lib/restclient/reque
st.rb:64:in `execute'
/Users/tim/.rvm/gems/ruby-1.9.2-p290/gems/rest-client-1.6.7/lib/restclient/reque
st.rb:33:in `execute'
/Users/tim/.rvm/gems/ruby-1.9.2-p290/gems/rest-client-1.6.7/lib/restclient.rb:68
:in `get'
Can someone tell me how to properly evaluate the response code and keep this exception from happening...?
See heading Exceptions on http://rubydoc.info/gems/rest-client/
for results code between 200 and 207 a RestClient::Response will be returned
for results code 301, 302 or 307 the redirection will be followed if the request is a get or a head
for result code 303 the redirection will be followed and the request transformed into a get
for other cases a RestClient::Exception holding the Response will be raised, a specific exception class will be thrown for know error codes
RestClient.get 'http://example.com/resource'
➔ RestClient::ResourceNotFound: RestClient::ResourceNotFound`
begin
RestClient.get 'http://example.com/resource'
rescue => e
e.response
end
➔ 404 Resource Not Found | text/html 282 bytes
Also in the same documentation #wich pointed to, you can pass a block to RestClient.get such that it will not throw an exception on non-200 response codes:
# Don't raise exceptions but return the response
RestClient.get('http://example.com/resource'){|response, request, result| response }
See the "Result Handling" section from the documentation.
rescue RestClient::ExceptionWithResponse => err
There are several errors that could happen, specific exception types like Errno::EHOSTUNREACH or the more generic ExceptionWithResponse. Check the readme for more info.
I believe the best way to handle exceptions of an API client is to get the original error message thrown by the API endpoint. Here is an example code to handle that with RestClient
require 'json'
def get_call
begin
standard_response = {body: nil, success: false, message: ''}
response = RestClient.get('https://example.com/api/v1/xx', headers={'Authorization' => 'AbcDef xxx'})
standard_response[:success] = true
standard_response[:body] = JSON.parse(response.body)
rescue RestClient::ExceptionWithResponse => e
http_body = JSON.parse(e.http_body) # This is the original response from the API endpoint. e.g. {'message': 'Reason for the failure'}
meaningful_error_message = http_body['message'].nil? ? e.message : http_body['message'] # if {'message': 'error message'} is the format of your API
standard_response[:message] = meaningful_error_message
end
standard_response
end
Beautiful way to handle the exceptions in rest client.
For more info do check rest-client#response-callbacks-error-handling
RestClient.get('http://example.com/resource') { |response, request, result, &block|
case response.code
when 200
p "It worked !"
response
when 423
raise SomeCustomExceptionIfYouWant
else
response.return!(&block)
end
}
I'm trying to read local image file, properly encode it and post to Tumbrl. According to the Tumblr API I can pass a parameter data which is Array (URL-encoded binary contents) Limit: 5 MB
I've tested my code with http://api.tumblr.com/v2/blog/#{BLOG}/info request. It is working. But I can't post a photo. Here is my code:
require 'oauth'
require 'oauth/consumer'
require 'open-uri'
require 'active_support'
CONSUMER = 'foo'
SECRET = 'foo'
TOKEN = 'foo'
TOKEN_SECRET = 'foo'
BLOG = 'foo'
consumer=OAuth::Consumer.new(CONSUMER, SECRET, {:site=>"http://tumblr.com"})
access_token = OAuth::AccessToken.new(consumer, TOKEN, TOKEN_SECRET)
# Here I tried one of two lines:
# data = Base64.encode64(IO.binread('./resized')) #first try
data = URI::encode(IO.binread('./resized')) #second try
# response = access_token.get "http://api.tumblr.com/v2/blog/#{BLOG}/info?api_key=#{CONSUMER}"
# puts response
response=access_token.post "http://api.tumblr.com/v2/blog/#{BLOG}/post?api_key=#{CONSUMER}&type=photo&data=#{data}&link=http://ya.ru&"
puts response
1st try:
% ruby ./w_oauth.rb
/usr/lib/ruby/1.9.1/uri/common.rb:176:in `split': bad URI(is not URI?): http://api.tumblr.com/v2/blog/foo/post?api_key=foo&type=photo&data=/9j/4AAQSkZJRgABAQEASABIAAD//gA7Q1JFQVRPUjogZ2QtanBlZyB2MS4w (URI::InvalidURIError)
ICh1c2luZyBJSkcgSlBFRyB2NjIpLCBxdWFsaXR5ID0gODAK/9sAQwAGBAUG
BQQGBgUGBwcGCAoQCgoJCQoUDg8MEBcUGBgXFBYWGh0lHxobIxwWFiAsICMm
(!!!long piece of image data skipped!!!)
FI/16HfTbyHPWurqdE+TGH4wx2js5SKQb+6b4bIj3aurqCrEtcXrf/4yf/dS
DLet/wCzEB6sa6uoomxJN2eaQj5mkYuerQj611dQM7Fx/wDLF/8AbXV1dTA/
/9k=
&link=http://ya.ru&
from /usr/lib/ruby/1.9.1/uri/common.rb:211:in `parse'
from /usr/lib/ruby/1.9.1/uri/common.rb:747:in `parse'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/access_token.rb:7:in `request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/access_token.rb:47:in `post'
from ./w_oauth.rb:23:in `<main>'
2nd try:
% ruby ./w_oauth.rb
/var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:14:in `force_encoding': can't modify frozen String (RuntimeError)
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:14:in `rescue in escape'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:12:in `escape'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:43:in `block (2 levels) in normalize'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:42:in `collect'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:42:in `block in normalize'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:37:in `map'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:37:in `normalize'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/request_proxy/base.rb:98:in `normalized_parameters'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/request_proxy/base.rb:113:in `signature_base_string'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/signature/base.rb:77:in `signature_base_string'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/signature/hmac/base.rb:12:in `digest'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/signature/base.rb:65:in `signature'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/signature.rb:23:in `sign'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/client/helper.rb:45:in `signature'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/client/helper.rb:75:in `header'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/client/net_http.rb:91:in `set_oauth_header'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/client/net_http.rb:30:in `oauth!'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/consumer.rb:224:in `sign!'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/consumer.rb:188:in `create_signed_request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/consumer.rb:159:in `request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/consumer_token.rb:25:in `request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/access_token.rb:12:in `request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/access_token.rb:47:in `post'
from ./w_oauth.rb:23:in `<main>'
UPD: ./resized is a proper JPEG file:
% file ./resized
./resized: JPEG image data, JFIF standard 1.01, comment: "CREATOR: gd-jpeg v1.0 (using IJG JPEG v62), quality = 80"
URI encoding is not enough. You also need to encode: , / ? : # & = + $ #.
Try:
URI.escape(IO.binread('./resized'), Regexp.new("[^#{URI::PATTERN::UNRESERVED}]"))