I'm writing a small cli tool, that should check my calendar and do some stuff according to my appointments.
I'm struggling a little bit with the OAuth2 authentication. I've checked the scope and the client_id with the curl tool like this:
curl -d "client_id=12345...&scope=scope=https://www.googleapis.com/auth/calendar.readonly" https://accounts.google.com/o/oauth2/device/code
This way, I get the right response.
{
"device_code" : "somestuff",
"user_code" : "otherstuff",
"verification_url" : "http://www.google.com/device",
"expires_in" : 1800,
"interval" : 5
}
But, when I try to use Net::HTTP in Ruby I just get HTTP state 200. I've done it this way:
res = Net::HTTP.post_form(uri, {'client_id' =>'1234....apps.googleusercontent.com', 'scope' => 'https://www.googleapis.com/auth/calendar.readonly' })
If I check the res variable afterwards I get the state 302, but I guess this is correct.
Can someone tell me what I'm, doing wrong so I don't get the JSON response? Should I try something different than Net::HTTP?
res is a variable containing all the response data, not just the text of the response. If you puts res.body after your post_form() call, you should find your JSON (which you can parse with the JSON module).
Related
I'm prototyping a Rails application to upload documents to FSCrawler (running the REST interface), to incorporate into an Elasticsearch index. Using their example, this works:
response = `curl -F "file=##{params[:document][:upload].tempfile.path}" "http://127.0.0.1:8080/fscrawler/_upload?debug=true"`
The file gets uploaded, and the content gets indexed. This is an example of what I get:
"{\n \"ok\" : true,\n \"filename\" : \"RackMultipart20200130-91061-16swulg.pdf\",\n \"url\" : \"http://127.0.0.1:9200/local/_doc/d661edecf3e28572676e97a6f0d1d\",\n \"doc\" : {\n \"content\" : \"\\n \\n \\n\\nBasically, what you need to know is that Dante is all IP-based, and makes use of common IT standards. Each Dante device behaves \\n\\nmuch like any other network device you would already find on your network. \\n\\nIn order to make integration into an existing network easy, here are some of the things that Dante does: \\n\\n▪ Dante...
When I run curl at the command line, I get EVERYTHING, like the "filename" being properly set. If I use it as above, in the Rails controller, as you can see, the filename is set to the Tempfile's filename. That's not a workable solution. Trying to use params[:document][:upload].tempfile (without .path) or just params[:document][:upload] both fail entirely.
I'm trying to do this "the right way," but every incarnation of using a proper HTTP client to do this fails. I can't figure out how to invoke an HTTP POST that will submit a file to FSCrawler the way curl (on the command line) does it.
In this example, I'm just trying to send the file by using the Tempfile file object. For some reason, FSCrawler gives me the error in the comment, and get a little metadata, but no content is indexed:
## Failed to extract [100000] characters of text for ...
## org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes
uri = URI("http://127.0.0.1:8080/fscrawler/_upload?debug=true")
request = Net::HTTP::Post.new(uri)
form_data = [['file', params[:document][:upload].tempfile,
{ filename: params[:document][:upload].original_filename,
content_type: params[:document][:upload].content_type }]]
request.set_form form_data, 'multipart/form-data'
response = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(request)
end
If I change the above to use params[:document][:upload].tempfile.path, then I don't get the error about the InputStream, but I also (still) do not get any content indexed. This is an example of what I get:
{"_index":"local","_type":"_doc","_id":"72c9ecf2a83440994eb87d28786e6","_version":3,"_seq_no":26,"_primary_term":1,"found":true,"_source":{"content":"/var/folders/bn/pcc1h8p16tl534pw__fdz2sw0000gn/T/RackMultipart20200130-91061-134tcxn.pdf\n","meta":{},"file":{"extension":"pdf","content_type":"text/plain; charset=ISO-8859-1","indexing_date":"2020-01-30T15:33:45.481+0000","filename":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf"},"path":{"virtual":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf","real":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf"}}}
If I try to use RestClient, and I try send the file by referencing the actual path to the Tempfile, then I get this error message, and I get nothing:
## Unsupported media type
response = RestClient.post 'http://127.0.0.1:8080/fscrawler/_upload?debug=true',
file: params[:document][:upload].tempfile.path,
content_type: params[:document][:upload].content_type
If I try to .read() the file, and submit that, then I break the FSCrawler form:
## Internal server error
request = RestClient::Request.new(
:method => :post,
:url => 'http://127.0.0.1:8080/fscrawler/_upload?debug=true',
:payload => {
:multipart => true,
:file => File.read(params[:document][:upload].tempfile),
:content_type => params[:document][:upload].content_type
})
response = request.execute
Obviously, I've been trying this every way I can, but I can't replicate whatever curl is doing with any known Ruby-based HTTP clients. I'm utterly lost as to how to get Ruby to submit data to FSCrawler in a way that will get the document contents indexed properly. I've been at this far longer than I care to admit. What am I missing here?
I finally tried Faraday, and, based on this answer, came up with the following:
connection = Faraday.new('http://127.0.0.1:8080') do |f|
f.request :multipart
f.request :url_encoded
f.adapter :net_http
end
file = Faraday::UploadIO.new(
params[:document][:upload].tempfile.path,
params[:document][:upload].content_type,
params[:document][:upload].original_filename
)
payload = { :file => file }
response = connection.post('/fscrawler/_upload', payload)
Using Fiddler helped me to see the results of my attempts, as I got closer and closer to the curl request. This snippet posts the request almost exactly as curl does. To route this call through the proxy, I just needed to add , proxy: 'http://localhost:8866' to the end of the connection setup.
I know that Snoo seems to be unmaintained, but I wanted to use a ruby framework since I'm trying to improve my Ruby skill.
I'm trying to add some functionality starting with subscribing and unsubscribing to subreddits. Link to API doc.
My first attempt was with the built-in post method which returned a 404 error
def subscribe(subreddit)
logged_in?
post('/api/subscribe.json',body:{uh: #modhash, action:'sub', sr: subreddit, api_type: 'json'})
end
Since the built-in post method was giving me a 404 I decided to try the HTTParty post method:
def subscribe(subreddit)
logged_in?
HTTParty.post('http://www.reddit.com/api/subscribe.json',body:{uh: #modhash, action:'sub', sr: subreddit, api_type: 'json'})
end
That returns this:
pry(main)> reddit.subscribe('/r/nba')
=> {"json"=>{"errors"=>[["USER_REQUIRED", "please login to do that", nil]]}}
Does anyone know if I need to pass more info in the body or if I'm just sending a badly formed request? Thanks!
Also, before running "reddit.subscribe" I have verified that I'm logged in with with a cookie, a modhash, can access my account info, etc.
Solution found:
def subscribe(subreddit)
#query the subreddit for it's 'about' info and get json back
subreddit_json = self.subreddit_info(subreddit)
#build the coded unique identifier for the targeted subreddit
subreddit_id = subreddit_json['kind'] + "_" + subreddit_json['data']['id']
#send post request to server
server_response = self.class.post('/api/subscribe.json',
body:{uh:#modhash, action:'sub', sr: subreddit_id, api_type:'json'})
end
The Reddit API doesn't accept the subreddit name as the value passed with 'sr', (e.g. sr:'/r/funny'). It requires the subreddit "type" (which is always 't5' for subreddits) and unique forum id. The parameter passed would look something like: sr: "t5_2qo4s". This information is available if you go to your target subreddit and add about.json, e.g., www.reddit.com/r/funny/about.json
I have been trying for days to pull down activity data from the Withings API using the OAuth Ruby gem. Regardless of what method I try I consistently get back a 503 error response (not enough params) even though I copied the example URI from the documentation, having of course swapped out the userid. Has anybody had any luck with this in the past. I hope it is just something stupid I am doing.
class Withings
API_KEY = 'REMOVED'
API_SECRET = 'REMOVED'
CONFIGURATION = { site: 'https://oauth.withings.com', request_token_path: '/account/request_token',
access_token_path: '/account/access_token', authorize_path: '/account/authorize' }
before do
#consumer = OAuth::Consumer.new API_KEY, API_SECRET, CONFIGURATION
#base_url ||= "#{request.env['rack.url_scheme']}://#{request.env['HTTP_HOST']}#{request.env['SCRIPT_NAME']}"
end
get '/' do
#request_token = #consumer.get_request_token oauth_callback: "#{#base_url}/access_token"
session[:token] = #request_token.token
session[:secret] = #request_token.secret
redirect #request_token.authorize_url
end
get '/access_token' do
#request_token = OAuth::RequestToken.new #consumer, session[:token], session[:secret]
#access_token = #request_token.get_access_token oauth_verifier: params[:oauth_verifier]
session[:token] = #access_token.token
session[:secret] = #access_token.secret
session[:userid] = params[:userid]
redirect "#{#base_url}/activity"
end
get '/activity' do
#access_token = OAuth::AccessToken.new #consumer, session[:token], session[:secret]
response = #access_token.get("http://wbsapi.withings.net/v2/measure?action=getactivity&userid=#{session[:userid]}&startdateymd=2014-01-01&enddateymd=2014-05-09")
JSON.parse(response.body)
end
end
For other API endpoints I get an error response of 247 - The userid provided is absent, or incorrect. This is really frustrating. Thanks
So I figured out the answer after copious amount of Googleing and grasping a better understanding of both the Withings API and the OAuth library I was using. Basically Withings uses query strings to pass in API parameters. I though I was going about passing these parameters correctly when I was making API calls, but apparently I needed to explicitly set the OAuth library to use the query string scheme, like so
http_method: :get, scheme: :query_string
This is appended to my OAuth consumer configuration and all worked fine immediately.
I'm doing a Post request to github at this url:
https://api.github.com/gists/2710948/comments
Theoretically, this should create a comment with the text being formed from what's in the request body. However, when I try to make that post, I get a 404 error. That leads me to believe that the gist is not being found, however, if you do a Get request at the same address it comes up just fine.
Is there an authentication thing I need to be doing? I've tried adding a username and password to my headers collection but I've got no idea if I'm using the right format. I've tried making this work via Ruby, HTTP Client, and curl, and I get the same error either way.
The curl command I'm using is this:
curl -X POST -d "This is my sample comment" https://api.github.com/gists/2710948/comments
I think that if I can get the curl command working, I'll be able to figure out the HTTP Client and then the Ruby. This will be my first attempt at consuming an API, so there's nothing too basic for me to double-check; all suggestions will be helpful.
curl -d '{ "body": "Test comment" }' -u "Username:Pass" -X POST https://api.github.com/gists/2710948/comments
Ruby code:
require 'net/http'
uri = URI("https://api.github.com/gists/2710948/comments")
req = Net::HTTP::Post.new(uri.to_s)
req.basic_auth("Username", "Pass")
req.body = '{"body": "Test message"}' # `to_json` can be used
req["content-type"] = "application/json"
Net::HTTP.start(uri.host, uri.port, :use_ssl => true) do |http|
p response = http.request(req)
end
See also http://developer.github.com/v3/gists/comments/
I am doing a video crawler in ruby. In there I have to log in to a page by enabling cookies and download pages. For that I am using the CURL library in ruby. I can successfully log in, but I can't download the pages inside that with curl. How can I fix this or download the pages otherwise?
My code is
curl = Curl::Easy.new(1st url)
curl.follow_location = true
curl.enable_cookies = true
curl.cookiefile = "cookie.txt"
curl.cookiejar = "cookie.txt"
curl.http_post(1st url,field)
curl.perform
curl = Curl::Easy.perform(2nd url)
curl.follow_location = true
curl.enable_cookies = true
curl.cookiefile = "cookie.txt"
curl.cookiejar = "cookie.txt"
curl.http_get
code = curl.body_str
What I've seen in writing my own similar "post-then-get" script is that ruby/Curb (I'm using version 0.7.15 with ruby 1.8) seems to ignore the cookiejar/cookiefile fields of a Curl::Easy object. If I set either of those fields and the http_post completes successfully, no cookiejar or cookiefile file is created. Also, curl.cookies will still be nil after your curl.http_post, however, the cookies ARE set within the curl object. I promise :)
I think where you're going wrong is here:
curl = Curl::Easy.perform(2nd url)
The curb documentation states that this creates a new object. That new object doesn't have any of your existing cookies set. If you change your code to look like the following, I believe it should work. I've also removed the curl.perform for the first url since curl.http_post already implicitly does the "perform". You were basically http_post'ing twice before trying your http_get.
curl = Curl::Easy.new(1st url)
curl.follow_location = true
curl.enable_cookies = true
curl.http_post(1st url,field)
curl.url = 2nd url
curl.http_get
code = curl.body_str
If this still doesn't seem to be working for you, you can verify if the cookie is getting set by adding
curl.verbose = true
Before
curl.http_post
Your Curl::Easy object will dump all the headers that it gets in the response from the server to $stdout, and somewhere in there you should see a line stating that it added/set a cookie. I don't have any example output right now but I'll try to post a follow-up soon.
HTTPClient automatically enables cookies, as does Mechanize.
From the HTTPClient docs:
clnt = HTTPClient.new
clnt.get_content(url1) # receives Cookies.
clnt.get_content(url2) # sends Cookies if needed.
Posting a form is easy too:
body = { 'keyword' => 'ruby', 'lang' => 'en' }
res = clnt.post(uri, body)
Mechanize makes this sort of thing really simple (It will handle storing the cookies, among other things).