I have written code in Ruby that will display the timeline for a specific user. I would like to write code to be able to just search twitter to just find every user that has mentioned a word. My code is currently:
require 'rubygems'
require 'oauth'
require 'json'
# Now you will fetch /1.1/statuses/user_timeline.json,
# returns a list of public Tweets from the specified
# account.
baseurl = "https://api.twitter.com"
path = "/1.1/statuses/user_timeline.json"
query = URI.encode_www_form(
"q" => "Obama"
)
address = URI("#{baseurl}#{path}?#{query}")
request = Net::HTTP::Get.new address.request_uri
# Print data about a list of Tweets
def print_timeline(tweets)
tweets.each do |tweet|
require 'date'
d = DateTime.parse(tweet['created_at'])
puts " #{tweet['text'].delete ","} , #{d.strftime('%d.%m.%y')} , #{tweet['user']['name']}, #{tweet['id']}"
end
end
# Set up HTTP.
http = Net::HTTP.new address.host, address.port
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
# If you entered your credentials in the first
# exercise, no need to enter them again here. The
# ||= operator will only assign these values if
# they are not already set.
consumer_key = OAuth::Consumer.new(
"")
access_token = OAuth::Token.new(
"")
# Issue the request.
request.oauth! http, consumer_key, access_token
http.start
response = http.request request
# Parse and print the Tweet if the response code was 200
tweets = nil
puts "Text,Date,Name,id"
if response.code == '200' then
tweets = JSON.parse(response.body)
print_timeline(tweets)
end
nil
How would I possibly change this code to search all of twitter for a specific word?
The easiest approach would be to use 'Twitter' gem. Refer to this Link for more information and the result type of the search results. Once you have all the correct authorization attribute in place (oAuth-Token,oAuth-secret, etc) you should be able to search as
Twitter.search('Obama')
or
Twitter.search('Obama', options = {})
Let us know, if that worked for you or not.
p.s. - Please mark the post as answered if it helped you. Else put a comment back with what is missing.
The Twitter API suggests the URI your should be using for global search is https://api.twitter.com/1.1/search/tweets.json and this means:
Your base_url component would be https://api.twitter.com
Your path component would be /1.1/search/tweets.json
Your query component would be the text you are searching for.
The query part takes a lot of values depending upon the API spec. Refer to the specification and you can change it as per your requirement.
Tip: Try to use irb (I'd recommend pry) REPL which makes it a lot easier to explore APIs. Also, checkout the Faraday gem which can be easier to use than the default HTTP library in Ruby IMO.
Related
This is a follow-up question to this post.
I am new to Ruby and want to create a script that will search a file for a pattern. However, I want to only replace part of it, i.e. remove all http:// patterns matches but only when they are followed by a valid url.
If "valid url" means that the string is parseable as an URL, then you might try using URI.parse. For example:
require 'uri'
IO.readlines(input_file).each do |line|
line.gsub(%r;(https?://\S+);) do |url|
URI.parse(url) && '' rescue url
end
end
However, the URI module is very lax. You'll find strings like not-an-uri are considered valid "generic" URIs.
You might want to check whether the captured URL can be fetched and returns a successful HTTP status. That is significantly more resource intensive, so operating over a large input file would be very slow. It also could be considered a security risk.
require 'uri'
require 'net/http'
def valid_url?(url)
uri = URI.parse(url)
Net::HTTP.get_response(uri).is_a? Net::HTTPSuccess
rescue
return false
end
IO.readlines(input_file).each do |line|
line.gsub(%r;(https?://\S+);) do |url|
valid_url?(url) ? '' : url
end
end
I'm trying to write a Ruby script to use the API on the image gallery site Piwigo, this requires you to login first with one HTTP post and upload an image with another post.
This is what I've got so far but it doesn't work, just returns a 401 error, can anyone see where I am going wrong?
require 'net/http'
require 'pp'
http = Net::HTTP.new('mydomain.com',80)
path = '/piwigo/ws.php'
data = 'method=pwg.session.login&username=admin&password=password'
resp, data = http.post(path, data, {})
if (resp.code == '200')
cookie = resp.response['set-cookie']
data = 'method=pwg.images.addSimple&image=image.jpg&category=7'
headers = { "Cookie" => cookie }
resp, data = http.post(path, data, headers)
puts resp.code
puts resp.message
end
Which gives this response when run;
$ ruby piwigo.rb
401
Unauthorized
There is a Perl example on their API page which I was trying to convert to Ruby http://piwigo.org/doc/doku.php?id=dev:webapi:pwg.images.addsimple
By using the nice_http gem: https://github.com/MarioRuiz/nice_http
NiceHttp will take care of your cookies so you don't have to do anything
require 'nice_http'
path = '/piwigo/ws.php'
data = '?method=pwg.session.login&username=admin&password=password'
http = NiceHttp.new('http://example.com')
resp = http.get(path+data)
if resp.code == 200
resp = http.post(path)
puts resp.code
puts resp.message
end
Also if you want you can add your own cookies by using http.cookies
You can use a gem called mechanize. It handles cookies transparently.
Current code works as long as there is no remote error:
def get_name_from_remote_url
cstr = "http://someurl.com"
getresult = open(cstr, "UserAgent" => "Ruby-OpenURI").read
doc = Nokogiri::XML(getresult)
my_data = doc.xpath("/session/name").text
# => 'Fred' or 'Sam' etc
return my_data
end
But, what if the remote URL times out or returns nothing? How I detect that and return nil, for example?
And, does Open-URI give a way to define how long to wait before giving up? This method is called while a user is waiting for a response, so how do we set a max timeoput time before we give up and tell the user "sorry the remote server we tried to access is not available right now"?
Open-URI is convenient, but that ease of use means they're removing the access to a lot of the configuration details the other HTTP clients like Net::HTTP allow.
It depends on what version of Ruby you're using. For 1.8.7 you can use the Timeout module. From the docs:
require 'timeout'
begin
status = Timeout::timeout(5) {
getresult = open(cstr, "UserAgent" => "Ruby-OpenURI").read
}
rescue Timeout::Error => e
puts e.to_s
end
Then check the length of getresult to see if you got any content:
if (getresult.empty?)
puts "got nothing from url"
end
If you are using Ruby 1.9.2 you can add a :read_timeout => 10 option to the open() method.
Also, your code could be tightened up and made a bit more flexible. This will let you pass in a URL or default to the currently used URL. Also read Nokogiri's NodeSet docs to understand the difference between xpath, /, css and at, %, at_css, at_xpath:
def get_name_from_remote_url(cstr = 'http://someurl.com')
doc = Nokogiri::XML(open(cstr, 'UserAgent' => 'Ruby-OpenURI'))
# xpath returns a nodeset which has to be iterated over
# my_data = doc.xpath('/session/name').text # => 'Fred' or 'Sam' etc
# at returns a single node
doc.at('/session/name').text
end
Is it possible to use OAuth with HTTParty? I'm trying to do this API call, but, contradictory to the documentation, it needs authentication.
Before you say "Use a Twitter-specific Gem", hear me out--I've tried. I've tried twitter, grackle, and countless others, but none support this specific API call. So, I've turned to HTTParty.
So, how could I use OAuth with HTTParty?
I've been using the vanilla OAuth gem to implement a few simple Twitter API calls. I didn't need a heavyweight gem to do everything, and I was already using OAuth, so a 'roll-your-own' approach seemed reasonable. I know that I haven't mentioned HTTParty, so please don't ding me for that. This may be useful to others for the essence of easy Twitter OAuth if you're already using the OAuth gem.
In case it is helpful, here is the pertinent code (sorry about mixing some constants and other variables / methods at the start - it was the easiest and most accurate way to extract this from my real code):
#Set up the constants, etc required for Twitter OAuth
OAUTH_SITE = "https://api.twitter.com"
TOKEN_REQUEST_METHOD = :post
AUTHORIZATION_SCHEME = :header
def app_request_token_path
"/oauth/request_token"
end
def app_authorize_path
"/oauth/authorize"
end
def app_access_token_path
"/oauth/access_token"
end
def consumer_key
"your twitter API key"
end
def consumer_secret
"your twitter API secret"
end
# Define the OAuth consumer
def consumer meth=:post
#consumer ||= OAuth::Consumer.new(consumer_key,consumer_secret, {
:site => "#{OAUTH_SITE}",
:request_token_path=>app_request_token_path,
:authorize_path=>app_authorize_path,
:access_token_path=>app_access_token_path,
:http_method=>:post,
:scheme=> :header,
:body_hash => ''
})
end
# Essential parts of a generic OAuth request method
def make_request url, method=:get, headers={}, content=''
if method==:get
res = #access_token.get(url, headers)
elsif method==:post
res = #access_token.post(url, content, headers)
end
if res.code.to_s=='200'
jres = ActiveSupport::JSON.decode(res.body)
if jres.nil?
#last_status_text = #prev_error = "Unexpected error making an OAuth API call - response body is #{res.body}"
end
return jres
else
#last_status_text = #prev_error = res if res.code.to_s!='200'
return nil
end
end
# Demonstrate the daily trends API call
# Note the use of memcache to ensure we don't break the rate-limiter
def daily_trends
url = "http://api.twitter.com/1/trends/daily.json"
#last_status_code = -1
#last_status_success = false
res = Rails.cache.fetch(url, :expires_in=> 5.minutes) do
res = make_request(url, :get)
unless res
#last_status_code = #prev_error.code.to_i
end
res
end
if res
#last_status_code = 200
#last_status_success = true
#last_status_text = ""
end
return res
end
I hope this, largely in context of broader use of the OAuth gem, might be useful to others.
I don't think that HTTParty supports OAuth (though I am no expert on HTTParty, it's way too high-level and slow for my taste).
I would just call the Twitter request directly using OAuth gem. Twitter API documentation even has an example of usage: https://dev.twitter.com/docs/auth/oauth/single-user-with-examples#ruby
I used a mix of the OAuth2 gem to get the authentication token and HTTParty to make the query
client = OAuth2::Client.new(apiKey, apiSecret,
:site => "https://SiteForAuthentication.com")
oauthResponse = client.password.get_token(username, password)
token = oauthResponse.token
queryAnswer = HTTParty.get('https://api.website.com/query/location',
:query => {"token" => token})
Not perfect by a long way but it seems to have done the trick so far
What is the best way to make an HTTP GET request in Ruby with modified headers?
I want to get a range of bytes from the end of a log file and have been toying with the following code, but the server is throwing back a response saying that "it is a request that the server could not understand" (the server is Apache).
require 'net/http'
require 'uri'
#with #address, #port, #path all defined elsewhere
httpcall = Net::HTTP.new(#address, #port)
headers = {
'Range' => 'bytes=1000-'
}
resp, data = httpcall.get2(#path, headers)
Is there a better way to define headers in Ruby?
Does anyone know why this would be failing against Apache? If I do a get in a browser to http://[address]:[port]/[path] I get the data I am seeking without issue.
Created a solution that worked for me (worked very well) - this example getting a range offset:
require 'uri'
require 'net/http'
size = 1000 #the last offset (for the range header)
uri = URI("http://localhost:80/index.html")
http = Net::HTTP.new(uri.host, uri.port)
headers = {
'Range' => "bytes=#{size}-"
}
path = uri.path.empty? ? "/" : uri.path
#test to ensure that the request will be valid - first get the head
code = http.head(path, headers).code.to_i
if (code >= 200 && code < 300) then
#the data is available...
http.get(uri.path, headers) do |chunk|
#provided the data is good, print it...
print chunk unless chunk =~ />416.+Range/
end
end
If you have access to the server logs, try comparing the request from the browser with the one from Ruby and see if that tells you anything. If this isn't practical, fire up Webrick as a mock of the file server. Don't worry about the results, just compare the requests to see what they are doing differently.
As for Ruby style, you could move the headers inline, like so:
httpcall = Net::HTTP.new(#address, #port)
resp, data = httpcall.get2(#path, 'Range' => 'bytes=1000-')
Also, note that in Ruby 1.8+, what you are almost certainly running, Net::HTTP#get2 returns a single HTTPResponse object, not a resp, data pair.