Want to scrape a bunch of tweets via the Twitter API, as an output I get cURL command, something like that
curl --get 'https://api.twitter.com/1.1/search/tweets.json' --data 'q=football' --header 'Authorization: OAuth oauth_consumer_key="**hidden**", oauth_nonce="**hidden**", oauth_signature="**hidden**", oauth_signature_method="HMAC-SHA1", oauth_timestamp="**hidden**", oauth_token="**hidden**", oauth_version="1.0"' --verbose
My question, is there a way to use this command into a Ruby script to scrape the tweets ?
Using the Twitter gem available here http://rdoc.info/gems/twitter with the following code you can get all the tweets from a ruby script.
require 'twitter'
client = Twitter::REST::Client.new do |config|
config.consumer_key ="hidden"
config.consumer_secret ="hidden"
config.access_token ="hidden"
config.access_token_secret ="hidden"
end
client.search("football").collect do |tweet|
puts tweet.text
end
you can wrap it in backticks and get the output like from any unix(like) command
script.rb
cmd=`echo 'hello world'`
puts cmd
ouptput: hello world
It is better to use existing API as #Hunter McMillen had suggested, but if you want to perform http-requests yourself, you can use net/http lib. Example below:
require 'net/http'
uri = URI('http://example.com/index.html')
params = { :limit => 10, :page => 3 }
uri.query = URI.encode_www_form(params)
res = Net::HTTP.get_response(uri)
puts res.body if res.is_a?(Net::HTTPSuccess)
Here is the info on how to set headers.
Related
I am new to both Ruby Language and Puppet. I want to run a ruby code which is a converted form of a cURL, from my puppet console.
I am putting the cURL and the converted ruby below.
curl -k -u stg_admin:password -sS -X POST https://www.something.com
As you can see, this is a very basic cURL, and the converted ruby code is-
require 'net/http'
require 'uri'
require 'openssl'
uri = URI.parse("https://www.something.com")
request = Net::HTTP::Post.new(uri)
request.basic_auth("stg_admin", "password")
req_options = {
use_ssl: uri.scheme == "https",
verify_mode: OpenSSL::SSL::VERIFY_NONE,
}
response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
# response.code
# response.body
I know how to exec cURL directly in my puppet class, but my question is "is there any way to call my ruby code directly in puppet's init.pp?"
I'll be greatfull for any suggestion.
In ruby, I want to get the same result than the code below but without using curl:
curl_output = `curl -X POST -s --form "input=##{png_image_file};type=image/png" http://pngcrush.com/crush > #{compressed_png_file}`
I tried this:
#!/usr/bin/env ruby
require "net/http"
require "uri"
# Image to crush
png_image_path = "./media/images/foo.png"
# Crush with http://pngcrush.com/
png_compress_uri = URI.parse("http://pngcrush.com/crush")
png_image_data = File.read(png_image_path)
req = Net::HTTP.new(png_compress_uri.host, png_compress_uri.port)
headers = {"Content-Type" => "image/png" }
response = req.post(png_compress_uri.path, png_image_data, headers)
p response.body
# => "Input is empty, provide a PNG image."
The problem with your code is you do not send required parameter to the server ("input" for http://pngcrush.com/crush). This works for me:
require 'net/http'
require 'uri'
uri = URI.parse('http://pngcrush.com/crush')
form_data = [
['input', File.open('filename.png')]
]
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new uri
# prepare request parameters
request.set_form(form_data, 'multipart/form-data')
response = http.request(request)
# save crushed image
open('crushed.png', 'wb') do |file|
file.write(response.body)
end
But I suggest you to use RestClient. It encapsulates net/http with cool features like multipart form data and you need just a few lines of code to do the job:
require 'rest_client'
resp = RestClient.post('http://pngcrush.com/crush',
:input => File.new('filename.png'))
# save crushed image
open('crushed.png', 'wb') do |file|
file.write(resp)
end
Install it with gem install rest-client
This is my hyperresource client:
require 'rubygems'
require 'hyperresource'
require 'json'
api = HyperResource.new(root: 'http://127.0.0.1:9393/todos',
headers: {'Accept' => 'application/vnd.127.0.0.1:9393/todos.v1+hal+json'})
string = '{"todo":{"title":"test"}}'
hash = JSON.parse(string)
api.post(hash)
puts hash
The hash output is: {"todo"=>{"title"=>"test"}}
At my Sinatra with Roar API I have this post function:
post "/todos" do
params.to_json
puts params
#todo = Todo.new(params[:todo])
if #todo.save
#todo.extend(TodoRepresenter)
#todo.to_json
else
puts 'FAIL'
end
end
My puts 'params' over here gets: {"{\"todo\":{\"title\":\"test\"}}"=>nil}
I found out, these are 'escaped strings' but I don't know where it goes wrong.
EDIT:
I checked my api with curl and postman google extension, both work fine. It's just hyperresource I guess
You are posting JSON, ergo you either need to register a Sinatra middleware that will automatically parse incoming JSON requests, or you need to do it yourself.
require 'rubygems'
require 'hyperresource'
require 'json'
api = HyperResource.new(root: 'http://127.0.0.1:9393/todos',
headers: {'Accept' => 'application/vnd.127.0.0.1:9393/todos.v1+hal+json'})
string = '{"todo":{"title":"test"}}'
hash = JSON.parse(string)
api.post({:data => hash})
puts hash
---
post "/todos" do
p = JSON.parse(params[:data])
puts p.inspect
#todo = Todo.new(p[:todo])
if #todo.save
#todo.extend(TodoRepresenter)
#todo.to_json
else
puts 'FAIL'
end
end
Should do what you need.
I have a Ruby script that sends a POST request with a cookie using:
curl.exe -H "Cookie: SomeCookie=#{cookie}" -d "SomaData=#{data}" http://somesite.com/post
I tried to rewrite this into native Ruby using Net::HTTP, but this code doesn't work:
Net::HTTP.post_form(URI('http://somesite.com/post'),
{'SomeData' => '#{data}',
'Cookie' => 'SomeCookie=#{cookie}'} )
How do I solve this problem?
I'am using MRI Ruby 1.9.3 on Windows 7.
Why not look into using Curb? It's a Ruby interface to libcurl, and has an interface that's closer to cURL than Net::HTTP.
This is from the documentation:
http = Curl.get("http://www.google.com/")
puts http.body_str
http = Curl.post("http://www.google.com/", {:foo => "bar"})
puts http.body_str
http = Curl.get("http://www.google.com/") do|http|
http.headers['Cookie'] = 'foo=1;bar=2'
end
puts http.body_str
Anyone got a working example of using ruby to post to a presigned URL on s3
I have used aws-sdk and right_aws both.
Here is the code to do this.
require 'rubygems'
require 'aws-sdk'
require 'right_aws'
require 'net/http'
require 'uri'
require 'rack'
access_key_id = 'AAAAAAAAAAAAAAAAA'
secret_access_key = 'ASDFASDFAS4646ASDFSAFASDFASDFSADF'
s3 = AWS::S3.new( :access_key_id => access_key_id, :secret_access_key => secret_access_key)
right_s3 = RightAws::S3Interface.new(access_key_id, secret_access_key, {:multi_thread => true, :logger => nil} )
bucket_name = 'your-bucket-name'
key = "your-file-name.ext"
right_url = right_s3.put_link(bucket_name, key)
right_scan_command = "curl -I --upload-file #{key} '#{right_url.to_s}'"
system(right_scan_command)
bucket = s3.buckets[bucket_name]
form = bucket.presigned_post(:key => key)
uri = URI(form.url.to_s + '/' + key)
uri.query = Rack::Utils.build_query(form.fields)
scan_command = "curl -I --upload-file #{key} '#{uri.to_s}'"
system(scan_command)
Can you provide more information on how a "presigned URL" works? Is it like this:
AWS::S3::S3Object.url_for(self.full_filename,
self.bucket_name, {
:use_ssl => true,
:expires_in => ttl_seconds
})
I use this code to send authenticated clients the URL to their S3 file. I believe this is the "presigned URL" that you're asking about. I haven't used this code for a PUT, so I'm not exactly sure if it's right for you, but it might get you close.
I know this is an older question, but I was wondering the same thing and found an elegant solution in the AWS S3 Documentation.
require 'net/http'
file = "somefile.ext"
url = URI.parse(presigned_url)
Net::HTTP.start(url.host) do |http|
http.send_request("PUT", url.request_uri, File.read(file), {"content-type" => "",})
end
This worked great for my Device Farm uploads.
Does anything on the s3 library page cover what you need? There are loads of examples there.
There are some generic REST libraries for Ruby; Google for "ruby rest client". See also HTTParty.
I've managed to sort it out. Turns out the HTTP:Net in Ruby is has some short comings. Lot of Monkeypatch later I got it working.. More details when I have time. thank