It is image tag on the page accessed by capybara via HTTPS protocol:
<img src="path">
Is it any way to get image file from the page using capybara with any kind of driver?
I can not use something like File.read('path') because image is also accessible via HTTPS only. My latest researches brought me to such kind of solution:
Visit page
Save page to png (webkit driver has such useful ability)
Crop image
But I do believe that pretty solution exists.
Edited 1:
I've tried out padde's solution, but here is response body:
<html><head><title>Object moved</title></head>
<body>
<h2>Object moved to here.</h2>
</body>
</html>
Edited 2:
> curl -I image_path
5860cf30abf5d5480
HTTP/1.1 302 Found
Cache-Control: private
Content-Length: 168
Content-Type: text/html; charset=utf-8
Location: /Bledy/Blad404.aspx?aspxerrorpath=/CaptchaType.ashx
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 03 Nov 2012 17:18:55 GMT
What you probably want is a HTTPS request from Ruby if i get this right. Try:
require 'net/https'
url = URI.parse('path')
Net::HTTP.start(url.host, url.port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
res = http.get(url.request_uri)
open("image.png", "wb") do |f|
f.write(res.body)
end
end
For cropping, you can either use chunky_png (pure Ruby) or rmagick (requires ImageMagick)
Edit: If you want to follow redirects you can do
require 'net/https'
def process_image( content )
# do your cropping here
open("image.png", "wb") do |f|
f.write(content)
end
end
def fetch( url )
Net::HTTP.start(url.host, url.port, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
response = http.get(url.request_uri)
case response.code
when Net::HTTPRedirection
fetch response['location']
else
process_image response.body
end
end
end
fetch URI.parse('path')
Related
I am trying to use ruby with a website's api. The instructions are to send a GET request with a header. These are the instructions from the website and the example php code they give. I am to calculate a HMAC hash and include it under an apisign header.
$apikey='xxx';
$apisecret='xxx';
$nonce=time();
$uri='https://bittrex.com/api/v1.1/market/getopenorders?apikey='.$apikey.'&nonce='.$nonce;
$sign=hash_hmac('sha512',$uri,$apisecret);
$ch = curl_init($uri);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('apisign:'.$sign));
$execResult = curl_exec($ch);
$obj = json_decode($execResult);
I am simply using an .rb file with ruby installed on windows from command prompt. I am using net/http in the ruby file. How can I send a GET request with a header and print the response?
Using net/http as suggested by the question.
References:
Net::HTTP https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html
Net::HTTP::get https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html#method-c-get
Setting headers: https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html#class-Net::HTTP-label-Setting+Headers
Net::HTTP::Get https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP/Get.html
Net::HTTPGenericRequest https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTPGenericRequest.html and Net::HTTPHeader https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTPHeader.html (for methods that you can call on Net::HTTP::Get)
So, for example:
require 'net/http'
uri = URI("http://www.ruby-lang.org")
req = Net::HTTP::Get.new(uri)
req['some_header'] = "some_val"
res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == 'https') { |http|
http.request(req)
}
puts res.body # <!DOCTYPE html> ... </html> => nil
Note: if your response has HTTP result state 301 (Moved permanently), see Ruby Net::HTTP - following 301 redirects
Install httparty gem, it makes requests way easier, then in your script
require 'httparty'
url = 'http://someexample.com'
headers = {
key1: 'value1',
key2: 'value2'
}
response = HTTParty.get(url, headers: headers)
puts response.body
then run your .rb file..
As of Ruby 3.0, Net::HTTP.get_response supports an optional hash for headers:
Net::HTTP.get_response(URI('http://www.example.com/index.html'), { 'Accept' => 'text/html' })
Unfortunately this does not work for Ruby 2 (up to 2.7).
I'm trying to scrape some data using the mechanize library in ruby and I have to first get past a "Terms and Conditions" page. To that end I'm clicking an "I agree" button.
require 'mechanize'
agent = Mechanize.new
agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE
agent.get('https://apply.hobartcity.com.au/Common/Common/terms.aspx')
form = agent.page.form_with(:id => 'aspnetForm')
button = form.button_with(:name => 'ctl00$ctMain$BtnAgree')
page = form.submit(button)
But when I run the above code I get this error on the form submission step:
Uncaught exception: unsupported content-encoding: gzip,gzip
When I access that second page with a browser the response headers are
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
X-UA-Compatible: IE=9,10,11
Date: Tue, 16 Feb 2016 22:44:27 GMT
Cteonnt-Length: 16529
Content-Encoding: gzip
Content-Length: 5436
I assume mechanize can work with gzip content encoding, so I'm not sure where the error is coming from. Any ideas what's going on here?
Ruby 2.1.7, mechanize 2.7.4.
I didn't figure out what the actual cause of the problem was, but I was able to work around it by overriding the content-encoding:
agent.content_encoding_hooks << lambda { |httpagent, uri, response, body_io|
response['Content-Encoding'] = 'gzip'
}
agent.submit(form, button)
No more error.
quite new to this been learning about API's in Ruby. Using an Emails service's API to create a user in a system.
This is an example of the POST:
POST http://localhost:8080/core/postgres-pages-xy/api/rest/v4/user/create?email=user003#test.invalid HTTP/1.1
Authorization: Basic bWFzdGVyQGVuubXVjLmVjaXJjbGUuZGU6aDhuc3d1cnN0
User-Agent: curl/7.29.0
Host: localhost:8080
Proxy-Connection: Keep-Alive
Content-Type:application/json
Accept:application/json
Content-Length: 86
[{"name":"user.FirstName","value":"Stan"}, {"name":"user.LastName", "value":"Laurel"}]
I think I am close(ish)? in Ruby was hoping someone would tell me how I send my authentication through. System requires login headers not sure how to do that, will be an email and a password:
require 'uri'
require 'net/http'
uri = URI("https://site.com/api/rest/v4/user/create?email=ruby1#ruby.com")
https = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.path)
request.basic_auth 'email', 'pass'
request["user.FirstName"] = 'Liam'
request["user.LastName"] = 'Coates'
response = https.request(request)
puts response
Thanks for feedback or learnings.
You can enter the credentials in the URL:
url = "http://username:password#localhost:8080/core/postgres-pages-xy/api/rest/v4/user/create"
If the username and password are there, it should automatically do HTTP basic auth (source).
However supposedly this is deprecated, so there is a longer solution:
req = Net::HTTP::Post.new(uri)
req.basic_auth 'user', 'pass'
res = Net::HTTP.start(uri.hostname, uri.port) {|http|
http.request(req)
}
puts res.body
I'd like to send a post from my Rails app to an API. I can get this to work using POSTMAN:
If I click on Preview in POSTMAN, it shows this as the request:
POST /api/users/status HTTP/1.1
Host:
Cache-Control: no-cache
----WebKitFormBoundaryE19zNvXGzXaLvS5C
Content-Disposition: form-data; name="params"
{"pgb": "sample_token", "token": "sample_token" }
----WebKitFormBoundaryE19zNvXGzXaLvS5C
Which is what I want to send. But I can't seem to replicate form-data when using Ruby's Net::HTTP::Post. Here's what I have so far, but this posts with x-www-form-urlencoded as the content-type:
url = URI.parse(ENV['URL'])
req = Net::HTTP::Post.new(url.path)
req.set_form_data({"params" => data.to_json})
https = Net::HTTP.new(url.host, url.port)
https.use_ssl = true
https.verify_mode = OpenSSL::SSL::VERIFY_NONE
https.set_debug_output $stdout
resp = https.request(req)
response = JSON.parse(resp.body)
Is there any way to post just form-data with ruby?
you could try using some of Ruby gems like Rest client. http://rubygems.org/gems/rest-client
Just type the following
gem install rest-client
The documentation can be found here http://rubygems.org/gems/rest-client.
In Ruby, how can I get hold of the HTTP Request Headers that will be sent by a net/http(s) or open-uri request BEFORE it actually makes the request.
In some cases, headers are used when creating a signed string in a URI. Surely there is some way to acquire the request headers that will be sent. These should include the "Host:" header for example.
see http://ruby-doc.org/stdlib-2.0/libdoc/net/http/rdoc/Net/HTTP.html#label-Setting+Headers
Works well in ruby 2.0.0 - but you are correct, different behavior in 1.9.3
Ruby 2.0.0
require 'net/http'
uri = URI('http://github.com/ruby')
http_request = Net::HTTP::Get.new(uri)
http_request.each_header { |header| puts header }
# => accept-encoding
# => accept
# => user-agent
# => host
http_response = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(http_request)
end
Ruby 1.9.3
require 'uri'
require 'net/http'
uri = URI.parse('http://github.com/ruby')
http_request = Net::HTTP::Get.new(uri.path)
http_request.each_header { |header| puts header }
# => accept
# => user-agent
http_response = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(http_request)
end
http_request.each_header { |header| puts header }
# => accept
# => user-agent
# => host
The Net::HTTP classes all seem to use Net::HTTPHeader as a mixin. You should be able to use to_hash() on the request object to get all headers at once, or each_header() / each() to iterate one header at a time.
Here's some code I wrote to help.
def get_request_headers(request)
http_method = request.method
path = request.path
request.to_hash.merge("method" => [http_method]).merge("path" => [path])
end
So now, you can run something like this.
url = URI("http://www.google.com")
request, response = Net::HTTP.start(uri(ico).host) do |http|
request = Net::HTTP::Get.new(uri(ico))
response = http.request request
[request, response]
end
get_request_headers(request)
=> {"accept-encoding"=>["gzip;q=1.0,deflate;q=0.6,identity;q=0.3"], "accept"=>["*/*"], "user-agent"=>["Ruby"], "host"=>["www.google.com"], "method"=>["GET"], "path"=>["/"]}
request.to_hash give us a few headers for free, but there's more information stored in the instance variables for the request and response classes.
With the following code, you can check if there's anything else you'd like to merge into the basic request hash.
request.instance_variables.each do |variable|
puts "#{variable}: #{request.instance_variable_get(variable)}"
end
=> #method: GET
=> #request_has_body: false
=> #response_has_body: true
=> #uri: http://www.google.com
=> #path: /
=> #decode_content: true
=> #header: {"accept-encoding"=>["gzip;q=1.0,deflate;q=0.6,identity;q=0.3"], "accept"=>["*/*"], "user-agent"=>["Ruby"], "host"=>["www.google.com"]}
=> #body:
=> #body_stream:
=> #body_data:
=> [:#method, :#request_has_body, :#response_has_body, :#uri, :#path, :#decode_content, :#header, :#body, :#body_stream, :#body_data]
Note that I've pulled out the method and path for the get_request_headers method.
Finally, you can do the same for the response.
def get_response_headers(response)
code = response.code
http_version = response.http_version
message = response.message
response.to_hash.merge("code" => [code]).merge("http_version" => [http_version]).merge("message" => [message])
end
get_response_headers(response)
=> {"date"=>["Thu, 06 May 2021 14:34:27 GMT"], "expires"=>["-1"], "cache-control"=>["private, max-age=0"], "content-type"=>["text/html; charset=ISO-8859-1"], "p3p"=>["CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\""], "server"=>["gws"], "content-length"=>["6067"], "x-xss-protection"=>["0"], "x-frame-options"=>["SAMEORIGIN"], "set-cookie"=>["NID=215=dYowhmNSD9_CnKYLtsFI3uWVGy8ca8PKJTE8VY6_92q7tU5Y_AOWLsaabXxlSPBjc2QjOr4xXVX5SGMCrccTCnBR9pbdsKkrpVTV5TMqrV6H09ChxGjBr5mHVdZkgjOxswiXu72TF3eAX0uhXqloDb-5gmZ6NJ4w1YDKQKNoDp4; expires=Fri, 05-Nov-2021 14:34:27 GMT; path=/; domain=.google.com; HttpOnly"], "code"=>["200"], "http_version"=>["1.1"], "message"=>["OK"]}