I have some code that looks like this:
while response.droplet.status != env["user_droplet_desired_state"] do
sleep 2
response = ocean.droplet.show env["droplet_id"]
say ".", nil, false
end
The idea being you can set the app to wait until the server is in a certain state (eg. restart it, then watch it until it's active again)
However, I'm using webmock in the tests, and I can't figure out a way to give a different response the second time.
For example, code like this:
stub_request(:get, "https://api.digitalocean.com/v2/droplets/6918990?per_page=200").
with(:headers => {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Authorization'=>'Bearer foo', 'Content-Type'=>'application/json', 'User-Agent'=>'Faraday v0.9.2'}).
to_return(:status => 200, :body => fixture('show_droplet_inactive'), :headers => {})
stub_request(:get, "https://api.digitalocean.com/v2/droplets/6918990?per_page=200").
with(:headers => {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Authorization'=>'Bearer foo', 'Content-Type'=>'application/json', 'User-Agent'=>'Faraday v0.9.2'}).
to_return(:status => 200, :body => fixture('show_droplet'), :headers => {})
With the idea being "First time mark as in-active so the loop goes through one-time, then mark as active afterwards"
The documentation says that stubs are just done as "Last one found will work":
Always the last declared stub matching the request will be applied
i.e:
stub_request(:get, "www.example.com").to_return(:body => "abc")
stub_request(:get, "www.example.com").to_return(:body => "def")
Net::HTTP.get('www.example.com', '/') # ====> "def"
Is it possible to model multiple calls to the same endpoint with different results in webmock?
If you pass multiple arguments to #to_return, it will respond each time with the next response, and then just keep returning the last one over and over. For example:
require 'webmock/rspec'
require 'uri'
describe "something" do
it "happens" do
stub_request(:get, 'example.com/blah').
to_return({status: 200, body: 'ohai'}, {status: 200, body: 'there'})
puts Net::HTTP.get(URI('http://example.com/blah'))
puts Net::HTTP.get(URI('http://example.com/blah'))
puts Net::HTTP.get(URI('http://example.com/blah'))
puts Net::HTTP.get(URI('http://example.com/blah'))
end
end
When run as rspec <file>, this will print:
ohai
there
there
there
Related
I’m making requests to the Reddit API. First, I set a subreddit top URL:
reddit_url = URI.parse('https://www.reddit.com/r/pixelart/top.json')
All of these correctly get the contents:
Net::HTTP.get(reddit_url, 'User-Agent' => 'My agent')
Open3.capture2('/usr/bin/curl', '--user-agent', 'My agent', reddit_url.to_s)[0]
URI.open(reddit_url, 'User-Agent' => 'My agent').read
But then I try it with a URL for a specific post:
reddit_url = URI.parse('https://reddit.com/r/PixelArt/comments/lkaiqf/another_watercolour_pixelart_tree.json')
And both Net::HTTP and Open3/curl fail, getting only empty strings. URI.open continues to work, as does opening the URL in a web browser.
Why doesn’t the second request work with two of the solutions? And why does it work with URI.open, when that’s supposed to be “an easy-to-use wrapper for Net::HTTP”? What does it do differently, and how to replicate it with Net::HTTP an curl?
Working with your example, and focussing on Net::HTTP for simplicity, the first example doesn't work as written:
require 'net/http'
reddit_url = URI.parse('https://www.reddit.com/r/pixelart/top.json')
Net::HTTP.get(reddit_url, 'User-Agent' => 'My agent')
# => Type Error - no implicit conversion of URI::HTTPS into String
Instead I used this as my starting point:
require 'net/http'
reddit_url = URI.parse('https://www.reddit.com/r/pixelart/top.json')
http = Net::HTTP.new(reddit_url.host, reddit_url.port)
http.use_ssl = true
result = http.get(reddit_url.request_uri, 'User-Agent' => 'My agent')
puts result
# => #<Net::HTTPOK:0x00007fc3ea8e7320>
puts result.body.size
# => 167,394
With that working we can try the second URL. Interestingly, I get different results depending on whether I re-use the initial connection or make a new one:
require 'net/http'
reddit_url = URI.parse('https://www.reddit.com/r/pixelart/top.json')
reddit_url_two = URI.parse('https://reddit.com/r/PixelArt/comments/lkaiqf/another_watercolour_pixelart_tree.json')
http = Net::HTTP.new(reddit_url.host, reddit_url.port)
http.use_ssl = true
result = http.get(reddit_url.request_uri, 'User-Agent' => 'My agent')
puts result
# => #<Net::HTTPOK:0x00007f931a143390>
puts result.body.size
# => 174,615
http_two = Net::HTTP.new(reddit_url_two.host, reddit_url_two.port)
http_two.use_ssl = true
result_two = http_two.get(reddit_url_two.request_uri, 'User-Agent' => 'My agent')
puts result_two
# => #<Net::HTTPMovedPermanently:0x00007f931a148818>
puts result_two.body.size
# => 0
result_reusing_connection = http.get(reddit_url_two.request_uri, 'User-Agent' => 'My agent')
puts result_reusing_connection
# => #<Net::HTTPOK:0x00007f931a0fb3b0>
puts result_reusing_connection.body.size
# => 141,575
So I suspect you're getting a 301 redirect sometimes and that's causing the confusion. There's another question and answer here for how to follow redirects.
With Ruby, my app:
checks if the page status is 200
Parses the PDF files if so
sends via email the result of scraping
Having tested all the parts of the code, everything works fine, except one thing, the mail that is sent doesn't contain the result of my scrpaing;
What is the issue, is it related to the variable #monscrape that may be not recongnised in the final party of the code ?
My code:
require 'open-uri'
require "net/http"
require 'rubygems'
require 'pdf/reader'
require 'mail'
options = { :address => "smtp.gmail.com",
:port => 587,
:domain => 'gmail.com',
:user_name => 'mail#gmail.com',
:password => 'pwd',
:authentication => 'plain',
:enable_starttls_auto => true
}
lien= "http://www.example.com"
url = URI.parse(lien)
req = Net::HTTP.new(url.host, url.port)
res = req.request_head(url.path)
if res.code == "200"
io = open('http://www.example.com')
reader = PDF::Reader.new(io)
reader.pages.each do |page|
res = page.text
#monscrape = res.scan(/text[\s\S]*text/)
end
Mail.defaults do
delivery_method :smtp, options
end
Mail.deliver do
to 'mail#hotmail.com'
from 'Author <mail#gmail.com>'
subject 'testing sendmail'
html_part do
content_type 'text/html; charset=UTF-8'
body '<h1>Please find below the scrape <%= #monscrape %></h1>'
end
end
else
puts "the link doenst work"
end
The problem is the Mail.deliver block is evaluated using instance_eval. Therefore no local instance #variables will be visible to the Mail block.
So #monscrape will always be nil inside the Mail.deliver block.
One solution is to use a local (non-instance) variable instead:
monscrape = "test"
Mail.deliver do
...
body "<h1>Please find below the scrape #{monscrape}</h1>"
...
end
Also note that Mail does not support ERB(!) therefore you cannot use something like <%= monscrape %> in the body. You have to treat it like a normal string using string expansion with double quotes " and not single quotes '.
See further discussion and options here:
Why can't the Mail block see my variable?
You can't use
res = req.request_head(url.path)
when url.path returns "". request_head expects a path of at least "/". That implies you need to fix up the URL being passed so it at least has the root path "/".
url = URI.parse('http://www.example.com')
url.path # => ""
req.request_head(url.path)
*** ArgumentError Exception: HTTP request path is empty
vs.
url = URI.parse('http://www.example.com/')
url.path # => "/"
req.request_head(url.path)
#<Net::HTTPOK 200 OK readbody=true>
The second problem is you're trying to read something as PDF that isn't a PDF file. Example.com returns HTML, which is text. You can't use:
io = open('http://www.example.com')
reader = PDF::Reader.new(io)
Trying to returns "PDF does not contain EOF marker".
It's really important that you understand what types of objects/resources are being returned by a site when you request a URL. You can't declare them willy-nilly and expect code to accept it without errors.
I'm trying to use webmock with rspec to stub out requests to Aws but I can't seem to get the regex to work for SQS polling. If I run rspec, webmock generates a 'correct' stub for me to use in a before(:each) block, in my spec_helper.rb like this:
You can stub this request with the following snippet:
stub_request(:post, "https://sqs.us-west-2.amazonaws.com/123456789012/backlog").
with(:body => "Action=ReceiveMessage&AttributeName.1=All&MaxNumberOfMessages=1&MessageAttributeName.1=All&QueueUrl=https%3A%2F%2Fsqs.us-west-2.amazonaws.com%2F123456789012%2Fbacklog&Version=2012-11-05&VisibilityTimeout=0&WaitTimeSeconds=20",
:headers => {'Accept'=>'*/*', 'Accept-Encoding'=>'', 'Authorization'=>'AWS4-HMAC-SHA256 Credential=MY_ACCESS_KEY/20150726/us-west-2/sqs/aws4_request, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=large_alpha-numeric-signature', 'Content-Length'=>'224', 'Content-Type'=>'application/x-www-form-urlencoded; charset=utf-8', 'Host'=>'sqs.us-west-2.amazonaws.com', 'User-Agent'=>'aws-sdk-ruby2/2.1.7 ruby/2.2.2 x86_64-darwin14', 'X-Amz-Content-Sha256'=>'69336339ae76cf370477d4dsaf667as0b5dd8d25762c7c78sad8a', 'X-Amz-Date'=>'20150726T143009Z'}).
to_return(:status => 200, :body => "", :headers => {})
So in my spec_helper.rb I have
RSpec.configure do |config|
config.before(:each) do
stub_request(:post, "https://sqs.us-west-2.amazonaws.com/123456789012/backlog").
with(:body => "Action=ReceiveMessage&AttributeName.1=All&MaxNumberOfMessages=1&MessageAttributeName.1=All&QueueUrl=https%3A%2F%2Fsqs.us-west-2.amazonaws.com%2F123456789012%2Fbacklog&Version=2012-11-05&VisibilityTimeout=0&WaitTimeSeconds=20",
:headers => {'Accept'=>'*/*',
'Accept-Encoding'=>'',
'Authorization'=>"AWS4-HMAC-SHA256 Credential=MY_ACCESS_KEY/20150726/us-west-2/sqs/aws4_request, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=" + /"^[a-zA-Z0-9]*$"/,
'Content-Length'=>'224',
'Content-Type'=>'application/x-www-form-urlencoded; charset=utf-8',
'Host'=>'sqs.us-west-2.amazonaws.com',
'User-Agent'=>'aws-sdk-ruby2/2.1.7 ruby/2.2.2 x86_64-darwin14',
'X-Amz-Content-Sha256'=>'694236339ae76cf370477d4dsaf667as0b5dd8d25762c7c78sad8a',
'X-Amz-Date'=>""+ /"^[a-zA-Z0-9]*$"/}).
to_return(:status => 200, :body => "", :headers => {})
end
The areas I'm trying to use regex against are the Signature and the X-Amz-Date because they're the only two that seem to change between different attempts to run the rspec.
The problem is the regex seems to not be working because even though I've added it into the spec_helper.rb, every time I run the suite, I get back the recommended stub from webmock instead of a passing or failing test. It should be passing at this point, from what I understand from the webmock docs and several tutorials.
How should I change this to get webmock to work for my test suite against Aws SQS polling?
I've been bashing my head against my desk for a few days now so any help is much appreciated.
Signature is likely generated using a byproduct of Time.now and I'm guessing you don't actually want to test for that. Instead simply do:
stub_request(:post, "https://sqs.us-west-2.amazonaws.com/123456789012/backlog").and_return(:status => 200, :body => "", :headers => {})
If you want to be even less specific on the URL (like ommiting that ID) you can even use a regex match:
stub_request(:post, /amazonaws.com/).and_return(:status => 200, :body => "", :headers => {})
I have the same problem and have the solution only for 'X-Amz-Date'.
Since it is date in special format, use Timecop.freeze block around your mock and method call.
Timecop.freeze do
stub_request(:get, "https://s3.eu-central-1.amazonaws.com/test_bucket/test").
with(headers: {'Accept'=>'*/*', 'Accept-Encoding'=>'', 'Authorization'=>'AWS4-HMAC-SHA256 Credential=access_key/20190814/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9e6a6a209a0cb05346a058c12ef706ebff185ae3b72f3e542e1becbc97e8ea7a', 'Content-Length'=>'0', 'Content-Type'=>'', 'Host'=>'s3.eu-central-1.amazonaws.com', 'User-Agent'=>'aws-sdk-ruby2/2.9.44 ruby/2.6.3 x86_64-darwin18 resources', 'X-Amz-Content-Sha256'=>'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'X-Amz-Date'=>"#{time.utc.iso8601(0).gsub(/[^\p{Alnum}]/, '')}"}).
to_return(status: 200, body: "", headers: {})
AwsService.bucket.object("test").get
end
The regexp leaves only alphanumerical characters and Timecop takes care of the time.
I have to upload multiple files as form request. I am using the Rest Client to post my request. I am able to upload single file but I am not sure how to add multiple files in a single request.
I searched/googled for such option and I am not finding any solution that solves my problem.
Below is my code.
It has variable argument (*yamlfile) which takes one or more files. I have to upload all the files together.
The issue now is , I am getting syntax error when I add the loop to extract the file within the payload.
my assumption is now to form this outside the payload and include it inside the payload block but I am not sure how to do it.
Can someone help me with that.
( I have tried net/http/post/multipart library too and I don't find much documents around it)
def uploadRest(endpoint,archive_file_path,,yaml_file_path,*yamlfile)
$arg_len=yamlfile.length
request = RestClient::Request.new(
:method => :post,
:url => endpoint,
:payload => {
:multipart => true,
:job_upload_archive => File.new(archive_file_path,'rb'),
:job_upload_path => "/tmp",
# Trying to add multiple file, but I get syntax error
yamlfile.each_with_index { |yaml, index|
:job_upload_yaml_file+index => File.new("#{yaml_file_path}/#{pmml}")
}
})
response=request.execute
puts response.code
end
uploadRest(endpoint,archive_file_path,yaml_file_path,*yamlfile)
#files=Array.new
yamlfile.each{ |yaml_file|
#files.push(File.new("#{yaml_file_path}/#{yaml_file}"))
}
request = RestClient::Request.new(
:method => :post,
:url => endpoint,
:payload => { :multipart => true, :job_upload_archive => File.new(archive_file_path,'rb'),
:job_upload_path => "/tmp", :job_upload_yaml_file => #files })
response=request.execute
end
I had a similar problem and was able to get this to work by passing an array of arrays as a requests.
file1 = File.new("#{yaml_file_path}/#{yaml_file1}", 'rb')
file2 = File.new("#{yaml_file_path}/#{yaml_file}", 'rb')
request_body = [["files", file1], ["files", file2]]
RestClient.post url, request_body, request_headers
There were two issues with your question code:
1) Attempt to add a symbol to an integer
2) Attempt to insert contents of yamlfile direct into the hash (because that is what yamlfile.each_with_index returns, as opposed to how it calls your block. The return value from the block is not used)
Both of these code issues read as if you have gained experience in HAML or another templating language, and are using structures/ideas that would work in that?
There are lots of possble solutions in Ruby, but a simple approach to build up the hash in parts, as opposed to generate it in one go with clever hash-returning routines embedded. Try something like this:
payload_hash = {
:multipart => true,
:job_upload_archive => File.new(archive_file_path,'rb'),
:job_upload_path => "/tmp",
}
# This does not use the return value from each_with_index, instead it relies
# on the block to make changes to the hash by adding new key/value pairs
yamlfile.each_with_index { |yaml, index|
# This concatenates two strings, and then converts the combined
# string into the symbol that you want
file_key = ("job_upload_yaml_file"+index.to_s).to_sym
payload_hash[file_key] = File.new("#{yaml_file_path}/#{yaml}")
}
request = RestClient::Request.new(
:method => :post,
:url => endpoint,
:payload => payload_hash
)
For added code cleanliness, you could make the first two parts a separate method, and call it where it currently has payload_hash.
This should get you over current syntax hurdles. However, I have made no attempt to check whether this will allow you to upload multiple files via RESTClient.
Section1:
#params = {
"FacialImage" => UploadIO.new(File.new('C:\temp\ATDD\Test\test\sachin.jpg'), "image/jpeg"),
"Login" => UploadIO.new(File.new('C:\temp\ATDD\Test\test\login.txt'), "application/json")
}
I need to send a POST request as an XML string but I get odd results. The code:
require 'rest_client'
response = RestClient.post "http://127.0.0.1:2000", "<tag1>text</tag1>", :content_type => "text/xml"
I expect to receive "<tag1>text</tag1>" as the parameter on the request server. Instead, I get "tag1"=>"text". It converts the XML to a hash. Why is that? Any way around this?
Try this:
response = RestClient.post "http://127.0.0.1:2000",
"<tag1>text</tag1>",
{:accept => :xml, :content_type => :xml}
I think you just needed to specify the ":accept" to let it know you wanted to receive it in the XML format. Assuming it's your own server, you can debug on the server and see the request format used is probably html.
Hope that helps.
Instead of using RestClient, use Ruby's built-in Open::URI for GET requests or something like Net::HTTP or the incredibly powerful Typhoeus:
uri = URI('http://www.example.com/search.cgi')
res = Net::HTTP.post_form(uri, 'q' => 'ruby', 'max' => '50')
In Typhoeus, you'd use:
res = Typhoeus::Request.post(
'http://localhost:3000/posts',
:params => {
:title => 'test post',
:content => 'this is my test'
}
)
Your resulting page, if it's in XML will be easy to parse using Nokogiri:
doc = Nokogiri::XML(res.body)
At that point you'll have a fully parsed DOM, ready to be searched, using Nokogiri's search methods, such as search and at, or any of their related methods.