Is there a cURL library for Ruby?
Curb and Curl::Multi provide cURL bindings for Ruby.
If you like it less low-level, there is also Typhoeus, which is built on top of Curl::Multi.
Use OpenURI and
open("http://...", :http_basic_authentication=>[user, password])
accessing sites/pages/resources that require HTTP authentication.
Curb-fu is a wrapper around Curb which in turn uses libcurl. What does Curb-fu offer over Curb? Just a lot of syntactic sugar - but that can be often what you need.
HTTP clients is a good page to help you make decisions about the various clients.
You might also have a look at Rest-Client
If you know how to write your request as a curl command, there is an online tool that can turn it into ruby (2.0+) code: curl-to-ruby
Currently, it knows the following options: -d/--data, -H/--header, -I/--head, -u/--user, --url, and -X/--request. It is open to contributions.
the eat gem is a "replacement" for OpenURI, so you need to install the gem eat in the first place
$ gem install eat
Now you can use it
require 'eat'
eat('http://yahoo.com') #=> String
eat('/home/seamus/foo.txt') #=> String
eat('file:///home/seamus/foo.txt') #=> String
It uses HTTPClient under the hood. It also has some options:
eat('http://yahoo.com', :timeout => 10) # timeout after 10 seconds
eat('http://yahoo.com', :limit => 1024) # only read the first 1024 chars
eat('https://yahoo.com', :openssl_verify_mode => 'none') # don't bother verifying SSL certificate
Here's a little program I wrote to get some files with.
base = "http://media.pragprog.com/titles/ruby3/code/samples/tutthreads_"
for i in 1..50
url = "#{ base }#{ i }.rb"
file = "tutthreads_#{i}.rb"
File.open(file, 'w') do |f|
system "curl -o #{f.path} #{url}"
end
end
I know it could be a little more eloquent but it serves it purpose. Check it out. I just cobbled it together today because I got tired of going to each URL to get the code for the book that was not included in the source download.
There's also Mechanize, which is a very high-level web scraping client that uses Nokogiri for HTML parsing.
Adding a more recent answer, HTTPClient is another Ruby library that uses libcurl, supports parallel threads and lots of the curl goodies. I use HTTPClient and Typhoeus for any non-trivial apps.
To state the maybe-too-obvious, tick marks execute shell code in Ruby as well. Provided your Ruby code is running in a shell that has curl:
puts `curl http://www.google.com?q=hello`
or
result = `
curl -X POST https://www.myurl.com/users \
-d "name=pat" \
-d "age=21"
`
puts result
A nice minimal reproducible example to copy/paste into your rails console:
require 'open-uri'
require 'nokogiri'
url = "https://www.example.com"
html_file = URI.open(url)
doc = Nokogiri::HTML(html_file)
doc.css("h1").text
# => "Example Domain"
Related
I was reading the documentation (which is pretty outdated). Net-ssh does not have .shell method anymore? how can i achieve signing into the shell, run command and get stdout until its done?
Can anyone point me to a good documentation or advice on how i can do it?
Thank you
Net::SSH has been updated a few days ago, and using the first example of the README on the first page of the project did exactly what you wanted to do :
require 'net/ssh'
Net::SSH.start('my_server', 'my_user') do |ssh|
output = ssh.exec!("ls")
puts output
end
#=>
# 20130402_083136_DSCF0923.jpg
# 20160715_113357_DSC_6050.jpg
# 20160715_121646_DSC_2.jpg
...
...
I am using the -O or --output-document option on wget to store get the http from a website. However the -O option requires a file for the output to be stored in and I would like to store it in a variable in my program so that I can manipulate it easier. Is there any way to do this without rereading it in from the file? In essence, I am manually creating a crude cache.
Sample code
#!/usr/bin/ruby
url= "http://www.google.com"
whereIWantItStored = `wget #{url} --output-document=outsideFile`
Reference:
I found this post helpful in using wget within my program: Using wget via Ruby on Rails
#!/usr/bin/ruby
url= "http://www.google.com"
whereIWantItStored = `wget #{url} -O -`
Be sure to sanitize your url to avoid shell injection. The - after -O means standard output, which gets captured by the ruby backticks.
https://www.owasp.org/index.php/Command_Injection explains shell injection.
http://apidock.com/ruby/Shellwords/shellescape For Ruby >=1.9 or the Escape Gem for ruby 1.8.x
I wouldn't use wget. I'd use something like HTTParty.
Then you could do:
require 'httparty'
url = 'http://www.google.com'
response = HTTParty.get(url)
whereIWantItStored = response.code = 200 ? response.body : nil
The following gets me one match:
query = http://0.0.0.0:9393/review?first_name=aoeu&last_name=rar
find = /(?<=(\?|\&)).*?(?=(\&|\z))/.match(query)
When I examine 'find' I get:
first_name=aoeu
I want to match everything between a '?' and a '&', so I tried
find = query.scan(/(?<=(\?|\&)).*?(?=(\&|\z))/)
But yet when I examine 'find' I now get:
[["?", "&"], ["&", ""]]
What do I need to do to get:
[first_name=aoeu][last_name=rar]
or
["first_name=aoeu","last_name=rar"]
?
Use String#split.
query.split(/[&?]/).drop(1)
or
query[/(?<=\?).*/].split("&")
But if your real purpose is to extract the parameters from url, then question and its answer.
Use other module provided by ruby or rails will make your code more maintainable and readable.
require 'uri'
uri = 'http://0.0.0.0:9393/review?first_name=aoeu&last_name=rar'
require 'rack'
require 'rack/utils'
Rack::Utils.parse_query(URI.parse(uri).query)
# => {"first_name"=>"aoeu", "last_name"=>"rar"}
# or CGI
require 'cgi'
CGI::parse(URI.parse(uri).query)
# => {"first_name"=>["aoeu"], "last_name"=>["rar"]}
If you need extract query params from URI, please, check thread "How to extract URL parameters from a URL with Ruby or Rails?". It contains a lot of solutions without using regexps.
We have an automatic build system that spits out packages, regression-tested & wrapped into a neat installer, ready for end-users to d/l & deploy.
We do tracking of end user support requests/bug reports via redmine. So far we uploaded the packages manually to the resp. 'Files' section of the redmine project, via the web interface.
What I'd like to do is to automate this step.
I imagine this would requires a few lines of Ruby to interface with redmine's db. I have zero knowledge about redmine's internals. :)
Basically I want the equivalent of a
mv package-x.y.z.tbz /usr/local/redmine/files/
as a Ruby (or whatever language suits the need) script that creates the right filename and registers the file in redmine's db so it shows up as if it had been uploaded through the Web interface, manually.
Cheers!
I've been frustrated with Redmine about things like this before. But before I go much further: is there a specific reason why you're using the Files section for this? It seems another tool (such as SSH/SFTP for uploading to someplace accessible to HTTP) might be a better fit for your needs. It would also be easily scriptable. Just point people to some constant URL like http://yourcompany.com/productname-current.zip.
If you really need to use Redmine for managing this, you might check out Mechanize: http://mechanize.rubyforge.org/. They should have a RESTful API also, but I've never used it.
I found this post, hope it can help you
Automating packaging and RedMine
I'm a bit late, but I've wrote a Redmine upload tool in Perl, using the WWW::Mechanize module.
Please find it on http://github.com/subogero/redgit
As already stated, you can use Mechanize for that.
There's a Python script written by Gert van Dijk's: https://github.com/gertvdijk/redmine-file-uploader
To use it you'll have to install Python Mechanize package first:
easy_install mechanize
If you prefer Ruby, you can use:
require 'mechanize'
# Replaces \ with / and removes "
ARGV.map!{|a|a.gsub('\\','/').gsub(/^"(.+)"$/,'\\1')}
filename = ARGV[0] || abort('Filename must be specified')
puts "File: #{filename}"
url = ARGV[1] || abort('Redmine URL must be specified')
puts "Redmine URL: #{url}"
username = ARGV[2] || abort('Redmine username must be specified')
puts "Username: #{username}"
password = ARGV[3] || abort('Redmine password must be specified')
puts "Password: #{'*' * password.length}"
project = ARGV[4] || abort('Redmine project must be specified')
puts "Project: #{project}"
login_page_path = '/login'
files_page_path = '/projects/' + project + '/files'
agent = Mechanize.new
# No certificate verification (I had to use this hack because our server is bound to custom port)
# agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE
agent.get(URI.join(url, login_page_path)) do |login_page|
login_page.form_with(:action => login_page_path) do |login_form|
login_form.username = username
login_form.password = password
end.submit
end
agent.get(URI.join(url, files_page_path + '/new')) do |upload_page|
upload_page.form_with(:action => files_page_path) do |upload_form|
upload_form.file_uploads.first.file_name = filename
end.submit
end
And don't forget to install gem first:
gem install mechanize
Something along the lines of:
def domain_exists?(domain)
# perform check
# return true|false
end
puts "valid!" if domain_exists?("example.com")
require 'socket'
def domain_exists?(domain)
begin
Socket.gethostbyname(domain)
rescue SocketError
return false
end
true
end
If you want to check whether a domain is registered or not, then you need to perform a Whois query.
http://www.ruby-whois.org/
With ruby-whois is pretty easy:
Install gem and require.
a = Whois.whois("google.com")
a.available?
=> false
There is also a CLI bundled if you install it via ruby gems: ruby-whois
web page at: ruby-whois.org
You could shell out to nslookup like this:
`nslookup #{domain}`
and parse the results as text with regexes etc.
Or you can use the Socket class, specifically Socket.getaddrinfo. See previous StackOverflow answer on this very question.