Ruby Escaping Arguments Inside Backticks Shell - ruby

I'm wanting to throw a Ruby variable filled with an HTML file that I have grabbed with open-uri and nokogiri, into a backticks system process to tidy it up. The nature of the variable is confusing the process. I am thinking I need to escape it but I am not sure. Any advice appreciated.
require 'open-uri'
require 'nokogiri'
url = 'http://www.wikihow.com/Bathe-a-Cat'
page = Nokogiri::HTML(open(url))
pagestring = page.to_s
result = `tidy --break-before-br no --char-encoding utf8 --clean yes --drop-empty-paras yes ' #{pagestring}'`
puts results.length
Here is the error I get:
sh: -c: line 144: syntax error near unexpected token `"Search","Search","Custom_search"'
sh: -c: line 144: ` <input type="submit" id="cse_sa" value="Search" class="search_button" onmouseover="button_swap(this);" onmouseout="button_unswap(this);" onclick='gatTrack("Search","Search","Custom_search");'>'
Cheers

You might want to use IO.popen instead. Then you can invoke the command with an array instead of stringifying it:
cmd = %w{ tidy --break-before-br no --char-encoding utf8 --clean yes --drop-empty-paras yes }
result = IO.popen(cmd, 'r+') {|io|
io.puts pagestring
io.close_write
io.read
}
assuming tidy reads HTML from stdin.

Instead of dumping all that HTML onto the command line, why not make a file?
require 'open-uri'
require 'nokogiri'
require 'tempfile'
url = 'http://www.wikihow.com/Bathe-a-Cat'
page = Nokogiri::HTML(open(url))
pagestring = page.to_s
file = Tempfile.new('blah')
file.write(pagestring)
file.close
result = `tidy --break-before-br no --char-encoding utf8 --clean yes --drop-empty-paras yes #{file.path}`
puts result.length
file.unlink
Seems to work with a quick test here...

For normal arguments like file paths and stuff like that, you could use "str".shellescape (http://apidock.com/ruby/Shellwords/shellescape).
args_array = [ ... ]
`tidy #{args_array.map(&:shellescape).join(' ')`
However, to pass a complete html file as an command line argument, something like what was suggested above might be better. I just though I'd mention this here for reference to others for normal cli arguments.

Related

Parse multiple command line options in Ruby using OptionParser

I've just started using OptionParser for Ruby and I wanted to use flags that would use more than just one argument.
For instance, I would like to be able to run:
script --move src dst
Note how src and dst are not separated using a coma.
My initial idea was:
opts.on("-m src dst", "--move src dst ", "move file from SRCto DST") do |src|
# do something
end
But this is not working. I assume that this is not the right approach. But how could this be done?
The example under the "Complete Example" section of the OptionParser details how a list of items can be accepted.
Here is a sample program based on that example. The third parameter Array in opts.on indicates that input src, dst should be used to create an array. To run this sample, you need to do gem install trollop.
# test.rb
require 'optparse'
options = {}
OptionParser.new do |opt|
opt.on("-m src, dst", "--move src, dst", Array, "Move from src to dst") do |list|
options[:src] = list[0]
options[:dst] = list[1]
end
end.parse!
puts options # It's a hash of parsed options
Sample run:
> ruby test.rb -m from,to
{:src=>"src", :dst=>"dst"}
>ruby test.rb -h
Usage: test [options]
-m, --move src, dst Move from src to dst
The above script forces one to separate the options using comma.
As indicated by "Really Cheap Command-Line Option Parsing in Ruby", there seems to be a gem, trollop, that can be quite easy to use for command-line parsing.
A sample program based on Trollop is given below, which allows usage of spaces for specifying options with multiple values
# test.rb
require "trollop"
opts = Trollop::options do
banner "Command line parsing using Trollop"
opt :move, "--move src dst', Move from src to dst", :short => "-m", :long => "--move", :type => :strings
end
# An array of option values
p opts.move
Sample run:
>ruby test.rb -m hello world
["hello", "world"]
>ruby test.rb -h
Command line parsing using Trollop
-m, --move=<s+> '--move src dst', Move from src to dst
-h, --help Show this message
There is a subtle difference between the help output between the two approaches. Trollop produces help text where --move=<s+> does not indicate clearly that it needs accepts two values, so I had to repeat the command syntax description.
OptionParser doesn't support that; It could be patched to do so, but I'm not sure it's worth the trouble.
Consider this code:
require 'optparse'
options = {}
OptionParser.new do |opt|
opt.on('-m', '--move') { |o| options[:move] = o }
end.parse!
from_name, to_name = ARGV
puts "Should move: #{ options.key?(:move) }"
puts "From: #{ from_name }"
puts "To: #{ to_name }"
Saving it and running it with various combinations of the parameters returns:
> ruby test.rb --move from to
Should move: true
From: from
To: to
> ruby test.rb from to
Should move: false
From:
To:
If the code is supposed to move files by default then don't bother with the --move flag, simply use:
test.rb from to
and consider removing the OptionParser block entirely.
If the code is supposed to normally copy with the option to move, then --move becomes more sensible to act as a flag that moving is desired.
ruby test.rb --move from to
I'd have code that tests for options[:move] and run the code to move instead of copy at that point.
In either case, the filenames shouldn't be tied to the flag, they should be supplied separately and retrieved from ARGV after OptionParser has finished parsing the command-line and removing entries it's handled.

OptionParser to parse arguments form file instead of command line

I am using Ruby to execute a code that takes command line arguments.
now i trying to use the same program with differnt options so i am putting the options in a file and i want the program to read each line interpret the options and execute the program accordingly.
but i get this error. "C:/Ruby193/lib/ruby/1.9.1/optparse.rb:1348:in block in parse_in_order': undefined methodshift' for "--c execue --query unix --Servername abc123":String (NoMethodError)"
i understand that its reading the file and treating the line as string. but wondering if there is a way to overcome this shift error and treat the line as if it was entered in command prompt. or any better solution.
here is my code.
require 'optparse'
require 'micro-optparse'
# --command execue --query unix command --Servername abc123
f =File.open("list_of_commands.txt", "r")
f.each_line { |line|
line= line.chomp
#line = "--c execue --query unix --Servername abc123"
#line = eval("\"#{line}\"")
puts line
options = {}
OptionParser.new do |opts|
opts.on("-c", "--command result,execue,chart,scpfile", String, "Single command to execute ") do |c|
options[:comd] = c
end
opts.on("-q", "--query remote command, unix command", String, "performs the command on local or remote machine") do |q|
options[:query] = q
end
opts.on("-s", "--servername CHSXEDWDC002 ", String, "server name to execute the command") do |v|
options[:hname] = v
end
opts.on_tail('-h', '--help', 'Show this message') do
puts opts
exit
end
end.parse!(line)
p options
}
the contents of the file are below
--c execue --query unix --Servername abc123
i also tried to use micro-optparse but facing same error. any workaround ?
Update:
as per suggestion from "#mu is too short" i tried below options.
end.parse!("#{Shellwords.shellsplit(line)}") and/or
end.parse!(Shellwords.shellsplit(line)).
but none of them worked.
i also tried to split the line as array using "line = line.split("\t")" and then
end.parse!(line). out put as
--c execue
--query unix
--Servername abc123
but now i get error as block in : invalid option --c execute
Update:#2
looking at the error, the issue is with the wrong parameter(-c. but thanks to user "#mu is too short" for suggesting to use Array.
Update: 3
passing the array only worked for short form of the arguments such as -c but when long form was supplied it failed with invalid argument erorr.
i dont see much documentation on the optparse. i even tried micro-parse but it requres default valuves and its not an option for me :(
The parse! method wants an array as its argument, not a string. You'll probable want to use Shellwords.shellsplit rather than String#split (or similar hand-rolled method) to convert your line to an array just in case you have to deal with quoting and whatnot. Something like this:
OptionParser.new do |opts|
#...
end.parse!(Shellwords.shellsplit(line))
While you can put your command-line arguments into a file, flags and all, there are better ways to remember configuration settings.
Instead of storing the flags, use a YAML file. YAML is a great data format, that translates easily to Ruby hashes and objects. "Yaml Cookbook" is a very useful page for learning the ins and outs of the format with Ruby. There are YAML parsers for a myriad other languages, making it easy to share the settings, which can be useful as a system grows.
With a little creative code you can use your YAML as the base settings, and let your CLI flags override the stored settings.
If you're not familiar with YAML, it's easy to get a start on the file using something like:
require 'yaml'
data = {
'command' => %w[result execute chart scpfile],
'query' => ['remote command', 'unix command'],
'servername' => 'CHSXEDWHDC002',
}
puts data.to_yaml
Which outputs:
---
command:
- result
- execute
- chart
- scpfile
query:
- remote command
- unix command
servername: CHSXEDWHDC002
Redirect that output to a file ending in .yaml and you're on your way.
To read it back into a script use:
require 'yaml'
data = YAML.load_file('path/to/data.yaml')
A quick round-trip test shows:
require 'yaml'
data = {
'command' => %w[result execute chart scpfile],
'query' => ['remote command', 'unix command'],
'servername' => 'CHSXEDWHDC002',
}
YAML.load(data.to_yaml)
Which looks like:
{"command"=>["result", "execute", "chart", "scpfile"],
"query"=>["remote command", "unix command"],
"servername"=>"CHSXEDWHDC002"}
If you want to have defaults, stored in the YAML file, and override them with command-line flags, read the data from the file then use that resulting object as the base for OptionParse:
require 'optparse'
require 'yaml'
# Note, YAML can deal with symbols as keys, but other languages might not like them.
options = {
:comd => %w[result execute chart scpfile],
:query => ['remote command', 'unix command'],
:hname => 'CHSXEDWHDC002',
}
# we'll overwrite the options variable to pretend we loaded it from a file.
options = YAML.load(options.to_yaml)
OptionParser.new do |opts|
opts.on("-c", "--Command result,execue,chart,scpfile", String, "Single command to execute ") do |c|
options[:comd] = c
end
opts.on("-q", "--query remote command, unix command", String, "performs the command on local or remote machine") do |q|
options[:query] = q
end
opts.on("-s", "--Servername CHSXEDWHDC002 ", String, "server name to execute the command") do |v|
options[:hname] = v
end
opts.on_tail('-h', '--help', 'Show this message') do
puts opts
exit
end
end.parse!
That's not tested, but we do similar things at work all the time, so save it to a file and poke at it with a stick for a while, and see what you come up with.

File Creation/Loop Problems in Ruby

EDIT: My original question was way off, my apologies. Mark Reed has helped me find out the real problem, so here it is.
Note that this code works:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
source_url = "www.flickr.com"
puts "Visiting #{source_url}"
page = Nokogiri::HTML(open("http://website/script.php?value=#{source_url}"))
textarea = page.css('textarea')
filename = source_url.to_s + ".txt"
create_file = File.open("#{filename}", 'w')
create_file.puts textarea
create_file.close
Which is really awesome, but I need it to do this to ~110 URLs, not just Flickr. Here's my loop that isn't working:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
File.open('sources.txt').each_line do |source_url|
puts "Visiting #{source_url}"
page = Nokogiri::HTML(open("http://website/script.php?value=#{source_url}"))
textarea = page.css('textarea')
filename = source_url.to_s + ".txt"
create_file = File.open("#{filename}", 'w')
create_file.puts "#{textarea}"
create_file.close
end
What am I doing wrong with my loop?
Ok, now you're looping over the lines of the input file. When you do that, you get strings that end in a newilne. So you're trying to create a file with a newline in the middle of its name, which is not legal in Windows.
Just chomp the string:
File.open('sources.txt').each_line do |source_url|
source_url.chomp!
# ... rest of code goes here ...
You can also use File#foreach instead of File#open.each_line:
File.foreach('sources.txt') do |source_url|
source_url.chomp!
# ... rest of code goes here
You're putting your parentheses in the wrong place:
create_file = File.open(variable, 'w')

Ruby Dir.mkdir Usage

I am pretty new to ruby and have a very simple ruby script that has the following purpose:
Read lines of file
Access jira instance using jira4r gem
Query jira instance for issue(s)
Create a directory using the issue key and issue summary
I've come to the conclusion after some tinkering that the Dir.mkdir command does not accept the object I am passing it as argument.
Findings:
If Dir.mkdir is passed a line, #{chompline}, from my textfile directory creation execute properly.
If Dir.mkdir is passed a string consisting of issue.key and issue.summary it chokes with the following error:
./readFile.rb:29:in `mkdir': No such file or directory - (Errno::ENOENT)
from ./readFile.rb:29
Based on point #1 and #2, it must be something about the string I create from issue key and summary.
I have the following theories/questions:
Is "#{keyPlusSummary}"the correct object type to pass into mkdir as argument ?
I believe it to be string, but perhaps I am assuming incorrectly.
Source:
#!/usr/bin/env ruby
require 'rubygems'
require 'jira4r'
require 'FileUtils'
jira = Jira4R::JiraTool.new(2, "http://jira.somejirainstance.com")
baseurl = jira.getServerInfo().baseUrl
puts "Base URL: " + baseurl , "\n"
jira.login("someUser", "somePassword")
file = File.new("awkOutput.txt", "r")
while (line = file.gets)
chompline = "#{line}".chomp!
issue = jira.getIssue("#{chompline}")
keyPlusSummary = "#{issue.key}"+"#{issue.summary}"
puts keyPlusSummary
Dir.mkdir "#{keyPlusSummary}"
end
file.close
It's a string, but you don't tell us what's in it.
# More canonical, both in var naming, and there's
# no need for concatenation in this case.
dir_name = "#{issue.key}#{issue.summary}"
Are you making the string "directory-name friendly"?
I would not use a JIRA issue summary as a directory name; IMO just the project/issue # would be enough. If you do use the summary, make it something that's directory-friendly by stripping out anything non-alphanumeric, and replacing spaces with underscores.
keyPlusSummary is a string, so it is of the right type. What may be the problem is slashes in the string. Like mkdir in UNIX, Dir.mkdir will not create parent directories for you, it will only create a single directory. If the key + summary includes a '/', then it will read it as a multi-level directory. You need to either escape the '/', or (better), use FileUtils.mkdir_p, or (best) do cleanup to replace ' ' with '_', and remove special characters that make using the directory harder :)
As an aside, your code doesn't need to have the interpolations it does:
#!/usr/bin/env ruby
require 'rubygems'
require 'jira4r'
require 'FileUtils'
jira = Jira4R::JiraTool.new(2, "http://jira.somejirainstance.com")
baseurl = jira.getServerInfo().baseUrl
puts "Base URL: #{baseurl}\n" #use it here!
jira.login("someUser", "somePassword")
File.new("awkOutput.txt", "r") do |file| #using the block form to ensure you close the file
while (line = file.gets)
chompline = line.chomp! #line is already a string, no need to interpolate
issue = jira.getIssue(chompline) #line is already a string, no need
keyPlusSummary = "#{issue.key}#{issue.summary}" #already interpolating, no need to add
puts keyPlusSummary
Dir.mkdir keyPlusSummary #already a string
end
end

Moving a file containing a space in ruby using FileUtils

I'm using Mac OS X and I'm trying to write a little script that moves a file to a specific folder. I'm using the FileUtils API since I don't want to run system specific commands (system("mv a b").
The script looks something like this:
#!/usr/bin/env ruby
require 'rubygems'
require 'escape'
require 'fileutils'
absolut_input_filename = Escape.shell_single_word ARGV[0]
move_folder = Escape.shell_single_word "/move/to/folder"
FileUtils.mv absolut_input_filename, move_folder
As long as the input filename contains no space, everything works fine. But as soon as I put in a file with a space the error output is something like this:
./scripts/success /path/to/file\ with\ space
/usr/local/Cellar/ruby/1.9.2-p0/lib/ruby/1.9.1/fileutils.rb:1418:in `stat': No such file or directory - '/path/to/file with space' (Errno::ENOENT)
from /usr/local/Cellar/ruby/1.9.2-p0/lib/ruby/1.9.1/fileutils.rb:1418:in `block in fu_each_src_dest'
from /usr/local/Cellar/ruby/1.9.2-p0/lib/ruby/1.9.1/fileutils.rb:1432:in `fu_each_src_dest0'
from /usr/local/Cellar/ruby/1.9.2-p0/lib/ruby/1.9.1/fileutils.rb:1416:in `fu_each_src_dest'
from /usr/local/Cellar/ruby/1.9.2-p0/lib/ruby/1.9.1/fileutils.rb:504:in `mv'
from ./scripts/success:8:in `<main>'
For escaping I use the 'escape' gem in version 0.0.4 in which the shell_single_word looks like this:
def shell_single_word(str)
if str.empty?
"''"
elsif %r{\A[0-9A-Za-z+,./:=#_-]+\z} =~ str
str
else
result = ''
str.scan(/('+)|[^']+/) {
if $1
result << %q{\'} * $1.length
else
result << "'#{$&}'"
end
}
result
end
end
you can just not use escape
require 'fileutils'
absolut_input_filename = ARGV[0]
move_folder = "/move/to/folder"
FileUtils.mv absolut_input_filename, move_folder
I don't actually know from Ruby, so take this with a grain of salt, but I know the underlying OS primitives inside and out, and from C you can do this with rename(2). Therefore, from Ruby, you should be able to do this with File.rename, which requires no quoting at all. Try this:
#! /usr/bin/env ruby
tomove = ARGV[0]
target = "/path/to/target/folder"
File.rename(tomove, File.join(target, File.basename(tomove)))
Solved using soft links:
ln -s ~/Folder\ with\ spaces/foo/ ./foo
now i can use FileUtils commands without problems:
FileUtils.cp("bar.txt", "foo/foobar.txt")
Hope will help!

Resources