Struggling with string interpolation while executing a shell command in ruby. Can you someone please help me identify what I'm missing here?
My one-liner ruby code follows redirects of any shortURL and returns the final URL. For example, this ruby code works perfectly fine.
curl -I https://bit.ly/1mJk8X7 | perl -n -e '/^Location: (.*)$/ && print "$1\n"'
It prints out the final URL.
I have a .txt file with a series of short URLs from which I'd like to derive a list of the final URLs. Say, it's called shortURLs.txt. I'm using IO.foreach to loop through each line in the file, but I don't know what I'm doing wrong to bring the variable 'x' into the ruby command. This is my first time working with string interpolation, and I've tried various combinations of it, but no luck yet.
IO.foreach("shortURLs.txt") { |x| system "curl -I #{x} | perl -n -e '/^Location: (.*)$/ && print \"$1\n\"' >> finalURLs.txt" }
I get an error message around the pipe '|' symbol:
sh: -c: line 1: syntax error near unexpected token|'
sh: -c: line 1: | perl -n -e '/^Location: (.*)$/ && print "https://bit.ly/1mJk8X7'
Other threads have been useful about string interpolation and running shell commands through ruby.
In order to pass two commands to a shell, you should run the system command twice (check method 8 in this post)
require 'shell'
sh = Shell.new
IO.foreach("shortURLs.txt") { |x| sh.system("curl -I #{x}") | sh.system("perl -n -e '/^Location: (.*)$/ && print \"$1\n\" ' ") >> "finalURLs.txt" }
IO.foreach yields the lines including the new line at the end so you're actually executing
curl -I https://bit.ly/1mJk8X7
| perl -n -e ...
Which is why you get the syntax error. You could use strip to remove the new line from. I think the \n in the call to print will also get substituted before the string is passed to system. You may be interested in shellwords which has functions for escaping strings before passing them to a shell.
You could of course dodge the issue entirely and use ruby to get the redirect locations
require 'net/http'
require 'uri'
IO.foreach("shortURLs.txt") do |url|
puts Net::HTTP.get_response(URI.parse(url))["Location"]
end
Related
There is a file and I would like to prepend at top of the file.
file name is xyz.csv
I am able to run the same command in terminal and getting the result as well but when running through ruby script I get an error
xyz.rb
#Script
file = "/home/sumeet/xyz.csv"
command = "sed -i '1s/^/resource_id,code,value,date\n/'"
full = "#{command} #{file}"
`full`
error
$ ruby xyz.rb
xyz.rb:4:in ``': No such file or directory - full (Errno::ENOENT)
from xyz.rb:4:in `<main>'
The file exists in system
You need to use string interpolation when you want to use a variable in system calls. Try:
#Script
file = "/home/sumeet/xyz.csv"
command = "sed -i '1s/^/resource_id,code,value,date\\n/'"
full = "#{command} #{file}"
`#{full}`
EDIT: remember to escape \ in \n using \\n. Otherwise it'll be interpreted as a newline and terminate sed command to early.
I'm currently writing a script that will ultimately parse some baseball player ID numbers from MLB's Gameday data. I have set up my script so far to input today's date into the URL and to accept a team name as a command line argument.
The next step is to pass the URL to curl to get a list of today's games and the find the appropriate one by using grep with the team name.
When I write this on the command line:
curl -s http://gd2.mlb.com/components/game/mlb/year_2017/month_05/day_20/ | grep tormlb
It works.
However, when I write this in my script:
mlb_dir=$(curl -s $url | grep $team)
echo $mlb_dir
I am returned a mess of HTML without any line breaks. $url is equivalent to the url in the code block above and $team is set to "tormlb" just like above.
I am confused how I can get the result I want when I use the command line but not when I run my script. Any clue where I am going wrong?
When you pass a variable to a command, the variable is expanded and then split into arguments by whitespace. echo then outputs its arguments separated by spaces.
To ensure the variable is treated as a single argument, wrap it in double quotes. You should also do this for the other variables you are using in your code.
mlb_dir=$(curl -s "$url" | grep "$team")
echo "$mlb_dir"
Further information: http://tldp.org/LDP/abs/html/quotingvar.html
I have a ruby bin I'd like to pass information to in this fashion:
some_text | ./bin/my_ruby_bin
where some_text will be accessible by ARGV
is this possible with ruby + shell or am I taking the wrong approach here?
Here is simple solution that works for my cause, but it appears there are many ways to do this:
# ./bin/my_ruby_bin
#!/usr/bin/env ruby -n
puts "hello: #{$_}"
notice the -n flag
from command line:
echo 'world' | ./bin/my_ruby_bin
# => hello world
More on ruby -n
ruby -h
-n assume 'while gets(); ... end' loop around your script
As per the title, is it possible to get the raw command line used to invoke a ruby script?
The exact behaviour I'm after is similar to SSH when invoking a command directly:
ssh somehost -- ls -l
SSH will run "ls -l" on the server. It needs to be unparsed because if the shell has already interpreted quotes and performed expansions etc the command may not work correctly (if it contains quotes and such). This is why ARGV is no good; quotes are stripped.
Consider the following example:
my-command -- sed -e"s/something/something else/g"
The ARGV for this contains the following:
--
sed
-es/something/something else/g
The sed command will fail as the quotes will have been stripped and the space in the substitution command means that sed will not see "else/g".
So, to re-iterate, is it possible to get the raw command line used to invoke a ruby script?
No, this is at the OS level.
You could try simply quoting the entire input:
my-command -- "sed -e\"s/something/something else/g\""
In Ruby, this could be used like this:
ruby -e "puts ARGV[0]" -- "sed -e\"s/something/something else/g\""
(output) sed -e"s/something/something else/g"
Or, in a file putsargv1.rb (with the contents puts ARGV[1]):
ruby -- "putsargv1.rb" "sed -e\"s/something/something else/g\""
(output) sed -e"s/something/something else/g"
Your example is misguided. ssh somehost -- ls * will expand * on localhost (into e.g. ls localfile1 localfile2 localfile3), then execute that on the remote host, with the result of lots and lots of ls: cannot access xxx: No such file or directory errors. ssh does not see the uninterpreted command line.
As you said, you would get -es/something/something else/g as a single parameter. That is exactly what sed would get, too. This is, in fact, identical to what you get if you write -e"s/something/something else/g" and to "-es/something/something else/g", and to -es/something/something\ else.
Using this fact, you can use Shellwords.shellescape to "protect" the spaces and other unmentionables before handing them off to an external process. You can't get the original line, but you can make sure that you preserve the semantics.
Shellescape on the args worked but didn't quite mimic SSH. Take the following example (see below for test.rb contents):
ruby test.rb -- ls -l / \| sed -e's/root/a b c/g'
This will fail using the shellescape approach but succeed with SSH. I opted for manually escaping quotes and spaces. There may be some edge cases this doesn't capture but it seems to work for the majority of cases.
require 'shellwords'
unparsed = if ARGV.index('--')
ARGV.slice(ARGV.index('--') + 1, ARGV.length)
end || []
puts "Unparsed args: #{unparsed}"
exit if unparsed.empty?
shellescaped = unparsed.map(&Shellwords.method(:shellescape)).join(" ")
quoted = unparsed.map do |arg|
arg.gsub(/(["' ])/) { '\\' + $1 }
end.join(" ")
puts "Shellescaped: #{shellescaped}"
puts `bash -c #{shellescaped.shellescape}`
puts "Quoted: #{quoted}"
puts `bash -c #{quoted.shellescape}`
Thanks for your answers :)
In Ruby, I know I can execute a shell command with backticks like so:
`ls -l | grep drw-`
However, I'm working on a script which calls for a few fairly long shell commands, and for readability's sake I'd like to be able to break it out onto multiple lines. I'm assuming I can't just throw in a plus sign as with Strings, but I'm curious if there is either a command concatenation technique of some other way to cleanly break a long command string into multiple lines of source code.
You can escape carriage returns with a \:
`ls -l \
| grep drw-`
You can use interpolation:
`#{"ls -l" +
"| grep drw-"}`
or put the command into a variable and interpolate the variable:
cmd = "ls -l" +
"| grep drw-"
`#{cmd}`
Depending on your needs, you may also be able to use a different method of running the shell command, such as system, but note its behavior is not exactly the same as backticks.
Use %x:
%x( ls -l |
grep drw- )
Another:
%x(
echo a
echo b
echo c
)
# => "a\nb\nc\n"
You can also do this with explicit \n:
cmd_str = "ls -l\n" +
"| grep drw-"
...and then put the combined string inside backticks.
`#{cmd_str}`