I am wondering how can I execute ruby code without saving a file. For example:
ruby -e 'Time.now'
However, the quotes are a problem, for example:
ruby -e 'puts 'A syntax error''
I could use doble quotes instead:
ruby -e 'puts "No error"'
Or:
ruby -e "puts 'No error'"
The issue is that I am running arbitrary code inside a container so I can't avoid the issue with a manual replacement and scaping the quotes also generates an error:
ruby -e 'puts \'A syntax error\''
So I wonder how can I execute ruby code without files and without worrying about quotes inside the code.
You can use a Shell HEREDOC. The exact syntax may differ between shells but in BASH it is:
Edit: All Shells have the same syntax and behave the same way (see IEEE standard quote below).
ruby <<'EOS'
$error = 'A syntax error'
puts $error
EOS
Note: The quotes of <<'EOS' prevent the Shell from expanding variables inside the HEREDOC, or else $error would be expanded before being passed to Ruby. Also, you can use whatever word you prefer instead of EOS, for ex:
ruby <<'end of ruby script'
puts 'example'
end of ruby script
The IEEE standard says:
2.7.4 Here-Document
The redirection operators << and <<- both allow redirection of lines contained in a shell input file, known as a "here-document", to the input of a command.
The here-document shall be treated as a single word that begins after
the next <newline> and continues until there is a line containing only
the delimiter and a <newline>, with no <blank>s in between. Then the
next here-document starts, if there is one. The format is as follows:
[n]<<word
here-document
delimiter
where the optional n represents the file descriptor number. If the number is omitted, the here-document refers
to standard input (file descriptor 0).
If any character in word is quoted, the delimiter shall be formed by
performing quote removal on word, and the here-document lines shall
not be expanded. Otherwise, the delimiter shall be the word itself.
If no characters in word are quoted, all lines of the here-document
shall be expanded for parameter expansion, command substitution, and
arithmetic expansion. In this case, the backslash in the input behaves
as the backslash inside double-quotes (see Double-Quotes). However,
the double-quote character ( '"' ) shall not be treated specially
within a here-document, except when the double-quote appears within
$(), ``, or ${}.
If the redirection symbol is <<-, all leading <tab>s shall be
stripped from input lines and the line containing the trailing
delimiter
Update
That said, it's impossible to completely prevent conflicts. For example this Ruby code would break your Shell script:
ruby <<'EOS'
puts <<-EOS
example
EOS
EOS
The only way to work around that is to use a word that is very unlikely to appear in the Ruby code, for example:
ruby <<'2eGRng8B6NEmNphkA2K4yc9Y782PQZpWzkUW7pwGdmybuMBG5 PVtkzZkeSKGLq4'
puts <<-EOS
example
EOS
2eGRng8B6NEmNphkA2K4yc9Y782PQZpWzkUW7pwGdmybuMBG5 PVtkzZkeSKGLq4
You have to escape the quotes properly, for example:
ruby -e 'puts '\''foo bar'\''; puts '\''foo bar'\''; '
# ^-----^
# quote this
# ^-------^
# quote that
# ^^
# escaped quote to pass to Ruby
Output:
foo bar
foo bar
Note that most quotes are needed to simply quote the Ruby code in the shell. The escaped quote \' is an actual quote that is passed to Ruby.
Use Ruby's String Literals
To avoid (or at least simplify) the issue of quoting String objects inside shell quotes, you can use a heredoc or a percent-string such as %q for single quotes or %Q for double-quotes. For example:
# double-quoted percent string
ruby -e 'puts %Q(No error.)'
# single-quoted percent string
ruby -e 'puts %q(A syntax error.)'
# quoted String with nested symbols
ruby -e 'puts %Q[You can use most [percent-literal] symbols safely.]'
Note that you can use almost any paired or unnpaired symbol character to open and close the percent literal. (), [], and {} are common choices, but anything other than single quotes and \' should be safe from the shell inside a single-quoted argument to -e.
Related
I have to implement a minishell written in C for my school homework,and I am currently working on the parsing of the user input and I have a question regarding single quote and double quote.
How are they parsed, I mean many combo are possible as long as there is more than two single quotes / double quote.
Let’s say I have : ""hello"".
If I echo this the bash output would be hello, is this because bash interpreted "hello" inside "" or because bash interpreted this as : "" then hello "".
How does bash parse multiple chained quotes :
" "Hello" " : let’s say a = Hello. Does bash sees "a" or bash see ""a""
Bash parses single- and double-quotes slightly differently. Single-quotes are simpler, so I'll cover them first.
A single-quoted string (or single-quoted section of a string -- I'll get to that) runs from a single-quote to the next single-quote. Anything other than a single-quote (including double-quotes, backslashes, newlines, etc) is just a literal character in the string. But the next single-quote ends the single-quoted section. There is no way to put a single-quote inside a single-quoted string, because the next single-quote will end the single-quoted section.
You can have differently-quoted sections within a single "word" (or string, or whatever you want to call it). So, for example, ''hello'' will be parsed as a zero-length single-quoted section, the unquoted section hello, then another zero-length single-quoted section. Since there's no whitespace between them, they're all treated as part of the same word (and since the single-quoted sections are zero-length, they have no effect at all on the resulting word/string).
Double-quotes are slightly different, in that some characters within them retain their special meanings. For example, $ can introduce variable or command substitution, etc (although the result won't be subject to word-splitting like it would be without the double-quotes). Backslashes also function as escapes inside double-quotes, so \$ will be treated as a literal dollar-sign, not as the start of a variable expansion or anything. Other characters that can have their special meaning removed by backslash-escaping are backslashes themselves, and... double-quotes! You can include a double-quote in a double-quote by escaping it. The double-quoted section ends a the next non-escaped double-quote.
So compare:
echo ""hello"" # just prints hello
echo "\"hello\"" # prints "hello", because the escaped
# quotes are part of the string
echo "$PATH" # prints the value of the PATH variable
echo "\$PATH" # prints $PATH
echo ""'$PATH'"" # prints $PATH, because it's in
# single-quotes (with zero-lenght
# double-quoted sections on each side
Also, single- and double-quotes have no special meaning within the other type of quote. So:
echo "'hello'" # prints 'hello', because the single-quotes
# are just ordinary characters in a
# double-quoted string
echo '"hello"' # similarly, prints "hello"
echo "'$PATH'" # prints the PATH variable with
# single-quotes around it (because
# $variable expands in double-quotes)
I'm trying to understand how the bash works from the inside, and I have a problem understanding why the heredoc's output gets expanded when we set a delimiter without quotes and get expanded with them.
For example :
Delimiter without quotes
bash-3.2$ cat << a
> test
> $SHELL
> a
test
/bin/zsh
Delimiter with quotes :
bash-3.2$ cat << 'a'
> test
> $SHELL
> a
test
$SHELL
Quoting the heredoc delimiter as 'a' or "a" or \a or any other way causes the heredoc to be treated literally with expansion disabled. This is a useful feature for cases where you don't want dollar signs and other special characters to be expanded: for example, when the heredoc contains an embedded shell script.
This behavior is described in the Bash manual (emphasis added):
3.6.6 Here Documents
This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input (or file descriptor n if n is specified) for a command.
The format of here-documents is:
[n]<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or filename expansion is performed on word. If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.
I am new to BASH and to Julia, and I am trying to do the following steps:
Save a filepath in a variable
Use this variable in a julia command, which i managed to execute through bash by using a heredoc function
Hard-coded command that works:
cat << "EOF" | julia --project=.
module test
ARGS=["/path/To/My/Directory"]
include("nameOfMyProject.jl")
end
EOF
The problem is, that I want to exchange the hardcoded path with the value of a variable.
I tried:
path="/path/To/My/Directory"
cat << "EOF" | julia --project=.
module test
ARGS=[$path]
include("nameOfMyProject.jl")
end
EOF
However, that does not work. I read i should try to use <<EOF instead of << "EOF" but then I receive this error: syntax: "/" is not a unary operator
I am not sure how to interprete this, since it might actually access the variable and find the "/", but it does work with it if I use "/" in the hard-coded way... so why not there?
I am not sure if my problem lies in the heredoc function or in something with this variable, but I would be grateful for any help!
Solution for this problem:
I am adding this for reference, if someone else has the same problem. The format that does the trick nicely is:
path="/path/To/My/Directory"
julia --project=. <<EOF
module test
ARGS=["$path"]
include("nameOfMyProject.jl")
end
EOF
Thanks to Jetchisel and Jens
From the bash manual (emphasis mine):
Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no trailing
blanks) is seen. All of the lines read up to that point are then used
as the standard input (or file descriptor n if n is specified) for a
command.
The format of here-documents is:
[n]<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic
expansion, or pathname expansion is performed on word. If any part of
word is quoted, the delimiter is the result of quote removal on word,
and the lines in the here-document are not expanded. If word is
unquoted, all lines of the here-document are subjected to parameter
expansion, command substitution, and arithmetic expansion, the
character sequence \<newline> is ignored, and \ must be used to quote
the characters \, $, and `.
So all you need is
julia --project=. << EOF
...$FOO...
EOF
I have read a ton of pages including the bash manual, but still find the "non-obvious" use of backslashes confusing.
If I do:
echo \*
it prints a single asterisks, this is normal as I am escaping the asterisks making it literal.
If I do:
echo \\*
it prints \*
This also seems normal, the first backslash escapes the second.
If I do
echo `echo \\*`
It prints the contents of the directory. But in my mind it should print the same as echo \\* because when that is substituted and passed to echo. I understand this is the non-obvious use of backslashes everyone talks about, but I am struggling to understand WHY it happens.
Also the bash manual says
When the old-style backquote form of substitution is used, backslash retains its literal meaning except when followed by ‘$’, ‘`’, or ‘\’.
But it doesn't define what the "literal meaning on backslash" is. Is it as an escape character, a continuation character, or just literally a backslash character?
Also, it says it retain it's literal meaning, except when followed by ... So when it's followed by one of those three characters what does it do? Does it only escape those three characters?
This is mostly for historical interest since `...` command substitution has been superseded by the cleaner $(...) form. No new script should ever use backticks.
Here's how you evaluate a $(command) substitution
Run the command
Here's how you evaluate a `string` command substitution:
Determine the span of the string, from the opening backtick to the closing unescaped backtick (behavior is undefined if this backtick is inside a string literal: the shell will typically either treat it as literal backtick or as a closing backtick depending on its parser implementation)
Unescape the string by removing backslashes that come before one of the three characters dollar, backtick or backslash. This following character is then inserted literally into the command. A backslash followed by any other character will be left alone.
E.g. Hello\\ World will become Hello\ World, because the \\ is replaced with \
Hello\ World will also become Hello\ World, because the backslash is followed by a character other than one of those three, and therefore retains its literal meaning of just being a backslash
\\\* will become \\* since the \\ will become just \ (since backslash is one of the three), and the \* will remain \* (since asterisk is not)
Evaluate the result as a shell command (this includes following all regular shell escaping rules on the result of the now-unescaped command string)
So to evaluate echo `echo \\*`:
Determine the span of the string, here echo \\*
Unescape it according to the backtick quoting rules: echo \*
Evaluate it as a command, which runs echo to output a literal *
Since the result of the substitution is unquoted, the output will undergo:
Word splitting: * becomes * (since it's just one word)
Pathname expansion on each of the words, so * becomes bin Desktop Downloads Photos public_html according to files in the current directory
Note in particular that this was not the same as replacing the the backtick command with the output and rerunning the result. For example, we did not consider escapes, quotes and expansions in the output, which a simple text based macro expansion would have.
Pass each of these as arguments to the next command (also echo): echo bin Desktop Downloads Photos public_html
The result is a list of files in the current directory.
I'm trying to understand why Bash removes double quotes (but not single quotes) when doing variable expansion with ${parameter:+word} (Use Alternate Value), in a here-document, for example:
% var=1
% cat <<EOF
> ${var:+"Hi there"}
> ${var:+'Bye'}
> EOF
Hi there
'Bye'
According to the manual, the "word" after :+ is processed with tilde expansion, parameter expansion, command substitution, and arithmetic expansion. None of these should do anything.
What am I missing? How can I get double quotes in the expansion?
tl;dr
$ var=1; cat <<EOF
"${var:+Hi there}"
${var:+$(printf %s '"Hi there"')}
EOF
"Hi there"
"Hi there"
The above demonstrates two pragmatic workarounds to include double quotes in the alternative value.
The embedded $(...) approach is more cumbersome, but more flexible: it allows inclusion of embedded double quotes and also gives you control over whether the value should be expanded or not.
Jens' helpful answer and Patryk Obara's helpful answer both shed light on and further demonstrate the problem.
Note that the problematic behavior equally applies to:
(as noted in the other answers) regular double-quoted strings (e.g., echo "${var:+"Hi there"}"; for the 1st workaround, you'd have to \-quote surrounding " instances; e.g., echo "\"${var:+Hi there}\""; curiously, as Gunstick points out in a comment on the question, using \" in the alternative value to produce " in the output does work in double-quoted strings - e.g., echo "${var:+\"Hi th\"ere\"}" - unlike in unquoted here-docs.)
related expansions ${var+...}, ${var-...} / ${var:-...}, and ${var=...} / ${var:=...}
Also, there's a related oddity with respect to \-handling inside double-quoted alternative values inside a double-quoted string / unquoted here-doc: bash and ksh unexpectedly remove embedded \ instances; e.g.,
echo "${nosuch:-"a\b"}" unexpectedly yields ab, even though echo "a\b" in isolation yields a\b - see this question.
I have no explanation for the behavior[1]
, but I can offer pragmatic solutions that work with all major POSIX-compatible shells (dash, bash, ksh, zsh):
Note that " instances are never needed for syntactic reasons inside the alternative value: The alternative value is implicitly treated like a double-quoted string: no tilde expansion, no word-splitting, and no globbing take place, but parameter expansions, arithmetic expansions and command substitutions are performed.
Note that in parameter expansions involving substitution or prefix/suffix-removal, quotes do have syntactic meaning; e.g.: echo "${BASH#*"bin"}" or echo "${BASH#*'bin'}" - curiously, dash doesn't support single quotes, though.
If you want to surround the entire alternative value with quotes, and it has no embedded quotes and you want it expanded,
quote the entire expansion, which bypasses the problem of " removal from the alternative value:
# Double quotes
$ var=1; cat <<EOF
"${var:+The closest * is far from $HOME}"
EOF
"The closest * is far from /Users/jdoe"
# Single quotes - but note that the alternative value is STILL EXPANDED,
# because of the overall context of the unquoted here-doc.
var=1; cat <<EOF
'${var:+The closest * is far from $HOME}'
EOF
'The closest * is far from /Users/jdoe'
For embedded quotes, or to prevent expansion of the alternative value,
use an embedded command substitution (unquoted, although it'll behave as if it were quoted):
# Expanded value with embedded quotes.
var=1; cat <<EOF
${var:+$(printf %s "We got 3\" of snow at $HOME")}
EOF
We got 3" of snow at /Users/jdoe
# Literal value with embedded quotes.
var=1; cat <<EOF
${var:+$(printf %s 'We got 3" of snow at $HOME')}
EOF
We got 3" of snow at $HOME
These two approaches can be combined as needed.
[1]
In effect, the alternative value:
behaves like an implicitly double-quoted string,
' instances, as in regular double-quoted strings, are treated as literals.
Given the above,
it would make sense to treat embedded " instances as literals too, and simply pass them through, just like the ' instances.
Instead, sadly, they are removed, and if you try to escape them as \", the \ is retained too (inside unquoted here-documents, but curiously not inside double-quoted strings), except in ksh - the laudable exception -, where the \ instances are removed. In zsh, curiously, trying to use \" breaks the expansion altogether (as do unbalanced unescaped ones in all shells).
More specifically, the double quotes have no syntactic function in the alternative value, but they are parsed as if they did: quote removal is applied, and you can't use (unbalanced) " instances in the interior without \"-escaping them (which, as stated, is useless, because the \ instances are retained).
Given the implicit double-quoted-string semantics, literal $ instances must either be \$-escaped, or a command substitution must be used to embed a single-quoted string ($(printf %s '...')).
The behavior looks deliberate--it is consistent across all Bourne shells I tried (e.g. ksh93 and zsh behave the same way).
The behavior is equivalent to treating the here-doc as double-quoted for these special expansions only. In other words, you get the same result for
$ echo "${var:+"hi there"}"
hi there
$ echo "${var:+'Bye'}"
'Bye'
There is only a very faint hint in the POSIX spec I found that something special happens for double quoted words in parameter expansions. This is from the informative "Examples" section of Parameter Expansion:
The double-quoting of patterns is different depending on where the double-quotes are placed.
"${x#*}"
The <asterisk> is a pattern character.
${x#"*"}
The literal <asterisk> is quoted and not special.
I would read the last line as suggesting that quote removal for double quotes applies to the word. This example would not make sense for single quotes, and by omission, there's no quote removal for single quotes.
Update
I tried the FreeBSD /bin/sh, which is derived from an Almquist Shell. This shell outputs single and double quotes. So the behavior is no longer consistent across all shells, only across most shells I tried.
As for getting double quotes in the expansion of the word after :+, my take is
$ var=1
$ q='"'
$ cat <<EOF
${var:+${q}hi there$q}
EOF
"hi there"
$ cat <<EOF
${var:+bare alt value is string already}
${var:+'and these are quotes within string'}
${var:+"these are double quotes within string"}
${var:+"which are removed during substitution"}
"${var:+but you can simply not substitute them away ;)}"
EOF
bare alt value is string already
'and these are quotes within string'
these are double quotes within string
which are removed during substitution
"but you can simply not substitute them away ;)"
Note, that here-document is not needed to reproduce this:
$ echo "${var:+'foo'}"
'foo'