I saw an expression a=($(cat)) which I am not able to understand from it's working mechanism perspective.
Functionally it takes input from the standard input and assigns it to variable a (which forms an array).
My understanding is , when shell executes the inner parenthesese it executes the cat command which brings the standard input, and when you type a few lines on the standard input and press CTRL+D it returns the lines to the outer parenthesese which then assign the lines to an array a.
My question is why this expression gives error when I remove the $ and write it as a=((cat)).
It is because $(..) is a command substitution syntax to run commands on. The cat in your example run in a sub-shell under this construct. Without it the command cat and ( are interpreted literally which the shell does not like
From the bash(1) - Linux man page
Command Substitution
Command substitution allows the output of a command to replace the command name. There are two forms:
$(command) (or) command
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.
The arithmetic operator in bash is $((..)) which is not the syntax you are using in your example
Related
So I generally create job files with a list of commands in it. Then I execute it like so
cat jobFile | while read a; do $a; done
Which always works in bash. However, I've just started working in Mac which apparently uses zsh. And this command fails with "no such file" etc. I've tested the job file by running few lines from it manually, so it should be fine.
I've found questions on zsh read inbut they tend to be reading in from variables e.g. $a=('a' 'b' 'c') or echo $a
Thank you for your answers!
In bash, unquoted parameter expansions always undergo word-splitting, so if a="foo bar", then $a expands to two words, foo and bar. As a command, this means running the command foo with an argument bar.
In zsh, parameter expansions to not undergo word-splitting by default, which means the same expansion $a would produce a single word foo bar, treated as the name of the command to execute.
In either case, relying on parameter expansion to "parse" a shell command is fragile; in addition to word-splitting, the expansion is subject to pathname expansion (globbing), and you are limited to simple commands and their arguments. No pipes, lists (&&, ||), or redirections allowed, as everything will be treated as the command name and a sequence of arguments.
What you want in both shells is to simply treat your job file as a shell script, which can be executed in the current shell using the . command:
. jobFile
Why are you executing it in such a cumbersome way? Assuming jobFile is a file holding a sequence of bash commands, you can simply run it as
bash jobFile
If it contains a sequence of zsh commands, you can likewise run it as
zsh jobFile
If you follow this approach, I would however reflect in the name of the job file, what shell it is intended for, i.e.
bash jobFile.bash
zsh jobFile.zsh
and, if you write a job file so that it is supposed to be compatible with either shell, I would name it jobFile.sh.
I often see key=value bash-script to pass variables to a bash script. Here is an example:
$ echo $0
-bash
$ cat foo.sh
#!/usr/bin/env bash
echo "key1: $key1"
$ key1=value1 ./foo.sh
key1: value1
I have checked Bash Reference Manual. But I can't a description related to this usage.
Section 3.7.4 "Environment"
The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments, as described in Shell Parameters. These assignment statements affect only the environment seen by that command.
Apparently the online version of the documentation fails to describe the syntax of a simple command completely. It reads:
3.2.1 Simple Commands
A simple command is the kind of command encountered most often. It’s just a sequence of words separated by blanks, terminated by one of the shell’s control operators (see Definitions). The first word generally specifies a command to be executed, with the rest of the words being that command’s arguments.
There is some information about the variable assignments that may precede a simple command on the section Simple Commands Expansion:
3.7.1 Simple Command Expansion
When a simple command is executed, the shell performs the following expansions, assignments, and redirections, from left to right.
The words that the parser has marked as variable assignments (those preceding the command name) and redirections are saved for later processing.
...
The text after the = in each variable assignment undergoes tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal before being assigned to the variable.
The manual page bundled with the shell seems to be better at this. It says (the emphasis is mine):
Simple Commands
A simple command is a sequence of optional variable assignments followed by blank-separated words and redirections, and terminated by a control operator.
You can always read the documentation of bash (and of any other CLI program installed on your computer) using man.
Type man bash on your terminal and use the usual keys (<spacebar>, b, /, q etc.) to navigate in the document. Behind the scenes, man uses your default pager program (which is, most probably, less) to put the information on screen.
Consider the following script:
#!/bin/bash
CMD="echo hello world > /tmp/hello.out"
${CMD}
The output for this is:
hello world > /tmp/hello.out
How can I modify CMD so that the output gets redirected to hello.out?
For my use case, it is not feasible to either do this:
${CMD} > /tmp/hello.out
or to add this at the top of the script:
exec > /tmp/hello.out
No, there is no way to make a redirection happen from a variable.
Why?
The first thing the shell does with a command line is:
Each line that the shell reads from the standard input or a script is called a pipeline; it contains one or more commands separated by zero or
more pipe characters (|). For each pipeline it reads, the shell breaks it up into commands, sets up the I/O for the pipeline, then does the following for each command (Figure 7-1):
From: Learning the bash Shell Unix Shell Programming . Chapter Preview / Figure . Pdf
That means that even before starting with the first word of a command line, the redirections are set up.
The "Parameter Expansion" happens quite a lot latter (in step 6 of the Figure).
There is no way to set up redirections after a variable is expanded.
Unless ...
The "command line is reprocessed" using eval.
eval "$CMD"
But this comes with a lot of danger.
The command line is changed by the first processing in the 12 steps detailed in the book (quotes are removed, variables expanded, words split, etc.).
It is usually quite difficult to estimate all the changes and consequences before the line is actually processed.
And then, it is processed again.
You can use eval to instruct the shell to reinterpret the variable content as a shell command:
eval $CMD
Consider this little shell script.
# Save the first command line argument
cmd="$1"
# Execute the command specified in the first command line argument
out=$($cmd)
# Do something with the output of the specified command
# Here we do a silly thing, like make the output all uppercase
echo "$out" | tr -s "a-z" "A-Z"
The script executes the command specified as the first argument, transforms the output obtained from that command and prints it to standard output. This script may be executed in this manner.
sh foo.sh "echo select * from table"
This does not do what I want. It may print something like the following,
$ sh foo.sh "echo select * from table"
SELECT FILEA FILEB FILEC FROM TABLE
if fileA, fileB and fileC is present in the current directory.
From a user perspective, this command is reasonable. The user has quoted the * in the command line argument, so the user doesn't expect the * to be globbed. But my script astonishes the user by using this argument in a command substitution which causes globbing of * as seen in the above output.
I want the output to be the following instead.
SELECT * FROM TABLE
The entire text in cmd actually comes from command line arguments to the script so I would like to preserve any * symbol present in the argument without globbing them.
I am looking for a solution that works for any POSIX shell.
One solution I have come up with is to disable globbing with set -o noglob just before the command substitution. Here is the complete code.
# Save the first command line argument
cmd="$1"
# Execute the command specified in the first command line argument
set -o noglob
out=$($cmd)
# Do something with the output of the specified command
# Here we do a silly thing, like make the output all uppercase
echo "$out" | tr -s "a-z" "A-Z"
This does what I expect.
$ sh foo.sh "echo select * from table"
SELECT * FROM TABLE
Apart from this, is there any other concept or trick (such as a quoting mechanism) I need to be aware of to disable globbing only within a command substitution without having to use set -o noglob.
I am not against set -o noglob. I just want to know if there is another way. You know, globbing can be disabled for normal command line arguments just by quoting them, so I was wondering if there is anything similar for command substiution.
If I understand correctly, you want the user to provide a shell command as a command-line argument, which will be executed by the script, and is expected to produce an SQL string, which will be processed (upper-cased) and echoed to stdout.
The first thing to say is that there is no point in having the user provide a shell command that the script just blindly executes. If the script applied some kind of modification/preprocessing of the command before it executed it then perhaps it could make sense, but if not, then the user might as well execute the command himself and pass the output to the script as a command-line argument, or via stdin.
But that being said, if you really want to do it this way, then there are two things that need to be said. Firstly, this is the proper form to use:
out=$(eval "$cmd");
A fairly advanced understanding of the shell grammer and expansion rules would be required to fully understand the rationale for using the above syntax, but basically executing $cmd and executing eval "$cmd" have subtle differences that render the $cmd form inappropriate for executing a given shell command string.
Just to give some detail that will hopefully clarify the above point, there are seven kinds of expansion that are performed by the shell in the following order when processing input: (1) brace expansion, (2) tilde expansion, (3) parameter and variable expansion, (4) arithmetic expansion, (5) command substitution, (6) word splitting, and (7) pathname expansion. Notice that variable expansion happens somewhat in the middle of that sequence, and thus the variable-expanded shell command (which was provided by the user) will not receive the benefit of the prior expansion types. Other issues are that leading variable assignments, pipelines, and command list tokens will not be executed correctly under the $cmd form, because they are parsed and processed prior to variable expansion (actually prior to all expansions) as well.
By running the command through eval, properly double-quoted, you ensure that the full shell parsing/processing/execution algorithm will be applied to the shell command string that was given by the user of your script.
The second thing to say is this: If you try the above proper form in your script, you will find that it has not solved your problem. You will still get SELECT FILEA FILEB FILEC FROM TABLE as output.
The reason is this: Since you've decided you want to accept an arbitrary shell command from the user of your script, it is now the user's responsibility to properly quote all metacharacters that may be embedded in that piece of code. It does not make sense for you to accept a shell command as a command-line argument, but somehow change the processing rules for shell commands so that certain metacharacters will no longer be metacharacters when the given shell command is executed. Actually, you could do something like that, perhaps using set -o noglob as you discovered, but then that must become a contract between the script and the user of the script; the user must be made aware of exactly what the precise processing rules will be when the command is executed so that he can properly use the script.
Under this design, the user could call the script as follows (notice the extra layer of quoting for the shell command string evaluation; could alternatively backslash-escape just the asterisk):
$ sh foo.sh "echo 'select * from table'";
I'd like to return to my earlier comment about the overall design; it doesn't really make sense to do it this way. It makes more sense to take the text-to-process itself, not a shell command that is expected to produce the text-to-process.
Here is how that could be done:
## take the text-to-process via a command-line argument
sql="$1";
## process and echo it
echo "$sql"| tr a-z A-Z;
(I also removed the -s option of tr, which really doesn't make sense here.)
Notice that the script is simpler now, and usage is also simpler:
$ sh foo.sh 'select * from table';
In bash, both () and $() create a subshell.
What's the difference between each other? What's their typical usages?
() just creates a compound command, running the commands inside the parentheses. $() does the same, but also substitutes the output.
From the docs:
(list)
list is executed in a subshell environment ... Variable assignments and builtin
commands that affect the shell's environment do not remain in effect after the command completes. The return status is
the exit status of list.
Command Substitution
Command substitution allows the output of a command to replace the command name. There are two forms:
$(command)
or
`command`
Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command,
with any trailing newlines deleted.