When using a POSIX shell, the following
touch {quick,man,strong}ly
expands to
touch quickly manly strongly
Which will touch the files quickly, manly, and strongly, but is it possible to dynamically create the expansion? For example, the following illustrates what I want to do, but does not work because of the order of expansion:
TEST=quick,man,strong #possibly output from a program
echo {$TEST}ly
Is there any way to achieve this? I do not mind constricting myself to Bash if need be. I would also like to avoid loops. The expansion should be given as complete arguments to any arbitrary program (i.e. the program cannot be called once for each file, it can only be called once for all files). I know about xargs but I'm hoping it can all be done from the shell somehow.
... There is so much wrong with using eval. What you're asking is only possible with eval, BUT what you might want is easily possible without having to resort to bash bug-central.
Use arrays! Whenever you need to keep multiple items in one datatype, you need (or, should use) an array.
TEST=(quick man strong)
touch "${TEST[#]/%/ly}"
That does exactly what you want without the thousand bugs and security issues introduced and concealed in the other suggestions here.
The way it works is:
"${foo[#]}": Expands the array named foo by expanding each of its elements, properly quoted. Don't forget the quotes!
${foo/a/b}: This is a type of parameter expansion that replaces the first a in foo's expansion by a b. In this type of expansion you can use % to signify the end of the expanded value, sort of like $ in regular expressions.
Put all that together and "${foo[#]/%/ly}" will expand each element of foo, properly quote it as a separate argument, and replace each element's end by ly.
In bash, you can do this:
#!/bin/bash
TEST=quick,man,strong
eval echo $(echo {$TEST}ly)
#eval touch $(echo {$TEST}ly)
That last line is commented out but will touch the specified files.
Zsh can easily do that:
TEST=quick,man,strong
print ${(s:,:)^TEST}ly
Variable content is splitted at commas, then each element is distributed to the string around the braces:
quickly manly strongly
Taking inspiration from the answers above:
$ TEST=quick,man,strong
$ touch $(eval echo {$TEST}ly)
Related
I have a list/array like so:
['path/to/folder/a', 'path/to/folder/b']
This is an example, the array can be of any length. But for each item in the array I'd like to set up the following as a single command:
$ someTool <command> --flag <item-1> --flag <item-2> ... --flag <item-N>
At the moment I am currently doing a loop over the array but I am just wondering if doing them individually has a different behaviour to doing them all at once (which the tool specifies I should do).
for i in "${array[#]}"; do
someTool command --flag $i
done
Whether passing all flag arguments to a single invocation of the tool does the same thing as passing them one-at-a-time to separate invocations depends entirely on the tool and what it does. Without more information, it's impossible to say for sure, but if the instructions recommend passing them all at once, I'd go with that.
The simplest way to do this in bash is generally to create a second array with the flags and arguments as they need to be passed to the tool:
flagsArray=()
for i in "${array[#]}"; do
flagsArray+=(--flag "$i")
done
someTool command "${flagsArray[#]}"
Note: all of the above syntax -- all the quotes, braces, brackets, parentheses, etc -- matter to making this run properly and robustly. Don't leave anything out unless you know why it's there, and that leaving it out won't cause trouble.
BTW, if the option (--flag) doesn't have to be passed as a separate argument (i.e. if the tool allows --flag=path/to/folder/a instead of --flag path/to/folder/a), then you can use a substitution to add the --flag= bit to each element of the array in a single step:
someTool command "${array[#]/#/--flag=}"
Explanation: the /# means "replace at the beginning (of each element)", then the empty string for the thing to replace, / to delimit that from the replacement string, and --flag= as the replacement (/addition) string.
Here is the code at the bash shell. How is the file mask supposed to be specified, if not this way? I expected both commands to find the search expression, but it's not happening. In this example, I know in advance that I prefer to restrict the search to python source code files only, because unqualified searches are silly time wasters.
So, this works as expected:
grep -rni '/home/ga/projects' -e 'def Pr(x,u,v)'
/home/ga/projects/anom/anom.py:27:def Pr(x,u,v): blah, blah, ...
but this won't work:
grep --include=\*.{py} -rni '/home/ga/projects' -e 'def Pr(x,u,v)'
I'm using GNU grep version 2.16.
--include=\*.{py} looks like a broken attempt to use brace expansion (an unquoted {...} expression).
However, for brace expansion
to occur in bash (and ksh and zsh), you must either have:
a list of at least 2 items, separated with ,; e.g. {py,txt}, which expands to 2 arguments, py and txt.
or, a range of items formed from two end points, separated with ..; e.g., {1..3}, which expands to 3 arguments, 1, 2, and 3.
Thus, with a single item, simply do not use brace expansion:
--include=\*.py
If you did have multiple extensions to consider, e.g., *.py as well as *.pyc files, here's a robust form that illustrates the underlying shell features:
'--include=*.'{py,pyc}
Here:
Brace expansion is applied, because {...} contains a 2-item list.
Since the {...} directly follows the literal (single-quoted) string --include=*., the results of the brace expansion include the literal part.
Therefore, 2 arguments are ultimately passed to grep, with the following literal content:
--include=*.py
--include=*.pyc
Your command fails because of the braces '{}'. It will search for it in the file name. You can create a file such as 'myscript.{py}' to convince yourself. You'll see it will appear in the results.
The correct option parameter would be '*.py' or the equivalent \*.py. Either way will protect it from being (mis)interpreted by the shell.
On the other side, I can only advise to use the command find for such jobs :
find /home/ga/projects -regex '.*\.py$' -exec grep -e "def Pr(x,u,v)" {} +
That will protect you from hard to understand shell behaviour.
Try like this (using quotes to be safe; also better readability than backslash escaping IMHO):
grep --include='*.py' ...
your \*.{py} brace expansion usage isn't supported at all by grep. Please see the comments below for the full investigation regarding this. For the record, blame this answer for the resulting brace wars ;)
By the way, the brace expansion works generally fine in Bash. See mklement0 answer for more details.
Ack. As an alternative, you might consider switching to ack instead from now on. It's a tool just like grep, but fully optimized for programmers.
It's a great fit for what you are doing. A nice quote about it:
Every once in a while something comes along that improves an idea so much, you can't ignore it. Such a thing is ack, the grep replacement.
I'm trying to figure out what I thought would be a trivial issue in BASH, but I'm having difficulty finding the correct syntax. I want to loop over an array of values, one of them being an asterisk (*), I do not wish to have any wildcard expansion happening during the process.
WHITELIST_DOMAINS="* *.foo.com *.bar.com"
for domain in $WHITELIST_DOMAINS
do
echo "$domain"
done
I have the above, and I'm trying to get the following output:
*
*.foo.com
*.bar.com
Instead of the above, I get a directory listing on the current directory, followed by *.foo.com and *.bar.com
I know I need some escaping or quoting somewhere.. the early morning haze is still thick on my brain.
I've reviewed these questions:
How to escape wildcard expansion in a variable in bash?
Stop shell wildcard character expansion?
Your problem is that you want an array, but you wrote a single string that contains the elements with spaces between them. Use an array instead.
WHITELIST_DOMAINS=('*' '*.foo.com' '*.bar.com')
Always use double quotes around variable substitutions (i.e. "$foo"), otherwise the shell splits the the value of the variable into separate words and treats each word as a filename wildcard pattern. The same goes for command substitution: "$(somecommand)". For an array variable, use "${array[#]}" to expand to the list of the elements of the array.
for domain in "${WHITELIST_DOMAINS[#]}"
do
echo "$domain"
done
For more information, see the bash FAQ about arrays.
You can use array to store them:
array=('*' '*.foo.com' '*.bar.com')
for i in "${array[#]}"
do
echo "$i"
done
I seem to have some difficulty getting what I want to work. Basically, I have a series of variables that are assigned strings with some quotes and \ characters. I want to remove the quotes to embed them inside a json doc, since json hates quotes using python dump methods.
I figured it would be easy. Just determine how to remove the characters easy and then write a simple for loop for the variable substitution, well it didn't work that way.
Here is what I want to do.
There is a variable called "MESSAGE23", it contains the following "com.centrify.tokend.cac", I want to strip out the quotes, which to me is easy, a simple echo $opt | sed "s/\"//g". When I do this from the command line:
$> MESSAGE23="com."apple".cacng.tokend is present"
$> MESSAGE23=`echo $MESSAGE23 | sed "s/\"//g"`
$> com.apple.cacng.tokend is present
This works. I get the properly formatted string.
When I then try to throw this into a loop, all hell breaks loose.
for i to {1..25}; do
MESSAGE$i=`echo $MESSAGE$i | sed "s/\"//g"`
done
This doesn't work (either it throws a bunch of indexes out or nothing), and I'm pretty sure I just don't know enough about arg or eval or other bash substitution variables.
But basically I want to do this for another set of variables with the same problems, where I strip out the quotes and incidentally the "\" too.
Any help would be greatly appreciated.
You can't do that. You could make it work using eval, but that introduces another level of quoting you have to worry about. Is there some reason you can't use an array?
MESSAGE=("this is MESSAGE[0]" "this is MESSAGE[1]")
MESSAGE[2]="I can add more, too!"
for (( i=0; i<${#MESSAGE[#]}; ++i )); do
echo "${MESSAGE[i]}"
done
Otherwise you need something like this:
eval 'echo "$MESSAGE'"$i"'"'
and it just gets worse from there.
First, a couple of preliminary problems: MESSAGE23="com."apple".cacng.tokend is present" will not embed double-quotes in the variable value, use MESSAGE23="com.\"apple\".cacng.tokend is present" or MESSAGE23='com."apple".cacng.tokend is present' instead. Second, you should almost always put double-quotes around variable expansions (e.g. echo "$MESSAGE23") to prevent parsing oddities.
Now, the real problems: the shell doesn't allow variable substitution on the left side of an assignment (i.e. MESSAGE$i=something won't work). Fortunately, it does allow this in a declare statement, so you can use that instead. Also, when the sees $MESSAGE$i it replaces it will the value of $MESSAGE followed by the value of $i; for this you need to use indirect expansion (`${!metavariable}').
for i in {1..25}; do
varname="MESSAGE$i"
declare $varname="$(echo "${!varname}" | tr -d '"')"
done
(Note that I also used tr instead of sed, but that's just my personal preference.)
(Also, note that #Mark Reed's suggestion of an array is really the better way to do this sort of thing.)
In my bash scripts, I regularly use file paths which may contain spaces:
FOO=/path\ with\ spaces/
Later, if I want to use FOO, I have to wrap it in quotes ("$FOO") or it will be interpreted as a list (/path, with, spaces/). Is there a better way to force a variable never to be interpreted as a list? It is cumbersome to have to constantly quote-wrap.
No. You must always use quotes or bash will word-split (except in [[, but that is a special case).
You can also change the internal field separator, IFS, as in:
ORIGIFS="$IFS"
IFS=$(echo -en "\n\b")
# do stuff...
IFS="$ORIGIFS"
However, this affects all situations where bash looks to do field splitting, which might be more broad than you'd like.