I seem to have some difficulty getting what I want to work. Basically, I have a series of variables that are assigned strings with some quotes and \ characters. I want to remove the quotes to embed them inside a json doc, since json hates quotes using python dump methods.
I figured it would be easy. Just determine how to remove the characters easy and then write a simple for loop for the variable substitution, well it didn't work that way.
Here is what I want to do.
There is a variable called "MESSAGE23", it contains the following "com.centrify.tokend.cac", I want to strip out the quotes, which to me is easy, a simple echo $opt | sed "s/\"//g". When I do this from the command line:
$> MESSAGE23="com."apple".cacng.tokend is present"
$> MESSAGE23=`echo $MESSAGE23 | sed "s/\"//g"`
$> com.apple.cacng.tokend is present
This works. I get the properly formatted string.
When I then try to throw this into a loop, all hell breaks loose.
for i to {1..25}; do
MESSAGE$i=`echo $MESSAGE$i | sed "s/\"//g"`
done
This doesn't work (either it throws a bunch of indexes out or nothing), and I'm pretty sure I just don't know enough about arg or eval or other bash substitution variables.
But basically I want to do this for another set of variables with the same problems, where I strip out the quotes and incidentally the "\" too.
Any help would be greatly appreciated.
You can't do that. You could make it work using eval, but that introduces another level of quoting you have to worry about. Is there some reason you can't use an array?
MESSAGE=("this is MESSAGE[0]" "this is MESSAGE[1]")
MESSAGE[2]="I can add more, too!"
for (( i=0; i<${#MESSAGE[#]}; ++i )); do
echo "${MESSAGE[i]}"
done
Otherwise you need something like this:
eval 'echo "$MESSAGE'"$i"'"'
and it just gets worse from there.
First, a couple of preliminary problems: MESSAGE23="com."apple".cacng.tokend is present" will not embed double-quotes in the variable value, use MESSAGE23="com.\"apple\".cacng.tokend is present" or MESSAGE23='com."apple".cacng.tokend is present' instead. Second, you should almost always put double-quotes around variable expansions (e.g. echo "$MESSAGE23") to prevent parsing oddities.
Now, the real problems: the shell doesn't allow variable substitution on the left side of an assignment (i.e. MESSAGE$i=something won't work). Fortunately, it does allow this in a declare statement, so you can use that instead. Also, when the sees $MESSAGE$i it replaces it will the value of $MESSAGE followed by the value of $i; for this you need to use indirect expansion (`${!metavariable}').
for i in {1..25}; do
varname="MESSAGE$i"
declare $varname="$(echo "${!varname}" | tr -d '"')"
done
(Note that I also used tr instead of sed, but that's just my personal preference.)
(Also, note that #Mark Reed's suggestion of an array is really the better way to do this sort of thing.)
Related
The goal is to get rid of (by changing) filenames that give headaches for scripting by translating them to something else. The reason is that in this nearly 30 year Unix / Linux environment, with a lot of existing scripts that may not be "written correctly", a new, large and important cache of files arrived that have to be managed, and so, a colleague has asked me to write a script to help with "problematic filenames" and translate them. They've got a list of chars to turn into dots, such as the comma, and another list to turn into underscores, such as whitespace, as but two examples and ran into problems which I asked about over here.
I was using tr to do it, but commenters to it said I should perhaps ask just about this instead of how to get tr to work. So, I have!
Parameter expansion can do this for you.
Note that unlike when using tr (as requested on your other question), when using parameter expansion you don't need to use backslashes inside your character class definitions: put the expansion in double quotes and bash will treat the results of that expansion as literal.
#!/usr/bin/env bash
toDots='\,;:|+##$%^&*~'
toUnderscores='}{]['"'"'="()`!'
# requires bash 5+: if debug=1, then print what we would do instead of doing it
runOrDebug() {
if (( debug )); then
printf '%s\n' "${*#Q}"
else
"$#"
fi
}
renameFiles() {
local name subDots subBoth
for name; do
subDots=${name//["$toDots"]/.}
subBoth=${subDots//["$toUnderscores"]/_}
if [[ $subBoth != "$name" ]]; then
runOrDebug mv -- "$name" "$subBoth"
fi
done
}
debug=1 renameFiles '[/a],/;[p:r|o\b+lem#a#t$i%c]/#(%$^!/(e^n&t*ry)~='
Note that toUnderscores is (except for the single quote in the middle) in single quotes, so all the backslashes in it are part of the variable's data rather than being syntax; because globs use character class syntax from REs, they're parsed as POSIX regular expression character class syntax.
See a demonstration of the technique running at https://ideone.com/kKE7IJ
I know that there are a number of ways to manually escape nested quotes of the same kind.
Example
echo 'this single quote '"'"' is escaped.'
There are several more ways to do this that are documented well on StackOverflow already, but I'm trying to take that concept and apply it as a function to escape everything. I have been unsuccessful several times to find an all in one escaping solution for Bash (ex. I've looked for a string library that includes escaping with no luck).
Here's one of my attempts:
function quote() {
printf '%s' "'${1//\'/\'"\'"\'}'"
}
The idea is that all single quotes will be replaced with the aforementioned escape style '"'"'. I've also tried doing the same thing using the escape style like so: \'. No luck.
Every response on this topic in my research that I've come across has made it sound like this isn't possible, but I'm not the type to accept that for an answer. To me, if you can echo a string like this: 'test \' test', then it should be expressible in a function too where the backslashes or otherwise are automatically added to escape the characters.
echo '~!##$%^&*()_+`1234567890-=qwertyuiop[]\QWERTYUIOP{}|ASDFGHJKL:"ZXCVBNM<>?zxcvbnm,./'
As shown above, none of the basic keys on a traditional English keyboard need to be escaped, but nesting the same type of quotes within itself requires it.
Can anyone shed some light on this? Am I missing something obvious or is this really that difficult?
You don't need to mimic the shell quoting, properly quoting the variables should be enough.
#! /bin/bash
tag () {
echo Setting tags to "$2".
}
while read tags ; do
tag --set "$tags"
done <<EOF
tag1,tag2,tag3
Tom's_Shoes
The_"best"
EOF
If the tags are in variables or an array and you use proper quoting, you should't need to do any extra escaping. See #choroba's answer.
But there are times when you need it and this will do the trick:
printf -v my_var '%q' "$myvar"
which replaces the contents of my_var with an shell-escaped version.
I need to rename the files inside the folder that has a space in it eg(Deco/main library/file1.txt )
code:
while IFS="," read orig new pat
do
mv -v $pat$new $pat$orig
done < new.csv
csv file:
newname,file1.txt,Deco/main\\\ library/
error:
mv: invalid option -- '\'
Welcome to Stackoverflow!
First: Use quotes around the use of variables. That means except in very rare occasions, you always should use "$foo" instead of $foo because if you are using the latter, the shell is supposed (and will) interpret spaces in the variables as word delimiters which you rarely want. Especially in your case you do not want it.
Second: Your CSV file seems to contain backslashes to quote the spaces. And some additional step seems to have added another level of quotation so than now you end up with three backslashes and a space for each original space. If this really is the case (please double check if what you wrote in your question is correct, otherwise my answer doesn't fit), you need to unquote this before you can use it.
There are security issues involved in using eval, so do not use it lightly (this disclaimer is necessary whenever proposing to use eval), but if you have trust in the input you are handling to not contain any nastinesses, then you can do this using this code:
while IFS="," read orig new pat
do
eval eval mv -v "$pat$new" "$pat$orig"
done < new.csv
Using this, two levels of quotation are evaluated (that's what eval does) before the mv command is executed.
I strongly suggest to do a dry run by adding echo before the mv first. Then instead of executing your commands they are merely printed first.
I'm trying to figure out what I thought would be a trivial issue in BASH, but I'm having difficulty finding the correct syntax. I want to loop over an array of values, one of them being an asterisk (*), I do not wish to have any wildcard expansion happening during the process.
WHITELIST_DOMAINS="* *.foo.com *.bar.com"
for domain in $WHITELIST_DOMAINS
do
echo "$domain"
done
I have the above, and I'm trying to get the following output:
*
*.foo.com
*.bar.com
Instead of the above, I get a directory listing on the current directory, followed by *.foo.com and *.bar.com
I know I need some escaping or quoting somewhere.. the early morning haze is still thick on my brain.
I've reviewed these questions:
How to escape wildcard expansion in a variable in bash?
Stop shell wildcard character expansion?
Your problem is that you want an array, but you wrote a single string that contains the elements with spaces between them. Use an array instead.
WHITELIST_DOMAINS=('*' '*.foo.com' '*.bar.com')
Always use double quotes around variable substitutions (i.e. "$foo"), otherwise the shell splits the the value of the variable into separate words and treats each word as a filename wildcard pattern. The same goes for command substitution: "$(somecommand)". For an array variable, use "${array[#]}" to expand to the list of the elements of the array.
for domain in "${WHITELIST_DOMAINS[#]}"
do
echo "$domain"
done
For more information, see the bash FAQ about arrays.
You can use array to store them:
array=('*' '*.foo.com' '*.bar.com')
for i in "${array[#]}"
do
echo "$i"
done
When using a POSIX shell, the following
touch {quick,man,strong}ly
expands to
touch quickly manly strongly
Which will touch the files quickly, manly, and strongly, but is it possible to dynamically create the expansion? For example, the following illustrates what I want to do, but does not work because of the order of expansion:
TEST=quick,man,strong #possibly output from a program
echo {$TEST}ly
Is there any way to achieve this? I do not mind constricting myself to Bash if need be. I would also like to avoid loops. The expansion should be given as complete arguments to any arbitrary program (i.e. the program cannot be called once for each file, it can only be called once for all files). I know about xargs but I'm hoping it can all be done from the shell somehow.
... There is so much wrong with using eval. What you're asking is only possible with eval, BUT what you might want is easily possible without having to resort to bash bug-central.
Use arrays! Whenever you need to keep multiple items in one datatype, you need (or, should use) an array.
TEST=(quick man strong)
touch "${TEST[#]/%/ly}"
That does exactly what you want without the thousand bugs and security issues introduced and concealed in the other suggestions here.
The way it works is:
"${foo[#]}": Expands the array named foo by expanding each of its elements, properly quoted. Don't forget the quotes!
${foo/a/b}: This is a type of parameter expansion that replaces the first a in foo's expansion by a b. In this type of expansion you can use % to signify the end of the expanded value, sort of like $ in regular expressions.
Put all that together and "${foo[#]/%/ly}" will expand each element of foo, properly quote it as a separate argument, and replace each element's end by ly.
In bash, you can do this:
#!/bin/bash
TEST=quick,man,strong
eval echo $(echo {$TEST}ly)
#eval touch $(echo {$TEST}ly)
That last line is commented out but will touch the specified files.
Zsh can easily do that:
TEST=quick,man,strong
print ${(s:,:)^TEST}ly
Variable content is splitted at commas, then each element is distributed to the string around the braces:
quickly manly strongly
Taking inspiration from the answers above:
$ TEST=quick,man,strong
$ touch $(eval echo {$TEST}ly)