Bash's build in command printf supports the %q format string, which escapes the content of a variable for shell input.
I have tried some options::q only escaped space, and gnu printf does not support %q.
Currently, I use below code:
set valq = `echo $val:q | bash -c 'read q;printf %q "$q"'`
/path/to/executable $valq
I do not like csh script having dependency of bash. Is there any csh native solution for this?
Thanks.
Here is a test code for illustrating the problem I have met.
wrapper.csh
#!/bin/csh -f
set i = 1
set tst1 = ""
set tst2 = ""
while ( $i <= $#argv )
set arg = "$argv[$i]"
set tst1 = ($tst1:q $arg:q)
set arg2 = `echo $arg:q | bash -c 'read q;printf %q "$q"'`
set tst2 = "$tst2:q $arg2:q"
# i = $i + 1
end
echo "====case 1===="
./test.csh $tst1:q
./test.csh $tst1
./test.csh $tst2
echo "====case 2===="
csh -cf "./test.csh $tst1"
csh -cf "./test.csh $tst1:q"
csh -cf "./test.csh $tst2"
test.csh
#!/bin/csh -f
echo -n "TEST ARG:"
set i = 1
while ($i <= $#argv)
echo -n "${i}:$argv[$i] "
# i = $i + 1
end
echo
Test Results 1:
>./wrapper.csh "a ()" b c
====case 1====
TEST ARG:1:a () 2:b 3:c
TEST ARG:1:a 2:() 3:b 4:c
TEST ARG:1:a\ 2:\(\) 3:b 4:c
====case 2====
Badly placed ()'s.
Badly placed ()'s.
TEST ARG:1:a () 2:b 3:c
Test Results 2:
bash>./wrapper.csh "'\"a ()" b c csh>./wrapper.csh "'"'"'"a ( ) " b c
====case 1====
TEST ARG:1:'"a () 2:b 3:c
TEST ARG:1:'"a 2:() 3:b 4:c
TEST ARG:1:\'\"a\ 2:\(\) 3:b 4:c
====case 2====
Unmatched '.
Unmatched '.
TEST ARG:1:'"a () 2:b 3:c
Summary for the test:
If commands is directly called inside csh, then $val:q is the proper usage.
If commands is passed by arguments, then printf %q is the proper usage.
Just use /path/to/executable "$val".
Update
If variables are expanded within " (as in csh -cf "test.csh $tst1") and if special characters and multiple words are to be preserved, the words must indeed be quoted. But the special printf of bash isn't indispensable for this; we could do it e. g. with:
set tst1q=`printf " '%s'" $tst1:q`
csh -cf "test.csh $tst1q"
(the normal printf without %q).
Update
To allow both " and ', you can after you initially do
set s='s/[] "$&-*;<>?`|~[]/\\&/g'
replace bash -c 'read q;printf %q "$q"' with sed "$s" in wrapper.csh.
The regular expression
[] "$&-*;<>?`|~[]
is a bracket expression, a list of characters enclosed in []. It matches a single character which is to be prepended with a backslash by the replacement \\& (the special character & refers to the matched character). I didn't include the characters , and ^ (they are escaped by printf %q, but that's not needed in csh), while I included ~ (which isn't escaped by printf %q, but needs to be in csh - try wrapper.csh "~").
Related
I'm trying to convert a string that has either - (hyphen) or _ (underscore) to Capital_Case string.
#!/usr/bin/env sh
function cap_case() {
[ $# -eq 1 ] || return 1;
_str=$1;
_capitalize=${_str//[-_]/_} | sed -E 's/(^|_)([a-zA-Z])/\u\2/g'
echo "Capitalize:"
echo $_capitalize
return 0
}
read string
echo $(cap_case $string)
But I don't get anything out.
First I am replacing any occurrence of - and _ with _ ${_str//[-_]/_}, and then I pipe that string to sed which finds the first letter, or _ as the first group, and then the letter after the first group in the second group, and I want to uppercase the found letter with \u\2. I tried with \U\2 but that didn't work as well.
I want the string some_string to become
Some_String
And string some-string to become
Some_String
I'm on a mac, using zsh if that is helpful.
EDIT: More generic solution here to make each field's first letter Capital.
echo "some_string_other" | awk -F"_" '{for(i=1;i<=NF;i++){$i=toupper(substr($i,1,1)) substr($i,2)}} 1' OFS="_"
Following awk may help you.
echo "some_string" | awk -F"_" '{$1=toupper(substr($1,1,1)) substr($1,2);$2=toupper(substr($2,1,1)) substr($2,2)} 1' OFS="_"
Output will be as follows.
echo "some_string" | awk -F"_" '{$1=toupper(substr($1,1,1)) substr($1,2);$2=toupper(substr($2,1,1)) substr($2,2)} 1' OFS="_"
Some_String
This being zsh, you don't need sed (or even a function, really):
$ s=some-string-bar
$ print ${(C)s:gs/-/_}
Some_String_Bar
The (C) flag capitalizes words (where "words" are defined as sequences of alphanumeric characters separated by other characters); :gs/-/_ replaces hyphens with underscores.
If you really want a function, it's cap_case () { print ${(C)1:gs/-/_} }.
pure bash:
#!/bin/bash
camel_case(){
local d display string
declare -a strings # = scope local
[ "$2" ] && d="$2" || d=" " # optional output delimiter
ifs_ini="$IFS"
IFS+='_-' # we keep initial IFS
strings=( "$1" ) # array
for string in ${strings[#]} ; do
display+="${string^}$d"
done
echo "${display%$d}"
IFS="$ifs_ini"
}
camel_case "some-string_here" "_"
camel_case "some-string_here some strings here" "+"
camel_case "some-string_here some strings here"
echo "$BASH_VERSION"
exit
output:
Some_String_Here
Some+String+Here+Some+Strings+Here
Some String Here Some Strings Here
4.4.18(1) release
You can try this gnu sed
echo 'some_other-string' | sed -E 's/(^.)/\u&/;s/[_-](.)/_\u\1/g'
Explains :
s/(^.)/\u&/
(^.) match the first char and \u& put the match in capital letter.
s/[_-](.)/_\u\1/g
[_-](.) capture a char preceded by _ or - and replace it by _ and the matched char in capital letter.
The g at the end tell sed to make the replacement for each char which meet the criteria
You didn't assign to _capitalize - you set a _capitalize environment variable for the empty command that you piped into sed.
You probably meant
_capitalize=$(<<<"${_str//[-_]/_}" sed -E 's/(^|_)([a-zA-Z])/\1\u\2/g')
Note also that ${//} isn't standard shell, so you really ought to specify an interpreter other than sh.
A simpler approach would be simply:
#!/bin/sh
cap_case() {
printf "Capitalize: "
echo "$*" | sed -e 'y/-/_/' -e 's/\(^\|_\)[[:alpha:]]/\U&/g'
}
echo $(cap_case "snake_case")
Note that the \u / \U replacement is a GNU extension to sed - if you're using a non-GNU implementation, check whether it supports this feature.
#!/bin/csh -f
foreach line ("`cat test`")
set x=`echo "$line" | awk '{split($0, b, " "); print b[1]}'`
echo "$x"
end
test file contains following contents:
How to Format
Stack[7] Overflow
Put returns between paragraphs
On executing the script I am getting following error:
set: No match.
How to store the string which contains special character like square brackets [] in a variable and then use them in code?
The problem is that you're doing:
set x = [some string with shell globbing characters]
This won't work for the same reason that set x = foo works, but set x = [foo] doesn't. You need to use set x = "[foo]" (or '[foo]') to escape the special shell globbing characters ([ and ] in this case).
Nesting quotes in the C shell is pretty hard, and it's one the reasons it's generally discouraged to use the C shell for scripting. It's perhaps possible for your command, but I'm not smart enough (or too lazy) to figure out how. My solution is typically to set the special noglob variable to prevent expansion of globbing characters:
set noglob
foreach line ("`cat test`")
set x = `echo "$line" | awk '{split($0, b, " "); print b[1]}'`
echo "$x"
end
outputs:
How
Stack[7]
Put
P.S. There is an easier way to echo the first word of every line; put it in a list:
set noglob
foreach line ("`cat test`")
set x = ($line)
echo "$x[1]"
end
I have a shell script to automate builds of my programm. I need to transform versioning numbers like V4_5_1-RC1 to 4.5.1-RC1. The V should be removed and the _ should be replaced with .. I tried several things, for example with sed:
$NAMEEXT = "V4_5_1-RC1"
$lffNameRSC = ${sed -e "s/V//g" <<< $NAMEEXT}
$lffNameRSC = ${sed -e "s/_/./g" <<< $lffNameRSC}
echo $lffNameRSC
but I'm getting errors.
./makerelease.sh: line 113: ${sed -e "s/V//g" <<< $NAMEEXT}: bad substitution
./makerelease.sh: line 114: ${sed -e "s/_/./g" <<< $lffNameRSC}: bad substitution
there should be no spaces around =
there should be $(..) instead of ${..} to evaluate the command
there should not be $ in variable assignment statement
With
#!/bin/bash
NAMEEXT="V4_5_1-RC1"
lffNameRSC=$(sed -e "s/V//g" <<< $NAMEEXT)
lffNameRSC=$(sed -e "s/_/./g" <<< $lffNameRSC)
echo $lffNameRSC
you will get
4.5.1-RC1
And, by the way, it could be done easier, like
$> echo "V4_5_1-RC1" | sed "s/V//g; s/_/./g"
4.5.1-RC1
Bash parameter expansion can do what you want without any external tools:
NAMEEXT="V4_5_1-RC1"
version=${NAMEEXT#V} # remove the leading V
version=${version//_/.} # replace all _ with .
echo $version # ==> 4.5.1-RC1
I'm trying to write a bash function that uses perl to find and replace characters. I've written the following function:
find_replace() {
perl -p -i -e "s/$1/$2/g" "$3"
}
It is not working right now, I think because $1 and $2 are being escaped by the quotation marks that surround them (which as far as I know, are a part of the perl syntax that needs to be there).
Any tips on how to make this function work (or a better way to write it that avoids this problem)?
EDIT:
Following Barmar's suggestion, here is the output when I attempt to run the function:
dholtz$ find_replace \001 , revenue_by_offer_tid
+ find_replace 001 , revenue_by_offer_tid
+ perl -p -i -e ''\''s/001/,/g'\''' revenue_by_offer_tid
++ update_terminal_cwd
++ local 'SEARCH= '
++ local REPLACE=%20
++ local PWD_URL=file://Dave-Mac-2.local/Users/dholtz
++ printf '\e]7;%s\a' file://Dave-Mac-2.local/Users/dholtz
dholtz$ head revenue_by_offer_tid
+ head revenue_by_offer_tid
Friday00228686050.0
Friday00228690410.0
Friday017438366585.040000000000004
Friday017438366591.3200000000000003
Friday017438366600.12
Friday0174383666114.759999999999962
Friday017438371407.440000000000006
Friday0174383815118.599999999999977
Friday017438382221.5600000000000005
Friday017438383663.480000000000002
Expected output is:
Friday,0,0,22,86860,50.0
Friday,0,0,22,86904,10.0
Friday,0,1,7438,36658,5.040000000000004
Friday,0,1,7438,36659,1.3200000000000003
Friday,0,1,7438,36660,0.12
Friday,0,1,7438,36661,14.759999999999962
Friday,0,1,7438,37140,7.440000000000006
Friday,0,1,7438,38151,18.599999999999977
Friday,0,1,7438,38222,1.5600000000000005
Friday,0,1,7438,38366,3.480000000000002
The backslashes were preventing the quotes from being processed properly
find_replace() {
perl -p -i -e "s/$1/$2/g" "$3"
}
The bash manual tells:
A non-quoted backslash (\) is the escape character. It preserves the
literal value of the next character that follows, ...
So, as we see in the output above, your command
find_replace \001 , revenue_by_offer_tid
is treated as (\ just unnecessarily preserving the 0)
find_replace 001 , revenue_by_offer_tid
- not as you wanted. To preserve the backslash, you must quote it in the command line you enter, e. g.
find_replace \\001 , revenue_by_offer_tid
I want to pipe the output of a "template" file into MySQL, the file having variables like ${dbName} interspersed. What is the command line utility to replace these instances and dump the output to standard output?
The input file is considered to be safe, but faulty substitution definitions could exist. Performing the replacement should avoid performing unintended code execution.
Update
Here is a solution from yottatsa on a similar question that only does replacement for variables like $VAR or ${VAR}, and is a brief one-liner
i=32 word=foo envsubst < template.txt
Of course if i and word are in your environment, then it is just
envsubst < template.txt
On my Mac it looks like it was installed as part of gettext and from MacGPG2
Old Answer
Here is an improvement to the solution from mogsie on a similar question, my solution does not require you to escale double quotes, mogsie's does, but his is a one liner!
eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
The power on these two solutions is that you only get a few types of shell expansions that don't occur normally $((...)), `...`, and $(...), though backslash is an escape character here, but you don't have to worry that the parsing has a bug, and it does multiple lines just fine.
Sed!
Given template.txt:
The number is ${i}
The word is ${word}
we just have to say:
sed -e "s/\${i}/1/" -e "s/\${word}/dog/" template.txt
Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation.
Use /bin/sh. Create a small shell script that sets the variables, and then parse the template using the shell itself. Like so (edit to handle newlines correctly):
File template.txt:
the number is ${i}
the word is ${word}
File script.sh:
#!/bin/sh
#Set variables
i=1
word="dog"
#Read in template one line at the time, and replace variables (more
#natural (and efficient) way, thanks to Jonathan Leffler).
while read line
do
eval echo "$line"
done < "./template.txt"
Output:
#sh script.sh
the number is 1
the word is dog
I was thinking about this again, given the recent interest, and I think that the tool that I was originally thinking of was m4, the macro processor for autotools. So instead of the variable I originally specified, you'd use:
$echo 'I am a DBNAME' | m4 -DDBNAME="database name"
Create rendertemplate.sh:
#!/usr/bin/env bash
eval "echo \"$(cat $1)\""
And template.tmpl:
Hello, ${WORLD}
Goodbye, ${CHEESE}
Render the template:
$ export WORLD=Foo
$ CHEESE=Bar ./rendertemplate.sh template.tmpl
Hello, Foo
Goodbye, Bar
template.txt
Variable 1 value: ${var1}
Variable 2 value: ${var2}
data.sh
#!/usr/bin/env bash
declare var1="value 1"
declare var2="value 2"
parser.sh
#!/usr/bin/env bash
# args
declare file_data=$1
declare file_input=$2
declare file_output=$3
source $file_data
eval "echo \"$(< $file_input)\"" > $file_output
./parser.sh data.sh template.txt parsed_file.txt
parsed_file.txt
Variable 1 value: value 1
Variable 2 value: value 2
Here's a robust Bash function that - despite using eval - should be safe to use.
All ${varName} variable references in the input text are expanded based on the calling shell's variables.
Nothing else is expanded: neither variable references whose names are not enclosed in {...} (such as $varName), nor command substitutions ($(...) and legacy syntax `...`), nor arithmetic substitutions ($((...)) and legacy syntax $[...]).
To treat a $ as a literal, \-escape it; e.g.:\${HOME}
Note that input is only accepted via stdin.
Example:
$ expandVarsStrict <<<'$HOME is "${HOME}"; `date` and \$(ls)' # only ${HOME} is expanded
$HOME is "/Users/jdoe"; `date` and $(ls)
Function source code:
expandVarsStrict(){
local line lineEscaped
while IFS= read -r line || [[ -n $line ]]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n
# Escape ALL chars. that could trigger an expansion..
IFS= read -r -d '' lineEscaped < <(printf %s "$line" | tr '`([$' '\1\2\3\4')
# ... then selectively reenable ${ references
lineEscaped=${lineEscaped//$'\4'{/\${}
# Finally, escape embedded double quotes to preserve them.
lineEscaped=${lineEscaped//\"/\\\"}
eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
done
}
The function assumes that no 0x1, 0x2, 0x3, and 0x4 control characters are present in the input, because those chars. are used internally - since the function processes text, that should be a safe assumption.
here's my solution with perl based on former answer, replaces environment variables:
perl -p -e 's/\$\{(\w+)\}/(exists $ENV{$1}?$ENV{$1}:"missing variable $1")/eg' < infile > outfile
I would suggest using something like Sigil:
https://github.com/gliderlabs/sigil
It is compiled to a single binary, so it's extremely easy to install on systems.
Then you can do a simple one-liner like the following:
cat my-file.conf.template | sigil -p $(env) > my-file.conf
This is much safer than eval and easier then using regex or sed
Here is a way to get the shell to do the substitution for you, as if the contents of the file were instead typed between double quotes.
Using the example of template.txt with contents:
The number is ${i}
The word is ${word}
The following line will cause the shell to interpolate the contents of template.txt and write the result to standard out.
i='1' word='dog' sh -c 'echo "'"$(cat template.txt)"'"'
Explanation:
i and word are passed as environment variables scopped to the execution of sh.
sh executes the contents of the string it is passed.
Strings written next to one another become one string, that string is:
'echo "' + "$(cat template.txt)" + '"'
Since the substitution is between ", "$(cat template.txt)" becomes the output of cat template.txt.
So the command executed by sh -c becomes:
echo "The number is ${i}\nThe word is ${word}",
where i and word are the specified environment variables.
If you are open to using Perl, that would be my suggestion. Although there are probably some sed and/or AWK experts that probably know how to do this much easier. If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily, but you might just as well put it into a standard Perl script at that point.
perl -p -e 's/\$\{dbName\}/testdb/s' yourfile | mysql
A short Perl script to do something slightly more complicated (handle multiple keys):
#!/usr/bin/env perl
my %replace = ( 'dbName' => 'testdb', 'somethingElse' => 'fooBar' );
undef $/;
my $buf = <STDIN>;
$buf =~ s/\$\{$_\}/$replace{$_}/g for keys %replace;
print $buf;
If you name the above script as replace-script, it could then be used as follows:
replace-script < yourfile | mysql
file.tpl:
The following bash function should only replace ${var1} syntax and ignore
other shell special chars such as `backticks` or $var2 or "double quotes".
If I have missed anything - let me know.
script.sh:
template(){
# usage: template file.tpl
while read -r line ; do
line=${line//\"/\\\"}
line=${line//\`/\\\`}
line=${line//\$/\\\$}
line=${line//\\\${/\${}
eval "echo \"$line\"";
done < ${1}
}
var1="*replaced*"
var2="*not replaced*"
template file.tpl > result.txt
I found this thread while wondering the same thing. It inspired me to this (careful with the backticks)
$ echo $MYTEST
pass!
$ cat FILE
hello $MYTEST world
$ eval echo `cat FILE`
hello pass! world
Lots of choices here, but figured I'd toss mine on the heap. It is perl based, only targets variables of the form ${...}, takes the file to process as an argument and outputs the converted file on stdout:
use Env;
Env::import();
while(<>) { $_ =~ s/(\${\w+})/$1/eeg; $text .= $_; }
print "$text";
Of course I'm not really a perl person, so there could easily be a fatal flaw (works for me though).
It can be done in bash itself if you have control of the configuration file format. You just need to source (".") the configuration file rather than subshell it. That ensures the variables are created in the context of the current shell (and continue to exist) rather than the subshell (where the variable disappear when the subshell exits).
$ cat config.data
export parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA
export parm_user=pax
export parm_pwd=never_you_mind
$ cat go.bash
. config.data
echo "JDBC string is " $parm_jdbc
echo "Username is " $parm_user
echo "Password is " $parm_pwd
$ bash go.bash
JDBC string is jdbc:db2://box7.co.uk:5000/INSTA
Username is pax
Password is never_you_mind
If your config file cannot be a shell script, you can just 'compile' it before executing thus (the compilation depends on your input format).
$ cat config.data
parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA # JDBC URL
parm_user=pax # user name
parm_pwd=never_you_mind # password
$ cat go.bash
cat config.data
| sed 's/#.*$//'
| sed 's/[ \t]*$//'
| sed 's/^[ \t]*//'
| grep -v '^$'
| sed 's/^/export '
>config.data-compiled
. config.data-compiled
echo "JDBC string is " $parm_jdbc
echo "Username is " $parm_user
echo "Password is " $parm_pwd
$ bash go.bash
JDBC string is jdbc:db2://box7.co.uk:5000/INSTA
Username is pax
Password is never_you_mind
In your specific case, you could use something like:
$ cat config.data
export p_p1=val1
export p_p2=val2
$ cat go.bash
. ./config.data
echo "select * from dbtable where p1 = '$p_p1' and p2 like '$p_p2%' order by p1"
$ bash go.bash
select * from dbtable where p1 = 'val1' and p2 like 'val2%' order by p1
Then pipe the output of go.bash into MySQL and voila, hopefully you won't destroy your database :-).
In place perl editing of potentially multiple files, with backups.
perl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : ""/eg' \
-i.orig \
-p config/test/*
I created a shell templating script named shtpl. My shtpl uses a jinja-like syntax which, now that I use ansible a lot, I'm pretty familiar with:
$ cat /tmp/test
{{ aux=4 }}
{{ myarray=( a b c d ) }}
{{ A_RANDOM=$RANDOM }}
$A_RANDOM
{% if $(( $A_RANDOM%2 )) == 0 %}
$A_RANDOM is even
{% else %}
$A_RANDOM is odd
{% endif %}
{% if $(( $A_RANDOM%2 )) == 0 %}
{% for n in 1 2 3 $aux %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/passwd field #$n: $(grep $USER /etc/passwd | cut -d: -f$n)
{% endfor %}
{% else %}
{% for n in {1..4} %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/group field #$n: $(grep ^$USER /etc/group | cut -d: -f$n)
{% endfor %}
{% endif %}
$ ./shtpl < /tmp/test
6535
6535 is odd
$myarray[0]: a
/etc/group field #1: myusername
$myarray[1]: b
/etc/group field #2: x
$myarray[2]: c
/etc/group field #3: 1001
$myarray[3]: d
/etc/group field #4:
More info on my github
To me this is the easiest and most powerful solution, you can even include other templates using the same command eval echo "$(<template.txt):
Example with nested template
create the template files, the variables are in regular bash syntax ${VARIABLE_NAME} or $VARIABLE_NAME
you have to escape special characters with \ in your templates otherwhise they will be interpreted by eval.
template.txt
Hello ${name}!
eval echo $(<nested-template.txt)
nested-template.txt
Nice to have you here ${name} :\)
create source file
template.source
declare name=royman
parse the template
source template.source && eval echo "$(<template.txt)"
the output
Hello royman!
Nice to have you here royman :)
envsubst
please don't use anything else (ie. don't eval)