I want to stick all the variables into one variable
A=('blah')
AA=('blah2')
AAA=('blah3')
AAB=('blah4')
AAC=('blah5')
#^^lets pretend theres 100 more of these ^^
#Variable composition
#after AAA, is AAB then AAC then AAD etc etc, does that 100 times
I want them all placed into this MASTER variable
#MASTER=${A}${AA}${AAA} (<-- insert AAB, AAC and 100 more variables here)
I obviously don't want to type 100 variables in this expression because there's probably an easier way to do this. Plus I'm gonna be doing more of these therefore I need it automated.
I'm relatively new to sed, awk, is there a way to append those 100 variables into the master variable?
For this specific purpose I DO NOT want an array.
You can use a simple one-liner, quite straightforward, though more expensive:
master=$(set | grep -E '^(A|AA|A[A-D][A-D])=' | sort | cut -f2- -d= | tr -d '\n')
set lists all the variables in var=name format
grep filters out the variables we need
sort puts them in the right order (probably optional since set gives a sorted output)
cut extracts the values, removing the variable names
tr removes the newlines
Let's test it.
A=1
AA=2
AAA=3
AAB=4
AAC=5
AAD=6
AAAA=99 # just to make sure we don't pick this one up
master=$(set | grep -E '^(A|AA|A[A-D][A-D])=' | sort | cut -f2- -d= | tr -d '\n')
echo "$master"
Output:
123456
With my best guess, how about:
#!/bin/bash
A=('blah')
AA=('blah2')
AAA=('blah3')
AAB=('blah4')
AAC=('blah5')
# to be continued ..
for varname in A AA A{A..D}{A..Z}; do
value=${!varname}
if [ -n "$value" ]; then
MASTER+=$value
fi
done
echo $MASTER
which yields:
blahblah2blah3blah4blah5...
Although I'm not sure whether this is what the OP wants.
echo {a..z}{a..z}{a..z} | tr ' ' '\n' | head -n 100 | tail -n 3
adt
adu
adv
tells us, that it would go from AAA to ADV to reach 100, or for ADY for 103.
echo A{A..D}{A..Z} | sed 's/ /}${/g'
AAA}${AAB}${AAC}${AAD}${AAE}${AAF}${AAG}${AAH}${AAI}${AAJ}${AAK}${AAL}${AAM}${AAN}${AAO}${AAP}${AAQ}${AAR}${AAS}${AAT}${AAU}${AAV}${AAW}${AAX}${AAY}${AAZ}${ABA}${ABB}${ABC}${ABD}${ABE}${ABF}${ABG}${ABH}${ABI}${ABJ}${ABK}${ABL}${ABM}${ABN}${ABO}${ABP}${ABQ}${ABR}${ABS}${ABT}${ABU}${ABV}${ABW}${ABX}${ABY}${ABZ}${ACA}${ACB}${ACC}${ACD}${ACE}${ACF}${ACG}${ACH}${ACI}${ACJ}${ACK}${ACL}${ACM}${ACN}${ACO}${ACP}${ACQ}${ACR}${ACS}${ACT}${ACU}${ACV}${ACW}${ACX}${ACY}${ACZ}${ADA}${ADB}${ADC}${ADD}${ADE}${ADF}${ADG}${ADH}${ADI}${ADJ}${ADK}${ADL}${ADM}${ADN}${ADO}${ADP}${ADQ}${ADR}${ADS}${ADT}${ADU}${ADV}${ADW}${ADX}${ADY}${ADZ
The final cosmetics is easily made by hand.
One-liner using a for loop:
for n in A AA A{A..D}{A..Z}; do str+="${!n}"; done; echo ${str}
Output:
blahblah2blah3blah4blah5
Say you have the input file inputfile.txt with arbitrary variable names and values:
name="Joe"
last="Doe"
A="blah"
AA="blah2
then do:
master=$(eval echo $(grep -o "^[^=]\+" inputfile.txt | sed 's/^/\$/;:a;N;$!ba;s/\n/$/g'))
This will concatenate the values of all variables in inputfile.txt into master variable. So you will have:
>echo $master
JoeDoeblahblah2
I'm trying to parse a lattice with grep and save the output in a variable in a cshell script. But somehow I always get the error message "Illegal variable name" when adding the FILE_IN_B variable. I tried different spacings after the variables name and also tried $() ,or %.* ,instead of sed to remove the last four letters but neither works in cshell. Also tried setting the declaration in "", to no avail. I'm really desperate here...
#!/bin/csh
set FILE_IN = file.ext
set SOURCE = home/Developer
set FILE_IN_B=`sed '/.\{4\}$//' >>> "$FILE_IN"`.lat
set REC = `grep -C 1 'I=11' "$SOURCE/Lattice/$FILE_IN_B" | cut -d ' ' -f 3 | cut -d= -f 2| sed 's/sp//'`
csh has some built-in variable manipulation, including one meant for extension removal:
% set FILE_IN=file.ext
% set FILE_IN_B="${FILE_IN:r}.lat"
% echo "$FILE_IN_B"
file.lat
If sed is required then printf can be used as the standard input, similar to portable sh:
% set FILE_IN=file.ext
% set FILE_IN_B=`printf %s "$FILE_IN" | sed 's/.\{4\}$//'`.lat
% echo "$FILE_IN_B"
file.lat
I am trying to write a shell which will take an SQL file as input. Example SQL file:
SELECT *
FROM %%DB.TBL_%%TBLEXT
WHERE CITY = '%%CITY'
Now the script should extract all variables, which in this case everything starting with %%. So the output file will be something as below:
%%DB
%%TBLEXT
%%CITY
Now I should be able to extract the matching values from the user's .profile file for these variables and create the SQL file with the proper values.
SELECT *
FROM tempdb.TBL_abc
WHERE CITY = 'Chicago'
As of now I am trying to generate the file1 which will contain all the variables. Below code sample -
sed "s/[(),']//g" "T:/work/shell/sqlfile1.sql" | awk '/%%/{print $NF}' | awk '/%%/{print $NF}' > sqltemp2.sql
takes me till
%%DB.TBL_%%TBLEXT
%%CITY
Can someone help me in getting to file1 listing the variables?
You can use grep and sort to get a list of unique variables, as per the following transcript:
$ echo "SELECT *
FROM %%DB.TBL_%%TBLEXT
WHERE CITY = '%%CITY'" | grep -o '%%[A-Za-z0-9_]*' | sort -u
%%CITY
%%DB
%%TBLEXT
The -o flag to grep instructs it to only print the matching parts of lines rather than the entire line, and also outputs each matching part on a distinct line. Then sort -u just makes sure there are no duplicates.
In terms of the full process, here's a slight modification to a bash script I've used for similar purposes:
# Define all translations.
declare -A xlat
xlat['%%DB']='tempdb'
xlat['%%TBLEXT']='abc'
xlat['%%CITY']='Chicago'
# Check all variables in input file.
okay=1
for key in $(grep -o '%%[A-Za-z0-9_]*' input.sql | sort -u) ; do
if [[ "${xlat[$key]}" == "" ]] ; then
echo "Bad key ($key) in file:"
grep -n "${key}" input.sql | sed 's/^/ /'
okay=0
fi
done
if [[ ${okay} -eq 0 ]] ; then
exit 1
fi
# Process input file doing substitutions. Fairly
# primitive use of sed, must change to use sed -i
# at some point.
# Note we sort keys based on descending length so we
# correctly handle extensions like "NAME" and "NAMESPACE",
# doing the longer ones first makes it work properly.
cp input.sql output.sql
for key in $( (
for key in ${!xlat[#]} ; do
echo ${key}
done
) | awk '{print length($0)":"$0}' | sort -rnu | cut -d':' -f2) ; do
sed "s/${key}/${xlat[$key]}/g" output.sql >output2.sql
mv output2.sql output.sql
done
cat output.sql
It first checks that the input file doesn't contain any keys not found in the translation array. Then it applies sed substitutions to the input file, one per translation, to ensure all keys are substituted with their respective values.
This should be a good start, though there may be some edge cases such as if your keys or values contain characters sed would consider important (like / for example). If that is the case, you'll probably need to escape them such as changing:
xlat['%%UNDEFINED']='0/0'
into:
xlat['%%UNDEFINED']='0\/0'
I am writing a shell script for some purpose. I have a variable of the form --
var1 = "policy=set policy"
Now I need to manipulate the variable var to get the string after index =. That is I should have "set policy". Also I need to to this for many other variables where the value of "=" is not constant. Like
var2 = "bgroup = set bgroup port"
var3 = "utm = set security utm"
Can you give me an idea how to do it, please?
${var#*=}
removes the shortest match of *= from the left. Note that this is not in place: if you want to save the result, you'll have to store the result in a variable.
On a side note, this is for bash. AFAIK it also works for ksh and zsh, but not csh or tcsh.
Other ways that is not dependent on what shell you have.
$ var1="policy=set policy"
$ echo $var1 | awk '{sub(/.[^=]*=/,"")}1'
set policy
$ echo $var1 | cut -d= -f2-
set policy
$ echo $var1 | ruby -e 'puts gets.split(/=/,2)[1]'
set policy
$ echo $var1 | sed 's/.[^=]*=//'
set policy
I want to pipe the output of a "template" file into MySQL, the file having variables like ${dbName} interspersed. What is the command line utility to replace these instances and dump the output to standard output?
The input file is considered to be safe, but faulty substitution definitions could exist. Performing the replacement should avoid performing unintended code execution.
Update
Here is a solution from yottatsa on a similar question that only does replacement for variables like $VAR or ${VAR}, and is a brief one-liner
i=32 word=foo envsubst < template.txt
Of course if i and word are in your environment, then it is just
envsubst < template.txt
On my Mac it looks like it was installed as part of gettext and from MacGPG2
Old Answer
Here is an improvement to the solution from mogsie on a similar question, my solution does not require you to escale double quotes, mogsie's does, but his is a one liner!
eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
The power on these two solutions is that you only get a few types of shell expansions that don't occur normally $((...)), `...`, and $(...), though backslash is an escape character here, but you don't have to worry that the parsing has a bug, and it does multiple lines just fine.
Sed!
Given template.txt:
The number is ${i}
The word is ${word}
we just have to say:
sed -e "s/\${i}/1/" -e "s/\${word}/dog/" template.txt
Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation.
Use /bin/sh. Create a small shell script that sets the variables, and then parse the template using the shell itself. Like so (edit to handle newlines correctly):
File template.txt:
the number is ${i}
the word is ${word}
File script.sh:
#!/bin/sh
#Set variables
i=1
word="dog"
#Read in template one line at the time, and replace variables (more
#natural (and efficient) way, thanks to Jonathan Leffler).
while read line
do
eval echo "$line"
done < "./template.txt"
Output:
#sh script.sh
the number is 1
the word is dog
I was thinking about this again, given the recent interest, and I think that the tool that I was originally thinking of was m4, the macro processor for autotools. So instead of the variable I originally specified, you'd use:
$echo 'I am a DBNAME' | m4 -DDBNAME="database name"
Create rendertemplate.sh:
#!/usr/bin/env bash
eval "echo \"$(cat $1)\""
And template.tmpl:
Hello, ${WORLD}
Goodbye, ${CHEESE}
Render the template:
$ export WORLD=Foo
$ CHEESE=Bar ./rendertemplate.sh template.tmpl
Hello, Foo
Goodbye, Bar
template.txt
Variable 1 value: ${var1}
Variable 2 value: ${var2}
data.sh
#!/usr/bin/env bash
declare var1="value 1"
declare var2="value 2"
parser.sh
#!/usr/bin/env bash
# args
declare file_data=$1
declare file_input=$2
declare file_output=$3
source $file_data
eval "echo \"$(< $file_input)\"" > $file_output
./parser.sh data.sh template.txt parsed_file.txt
parsed_file.txt
Variable 1 value: value 1
Variable 2 value: value 2
Here's a robust Bash function that - despite using eval - should be safe to use.
All ${varName} variable references in the input text are expanded based on the calling shell's variables.
Nothing else is expanded: neither variable references whose names are not enclosed in {...} (such as $varName), nor command substitutions ($(...) and legacy syntax `...`), nor arithmetic substitutions ($((...)) and legacy syntax $[...]).
To treat a $ as a literal, \-escape it; e.g.:\${HOME}
Note that input is only accepted via stdin.
Example:
$ expandVarsStrict <<<'$HOME is "${HOME}"; `date` and \$(ls)' # only ${HOME} is expanded
$HOME is "/Users/jdoe"; `date` and $(ls)
Function source code:
expandVarsStrict(){
local line lineEscaped
while IFS= read -r line || [[ -n $line ]]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n
# Escape ALL chars. that could trigger an expansion..
IFS= read -r -d '' lineEscaped < <(printf %s "$line" | tr '`([$' '\1\2\3\4')
# ... then selectively reenable ${ references
lineEscaped=${lineEscaped//$'\4'{/\${}
# Finally, escape embedded double quotes to preserve them.
lineEscaped=${lineEscaped//\"/\\\"}
eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
done
}
The function assumes that no 0x1, 0x2, 0x3, and 0x4 control characters are present in the input, because those chars. are used internally - since the function processes text, that should be a safe assumption.
here's my solution with perl based on former answer, replaces environment variables:
perl -p -e 's/\$\{(\w+)\}/(exists $ENV{$1}?$ENV{$1}:"missing variable $1")/eg' < infile > outfile
I would suggest using something like Sigil:
https://github.com/gliderlabs/sigil
It is compiled to a single binary, so it's extremely easy to install on systems.
Then you can do a simple one-liner like the following:
cat my-file.conf.template | sigil -p $(env) > my-file.conf
This is much safer than eval and easier then using regex or sed
Here is a way to get the shell to do the substitution for you, as if the contents of the file were instead typed between double quotes.
Using the example of template.txt with contents:
The number is ${i}
The word is ${word}
The following line will cause the shell to interpolate the contents of template.txt and write the result to standard out.
i='1' word='dog' sh -c 'echo "'"$(cat template.txt)"'"'
Explanation:
i and word are passed as environment variables scopped to the execution of sh.
sh executes the contents of the string it is passed.
Strings written next to one another become one string, that string is:
'echo "' + "$(cat template.txt)" + '"'
Since the substitution is between ", "$(cat template.txt)" becomes the output of cat template.txt.
So the command executed by sh -c becomes:
echo "The number is ${i}\nThe word is ${word}",
where i and word are the specified environment variables.
If you are open to using Perl, that would be my suggestion. Although there are probably some sed and/or AWK experts that probably know how to do this much easier. If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily, but you might just as well put it into a standard Perl script at that point.
perl -p -e 's/\$\{dbName\}/testdb/s' yourfile | mysql
A short Perl script to do something slightly more complicated (handle multiple keys):
#!/usr/bin/env perl
my %replace = ( 'dbName' => 'testdb', 'somethingElse' => 'fooBar' );
undef $/;
my $buf = <STDIN>;
$buf =~ s/\$\{$_\}/$replace{$_}/g for keys %replace;
print $buf;
If you name the above script as replace-script, it could then be used as follows:
replace-script < yourfile | mysql
file.tpl:
The following bash function should only replace ${var1} syntax and ignore
other shell special chars such as `backticks` or $var2 or "double quotes".
If I have missed anything - let me know.
script.sh:
template(){
# usage: template file.tpl
while read -r line ; do
line=${line//\"/\\\"}
line=${line//\`/\\\`}
line=${line//\$/\\\$}
line=${line//\\\${/\${}
eval "echo \"$line\"";
done < ${1}
}
var1="*replaced*"
var2="*not replaced*"
template file.tpl > result.txt
I found this thread while wondering the same thing. It inspired me to this (careful with the backticks)
$ echo $MYTEST
pass!
$ cat FILE
hello $MYTEST world
$ eval echo `cat FILE`
hello pass! world
Lots of choices here, but figured I'd toss mine on the heap. It is perl based, only targets variables of the form ${...}, takes the file to process as an argument and outputs the converted file on stdout:
use Env;
Env::import();
while(<>) { $_ =~ s/(\${\w+})/$1/eeg; $text .= $_; }
print "$text";
Of course I'm not really a perl person, so there could easily be a fatal flaw (works for me though).
It can be done in bash itself if you have control of the configuration file format. You just need to source (".") the configuration file rather than subshell it. That ensures the variables are created in the context of the current shell (and continue to exist) rather than the subshell (where the variable disappear when the subshell exits).
$ cat config.data
export parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA
export parm_user=pax
export parm_pwd=never_you_mind
$ cat go.bash
. config.data
echo "JDBC string is " $parm_jdbc
echo "Username is " $parm_user
echo "Password is " $parm_pwd
$ bash go.bash
JDBC string is jdbc:db2://box7.co.uk:5000/INSTA
Username is pax
Password is never_you_mind
If your config file cannot be a shell script, you can just 'compile' it before executing thus (the compilation depends on your input format).
$ cat config.data
parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA # JDBC URL
parm_user=pax # user name
parm_pwd=never_you_mind # password
$ cat go.bash
cat config.data
| sed 's/#.*$//'
| sed 's/[ \t]*$//'
| sed 's/^[ \t]*//'
| grep -v '^$'
| sed 's/^/export '
>config.data-compiled
. config.data-compiled
echo "JDBC string is " $parm_jdbc
echo "Username is " $parm_user
echo "Password is " $parm_pwd
$ bash go.bash
JDBC string is jdbc:db2://box7.co.uk:5000/INSTA
Username is pax
Password is never_you_mind
In your specific case, you could use something like:
$ cat config.data
export p_p1=val1
export p_p2=val2
$ cat go.bash
. ./config.data
echo "select * from dbtable where p1 = '$p_p1' and p2 like '$p_p2%' order by p1"
$ bash go.bash
select * from dbtable where p1 = 'val1' and p2 like 'val2%' order by p1
Then pipe the output of go.bash into MySQL and voila, hopefully you won't destroy your database :-).
In place perl editing of potentially multiple files, with backups.
perl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : ""/eg' \
-i.orig \
-p config/test/*
I created a shell templating script named shtpl. My shtpl uses a jinja-like syntax which, now that I use ansible a lot, I'm pretty familiar with:
$ cat /tmp/test
{{ aux=4 }}
{{ myarray=( a b c d ) }}
{{ A_RANDOM=$RANDOM }}
$A_RANDOM
{% if $(( $A_RANDOM%2 )) == 0 %}
$A_RANDOM is even
{% else %}
$A_RANDOM is odd
{% endif %}
{% if $(( $A_RANDOM%2 )) == 0 %}
{% for n in 1 2 3 $aux %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/passwd field #$n: $(grep $USER /etc/passwd | cut -d: -f$n)
{% endfor %}
{% else %}
{% for n in {1..4} %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/group field #$n: $(grep ^$USER /etc/group | cut -d: -f$n)
{% endfor %}
{% endif %}
$ ./shtpl < /tmp/test
6535
6535 is odd
$myarray[0]: a
/etc/group field #1: myusername
$myarray[1]: b
/etc/group field #2: x
$myarray[2]: c
/etc/group field #3: 1001
$myarray[3]: d
/etc/group field #4:
More info on my github
To me this is the easiest and most powerful solution, you can even include other templates using the same command eval echo "$(<template.txt):
Example with nested template
create the template files, the variables are in regular bash syntax ${VARIABLE_NAME} or $VARIABLE_NAME
you have to escape special characters with \ in your templates otherwhise they will be interpreted by eval.
template.txt
Hello ${name}!
eval echo $(<nested-template.txt)
nested-template.txt
Nice to have you here ${name} :\)
create source file
template.source
declare name=royman
parse the template
source template.source && eval echo "$(<template.txt)"
the output
Hello royman!
Nice to have you here royman :)
envsubst
please don't use anything else (ie. don't eval)