I have a bash script which uses one multi-line sed command to change a file. From the command line, this line works:
sed -e '1a\' -e '/DELIMITER="|" \' -e '/RESTRICT_ADD_CHANGE=1 \' -e '/FIELDS=UPDATE_MODE|PRIMARYKEYVALUE|PATRONFLAGS' -e '1 d' general_update_01 > general_update_01.mp
I use the same bash script for a variety of files. So I need to pass all of the sed commands from the sending application to the bash script as a single parameter. However, when it passes in from the application, I get only -e.
In the bash script, I have tried a variety of ways to receive the variable as a complete string. None of these store the variable.
sed_instructions=$(echo $6)
sed_instructions=$6
sed_instructions=$(eval "$6")
and a few other configurations.
My command line would use the variable like this:
sed $sed_instructions $filename > $filename.mp
I assume you invoke your shell script like this
script.sh -e '1a' -e '/DELIMITER="|" ' -e '/RESTRICT_ADD_CHANGE=1 ' -e '/FIELDS=UPDATE_MODE|PRIMARYKEYVALUE|PATRONFLAGS' -e '1 d' general_update_01
What you want to do here is store the nth parameter as the filename, and the first n-1 parameters in an array
#!/usr/bin/env
n=$#
filename=${!n}
sed_commands=("${#:1:n-1}")
# Now, call sed
sed "${sed_commands[#]}" "$filename" > "${filename}.mp"
To demonstrate that code in action:
$ set -- one two three four
$ n=$#
$ filename=${!n}
$ args=("${#:1:n-1}")
$ declare -p filename args
declare -- filename="four"
declare -a args=([0]="one" [1]="two" [2]="three")
Related
I have a properties file that looks like this:
mysql.username=USERNAME
mysql.pass=PASS
I need to change USERNAME and PASS with variable values passed to shell script. I cannot use sed since that will work fine the first time to replace USERNAME and PASS with "values" but once they are replaced, sed will not find the variable names in the file the second time the script runs and thus an issue.
How can this be handled?
I believe the easiest way to achieve your goal is to always execute the substitutions on the untouched template file rather than trying to work on the previous version of the resulting file :
# produce a first version of the file.properties
sed -e 's/USERNAME/user1/' -e 's/PASS/pass1/' file.properties.template > file.properties
# succesfully produce a second version of the file.properties
sed -e 's/USERNAME/user2/' -e 's/PASS/pass2/' file.properties.template > file.properties
Another alternative would be to change the input (or code?) of your script so that you would base your search&replace operation not on placeholder values but rather on the properties names :
$ cat test.sh
#!/bin/bash
target=$1
shift
while key=$1 && value=$2 && shift 2; do
sed -i "s/^$key=.*/$key=$value/" $target
done
$ cat test.props
mysql.hostname=HOSTNAME
mysql.username=USERNAME
mysql.pass=PASS
$ ./test.sh test.props mysql.username sa mysql.pass clearTextPasswordsAreTerrible
$ cat test.props
mysql.hostname=HOSTNAME
mysql.username=sa
mysql.pass=clearTextPasswordsAreTerrible
$ ./test.sh test.props mysql.username secondTry mysql.pass successfullyModified
$ cat test.props
mysql.hostname=HOSTNAME
mysql.username=secondTry
mysql.pass=successfullyModified
I'm trying to export some environment variables for use by a TomCat process.
There's a few ways to do this (I know how to solve the overall problem), but it bugged me that I didn't know how to do this particular shell task.
Tomcat recommends that all your environment customizations should be exported by "$CATALINA_HOME/bin/setenv.sh".
This whole thing is gonna be stuffed into a Docker container, so the only parameterizability will be via Docker env variables (let's assume for this task that I don't want to use volume mounts or create setenv.sh during the build process).
First, observe that docker run -e can be used to pass environment into the container:
🍔 docker run -eMY_VAR=SUP alpine env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=a528b6fc264b
MY_VAR=SUP
no_proxy=*.local, 169.254/16
HOME=/root
If we wanted to copy all of that env into setenv.sh, it's as simple as:
SETENV="/usr/local/tomcat/bin/setenv.sh"
echo '#!/bin/sh' > "$SETENV"
echo 'export -p' >> "$SETENV"
env >> "$SETENV"
But copying everything somewhat defeats the point of setenv.sh -- which is, to give your tomcat process a clean environment, with only intentional customizations.
So, we can agree on a convention for "which env vars are ones that we want to pass through to setenv.sh". Everything prefixed with MY_.
And now we get to an interesting shell problem.
env | grep '^MY_' | sed 's/^MY_/EXPORT /'
This gets us pretty close. Output looks like:
🍔 docker run -e MY_VAR=hey alpine sh -c "env | grep '^MY_' | sed 's/^MY_/EXPORT /'"
EXPORT VAR=hey
So, we've selected from the env command: only env vars prefixed with MY_. And we can redirect that output to setenv.sh.
Why do I say "pretty close"? Looks like we're done, right?
Try this for size:
🍔 docker run -e MY_VAR='multi
quote> line
quote> string' alpine sh -c "env | grep '^MY_' | sed 's/^MY_/EXPORT /'"
EXPORT VAR=multi
The script only worked for a simple subset of possibilities. i.e. we only managed to export the first line of our multi-line string.
For your convenience: env output for multi-line strings looks like this:
🍔 docker run -e MY_VAR='multi
line
string' alpine env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=0d0afaac6bec
MY_VAR=multi
line
string
no_proxy=*.local, 169.254/16
HOME=/root
I hesitate to try and tackle this using awk; there may be further string escaping complications that I have not considered.
I wonder whether there's a better way altogether to select & serialize a subset of exported environment?
EDIT: I negligently tagged this as a bash question, when really my intention was to pose an sh question. Specifically my intention is to get something that will work with no dependencies other than those that come with the alpine docker image. i.e. BusyBox sh, sed, grep, awk, env.
I've retained the bash tag so as not to punish the initial answer that was submitted when this was a bash-only question.
But I will give preference to an sh-compatible answer, and in particular to one that works with just the BusyBox UNIX utils.
So you need several things:
Enumerate the environment variables and select a subset.
For each selected environment variable, emit sh code that sets the variable to the desired value.
You can use export -p if you want to export all variables in a form that can be read back in, but parsing it to select only certain variables is harder. One way to make use of export -p is to unset the other variables. This only works if none of the environment variables is read-only, but you can work around that by running a separate shell instance (as opposed to a subshell).
To gather the list of variables to unset, you only need to get a superset of the list of all environment variables, and remove the ones you want to keep. You can easily do that by filtering the env output. I do that with a simple grep, you may want to use more complex code if your criteria for inclusion are more complex than “begins with a specific prefix”.
The occasional false positive due to a variable containing a newline followed by a valid variable name and an equal sign will only lead to calling unset on a non-existent variable, which does nothing. The desired variables are removed from the exclusion list, so the final output will never omit a desired variable.
excluded=$(env | LC_ALL=C sed -n 's/^\([A-Z_a-z][0-9A-Z_a-z]*\)=.*/\1/p' |
grep -v 'MY_')
sh -c 'unset $1; export -p' sh "$excluded" >setenv.sh
Dash prints an extra export PATH (with no value) if PATH was in the environment when it was invoked. If that bothers you, change sh -c … to (unset PATH; sh -c …).
Assuming GNU grep:
grep --null '^MY_' </proc/self/environ
...will emit your environment variables in NUL-delimited form (newlines intact).
Similarly, if you have bash:
while IFS= read -r -d '' vardef; do
[[ $vardef = MY_* ]] && printf '%s\0' "$vardef"
done </proc/self/environ
Note that if these variables were set in the same shell session, you may need to create a subprocess for /proc/self/environ to be updated:
(while IFS= read -r -d '' vardef; do
[[ $vardef = MY_* ]] && printf '%s\0' "$vardef"
done </proc/self/environ)
alpine image doesn't ship with bash.
You can use this script to extract all MY_* variables including newline variables:
docker run -e MY_FOO=bar -e MY_VAR="multi' export MY_INJECTED='val" -e MY_VAR2=$'multi
0MY_line=val
string' alpine sh -c "awk -v RS='\06' -F= '/^MY_/{k=\$1; sub(/^[^=]+=/, \"\");
gsub(/\047/, \"\047\\\\\\047\047\"); printf \"export %s=\047%s\047\n\", k, \$0
}' /proc/self/environ"
This will output:
export MY_FOO='bar'
export MY_VAR='multi'\'' export MY_INJECTED='\''val'
export MY_VAR2='multi
0MY_line=val
string'
Here is how awk works:
-v RS='\6': sets record separator as \6 works for nul byte as well (assuming you don't have \6 in value)
-F=: sets field separator as =
/^MY_/: Only process records starting with MY_
store variable name or $1 in variable k
Using sub function get part after = in $0
Using print format output so that it can be used in $CATALINA_HOME/bin/setenv.sh file.
\047 is for printing single quote
what about
declare -p ${!MY_*}
and
declare -p ${!MY_*} | sed -r 's/^declare (-[^ ]*)* MY_/export /'
or
declare -p ${!MY_*} | sed 's/^declare \(-[^ ]*\)* MY_/export /'
EDIT posix compliant version :
some env or printenv accept -0 option to end each output line by \0 rather than a newline. Thus
env -0 | perl -ne 'BEGIN{$/="\0";$\="\n";$q="\047"}next unless /^MY_/;chomp;s/$q/$q\\$q$q/;s/=/=$q/;s/$/$q/;print'
How it works
$/ : input record separator
$\ : output record separator
$q : variable to store single quote (\047) because of surrounding single quotes in command
next : to filter "MY_" variables
chomp : removes the input separator
s/// : quote substitution
EDIT: variation of perl version in posix shell
env -0 | xargs -0 sh -c 'for entry; do [[ $entry = MY_* ]] || continue; printf "%s=\047%s\047\n" "${entry%%=*}" "$(echo "${entry#*=}" | sed '\''s/\x27/\x27\\\x27\x27/g'\'' )"; done' -
I want to be able to add newline characters before every occurences of some tokens appearing in some .tex files that I possess, some of those tokens are '\itemQ', '\pagebreakQ'. I created a procedure that ends up creating a command for sed stored in $sedInpt:
~$ echo "$sedInpt"
-e s/\(\\itemQ\)/\n\1/ -e s/\(\\pagebreakQ\)/\n\1/
I want to use "$sedInpt" as a command for sed:
echo "$inputText" | eval "sed ${sedInpt}"
but if I do the following as a test:
echo 'hello\itemQ' | eval "sed ${sedInpt}"
hello\itemQ
you can see there ain't any newline that has been added before \itemQ.
So I've tried debugging this way of doing thing by calling bash -x to see what's happened in detail:
~$ bash -x
~$ echo "hello\itemQ" | eval "sed ${sedInpt}"
+ echo 'hello\itemQ'
+ eval 'sed -e s/\(\\itemQ\)/\n\1/ -e s/\(\\pagebreakQ\)/\n\1/'
++ sed -e 's/(\itemQ)/n1/' -e 's/(\pagebreakQ)/n1/'
hello\itemQ
you can see that the backslashes of \n and \1 and even the ones before ( and ) that I had placed in "$sedInpt" seem to have disappeared when parsed by eval.
So I am bit lost on what to do next to do what I want.. any ideas?
You could also just combine them into a single command, which in my opinion is more straightforward:
$ cat /tmp/sed.sh
sedInpt='s/\(\\itemQ\)/\n\1/; s/\(\\pagebreakQ\)/\n\1/'
echo "hello\itemQ" | sed "$sedInpt"
$ /tmp/sed.sh
hello
\itemQ
Edit: As #123 rightly points out, storing commands in variables is dangerous and should be avoided if possible. If you have complete control over what is stored, it should be safe, but if it comes from any sort of user input, it is a "Command Injection" vulnerability.
Following #Inian advice I managed to achieve what I wanted to do in this way:
~$ sedInpt=( -e 's/\(\\itemQ\)/\n\1/' -e 's/\(\\pagebreakQ\)/\n\1/' )
~$ echo "hello\itemQ" | sed "${sedInpt[#]}"
hello
\itemQ
Im new bee to Shell script. Im trying to do search and replace for Command line argument value.
Ex: Sh script.sh 'ID,EmpName,Address'
In command line for $1 I have value like ID,EmpName,Address
Expected output : ID: chararray,EmpName: chararray,Address: chararray
Code tried
Sh script.sh 'ID,EmpName,Address'
Print 'sed -e 's/,/: chararray,/g' '"$2"''
You need:
echo "$2" | sed -e 's/,/: chararray,/'
or similar instead of:
sed -e 's/,/: chararray,/g' "$2"
As written sed thinks you are asking it to run on a file named ID,EmpName,Address, not on a string with that value.
Caveat: I have no idea what hadoop or apache-pig as tagged in your question are, the above is how to write the code in UNIX shell.
I want to set a command as configuration option of a program.
There are parameters passed to the command.
But I want the output of the command to be replaced with sed.
In principle I want to to do something like this:
option = cmd <arg1> <arg2> | sed s/x/y/
But the the option can only set this way:
option = cmd
and the arguments are automatically append by the program.
So I want to "reverse" the pipe like this:
option = sed s/x/y/ < <(cmd)
But since the program appends the arguments the following would
be exectuted:
sed s/x/y/ < <(cmd) <arg1> <arg2>
but I wanted
sed s/x/y/ < <(cmd <arg1> <arg2>)
Since there are no placeholders in the option for the arguments I am not able to close the parenthese properly.
Is there any way around this without writing a wrapper script?
If you're careful with quoting, you can do it like this:
option = bash -c 'cmd "$1" "$2" | sed "s/x/y/"' sub
The word "sub" at the end is arbitrary but necessary; it's there because $1 in the subshell created with bash -c is actually the second positional argument, not the first one.