I have a properties file that looks like this:
mysql.username=USERNAME
mysql.pass=PASS
I need to change USERNAME and PASS with variable values passed to shell script. I cannot use sed since that will work fine the first time to replace USERNAME and PASS with "values" but once they are replaced, sed will not find the variable names in the file the second time the script runs and thus an issue.
How can this be handled?
I believe the easiest way to achieve your goal is to always execute the substitutions on the untouched template file rather than trying to work on the previous version of the resulting file :
# produce a first version of the file.properties
sed -e 's/USERNAME/user1/' -e 's/PASS/pass1/' file.properties.template > file.properties
# succesfully produce a second version of the file.properties
sed -e 's/USERNAME/user2/' -e 's/PASS/pass2/' file.properties.template > file.properties
Another alternative would be to change the input (or code?) of your script so that you would base your search&replace operation not on placeholder values but rather on the properties names :
$ cat test.sh
#!/bin/bash
target=$1
shift
while key=$1 && value=$2 && shift 2; do
sed -i "s/^$key=.*/$key=$value/" $target
done
$ cat test.props
mysql.hostname=HOSTNAME
mysql.username=USERNAME
mysql.pass=PASS
$ ./test.sh test.props mysql.username sa mysql.pass clearTextPasswordsAreTerrible
$ cat test.props
mysql.hostname=HOSTNAME
mysql.username=sa
mysql.pass=clearTextPasswordsAreTerrible
$ ./test.sh test.props mysql.username secondTry mysql.pass successfullyModified
$ cat test.props
mysql.hostname=HOSTNAME
mysql.username=secondTry
mysql.pass=successfullyModified
Related
I have a bash script which uses one multi-line sed command to change a file. From the command line, this line works:
sed -e '1a\' -e '/DELIMITER="|" \' -e '/RESTRICT_ADD_CHANGE=1 \' -e '/FIELDS=UPDATE_MODE|PRIMARYKEYVALUE|PATRONFLAGS' -e '1 d' general_update_01 > general_update_01.mp
I use the same bash script for a variety of files. So I need to pass all of the sed commands from the sending application to the bash script as a single parameter. However, when it passes in from the application, I get only -e.
In the bash script, I have tried a variety of ways to receive the variable as a complete string. None of these store the variable.
sed_instructions=$(echo $6)
sed_instructions=$6
sed_instructions=$(eval "$6")
and a few other configurations.
My command line would use the variable like this:
sed $sed_instructions $filename > $filename.mp
I assume you invoke your shell script like this
script.sh -e '1a' -e '/DELIMITER="|" ' -e '/RESTRICT_ADD_CHANGE=1 ' -e '/FIELDS=UPDATE_MODE|PRIMARYKEYVALUE|PATRONFLAGS' -e '1 d' general_update_01
What you want to do here is store the nth parameter as the filename, and the first n-1 parameters in an array
#!/usr/bin/env
n=$#
filename=${!n}
sed_commands=("${#:1:n-1}")
# Now, call sed
sed "${sed_commands[#]}" "$filename" > "${filename}.mp"
To demonstrate that code in action:
$ set -- one two three four
$ n=$#
$ filename=${!n}
$ args=("${#:1:n-1}")
$ declare -p filename args
declare -- filename="four"
declare -a args=([0]="one" [1]="two" [2]="three")
I have a script that takes a file name as input in $1, and processes it...and creates an output file as ${1}.output.log and it works fine. e.g. if i tried
./myscript filename.txt
It'll process and generate output file with name: filename.txt.output.log
But When I tried to substitute a process to give input to this script like
./myscript <(echo something), it failed as it cannot create a file anymore with ${1}. output.log ; because now $1 is not an actual file and doesn't exist in my working directory where script is suppose to create an output.
Any suggestions to work around this problem?
The problem is probably that when using process substitution you are trying to create a file in /dev, more specifically, /dev/fd/63.output.log
I recommend doing this:
output_file="$( sed 's|/dev/fd/|./process_substitution-|' <<< ${1} ).output.log"
echo "my output" >> "$output_file"
We use sed to replace /dev/fd/ to ./process_substitution- so the file gets created in the current working directory (pwd) with the name process_substitution-63.output.log
I am creating a bash script that will simply use grep to look through a bunch of logs for a certain string.
Something interesting happens though.
For the purpose of testing all of the log files the files are named test1.log, test2.log, test3.log, etc.
When using the grep command:
grep -oHnR TEST Logs/test*
The output contains all instances from all files in the folder as expected.
But when using a command but contained in the bash script below:
#!/bin/bash
#start
grep -oHnR $1 $2
#end
The output displays the instances from only 1 file.
When running the script I am using the following command:
bash test.bash TEST Logs/test*
Here is an example of the expected output (what occurs when simply using grep):
Logs/test2.log:8:TEST
Logs/test2.log:20:TEST
Logs/test2.log:41:TEST
Logs/test.log:2:TEST
Logs/test.log:18:TEST
and here is an example of the output received when using the bash script:
Logs/test2.log:8:TEST
Logs/test2.log:20:TEST
Logs/test2.log:41:TEST
Can someone explain to me why this happens?
When you call the line
bash test.bash TEST Logs/test*
this will be translated by the shell to
bash test.bash TEST Logs/test1.log Logs/test2.log Logs/test3.log Logs/test4.log
(if you have four log files).
The command line parameters TEST, Logs/test1.log, Logs/test2.log, etc. will be given the names $1, $2, $3, etc.; $1 will be TEST, $2 will be Logs/test1.log.
You just ignore the remaining parameters and use just one log file when you use $2 only.
A correct version would be this:
#!/bin/bash
#start
grep -oHnR "$#"
#end
This will pass all the parameters properly and also take care of nastinesses like spaces in file names (your version would have had trouble with these).
To understand what's happening, you can use a simpler script:
#!/bin/bash
echo $1
echo $2
That outputs the first two arguments, as you asked for.
You want to use the first argument, and then use all the rest as input files. So use shift like this:
#!/bin/bash
search=$1
shift
echo "$1"
echo "$#"
Notice also the use of double quotes.
In your case, because you want the search string and the filenames to be passed to grep in the same order, you don't even need to shift:
#!/bin/bash
grep -oHnR -e "$#"
(I added the -e in case the search string begins with -)
The unquoted * is being affected by globbing when you are calling the script.
Using set -x to output what is running from the script makes this more clear.
$ ./greptest.sh TEST test*
++ grep -oHnR TEST test1.log
$ ./greptest.sh TEST "test*"
++ grep -oHnR TEST test1.log test2.log test3.log
In the first case, bash is expanding the * into the list of file names versus the second case it is being passed to grep. In the first case you actually have >2 args (as each filename expanded would become an arg) - adding echo $# to the script shows this too:
$ ./greptest.sh TEST test*
++ grep -oHnR TEST test1.log
++ echo 4
4
$ ./greptest.sh TEST "test*"
++ grep -oHnR TEST test1.log test2.log test3.log
++ echo 2
2
You probably want to escape the wildcard on your bash invocation:
bash test.bash TEST Logs/test\*
That way it'll get passed through to grep as an *, otherwise the shell will have expanded it to every file in the Logs dir whose name starts with test.
Alternatively, change your script to allow more than one file on the command line:
#!/bin/bash
hold=$1
shift
grep -oHnR $hold $#
I designed a custom script to grep a concatenated list of .bash_history backup files. In my script, I am creating a temporary file with mktemp and saving it to a variable temp. Next, I am redirecting output to that file using the cat command.
Is there a means to create a temporary file (using mktemp), redirect output to it, then store it in a variable in one command, while preserving newline characters?
The below snippet of code works just fine, but I have a feeling there is a more terse and canonical way to achieve this in one line – maybe using process substitution or something of the like.
# Concatenate all .bash_history files into a temporary file `temp`.
temp="$(mktemp)"
cat "$HOME/.bash_history."* > $temp
trap 'rm -f $temp' 0
# Set `HISTFILE` shell variable to the `temp` file.
HISTFILE="$temp"
keyword="$1"
# Search for `keyword` using the `history` command
if [[ "$keyword" ]]; then
# Enable history
set -o history
history | grep "$keyword"
# Disable history
set +o history
else
echo -e "usage: search <keyword>"
exit 0
fi
If you're comfortable with the side effect of making the assignment conditional on tempfile not previously having a nonempty value, this is straightforward via the ${var:=value} expansion:
cat "$HOME/.bash_history" >"${tempfile:=$(mktemp)}"
cat myfile.txt | f=`mktemp` && cat > "${f}"
I guess there is more than one way to do it. I found following to be working for me:
cat myfile.txt > $(echo "$(mktemp)")
Also don't forget about tee:
cat myfile.txt | tee "$(mktemp)" > /dev/null
Consider a ASCII text file (lets say it contains code of a non-shell scripting language):
Text_File.msh:
spool on to '$LOG_FILE_PATH/logfile.log';
login 'username' 'password';
....
Now if this were a shell script I could run it as $ sh Text_File.msh and the shell would automatically expand the variables.
What I want to do is have shell expand these variables and then create a new file as Text_File_expanded.msh as follows:
Text_File_expanded.msh:
spool on to '/expanded/path/of/the/log/file/../logfile.log';
login 'username' 'password';
....
Consider:
$ a=123
$ echo "$a"
123
So technically this should do the trick:
$ echo "`cat Text_File.msh`" > Text_File_expanded.msh
...but it doesn't work as expected and the output-file while is identical to the source.
So I am unsure how to achieve this.. My goal is make it easier to maintain the directory paths embedded within my non-shell scripts. These scripts cannot contain any UNIX code as it is not compiled by the UNIX shell.
This question has been asked in another thread, and this is the best answer IMO:
export LOG_FILE_PATH=/expanded/path/of/the/log/file/../logfile.log
cat Text_File.msh | envsubst > Text_File_expanded.msh
if on Mac, install gettext first: brew install gettext
see:
Forcing bash to expand variables in a string loaded from a file
This solution is not elegant, but it works. Create a script call shell_expansion.sh:
echo 'cat <<END_OF_TEXT' > temp.sh
cat "$1" >> temp.sh
echo 'END_OF_TEXT' >> temp.sh
bash temp.sh >> "$2"
rm temp.sh
You can then invoke this script as followed:
bash shell_expansion.sh Text_File.msh Text_File_expanded.msh
If you want it in one line (I'm not a bash expert so there may be caveats to this but it works everywhere I've tried it):
when test.txt contains
${line1}
${line2}
then:
>line1=fark
>line2=fork
>value=$(eval "echo \"$(cat test.txt)\"")
>echo "$value"
line1 says fark
line2 says fork
Obviously if you just want to print it you can take out the extra value=$() and echo "$value".
If a Perl solution is ok for you:
Sample file:
$ cat file.sh
spool on to '$HOME/logfile.log';
login 'username' 'password';
Solution:
$ perl -pe 's/\$(\w+)/$ENV{$1}/g' file.sh
spool on to '/home/user/logfile.log';
login 'username' 'password';
One limitation of the above answers is that they both require the variables to be exported to the environment. Here's what i came up with that would allow the variables to be local to the current shell script:
#!/bin/sh
FOO=bar;
FILE=`mktemp`; # Let the shell create a temporary file
trap 'rm -f $FILE' 0 1 2 3 15; # Clean up the temporary file
(
echo 'cat <<END_OF_TEXT'
cat "$#"
echo 'END_OF_TEXT'
) > $FILE
. $FILE
The above example allows the variable $FOO to be substituted in the files named on the command line. I'm sure it can be improved, but this works for me so far.
Thanks to both previous answers for their ideas!
If the variables you want to translate are known and limited in number, you can always do the translation yourself:
sed "s/\$LOG_FILE_PATH/$LOG_FILE_PATH/g" input > output
And also assuming the variable itself is already known
This solution allows you to keep the same formatting in the ouput file
Copy and paste the following lines in your script
cat $1 | while read line
do
eval $line
echo $line
eval echo $line
done | uniq | grep -v '\$'
this will read the file passed as argument line by line, and then process to try and print each line twice:
- once without substitution
- once with substitution of the variables.
then remove the duplicate lines
then remove the lines containing visible variables ($)
Yes eval should be used carefully, but it provided me this simple oneliner for my problem. Below is an example using your filename:
eval "echo \"$(<Text_File.msh)\""
I use printf instead of echo for my own purposes, but that should do the trick. Thank you abyss.7 providing the link that solve my problem. Hope it helps.
Create an ascii file test.txt with the following content:
Try to replace this ${myTestVariable1}
bla bla
....
Now create a file “sub.sed” containing variable names, eg
's,${myTestVariable1},'"${myTestVariable1}"',g;
s,${myTestVariable2},'"${myTestVariable2}"',g;
s,${myTestVariable3},'"${myTestVariable3}"',g;
s,${myTestVariable4},'"${myTestVariable4}"',g'
Open a terminal move to the folder containing test.txt and sub.sed.
Define the value of the varible to be replaced
myTestVariable1=SomeNewText
Now call sed to replace that variable
sed "$(eval echo $(cat sub.sed))" test.txt > test2.txt
The output will be
$cat test2.txt
Try to replace this SomeNewText
bla bla
....
#logfiles.list:
$EAMSROOT/var/log/LinuxOSAgent.log
$EAMSROOT/var/log/PanacesServer.log
$EAMSROOT/var/log/PanacesStrutsGUI.log
#My Program:
cat logfiles.list | while read line
do
eval Eline=$line
echo $Eline
done