Shell script updating same file code is looping through - shell

The below code is currently working to update a property file with values from set environment variables. However, I'm wondering are there any issues/problems with updating the same file the code is looping through (at the same time)? https://github.com/koalaman/shellcheck/wiki/SC2094
The below code:
Loops through each line of a property file
Finds the key=value pair
Converts the key property to an ENVIRONMENT_VARIABLE name
Updates the same file with the new value from the ENVIRONMENT_VARIABLE so that now -- > key=ENVIRONMENT_VARIABLE
Before code is run:
MY_COLOR_PROP=yellow
my.color.prop=red
After code is run:
my.color.prop=yellow
FILE_NAME="./path/to/folder/my-app-${ENV}.properties"
echo "Update $FILE_NAME..."
OIFS=$IFS
while IFS="=" read -r key value; do
case "$key" in
'#'*)
echo "Skipping comment line..."
;;
*)
echo "Property from file: $key=$value"
ENV_VAR=$(echo "$key" | sed 's/\./_/g' | tr '[:lower:]' '[:upper:]')
ENV_VAR_VALUE=$(printf '%s\n' "${!ENV_VAR}")
echo "Environment variable: $ENV_VAR=$ENV_VAR_VALUE"
if [ -n "$ENV_VAR_VALUE" ]; then
echo "Setting $key=$ENV_VAR_VALUE in $FILE_NAME..."
sed -i "/$key=/ s^=.*^=$ENV_VAR_VALUE^" "$FILE_NAME"
else
echo "Skipping $key property - No value set for $ENV_VAR..."
fi
;;
esac
done <"$FILE_NAME"
IFS=$OIFS
***** UPDATE with what I implemented from below answer *****
My property file has comments in it and only a subset of the properties will need to be updated, the rest should remain untouched.
input="path/to/file/my.properties"
while IFS="=" read -r key value; do
case "$key" in
'#'*)
echo "$key"
;;
*)
ENV_VAR=$(echo "$key" | sed 's/\./_/g' | tr '[:lower:]' '[:upper:]')
echo "$key"="${!ENV_VAR-$value}"
;;
esac
done < "$input" > tmp; mv tmp "$input"
where the my.properties file looks like...
# Comment number one
bob=tom
organization.name=Chris Peter Org
# Comment number two
# Comment number three
db.url=jdbc:mysql://dburl/dbname
# Comment number four
test.parameter.ccf=always

Just do:
while IFS== read k v ; do echo $k=${!k-$v}; done < input > tmp && mv tmp input
eg:
$ cat input
foo=bar
baz=qux
$ unset baz
$ foo=foovalue
$ while IFS== read k v ; do echo $k=${!k-$v}; done < input > tmp && mv tmp input
$ cat input
foo=foovalue
baz=qux
Note that $!{k} is a bashism, and if you're not using bash you might want to do something like:
while IFS== read k v ; do eval echo \$k=\${$k:-$v}; done < input

Related

Bash read dictionary file

I have a dictionary file, call it 1.txt with the following content:
app1=1
app2=10
app3=2
app1=8
and so on.
From a bash script I would like to:
call 1.txt
read its content line by line
get key into variable A
get value into variable B
I tried:
var1=${for KEY in "${!dict[#]}"; do echo $KEY; done
var2=${for KEY in "${!dict[#]}"; do echo $dict[$KEY]}; done
which works just fine but I do not know how to wrap this into a bash function that calls the dictionary set on another file.
Would you please point out on this?
Probably this is what you want.
#!/bin/bash
declare -A dict
# Read the file into associative array "dict"
read_into_dict () {
while read -r line; do
[[ $line = *=* ]] && dict[${line%%=*}]=${line#*=}
done < "$1"
}
read_into_dict 1.txt
# Print out the dict
for key in "${!dict[#]}"; do
printf '%s=%s\n' "$key" "${dict[$key]}"
done
#M. Nejat Aydin,
I tried the following at localhost.
``
$ cat 73095065.sh
#!/bin/bash
declare -A dict
for EACHLN in $(cat 1.txt)
do
KEY=$(echo "$EACHLN" | sed "s/=.//;")
VALUE=$(echo "$EACHLN" | sed "s/.=//;")
if [ "${dict[$KEY]}" = "" ]
then
dict["$KEY"]="$VALUE"
else
echo "FOUND DUPLICATE KEY AT 1.txt"
echo "HENCE NOT SETTING dict["$KEY"] TO $VALUE AGAIN."
echo "CURRENT LINE: $EACHLN"
echo "OLD dict value: $KEY=${dict["$KEY"]}"
fi
done
echo "EXISTING DICT VALUES:"
for KEY in "${!dict[#]}"
do
echo "KEY $KEY VALUE ${dict[$KEY]}"
done
Sample output:
$ ./73095065.sh
FOUND DUPLICATE KEY AT 1.txt
HENCE NOT SETTING dict[app1] TO 8 AGAIN.
CURRENT LINE: app1=8
OLD dict value: app1=1
EXISTING DICT VALUES:
KEY app3 VALUE 2
KEY app2 VALUE 10
KEY app1 VALUE 1

Generate multiple output files for loop

i'm trying to generate a new output file from each existing file in a directory of .txt files. I want to check line by line in each file for two substrings. And append the lines that match that substring to each new output file.
I'm having trouble generating the new files.
This is what i currently have:
#!/bin/sh
# My first Script
success="(Compiling)\s\".*\"\s\-\s(Succeeded)"
failure="(Compiling)\s\".*\"\s\-\s(Failed)"
count_success=0
count_failure=0
for i in ~/Documents/reports/*;
do
while read -r line;
do
if [[$success=~$line]]; then
echo $line >> output_$i
count_success++
elif [[$failure=~$]]; then
echo $line >> output_$i
count_failure++
fi
done
done
echo "$count_success of jobs ran succesfully"
echo "$count_failure of jobs didn't work"
~
Any help would be appreciated, thanks
Please, use https://www.shellcheck.net/ to check your shell scripts.
If you use Visual Studio Code, you could install "ShellCheck" (by Timon Wong) extension.
About your porgram.
Assume bash
Define different extensions for input and output files (really important if there are in the same directory)
Loop on report, input, files only
Clear output file
Read input file
if sequence:
if [[ ... ]] with space after [[ and before ]]
spaces before and after operators (=~)
reverse operands order for operators =~
Prevent globbing with "..."
#! /bin/bash
# Input file extension
declare -r EXT_REPORT=".txt"
# Output file extension
declare -r EXT_OUTPUT=".output"
# RE
declare -r success="(Compiling)\s\".*\"\s\-\s(Succeeded)"
declare -r failure="(Compiling)\s\".*\"\s\-\s(Failed)"
# Counters
declare -i count_success=0
declare -i count_failure=0
for REPORT_FILE in ~/Documents/reports/*"${EXT_REPORT}"; do
# Clear output file
: > "${REPORT_FILE}${EXT_OUTPUT}"
# Read input file (see named file in "done" line)
while read -r line; do
# does the line match the success pattern ?
if [[ $line =~ $success ]]; then
echo "$line" >> "${REPORT_FILE}${EXT_OUTPUT}"
count_success+=1
# does the line match the failure pattern ?
elif [[ $line =~ $failure ]]; then
echo "$line" >> "${REPORT_FILE}${EXT_OUTPUT}"
count_failure+=1
fi
done < "$REPORT_FILE"
done
echo "$count_success of jobs ran succesfully"
echo "$count_failure of jobs didn't work"
What about using grep?
success='Compiling\s".*"\s-\sSucceeded'
failure='Compiling\s".*"\s-\sFailed'
count_success=0
count_failure=0
for i in ~/Documents/reports/*; do
(( count_success += $(grep -E "$success" "$i" | tee "output_$i" | wc -l) ))
(( count_failure += $(grep -E "$failure" "$i" | tee -a "output_$i" | wc -l) ))
done
echo "$count_success of jobs ran succesfully"
echo "$count_failure of jobs didn't work"

How to rename duplicate word as dup,dup(1),dup(2)

I have a string like lisp,pascal,lisp,bash,bash,bash,lisp
I want the result as lisp, pascal,lisp(1), bash,bash(1),bash(2),lisp(2)
My approace -
# converting string to array
string="lisp,pascal,lisp,bash,bash,bash,lisp"
set -f
array=(${string//,/ })
for i in "${!array[#]}"
do
echo "$i=>${array[i]}"
done
I don't know much bash so I need a help...
I would:
string="lisp,pascal,lisp,bash,bash,bash,lisp"
array=(${string//,/ })
output=()
for i in "${array[#]}"; do
# if the element already exists in the array
if ! printf "%s\n" "${output[#]}" | grep -Fxq "$i"; then
output+=("$i")
else
# extract the last number of that element
if
last_num=$(
printf "%s\n" "${output[#]}" |
sed '/'"$i"'(\([0-9]\{1,\}\))$/!d; s//\1/'
) && [[ -n "$last_num" ]]
then
# increment it and output
output+=("$i($((last_num+1)))")
else
# start with (1)
output+=("$i(1)")
fi
fi
done
declare -p output
would output:
declare -a output=([0]="lisp" [1]="pascal" [2]="lisp(1)" [3]="bash" [4]="bash(1)" [5]="bash(2)" [6]="lisp(2)")

Increment a column value every time in file

I am having a requirement to split a file based on segment. I need to split the files based on CLP segment in the file. Header and trailer is same for all the files and i need to increment the value of header and trailer starting from second file. Attached the example file and output files.(). I have designed the below code and stuck with the increment section. In my code i have assigned header value to (LINE1 and LINE2) and Trailer value to (TRAILER) variables respectively. How increment the header and trailer value inside a loop.
Code:-
if [[ $# -lt 1 ]] ; then
echo "Usage: $(basename $0) File2Split [MaxLines]" >&2
exit 1
fi
FileName="$1"
if [[ ! -r $FileName ]] ; then
echo "ERROR: could not read $FileName" >&2
exit 1
fi
MAXCLP="$2"
if [[ -z "${MAXCLP}" ]] ; then
echo "INFO: no value for MAXCLP'S. Setting it to 100." >&2
MAXCLP=100
fi
echo "File name = ${FileName} "
echo "No of CLPs = ${MAXCLP} "
#take a back up of file
echo "$(date +'%m/%d/%Y %H:%M:%S:%3N') : Creating a backup..."
cp -v $FileName ${FileName}_backup
#Format the file with separated lines
echo "$(date +'%m/%d/%Y %H:%M:%S:%3N') : Separating by line..."
LineSeparator="$(head -q -n 1 "$FileName" | cut -c106)"
sed -i -r -e "s#${LineSeparator}([^\n])#${LineSeparator}\n\1#g" -e "s/^[ \t]*//" "$FileName"
#Initialize variables
SAVEIFS=$IFS
Cnt=0;
FileNum=1;
#Get all lines with CLP segment and their corresponding line numbers
LINES="$(cat $FileName | grep -n ^CLP | awk "NR % $MAXCLP == 0" )"
LINE1="$(head -n 1 "$FileName")
LINE2="$(head -n 2 "$FileName" | tail -1 )
TRAILER="$(tail -n 2 "$FileName")
MID="$(sed -n '3,/CLP/p' "$FileName" | head -n -1)
StartLine=1;
EndLine="$";
CurrentLine=1;
for LN in $LINES ; do
CurrentLine=$(echo ${LN} | cut -d':' -f1);
EndLine=$(($CurrentLine-1));
sed -n "${LINE1},${LINE2},${MID},${StartLine},${EndLine},${TRAILER}p" $FileName > "${FileName}_${FileNum}";
--- increment section comes here----
StartLine=$CurrentLine;
EndLine="$";
((FileNum++));
Cnt=1;
done
*************************************************************
Input File Details:-
ISA*00**200402*0131*^*00501*000000234*1*P*|~
GS*HC*580977458*12345678*20200402*013121*000000001*X*00903943AX2~
TS*837*000000004*09345454DERTD~
TBH*0019*00*600920001*20200401*070634*CH~
ND*2001630~
XR**COUNTY*****PI*9353~
RT*PO BO6~
CVD*TWIN *ID*83303~
CLP*XXXXXX***HANDICAP**A*Y*Y~
PTD*VVVV*ASX*1000~
PTD*434*RD8*~
PTD*435*DT*~
CL1*3*1*01~
FER*D9*CDSKSLS~
FER*EA*AAAAAA~
CLP*XXXXXX***HANDICAP**A*Y*Y~
PTD*VVVV*ASX*1000~
PTD*434*RD8*~
PTD*435*DT*~
CL1*3*1*01~
FER*D9*CDSKSLS~
FER*EA*AAAAAA~
CLP*XXXXXX***HANDICAP**A*Y*Y~
PTD*VVVV*ASX*1000~
PTD*434*RD8*~
PTD*435*DT*~
CL1*3*1*01~
FER*D9*CDSKSLS~
FER*EA*AAAAAA~
GE*211*000000001~
IEA*1*000000001~
Out Put File 1:-
ISA*00**200402*0131*^*00501*000000235*1*P*|~
GS*HC*580977458*12345678*20200402*013121*000000002*X*00903943AX2~
TS*837*000000004*09345454DERTD~
TBH*0019*00*600920001*20200401*070634*CH~
ND*2001630~
XR**COUNTY*****PI*9353~
RT*PO BO6~
CVD*TWIN *ID*83303~
CLP*XXXXXX***HANDICAP**A*Y*Y~
PTD*VVVV*ASX*1000~
PTD*434*RD8*~
PTD*435*DT*~
CL1*3*1*01~
FER*D9*CDSKSLS~
FER*EA*AAAAAA~
GE*211*000000002~
IEA*1*000000002~
Note:-In the output file ISA segment value 000000235, GS segment
value 000000002 , IEA and GE segent value 000000002 got incremented
by 1.

How can we delete space with a shell script?

I did a script in order to display different things, in different case.
The script is :
#!/usr/bin/env bash
# declare an array, to store stuff in
declare -a myArray
shopt -s nocasematch
# read the full file into the array
# This while loop terminates when pressing CTRL-D
i=1
while read -r line; do
myArray[i]="${line}"
((i++))
done < /dev/stdin
# Process the array
for ((j=1;j<i;++j)); do
# perform your actions here on myArray[j]
case "${myArray[j]}" in
bob)
echo "boy"
;;
alicia)
echo "girl"
;;
cookie)
echo "dog"
;;
*)
echo "unknown" "${myArray[j]}"
;;
esac
done
But I have a problem, when I execute the code with this command:
cat input.txt | ./prog.sh > file.txt
I have the following input:
bob
alicia
amhed
cookie
daniel
In this input I have so space, but when I run my program I don't obtain this right result. I need my code not to take into account spaces, but if it take care about the space, is wrote "unknown" on the OUTPOUT file.txt
and I obtain the result :
boy
girl
unknown amhed
dog
unknown
unknown
unknown daniel
So can I eliminate/delete the space without touching the input file?
why do this in bash?
with awk
$ awk 'BEGIN{n=split("bob boy alicia girl cookie dog",x);
for(i=1;i<n;i+=2) a[x[i]]=x[i+1]} # build the lookup table
{print $1 in a?a[$1]:"unknown "$1}' file
boy
girl
unknown amhed
dog
unknown
unknown
unknown daniel
you can externalize the lookup map to another file as well, so that the code doesn't need to be modified if either values change.
If you want to do nothing when the input line is empty, you can add that to your case:
#!/usr/bin/env bash
# declare an array, to store stuff in
declare -a myArray
shopt -s nocasematch
# read the full file into the array
# This while loop terminates when pressing CTRL-D
i=1
while read -r line; do
myArray[i]="${line}"
((i++))
done < /dev/stdin
# Process the array
for ((j=1;j<i;++j)); do
# perform your actions here on myArray[j]
case "${myArray[j]}" in
"") # This is an empty line, skip it
;;
bob)
echo "boy"
;;
alicia)
echo "girl"
;;
cookie)
echo "dog"
;;
*)
echo "unknown" "${myArray[j]}"
;;
esac
done
Alternatively, check whether the line you read was empty before adding it to the array.
You can read from stdin and process each line and avoid array creation.
Use && [[ -n $line ]] to make sure you process only non-empty lines.
By default read command reads it from /dev/stdin so you can omit < /dev/stdin from your code.
Code:
shopt -s nocasematch
while read -r line && [[ -n $line ]]; do
case "$line" in
bob)
echo "boy"
;;
alicia)
echo "girl"
;;
cookie)
echo "dog"
;;
*)
echo "unknown $line"
;;
esac
done
Run it as:
./prog.sh < input.txt
Output:
boy
girl
unknown amhed
dog

Resources