Updating a config file based on the presence of a specific string - bash

I want to be able to comment and uncomment lines which are "managed" using a bash script.
I am trying to write a script which will update all of the config lines which have the word #managed after them and remove the preceeding # if it exists.
The rest of the config file needs to be left unchanged. The config file looks like this:
configFile.txt
#config1=abc #managed
#config2=abc #managed
config3=abc #managed
config3=abc
This is the script I have created so far. It iterates the file, finds lines which contain "#managed" and detects if they are currently commented.
I need to then write this back to the file, how do I do that?
manage.sh
#!/bin/bash
while read line; do
STR='#managed'
if grep -q "$STR" <<< "$line"; then
echo "debug - this is managed"
firstLetter=${$line:0:1}
if [ "$firstLetter" = "#" ]; then
echo "Remove the initial # from this line"
fi
fi
echo "$line"
done < configFile.txt

With your approach using grep and sed.
str='#managed$'
file=ConfigFile.txt
grep -q "^#.*$str" "$file" && sed "/^#.*$str/s/^#//" "$file"
Looping through files ending in *.txt
#!/usr/bin/env bash
str='#managed$'
for file in *.txt; do
grep -q "^#.*$str" "$file" &&
sed "/^#.*$str/s/^#//" "$file"
done
In place editing with sed requires the -i flag/option but that varies from different version of sed, the GNU version does not require an -i.bak args, while the BSD version does.
On a Mac, ed should be installed by default, so just replace the sed part with.
printf '%s\n' "g/^#.*$str/s/^#//" ,p Q | ed -s "$file"
Replace the Q with w to actually write back the changes to the file.
Remove the ,p if no output to stdout is needed/required.
On a side note, embedding grep and sed in a shell loop that reads line-by-line the contents of a text file is considered a bad practice from shell users/developers/coders. Say the file has 100k lines, then grep and sed would have to run 100k times too!

This sed one-liner should do the trick:
sed -i.orig '/#managed/s/^#//' configFile.txt
It deletes the # character at the beginning of the line if the line contains the string #managed.
I wouldn't do it in bash (because that would be slower than sed or awk, for instance), but if you want to stick with bash:
#! /bin/bash
while IFS= read -r line; do
if [[ $line = *'#managed'* && ${line:0:1} = '#' ]]; then
line=${line:1}
fi
printf '%s\n' "$line"
done < configFile.txt > configFile.tmp
mv configFile.txt configFile.txt.orig && mv configFile.tmp configFile.txt

Related

'sed: no input files' when using sed -i in a loop

I checked some solutions for this in other questions, but they are not working with my case and I'm stuck so here we go.
I have a csv file that I want to convert all to uppercase. It has to be with a loop and occupate 7 lines of code minimum. I have to run the script with this command:
./c_bash.sh student-mat.csv
So I tried this Script:
#!/bin/bash
declare -i c=0
while read -r line; do
if [ "$c" -gt '0' ]; then
sed -e 's/\(.*\)/\U\1/'
else
echo "$line"
fi
((c++))
done < student-mat.csv
I know that maybe there are a couple of unnecessary things on it, but I want to focus in the sed command because it looks like the problem here.
That script shows this output:(first 5 lines):
school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3
GP,F,17,U,GT3,T,1,1,AT_HOME,OTHER,COURSE,FATHER,1,2,0,NO,YES,NO,NO,NO,YES,YES,NO,5,3,3,1,1,3,4,5,5,6
GP,F,15,U,LE3,T,1,1,AT_HOME,OTHER,OTHER,MOTHER,1,2,3,YES,NO,YES,NO,YES,YES,YES,NO,4,3,2,2,3,3,10,7,8,10
GP,F,15,U,GT3,T,4,2,HEALTH,SERVICES,HOME,MOTHER,1,3,0,NO,YES,YES,YES,YES,YES,YES,YES,3,2,2,1,1,5,2,15,14,15
GP,F,16,U,GT3,T,3,3,OTHER,OTHER,HOME,FATHER,1,2,0,NO,YES,YES,NO,YES,YES,NO,NO,4,3,2,1,2,5,4,6,10,10
GP,M,16,U,LE3,T,4,3,SERVICES,OTHER,REPUTATION,MOTHER,1,2,0,NO,YES,YES,YES,YES,YES,YES,NO,5,4,2,1,2,5,10,15,15,15
Now that I see that it works, I want to apply that sed command permanently to the csv file, so I put -i after it:
#!/bin/bash
declare -i c=0
while read -r line; do
if [ "$c" -gt '0' ]; then
sed -i -e 's/\(.*\)/\U\1/'
else
echo "$line"
fi
((c++))
done < student-mat.csv
But the output instead of applying the changes, shows this:(first 5 lines)
school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3
sed: no input files
sed: no input files
sed: no input files
sed: no input files
sed: no input files
So checking a lot of different solutions on the internet, I also tried to change single quoting to double quoting.
#!/bin/bash
declare -i c=0
while read -r line; do
if [ "$c" -gt '0' ]; then
sed -i -e "s/\(.*\)/\U\1/"
else
echo "$line"
fi
((c++))
done < student-mat.csv
But in this case, instead of applying the changes, it generate a file with 0 bytes. So no output when I do this:
cat student-mat.csv
My expected solution here is that, when I apply this script, it changes permanently all the data to uppercase. And after applying the script, it should show this with the command cat student-mat.csv: (first 5 lines)
school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3
GP,F,17,U,GT3,T,1,1,AT_HOME,OTHER,COURSE,FATHER,1,2,0,NO,YES,NO,NO,NO,YES,YES,NO,5,3,3,1,1,3,4,5,5,6
GP,F,15,U,LE3,T,1,1,AT_HOME,OTHER,OTHER,MOTHER,1,2,3,YES,NO,YES,NO,YES,YES,YES,NO,4,3,2,2,3,3,10,7,8,10
GP,F,15,U,GT3,T,4,2,HEALTH,SERVICES,HOME,MOTHER,1,3,0,NO,YES,YES,YES,YES,YES,YES,YES,3,2,2,1,1,5,2,15,14,15
GP,F,16,U,GT3,T,3,3,OTHER,OTHER,HOME,FATHER,1,2,0,NO,YES,YES,NO,YES,YES,NO,NO,4,3,2,1,2,5,4,6,10,10
GP,M,16,U,LE3,T,4,3,SERVICES,OTHER,REPUTATION,MOTHER,1,2,0,NO,YES,YES,YES,YES,YES,YES,NO,5,4,2,1,2,5,10,15,15,15
Sed works on files, not on lines. Do not read lines, use sed on the file. Sed can exclude the first line by itself. See sed manual.
You want:
sed -i -e '2,$s/\(.*\)/\U\1/' student-mat.csv
You can do shorter with s/.*/\U&/.
Your code does not work as you think it does. Note that your code removes the second line from the output. Your code:
reads first line with read -r line
echo "$line" first line is printed
c++ is incremented
read -r line reads second line
then sed processes the rest of the file (from line 3 till the end) and prints them in upper case
then c++ is incremented
then read -r line fails, and the loop exits

Removing punctuation using sed

I am trying to write a script that removes punctuation from a text file.
I tried using sed, however am open to other suggestions (like awk)
This is my code so far
declare -a marks=('\.' '\,' '\;' '\:')
for i in {0..3}
do
sed -i 's/${marks[i]}//g' test.txt
done
cat test.txt`
I think my main problem is am not using escape keys correctly.
The command tr is great for that:
tr -d '[:punct:]' < test.txt > tmp.txt && mv -f tmp.txt test.txt
-d stands for delete.
Choose a non-existing file tmp.txt; to generate a temporary file a solution is mktemp -u.
Here is a small script which removes any punctuation in the files passed as arguments:
#! /bin/bash
t=$(mktemp -u)
for f ; do
tr -d '[:punct:]' < "$f" > "$t" && mv -f "$t" "$f"
done
for f is a shortcut for for f in "$#", which iterates over each argument without word splitting.
Using ed instead:
printf "%s\n" 'g/[[:punct:]]/s/[[:punct:]]//g' w | ed -s test.txt
removes all punctuation characters from a file and saves the remaining text.

bash to capture specific instance of pattern and exclude others

I am trying to capture and read into $line the line or lines in file that have only del in them (line 2 is an example). Line 3 has del in it but it also has ins and the bash when executed currently captures both. I am not sure how to exclude anything but del and only capture those lines. Thank you :).
file
NM_003924.3:c.765_779dupGGCAGCGGCGGCAGC
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
NM_003924.3:c.765_779delGGCAGCinsGGCGGCAGC
NM_003924.3:c.765_779insGGCAGCGGCGGCAGC
desired output
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
bash w/ current output
while read line; do
if [[ $line =~ del ]] ; then echo $line; fi
done < file
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
NM_003924.3:c.765_779delGGCAGCinsGGCGGCAGC
Could you please try following(if ok with awk).
awk '/del/ && !/ins/' Input_file
Try:
while read -r line; do
[[ $line =~ del && ! $line =~ ins ]] && printf '%s\n' "$line"
done < file
The revised code is also ShellCheck clean and avoids BashPitfall #14.
This solution may fail if the last line in the file does not have a terminating newline. If that is a concern, see the accepted answer to Read last line of file in bash script when reading file line by line for a fix.
Here is a sed solution. It negates the match del followed by ins and prints everything that has a del in it. -n to silent every other output.
$ sed -n -e '/del.*ins/!{/.*del.*/p}' inputFile
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
Here is another answer using PCRE enabled grep. This should work with -P option in GNU grep
$ grep -P 'del\.*(?!.*ins)' inputFile
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
Split it in 2 steps. You do not need a loop:
grep "del" file | grep -v "ins"

Deleting every line with a specific string in bash

Edit:
I realized that my current code is garbage, but other ways I tried before also did not work. The problem was that I edited the files in Notepad++ on Windows, and had them running on Linux. The programm dos2unix does the trick.
Solution:
I have used Notepad++ in Windows to write my files which caused the problem. Running the files trough dos2unix fixed it.
I have written a little bash script which should delete every line of $2 which contains a word which is specified in $1 and writes the output to $3. But somehow it does not work like it should.
#!/bin/bash
set -f
while IFS='' read -r i || [[ -n "$i" ]]; do
sed -i "/$i/d" "$2"
done < "$1"
Edit:
Example
file 1.test:
123
678
456
file 2.test:
dasdas123dasd
3fsef344
678 3423423
r23r23rfsad
456 dasdasd
running the script:
./script.sh 1.test 2.test
The output should be:
3fsef344
r23r23rfsad
but instead it is:
dasdas123dasd
3fsef344
678 3423423
r23r23rfsad
Why are you using a shell loop and sed for this?
$ grep -vFf file1 file2
3fsef344
r23r23rfsad
If that doesn't do what you need then clarify your question with a more truly representative example because "use a shell loop calling sed multiple times" is not the answer to any question. See https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice for some of the reasons I say that.
You're reloading $2 and overwriting $3 each time you call sed. You should apply all operations to the file instead of reloading.
#!/bin/bash
set -f
cat "$2" > "$3"
while IFS='' read -r i || [[ -n "$i" ]]; do
sed -i "/$i/d" "$3"
done < "$1"

Using sed in a for loop with variables and regex

I'm trying to build a script where a portion of it utilizes 'sed' to tag the filename onto the end of each line in that file, then dumps the output to a master list.
The part of the script giving me trouble is sed here:
DIR=/var/www/flatuser
FILES=$DIR/*
for f in $FILES
do
echo "processing $f file...."
sed -i "s/$/:$f/" $f
cat $f >> $DIR/master.txt
done
The issue is that the 'sed' statement works fine outside of the for loop, but when I place it in the script, I believe it's having issues interpreting the dollar signs. I've tried nearly every combo of " and ' that I can think of to get it to interpret the variable and it continuously either puts "$f" at the end of each line, or it fails outright.
Thanks for any input!
You just need to escape the dollar sign:
sed -i "s/\$/:$f/" "$f"
so that the shell passes it literally to sed.
To expand on Charles Duffy's point about quoting variables:
DIR=/var/www/flatuser
for f in "$DIR"/*
do
echo "processing $f file...."
sed -i "s/\$/:${f##*/}/" "$f"
cat "$f" >> "$DIR/master.txt"
done
If any file names contain a space, it's too late to do anything about it if you assign the list of file names to $FILES; you can no longer distinguish between spaces that belong to file names and spaces that separate file names. You could use an array instead, but it's simpler to just put the glob directly in the for loop. Here's how you would use an array:
DIR=/var/www/flatuser
FILES=( "$DIR"/* )
for f in "${FILES[#]}"
do
echo "processing $f file...."
sed -i "s/\$/:${f##*/}/" "$f"
cat "$f" >> "$DIR/master.txt"
done
For versions of sed that don't use -i, here's a way to explicitly handle the temp file needed to simulate in-place editing:
t=$(mktmp sXXXX); sed "s/\$/:$f/" "$f" > "$t"; mv "$t" "$f" && rm "$t"
Personally, I'd do this like so:
dir=/var/www/flatuser
for f in "$dir"/*; do
[[ $f = */master.txt ]] && continue
while read -r; do printf '%s:%s\n' "$REPLY" "${f##*/}"; done <"$f"
done >/var/www/flatuser/master.txt
It doesn't modify your files in-place the way sed -i does, so it's safe to run more than one time (the sed -i version will add the names to your files in-place every time it runs, so you'll end up with each line having more than one copy of the filename on it).
Also, sed -i isn't specified by POSIX, so not all operating systems will have it.
The problem is NOT the dollar sign. It's that the variable $f contains a "/" character, and sed is using that to separate expressions. Try using "#" as the separator.
DIR=/var/www/flatuser
FILES=$DIR/*
for f in $FILES
do
echo "processing $f file...."
sed -i s#"$"#:"$f"# $f
cat $f >> $DIR/master.txt
done
it's old, but maybe it helps someone.
Why not basename the file to get rid of leading directory
DIR=/var/www/flatuser
FILES=( "$DIR"/* )
for f in "${FILES[#]}"
do
echo "processing $f file...."
b=`basename $f`
sed -i "s/\$/:${b##*/}/" "$b"
cat "$f" >> "$DIR/master.txt"
done
not tested ...

Resources