'sed: no input files' when using sed -i in a loop - bash

I checked some solutions for this in other questions, but they are not working with my case and I'm stuck so here we go.
I have a csv file that I want to convert all to uppercase. It has to be with a loop and occupate 7 lines of code minimum. I have to run the script with this command:
./c_bash.sh student-mat.csv
So I tried this Script:
#!/bin/bash
declare -i c=0
while read -r line; do
if [ "$c" -gt '0' ]; then
sed -e 's/\(.*\)/\U\1/'
else
echo "$line"
fi
((c++))
done < student-mat.csv
I know that maybe there are a couple of unnecessary things on it, but I want to focus in the sed command because it looks like the problem here.
That script shows this output:(first 5 lines):
school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3
GP,F,17,U,GT3,T,1,1,AT_HOME,OTHER,COURSE,FATHER,1,2,0,NO,YES,NO,NO,NO,YES,YES,NO,5,3,3,1,1,3,4,5,5,6
GP,F,15,U,LE3,T,1,1,AT_HOME,OTHER,OTHER,MOTHER,1,2,3,YES,NO,YES,NO,YES,YES,YES,NO,4,3,2,2,3,3,10,7,8,10
GP,F,15,U,GT3,T,4,2,HEALTH,SERVICES,HOME,MOTHER,1,3,0,NO,YES,YES,YES,YES,YES,YES,YES,3,2,2,1,1,5,2,15,14,15
GP,F,16,U,GT3,T,3,3,OTHER,OTHER,HOME,FATHER,1,2,0,NO,YES,YES,NO,YES,YES,NO,NO,4,3,2,1,2,5,4,6,10,10
GP,M,16,U,LE3,T,4,3,SERVICES,OTHER,REPUTATION,MOTHER,1,2,0,NO,YES,YES,YES,YES,YES,YES,NO,5,4,2,1,2,5,10,15,15,15
Now that I see that it works, I want to apply that sed command permanently to the csv file, so I put -i after it:
#!/bin/bash
declare -i c=0
while read -r line; do
if [ "$c" -gt '0' ]; then
sed -i -e 's/\(.*\)/\U\1/'
else
echo "$line"
fi
((c++))
done < student-mat.csv
But the output instead of applying the changes, shows this:(first 5 lines)
school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3
sed: no input files
sed: no input files
sed: no input files
sed: no input files
sed: no input files
So checking a lot of different solutions on the internet, I also tried to change single quoting to double quoting.
#!/bin/bash
declare -i c=0
while read -r line; do
if [ "$c" -gt '0' ]; then
sed -i -e "s/\(.*\)/\U\1/"
else
echo "$line"
fi
((c++))
done < student-mat.csv
But in this case, instead of applying the changes, it generate a file with 0 bytes. So no output when I do this:
cat student-mat.csv
My expected solution here is that, when I apply this script, it changes permanently all the data to uppercase. And after applying the script, it should show this with the command cat student-mat.csv: (first 5 lines)
school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3
GP,F,17,U,GT3,T,1,1,AT_HOME,OTHER,COURSE,FATHER,1,2,0,NO,YES,NO,NO,NO,YES,YES,NO,5,3,3,1,1,3,4,5,5,6
GP,F,15,U,LE3,T,1,1,AT_HOME,OTHER,OTHER,MOTHER,1,2,3,YES,NO,YES,NO,YES,YES,YES,NO,4,3,2,2,3,3,10,7,8,10
GP,F,15,U,GT3,T,4,2,HEALTH,SERVICES,HOME,MOTHER,1,3,0,NO,YES,YES,YES,YES,YES,YES,YES,3,2,2,1,1,5,2,15,14,15
GP,F,16,U,GT3,T,3,3,OTHER,OTHER,HOME,FATHER,1,2,0,NO,YES,YES,NO,YES,YES,NO,NO,4,3,2,1,2,5,4,6,10,10
GP,M,16,U,LE3,T,4,3,SERVICES,OTHER,REPUTATION,MOTHER,1,2,0,NO,YES,YES,YES,YES,YES,YES,NO,5,4,2,1,2,5,10,15,15,15

Sed works on files, not on lines. Do not read lines, use sed on the file. Sed can exclude the first line by itself. See sed manual.
You want:
sed -i -e '2,$s/\(.*\)/\U\1/' student-mat.csv
You can do shorter with s/.*/\U&/.
Your code does not work as you think it does. Note that your code removes the second line from the output. Your code:
reads first line with read -r line
echo "$line" first line is printed
c++ is incremented
read -r line reads second line
then sed processes the rest of the file (from line 3 till the end) and prints them in upper case
then c++ is incremented
then read -r line fails, and the loop exits

Related

Updating a config file based on the presence of a specific string

I want to be able to comment and uncomment lines which are "managed" using a bash script.
I am trying to write a script which will update all of the config lines which have the word #managed after them and remove the preceeding # if it exists.
The rest of the config file needs to be left unchanged. The config file looks like this:
configFile.txt
#config1=abc #managed
#config2=abc #managed
config3=abc #managed
config3=abc
This is the script I have created so far. It iterates the file, finds lines which contain "#managed" and detects if they are currently commented.
I need to then write this back to the file, how do I do that?
manage.sh
#!/bin/bash
while read line; do
STR='#managed'
if grep -q "$STR" <<< "$line"; then
echo "debug - this is managed"
firstLetter=${$line:0:1}
if [ "$firstLetter" = "#" ]; then
echo "Remove the initial # from this line"
fi
fi
echo "$line"
done < configFile.txt
With your approach using grep and sed.
str='#managed$'
file=ConfigFile.txt
grep -q "^#.*$str" "$file" && sed "/^#.*$str/s/^#//" "$file"
Looping through files ending in *.txt
#!/usr/bin/env bash
str='#managed$'
for file in *.txt; do
grep -q "^#.*$str" "$file" &&
sed "/^#.*$str/s/^#//" "$file"
done
In place editing with sed requires the -i flag/option but that varies from different version of sed, the GNU version does not require an -i.bak args, while the BSD version does.
On a Mac, ed should be installed by default, so just replace the sed part with.
printf '%s\n' "g/^#.*$str/s/^#//" ,p Q | ed -s "$file"
Replace the Q with w to actually write back the changes to the file.
Remove the ,p if no output to stdout is needed/required.
On a side note, embedding grep and sed in a shell loop that reads line-by-line the contents of a text file is considered a bad practice from shell users/developers/coders. Say the file has 100k lines, then grep and sed would have to run 100k times too!
This sed one-liner should do the trick:
sed -i.orig '/#managed/s/^#//' configFile.txt
It deletes the # character at the beginning of the line if the line contains the string #managed.
I wouldn't do it in bash (because that would be slower than sed or awk, for instance), but if you want to stick with bash:
#! /bin/bash
while IFS= read -r line; do
if [[ $line = *'#managed'* && ${line:0:1} = '#' ]]; then
line=${line:1}
fi
printf '%s\n' "$line"
done < configFile.txt > configFile.tmp
mv configFile.txt configFile.txt.orig && mv configFile.tmp configFile.txt

Bash : reading regex from file and subsitute them into sed inline as variable

I am stuck with how sed interacts with variables. I am reading a list of regex from a file then substitute it into SED to mask certain sensitive information within a log file. if I hard coded the regex, the SED work perfectly, however it behave differently when used with variable.
con-list.txt contain below:
(HTTP\/)(.{2})(.*?)(.{2})(group\.com)
(end\sretrieve\sfacility\s)(.{2})(.*?)(.{3})$
Not sure if the dollar sign for regex is interfering with the SED command.
input="/c/Users/con-list.txt"
inputfiles="/c/Users/test.log"
echo $inputfiles
while IFS= read -r var
do
#echo "Searching $var"
count1=`zgrep -E "$var" "$inputfiles" | wc -l`
if [ ${count1} -ne 0 ]
then
echo "total:${count1} ::: ${var}"
sed -r -i "s|'[$]var'|'\1\2XXXX\4\5'|g" $inputfiles #this doesnt work
sed -r -i "s/(HTTP\/)(.{2})(.*?)(.{2})(group\.com)/'\1\2XXXX\4\5'/g" $inputfiles #This works
egrep -in "${var}" $inputfiles
fi
done < "$input"
I need the SED to accept the regex as variable read from the file. So I could automate masking for sensitive information within logs.
$ ./zgrep2.sh
/c/Users/test.log
total:4 ::: (HTTP\/)(.{2})(.*?)(.{2})(group\.comp\.com\#GROUP\.COM)
sed: -e expression #1, char 30: invalid reference \5 on `s' command's RHS
Your idea was right, but you forgot to leave the regex in the sed command to be under double quotes for $var to be expanded.
Also you don't need to use wc -l to count the match of occurrences. The family of utilities under grep all implement a -c flag that returns a count of matches. That said, you don't even need to count the matches, but use the return code of the command (if the match was found or not) simply as
if zgrep -qE "$var" "$inputfiles" ; then
Assuming you might need the count for debug purposes, you can continue with your approach with modifications to your script done as below
Notice how the var is interpolated in the sed substitution, leaving it expanded under double-quotes and once expanded preserving the literal values using the single-quote.
while IFS= read -r var
do
count1=$(zgrep -Ec "$var" "$inputfiles")
if [ "${count1}" -ne 0 ]
then
sed -r -i 's|'"$var"'|\1\2XXXX\4\5|g' "$inputfiles"
sed -r -i "s/(HTTP\/)(.{2})(.*?)(.{2})(group\.com)/'\1\2XXXX\4\5'/g" "$inputfiles"
egrep -in "${var}" "$inputfiles"
fi
done < "$input"
You need:
sed -r -i "s/$var"'/\1\2XXXX\4\5/g' $inputfiles
You also need to provide sample input (a useful bit of the log file) so that we can verify our solutions.
EDIT: a slight change to $var and I think this is what you want:
$ cat ~/tmp/j
Got creds for HTTP/PPCKSAPOD81.group.com
Got creds for HTTP/PPCKSAPOD21.group.com
Got creds for HTTP/PPCKSAPOD91.group.com
Got creds for HTTP/PPCKSWAOD81.group.com
Got creds for HTTP/PPCKSDBOD81.group.com
Got creds for HTTP/PPCKSKAOD81.group.com
$ echo $var
(HTTP\/)(.{2})(.*?)(.{2})(.group\.com)
$ sed -r "s/$var"'/\1\2XXXX\4\5/' ~/tmp/j
Got creds for HTTP/PPXXXX81.group.com
Got creds for HTTP/PPXXXX21.group.com
Got creds for HTTP/PPXXXX91.group.com
Got creds for HTTP/PPXXXX81.group.com
Got creds for HTTP/PPXXXX81.group.com
Got creds for HTTP/PPXXXX81.group.com

Deleting every line with a specific string in bash

Edit:
I realized that my current code is garbage, but other ways I tried before also did not work. The problem was that I edited the files in Notepad++ on Windows, and had them running on Linux. The programm dos2unix does the trick.
Solution:
I have used Notepad++ in Windows to write my files which caused the problem. Running the files trough dos2unix fixed it.
I have written a little bash script which should delete every line of $2 which contains a word which is specified in $1 and writes the output to $3. But somehow it does not work like it should.
#!/bin/bash
set -f
while IFS='' read -r i || [[ -n "$i" ]]; do
sed -i "/$i/d" "$2"
done < "$1"
Edit:
Example
file 1.test:
123
678
456
file 2.test:
dasdas123dasd
3fsef344
678 3423423
r23r23rfsad
456 dasdasd
running the script:
./script.sh 1.test 2.test
The output should be:
3fsef344
r23r23rfsad
but instead it is:
dasdas123dasd
3fsef344
678 3423423
r23r23rfsad
Why are you using a shell loop and sed for this?
$ grep -vFf file1 file2
3fsef344
r23r23rfsad
If that doesn't do what you need then clarify your question with a more truly representative example because "use a shell loop calling sed multiple times" is not the answer to any question. See https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice for some of the reasons I say that.
You're reloading $2 and overwriting $3 each time you call sed. You should apply all operations to the file instead of reloading.
#!/bin/bash
set -f
cat "$2" > "$3"
while IFS='' read -r i || [[ -n "$i" ]]; do
sed -i "/$i/d" "$3"
done < "$1"

bash not listing files in while loop

The below bash seems to run but no file names are displayed on the terminal screen, rather it just stalls. I can not seem to figure out why it is not working now as it used to. Thank you :).
bash
while read line; do
sed -i -e 's|https://www\.example\.com/xx/x/xxx/||' /home/file
echo $line
done
file
Auto_user_xxx-39-160506_file_name_x-x_custom_xx_91-1.pdf
Auto_user_xxx-48-160601_V4-2_file_name_x-x_custom_xx_101.pdf
coverageAnalysisReport(10).zip
The read command is waiting for input, since nothing is specified it will read from stdin. If you type a few lines and press you will see thats the input for the loop.
But you most likely want to redirect a file to the loop:
while IFS= read -r line; do
printf "%s\n" "$line"
done < /home/file
But afai can understand you have a file with other file names which you would like to run the substitution on, in that case you should use xargs:
xargs -n 1 -I {} sed -i -e 's|https://www\.example\.com/xx/x/xxx/||' {} < /home/file

Remove commas post second occurrence of comma only in last line and check for flag

I have a bunch of files in a specified path, in which I want to remove all the , post second occurrence of , in the last line only, in an efficient way.
I don't want process to read each line, instead just go directly to the last line and remove all , post second occurrence of ,.
Also, I want a check to be made if last line has EOF in it or not; if it is not available, no changes are to be applied, move to next file.
Sample file:
A,111,aaa,A
B,222,bbb,B
X,EOF,,,,x,X
Output:
A,111,aaa,A
B,222,bbb,B
X,EOF,xX
Example:
for i in $(ls /mypath/*.csv); do
sed '$s/,$//' < $i
done
This should do what you are looking for.
Note: apparently sed is not providing the "-i" option for all
platforms. If this is the case for your platform you have to use a
temporary file
Note also (thanks for glenn jackman's comment on this): This might
only work for the GNU sed implementation. You might need to adapt the
solution for other implementations
for i in $(ls /mypath/*.csv); do
if [[ `tail -n 1 $i | sed -n /EOF/p` != '' ]]; then
sed -i '$s/\([,]\)//3g' $i
fi
done
Use head to copy everything except the last line to a temporary file. Get the last line with tail, process it with sed and append it to the temporary file. Last but not least, replace the original file with the processed one.
for FILE in /mypath/*.csv ;
do
TMP_FILE="${FILE}.processed"
head -n "-1" "$FILE" > "$TMP_FILE"
tail -n "1" "$FILE" | sed 's/,\+/,/g' >> "$TMP_FILE"
mv -f "$TMP_FILE" "$FILE"
done
There is probably a more efficient inplace solution, but it does the job.

Resources