bash not listing files in while loop - bash
The below bash seems to run but no file names are displayed on the terminal screen, rather it just stalls. I can not seem to figure out why it is not working now as it used to. Thank you :).
bash
while read line; do
sed -i -e 's|https://www\.example\.com/xx/x/xxx/||' /home/file
echo $line
done
file
Auto_user_xxx-39-160506_file_name_x-x_custom_xx_91-1.pdf
Auto_user_xxx-48-160601_V4-2_file_name_x-x_custom_xx_101.pdf
coverageAnalysisReport(10).zip
The read command is waiting for input, since nothing is specified it will read from stdin. If you type a few lines and press you will see thats the input for the loop.
But you most likely want to redirect a file to the loop:
while IFS= read -r line; do
printf "%s\n" "$line"
done < /home/file
But afai can understand you have a file with other file names which you would like to run the substitution on, in that case you should use xargs:
xargs -n 1 -I {} sed -i -e 's|https://www\.example\.com/xx/x/xxx/||' {} < /home/file
Related
Updating a config file based on the presence of a specific string
I want to be able to comment and uncomment lines which are "managed" using a bash script.
I am trying to write a script which will update all of the config lines which have the word #managed after them and remove the preceeding # if it exists.
The rest of the config file needs to be left unchanged. The config file looks like this:
configFile.txt
#config1=abc #managed
#config2=abc #managed
config3=abc #managed
config3=abc
This is the script I have created so far. It iterates the file, finds lines which contain "#managed" and detects if they are currently commented.
I need to then write this back to the file, how do I do that?
manage.sh
#!/bin/bash
while read line; do
STR='#managed'
if grep -q "$STR" <<< "$line"; then
echo "debug - this is managed"
firstLetter=${$line:0:1}
if [ "$firstLetter" = "#" ]; then
echo "Remove the initial # from this line"
fi
fi
echo "$line"
done < configFile.txt
With your approach using grep and sed. str='#managed$' file=ConfigFile.txt grep -q "^#.*$str" "$file" && sed "/^#.*$str/s/^#//" "$file" Looping through files ending in *.txt #!/usr/bin/env bash str='#managed$' for file in *.txt; do grep -q "^#.*$str" "$file" && sed "/^#.*$str/s/^#//" "$file" done In place editing with sed requires the -i flag/option but that varies from different version of sed, the GNU version does not require an -i.bak args, while the BSD version does. On a Mac, ed should be installed by default, so just replace the sed part with. printf '%s\n' "g/^#.*$str/s/^#//" ,p Q | ed -s "$file" Replace the Q with w to actually write back the changes to the file. Remove the ,p if no output to stdout is needed/required. On a side note, embedding grep and sed in a shell loop that reads line-by-line the contents of a text file is considered a bad practice from shell users/developers/coders. Say the file has 100k lines, then grep and sed would have to run 100k times too!
This sed one-liner should do the trick:
sed -i.orig '/#managed/s/^#//' configFile.txt
It deletes the # character at the beginning of the line if the line contains the string #managed.
I wouldn't do it in bash (because that would be slower than sed or awk, for instance), but if you want to stick with bash:
#! /bin/bash
while IFS= read -r line; do
if [[ $line = *'#managed'* && ${line:0:1} = '#' ]]; then
line=${line:1}
fi
printf '%s\n' "$line"
done < configFile.txt > configFile.tmp
mv configFile.txt configFile.txt.orig && mv configFile.tmp configFile.txt
'sed: no input files' when using sed -i in a loop
I checked some solutions for this in other questions, but they are not working with my case and I'm stuck so here we go. I have a csv file that I want to convert all to uppercase. It has to be with a loop and occupate 7 lines of code minimum. I have to run the script with this command: ./c_bash.sh student-mat.csv So I tried this Script: #!/bin/bash declare -i c=0 while read -r line; do if [ "$c" -gt '0' ]; then sed -e 's/\(.*\)/\U\1/' else echo "$line" fi ((c++)) done < student-mat.csv I know that maybe there are a couple of unnecessary things on it, but I want to focus in the sed command because it looks like the problem here. That script shows this output:(first 5 lines): school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3 GP,F,17,U,GT3,T,1,1,AT_HOME,OTHER,COURSE,FATHER,1,2,0,NO,YES,NO,NO,NO,YES,YES,NO,5,3,3,1,1,3,4,5,5,6 GP,F,15,U,LE3,T,1,1,AT_HOME,OTHER,OTHER,MOTHER,1,2,3,YES,NO,YES,NO,YES,YES,YES,NO,4,3,2,2,3,3,10,7,8,10 GP,F,15,U,GT3,T,4,2,HEALTH,SERVICES,HOME,MOTHER,1,3,0,NO,YES,YES,YES,YES,YES,YES,YES,3,2,2,1,1,5,2,15,14,15 GP,F,16,U,GT3,T,3,3,OTHER,OTHER,HOME,FATHER,1,2,0,NO,YES,YES,NO,YES,YES,NO,NO,4,3,2,1,2,5,4,6,10,10 GP,M,16,U,LE3,T,4,3,SERVICES,OTHER,REPUTATION,MOTHER,1,2,0,NO,YES,YES,YES,YES,YES,YES,NO,5,4,2,1,2,5,10,15,15,15 Now that I see that it works, I want to apply that sed command permanently to the csv file, so I put -i after it: #!/bin/bash declare -i c=0 while read -r line; do if [ "$c" -gt '0' ]; then sed -i -e 's/\(.*\)/\U\1/' else echo "$line" fi ((c++)) done < student-mat.csv But the output instead of applying the changes, shows this:(first 5 lines) school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3 sed: no input files sed: no input files sed: no input files sed: no input files sed: no input files So checking a lot of different solutions on the internet, I also tried to change single quoting to double quoting. #!/bin/bash declare -i c=0 while read -r line; do if [ "$c" -gt '0' ]; then sed -i -e "s/\(.*\)/\U\1/" else echo "$line" fi ((c++)) done < student-mat.csv But in this case, instead of applying the changes, it generate a file with 0 bytes. So no output when I do this: cat student-mat.csv My expected solution here is that, when I apply this script, it changes permanently all the data to uppercase. And after applying the script, it should show this with the command cat student-mat.csv: (first 5 lines) school,sex,age,address,famsize,Pstatus,Medu,Fedu,Mjob,Fjob,reason,guardian,traveltime,studytime,failures,schoolsup,famsup,paid,activities,nursery,higher,internet,romantic,famrel,freetime,goout,Dalc,Walc,health,absences,G1,G2,G3 GP,F,17,U,GT3,T,1,1,AT_HOME,OTHER,COURSE,FATHER,1,2,0,NO,YES,NO,NO,NO,YES,YES,NO,5,3,3,1,1,3,4,5,5,6 GP,F,15,U,LE3,T,1,1,AT_HOME,OTHER,OTHER,MOTHER,1,2,3,YES,NO,YES,NO,YES,YES,YES,NO,4,3,2,2,3,3,10,7,8,10 GP,F,15,U,GT3,T,4,2,HEALTH,SERVICES,HOME,MOTHER,1,3,0,NO,YES,YES,YES,YES,YES,YES,YES,3,2,2,1,1,5,2,15,14,15 GP,F,16,U,GT3,T,3,3,OTHER,OTHER,HOME,FATHER,1,2,0,NO,YES,YES,NO,YES,YES,NO,NO,4,3,2,1,2,5,4,6,10,10 GP,M,16,U,LE3,T,4,3,SERVICES,OTHER,REPUTATION,MOTHER,1,2,0,NO,YES,YES,YES,YES,YES,YES,NO,5,4,2,1,2,5,10,15,15,15
Sed works on files, not on lines. Do not read lines, use sed on the file. Sed can exclude the first line by itself. See sed manual. You want: sed -i -e '2,$s/\(.*\)/\U\1/' student-mat.csv You can do shorter with s/.*/\U&/. Your code does not work as you think it does. Note that your code removes the second line from the output. Your code: reads first line with read -r line echo "$line" first line is printed c++ is incremented read -r line reads second line then sed processes the rest of the file (from line 3 till the end) and prints them in upper case then c++ is incremented then read -r line fails, and the loop exits
need to clean file via SED or GREP
I have these files NotRequired.txt (having lines which need to be remove) Need2CleanSED.txt (big file , need to clean) Need2CleanGRP.txt (big file , need to clean) content: more NotRequired.txt [abc-xyz_pqr-pe2_123] [lon-abc-tkt_1202] [wat-7600-1_414] [indo-pak_isu-5_761] I am reading above file and want to remove lines from Need2Clean???.txt, trying via SED and GREP but no success. myFile="NotRequired.txt" while IFS= read -r HKline do sed -i '/$HKline/d' Need2CleanSED.txt done < "$myFile" myFile="NotRequired.txt" while IFS= read -r HKline do grep -vE \"$HKline\" Need2CleanGRP.txt > Need2CleanGRP.txt done < "$myFile" Looks as if the Variable and characters [] making some problem.
What you're doing is extremely inefficient and error prone. Just do this: grep -vF -f NotRequired.txt Need2CleanGRP.txt > tmp && mv tmp Need2CleanGRP.txt Thanks to grep -F the above treats each line of NotRequired.txt as a string rather than a regexp so you don't have to worry about escaping RE metachars like [ and you don't need to wrap it in a shell loop - that one command will remove all undesirable lines in one execution of grep. Never do command file > file btw as the shell might decide to execute the > file first and so empty file before command gets a chance to read it! Always do command file > tmp && mv tmp file instead.
Your assumption is correct. The [...] construct looks for any characters in that set, so you have to preface ("escape") them with \. The easiest way is to do that in your original file:
sed -i -e 's:\[:\\[:' -e 's:\]:\\]:' "${myFile}"
If you don't like that, you can probably put the sed command in where you're directing the file in:
done < replace.txt|sed -e 's:\[:\\[:' -e 's:\]:\\]:'
Finally, you can use sed on each HKline variable:
HKline=$( echo $HKline | sed -e 's:\[:\\[:' -e 's:\]:\\]:' )
try gnu sed: sed -Ez 's/\n/\|/g;s!\[!\\[!g;s!\]!\\]!g; s!(.*).!/\1/d!' NotRequired.txt| sed -Ef - Need2CleanSED.txt Two sed process are chained into one by shell pipe NotRequired.txt is 'slurped' by sed -z all at once and substituted its \n and [ meta-char with | and \[ respectively of which the 2nd process uses it as regex script for the input file, ie. Need2CleanSED.txt. 1st process output; /\[abc-xyz_pqr-pe2_123\]|\[lon-abc-tkt_1202\]|\[wat-7600-1_414\]|\[indo-pak_isu-5_761\]/d add -u ie. unbuffered, option to evade from batch process, sort of direct i/o
How can I evaluate the filename passed to a bash script via stdin
I would like to evaluate a filename passed to a script via stdin and then use this in a sed command. At the moment I have the following code:
#!/bin/bash
eval file="${1:-/dev/stdin}"
echo "$file"
sed -i -r ':a; s%(.*)/\*.*\*/%\1%; ta; /\/\*/ !b; N; ba' "$file"
Calling this as:
bash callfile < file.txt
results in the following error:
/dev/stdin
sed: couldn't open temporary file /dev/sedSVVTdr: Permission denied
Why is this code unable to read the filename I pass to it?
That data is getting passed to you as a stream redirected from the file. Your script won't have any knowledge of that file - only the stream. As a further example, it could be data piped in from the output of another process, and so you wouldn't have any file to begin with.
As #Brian Agnew said, you are reading from your document, so a good use of your command line is:
while read -r line; do
echo ${line} | sed -r ':a; s%(.*)/\*.*\*/%\1%; ta; /\/\*/ !b; N; ba'
done < ${1:-/dev/stdin}
Note that is you have a list of files in a file, you can then run your process as intended:
./callfile < files_list.txt
while read -r file; do
sed -i -r ':a; s%(.*)/\*.*\*/%\1%; ta; /\/\*/ !b; N; ba' "${file}"
done < ${1:-/dev/stdin}
but at the moment, there is no advantage of reading from stdin, so:
./callfile "file.txt"
EDIT: replaced ${line} in the read by line, and ${file} with file
I hope we understood your question.
Extracting a pattern (grep output) in Linux from shell?
Grep output is usually like this: after/ftplugin/python.vim:49: setlocal number Is it possible for me extract the file name and line number from this result using standard linux utilities ? Looking for a generic solution that works pretty well . I can think of using awk to get the first string like : Input echo 'after/ftplugin/python.vim:49: setlocal number' | awk 'print $1' 'after/ftplugin/python.vim:49:' $ Expected after/ftplugin/python.vim and 49 Goal : Open in Vim I am writing a small function that transforms the grep output to something vim can understand - mostly for academic purpose . I know there are thinks like Ack.vim out there which does something similar . What are the standard light weight utils out there ? Edit: grep -n "text to find" file.ext |cut -f1 -d: seems to do it if you dont mind double parsing the string . Sed though needs to be used !
If you're using Bash you can do it this way: IFS=: read FILE NUM __ < <(exec grep -Hn "string to find" file) vim "+$NUM" "$FILE" Or POSIX: IFS=: read FILE NUM __ <<EOD $(grep -Hn "string to find" file) EOD vim "+$NUM" "$FILE" Style © konsolebox :)
This will do:
echo 'after/ftplugin/python.vim:49: setlocal number' | awk -F: '{print $1,"and",$2}'
after/ftplugin/python.vim and 49
But give us data before grep. It may be that we can cut it more down. No need for both grep and awk
If by "reverse parse" you mean you want to start from the end (and can safely assume that the file content contains no colons), parameter expansion makes that easy:
line='after/ftplugin/python.vim:49: setlocal number'
name_and_lineno=${line%:*}
name=${name_and_lineno%:*}
lineno=${name_and_lineno##*:}
Being all in-process (using shell built-in functionality), this is much faster than using external tools such as sed, awk, etc.
To connect it all together, consider a loop such as the following:
while read -r line; do
...
done < <(grep ...)
Now, to handle all possible filenames (including ones with colons) and all possible content (including strings with colons), you need a grep with GNU extensions:
while IFS='' read -u 4 -r -d '' file \
&& read -u 4 -r -d ':' lineno \
&& read -u 4 -r line; do
vim "+$lineno" "$file"
done 4< <(grep -HnZ -e "string to find" /dev/null file)
This works as follows:
Use grep -Z (a GNU extension) to terminate each filename with a NUL rather than a :
Use IFS='' read -r -d '' to read until the first NUL when reading filenames
Use read -r -d ':' lineno to read until a colon when reading line numbers
Read until the next newline when reading lines
Redirect contents on FD #4 to avoid overriding stdin, stdout or stderr (so vim will still work properly)
Use the -u 4 argument on all calls to read to handle contents from FD #4
How about this?: echo 'after/ftplugin/python.vim:49: setlocal number' | cut -d: -f1-2 | sed -e 's/:/ and /' Result: after/ftplugin/python.vim and 49