Removing current line in bash during read - bash

Question

I'm using bash to read a file and after doing opeation on particular line ,i need to delete that line from input file.
Can you please suggest some way to do so using sed or any other way ?
i've tried using sed command like this :-
#!/bin/sh
file=/Volumes/workplace/GeneratedRules.ion
while read line;do
printf "%s\n" "$line"
sed '1d' $file
done <$file
my aim in this program is to read one line and then deleting it.
Input :-
AllIsWell
LetsHopeForBest
YouCanMakeIt
but the output , i got is more weird than i thought.
output :-
AllIsWell
LetsHopeForBest
YouCanMakeIt
LetsHopeForBest
LetsHopeForBest
YouCanMakeIt
YouCanMakeIt
LetsHopeForBest
YouCanMakeIt
but i need to output as :
AllIsWell
LetsHopeForBest
YouCanMakeIt
as i want to delete line after reading it.
NOTE :- i have simplified my problem here . The actual usecase is :-
I need to perform some bunch of operation on line except reading that and the input file is too long and my operation got fails in some way in between .So i want those lines which i have read to be deleted so that if i start the process again , it will not start from the beginning but at the point where it got stuck.
please help.

Answer 1

You effectively said you want your process to be restartable. Depending upon how you define the successful completion of an iteration of your while loop, you should store a line number in a separate file, x, that indicates how many lines you have successfully processed. If the file doesn't exist, then assume you would start reading at line 1.
Otherwise, you would get the content of x into variable n and then you would start reading $file at line $n + 1.
How you start reading at a particular line depends on constraints we don't know yet.
One way you could do it is to use sed to put lines $n + 1 into a temporary file, remove $file and then move the temporary file to $file before your while loop begins.
There are other solutions but each one might not elegantly satisfy your constraints.
But you'll want to carefully consider what happens if some other process is modifying the content of $file and when it is changing the content. If it only changes the content before or after your bash script is running, then you're probably ok to continue down this path. Otherwise, you have a synchronization problem to solve.

Answer 2

As stated in comments, there are many issues with altering the file you are currently reading from. Don't do it.
You could just keep track of which lines you have dealt with in the first loop (with a counter) then use sed to delete those lines after your first loop has processed them.
A simple example:
cd /tmp
echo 'Line 1
Line 2
Line 3
Line 4
Line 5' >file
echo "file before:"
cat file
cnt=1
while IFS= read -r line || [[ -n $line ]]; do
printf "'%s' processed\n" "$line";
if [ "$cnt" -ge 3 ]; then
break
fi
let "cnt+=1"
done <file
sed -i '' "1,${cnt}d" file
echo "file after:"
cat file
Prints:
file before:
Line 1
Line 2
Line 3
Line 4
Line 5
'Line 1' processed
'Line 2' processed
'Line 3' processed
file after:
Line 4
Line 5
Another method is to use something like awk, ruby or perl to 'slurp' the file and then feed that slurpped file line-by-line to your Bash while loop. The file can then be modified in your loop since the other process has already fully read and closed the file:
# Note: This is SLOWER and USES MORE MEMORY...
echo "file before:"
cat file
while IFS= read -r line; do
printf "'%s' processed\n" "$line";
sed -i '' "1d" file
done < <(awk -v cnt=3 'NR>cnt{next}
{arr[NR]=$0}
END { for(i=1;i<=cnt;i++) print arr[i] }' file)
echo "file after:"
cat file
# same output
Note:
Please make sure you polish up on how to use bash to read a stream line-by-line for less surprises.
Read HERE and HERE for more.

Answer 3

It needs a option -i, and if you need backup the file, just assign a suffix to the option. see the man sed
#!/bin/sh
file=/Volumes/workplace/GeneratedRules.ion
while read line;do
printf "%s\n" "$line"
sed -i '1d' $file
done <$file