This question already has answers here:
Remove the last line from a file in Bash
(16 answers)
Closed 5 years ago.
I asked a question regarding popping the last line of a text file in PHP, and now, is it possible to re-write the logic in shell script?
I tried this to obtain the last line:
tail -n 1 my_log_file.log
but I am not sure how can I remove the last line and save the file.
P.S. given that I use Ubuntu server.
To get the file content without the last line, you can use
head -n-1 logfile.log
(I am not sure this is supported everywhere)
or
sed '$d' logfile.log
What you want is truncate the file just before the last line without having to read the file entirely.
truncate -s -"$(tail -n1 file | wc -c)" file
That's assuming the file is not currently being written to.
truncate is part of the GNU coreutils (so generally found on recent Linux distributions) and is not a standardized Unix or POSIX command. Many "dd" implementations can be used to truncate a file as well.
(Solution is based on sch's answer so credit should go to him/her)
This approach will allow you to efficiently retrieve the last line of the file and truncate the file to remove that line. This can better deal with large inputs as the file is not read sequentially.
# retrieve last line from file
LAST=$(tail -n 1 my_log_file.log)
# truncate file
let TRUNCATE_SIZE="${#LAST} + 1"
truncate -s -"$TRUNCATE_SIZE" my_log_file.log
# ... $LAST contains 'popped' last line
Note that this will not work as expected if the file is modified between the calls to tail and truncate.
One way is:
sed '$d' < f1 > f2 ; mv f2 f1
Related
This question already has answers here:
How can I use a file in a command and redirect output to the same file without truncating it?
(14 answers)
Closed 8 months ago.
I am using this command
sort -k1 -n source-g5.txt
to sort the content of file tmp-source-g5.txt (n rows, 2 columns) according to the numerical value of the first column.
When I run that line, the terminal prints out the desired result, but as I try to save the result into the same file,
sort -k1 -n source-g5.txt > source-g5.txt
the file shows no difference from before.
What am I doing wrong?
SOLVED
From this thread it turns out that redirecting the output of sort into the same file from which sort reads as source will not work since
the shell is makes the redirections (not the sort(1) program) and the
input file (as being the output also) will be erased just before
giving the sort(1) program the opportunity of reading it.
So I have split my command into two
sort -k1 -n source-g5.txt > tmp-source-g5.txt
mv tmp-source-g5.txt > source-g5.txt
I have a small script which basically generates a menu of all the scripts in my ~/scripts folder and next to each of them displays a sentence describing it, that sentence being the third line within the script commented out. I then plan to pipe this into fzf or dmenu to select it and start editing it or whatever.
1 #!/bin/bash
2
3 # a script to do
So it would look something like this
foo.sh a script to do X
bar.sh a script to do Y
Currently I have it run a for loop over all the files in the scripts folder and then run sed -n 3p on all of them.
for i in $(ls -1 ~/scripts); do
echo -n "$i"
sed -n 3p "~/scripts/$i"
echo
done | column -t -s '#' | ...
I was wondering if there is a more efficient way of doing this that did not involve a for loop and only used sed. Any help will be appreciated. Thanks!
Instead of a loop that is parsing ls output + sed, you may try this awk command:
awk 'FNR == 3 {
f = FILENAME; sub(/^.*\//, "", f); print f, $0; nextfile
}' ~/scripts/* | column -t -s '#' | ...
Yes there is a more efficient way, but no, it doesn't only use sed. This is probably a silly optimization for your use case though, but it may be worthwhile nonetheless.
The inefficiency is that you're using ls to read the directory and then parse its output. For large directories, that causes lots of overhead for keeping that list in memory even though you only traverse it once. Also, it's not done correctly, consider filenames with special characters that the shell interprets.
The more efficient way is to use find in combination with its -exec option, which starts a second program with each found file in turn.
BTW: If you didn't rely on line numbers but maybe a tag to mark the description, you could also use grep -r, which avoids an additional process per file altogether.
This might work for you (GNU sed):
sed -sn '1h;3{H;g;s/\n/ /p}' ~/scripts/*
Use the -s option to reset the line number addresses for each file.
Copy line 1 to the hold space.
Append line 3 to the hold space.
Swap the hold space for the pattern space.
Replace the newline with a space and print the result.
All files in the directory ~/scripts will be processed.
N.B. You may wish to replace the space delimiter by a tab or pipe the results to the column command.
This question already has answers here:
Bash tool to get nth line from a file
(22 answers)
How do I set a variable to the output of a command in Bash?
(15 answers)
Closed 3 years ago.
So, I am creating a script in bash and I am trying to read a text file which is saved in the same directory. I need to read the nth line of that file and then save it to a variable to be used later but I'm not sure how I can do this
What I have currently tried is listed below but it essentially reads the line from the file, saves it a variable and then deletes that line from the file and repeats. This is a hack and although it works, isn't what I want, I can't get the nth value and it's deleting from the file which I definitely don't want.
read -r first<"test.txt" // reads first line and stores in first
sed -i -e "1d" "test.txt" . // removes first line
read -r second<"test.txt" // reads first line and stores in second
sed -i -e "1d" "test.txt" . // removes first line
If I wanted to get the 2nd line for example, I have seen sed '2q;d' file but not sure how/where the result is saved. It gets printed in terminal? Any help appreciated, thanks!w
sed '2q;d' file
prints the second line in file to the terminal.
To populate a variable with it, use bash's command expansion feature:
$ var=$(sed '2q;d' file)
$ echo "$var"
this is second line
Simple solution using head and tail:
a=$(head -2 test.txt | tail -1 )
Saves the second line of test.txt to the variable $a.
I would like to extract the first line from a file, read into a variable and delete right afterwards, with a single command. I know sed can read the first line as follows:
sed '1q' file.txt
or delete it as follows:
sed '1q;d' file.txt
but can I somehow do both with a single command?
The reason for this is that multiple processes will be reading the first line of the file, and I want to minimize the chances of them getting the same line.
It's impossible.
Except you read the manpage, and have Gnu-sed:
echo -e {1..3}"\n" > input
cat input
1
2
3
sed -n '1p;2,$ Woutput' input
1
cat output
2
3
Explanation:
sed -n '1p;2,$ Woutput' input
-n no output by default
1p; print line 1
2,$ from line 2 until $ last line
W (non posix) Write buffer to file
From the man page gnu sed:
w filename
Write the current pattern space to filename.
W filename
Write the first line of the current pattern space to filename. This is a GNU extension.
However, reading and experimenting takes longer, than opening the file in a full blown office suite and deleting the line by hand, or invoking a text-to-speech framework and training it, to do the job.
It doesn't work if invoked in posix style:
sed -n --posix '1p;2,$ Woutput' input
And you still have the hard hanwork of renaming output to input again.
I didn't try to write to input in place, because that could damage my carefully crafted input file - try it on own risk:
sed -n '1p;2,$ Winput' input
However, you might set up a filesystem notify job, which always rename freshly created output files to input again. But I fear you can't do it from within the sed command. Except ... (to be continued)
This question already has answers here:
Find and replace in file and overwrite file doesn't work, it empties the file
(12 answers)
Sed to delete a range of lines from a specific match line TILL a specific match line (not including last line)
(3 answers)
Closed 8 years ago.
I am writing a bash shell script, and in it I am trying to delete lines from a text file between 2 markers
START
...
...
END
Do eliminate this I have tried a few things, and every time it leave my text file blank.
sed '/START/,/END/d' myfile.txt > myfile.txt
sed '/START/d' myfile.txt > myfile.txt
As long as I have a sed command in my code, my entire file gets erased and not just the section I am looking to erase.
You are redirecting stdout to the same file as stdin. When you do this, your redirection is interpreted buy the shell and it opens a new file for write with that name. Your existing file is overwritten by the newly created blank file. You will need to redirect the output to a different file or you can edit the file by using the -i option to sed.
You can't redirect to a file that you are reading. That will delete your file's contents, as you've noticed.
Instead, either redirect to a different file, or edit in place:
sed -i ... myfile.txt
A way in awk.
awk '/START/{x=1}/END/{x=0}!x{print > ARGV[1]}' myfile.txt
Portable
awk '/START/{x=1}/END/{x=0}!x{print > "myfile.txt"}' myfile.txt