This question already has answers here:
Unix command to prepend text to a file
(21 answers)
Closed 4 years ago.
I want to write a bash script that takes my file:
READ_ME.MD
two
three
four
and makes it
READ_ME.MD
one
two
three
four
There are a bunch of similar StackOverflow questions, but I tried their answers and haven't been successful.
These are the bash scripts that I have tried and failed with:
test.sh
sed '1s/^/one/' READ_ME.md > READ_ME.md
Result: Clears the contents of my file
test.sh
sed '1,1s/^/insert this /' READ_ME.md > READ_ME.md
Result: Clears the contents of my file
test.sh
sed -i '1s/^/one\n/' READ_ME.md
Result: sed: 1: "READ_ME.md": invalid command code R
Any help would be appreciated.
You can use this BSD sed command:
sed -i '' '1i\
one
' file
-i will save changes inline to file.
If you want to add a line at the top if same line is not already there then use BSD sed command:
line='one'
sed -i '' '1{/'"$line"'/!i\
'"$line"'
}' file
Your last example works for me with GNU sed. Based on the error message you added, I'd guess you're working on a Mac system? According to this blog post, a suffix argument may be required on Mac versions of sed:
sed -i ' ' '1s/^one\n/' READ_ME.md
If this is bash or zsh, you can use process substitution like so.
% cat x
one
two
three
% cat <(echo "zero") x
zero
one
two
three
Redirect this into a temp file and copy it back to the original
there is always ed
printf '%s\n' H 1i "one" . w | ed -s READ_ME.MD
This question already has answers here:
How to merge every two lines into one from the command line?
(21 answers)
Closed 6 years ago.
I'm looking for a shell command to delete the return chariot on one line out of two.
I have a file like this :
1.32640997
;;P
1.14517534
;;P
1.16120958
;;P
...
And I would like something like this:
1.32640997;;P
1.14517534;;P
1.16120958;;P
...
Is it possible?
Thanks
Using GNU paste
paste -d '' - - < file
Using BSD paste
paste -d '\0' - - < file
paste produces two columns from stdin with - - as parameters, 3 columns with - - - as parameters, and so on.
-d is to specify a column separator, use '\0' for no separator.
Using Perl
perl -ne 'chomp($prev = $_); print $prev, scalar <>' < file
Using awk
$ awk '{printf "%s%s",$0,(NR%2==0?ORS:"")}' File
1.32640997;;P
1.14517534;;P
1.16120958;;P
This prints each line followed by nothing for odd lines or followed by the output record separator for even lines.
Using sed
This works by reading in lines in pairs:
$ sed 'N; s/\n//' File
1.32640997;;P
1.14517534;;P
1.16120958;;P
N reads in the next line and s/\n// removes the newline.
Using xargs:
xargs -n 2 -d '\n' printf '%s%s\n' <file
This question already has answers here:
How to make "grep" read patterns from a file?
(2 answers)
Closed 6 years ago.
I'm trying to write a small unix script to search one file for the line that contain a certain term, found in a different file. I would like to save the output of this command to a new file.
I have a file containing terms (terms.txt) which has a term on each line:
term1
term2
term3
term4
For each of these terms, I want to find the line that contains this term in another file (scores.txt) and append the output of this to a new file (output.txt).
The script I have come up with thus far:
#!/bin/bash
for f in `cat terms.txt`;
do
grep -i $f scores.txt >> output.txt;
done
Somehow this does not seem to work properly.
Running just the grep command with the term hard coded does indeed give me the right line I'm searching for:
grep -i "term1" scores.txt
Also, a simple echo does give me the right terms:
for f in `cat terms.txt`; do echo $f; done
However, when I try to repeat this with the $f variable, to repeat the same command for every term in my terms.txt, it does not work.
Could someone help me out on this one?
can you try:
grep -if terms.txt scores.txt > output.txt
basically grep's option f treats the strings in the terms.txt file as patterns to search for in scores.txt
If your terms.txt has CRLF line endings, try this:
grep -if <(tr -d '\r' < terms.txt) scores.txt > output.txt
This question already has answers here:
How can I extract a predetermined range of lines from a text file on Unix?
(28 answers)
Closed 7 years ago.
This seems pretty silly, but I haven't found a tool that does this, so I figured I'd ask just to make sure one doesn't exist before trying to code it up myself:
Is there any easy way to cat or less specific lines of a file?
I'm hoping for behavior like this:
# -s == start && -f == finish
# we want to print lines 5 - 10 of file.txt
cat -s 5 -f 10 file.txt
Even something a little simpler would be appreciated, but I just don't know of any tool that does this:
# print from line 10 to the end of the file
cat -s 10 file.txt
I'm thinking that both of these functionalities could be easily created with a mixture of head, tail, and wc -l, but maybe there are builtins that do this of which I'm unaware?
yes awk and sed can help
for lines 5 to 10
awk 'NR>4&&NR<11' file.txt
sed -n '5,10p' file.txt
for lines 10 to last line
awk 'NR>9' file.txt
sed -n '10,$p' file.txt
I need to repeatedly remove the first line from a huge text file using a bash script.
Right now I am using sed -i -e "1d" $FILE - but it takes around a minute to do the deletion.
Is there a more efficient way to accomplish this?
Try tail:
tail -n +2 "$FILE"
-n x: Just print the last x lines. tail -n 5 would give you the last 5 lines of the input. The + sign kind of inverts the argument and make tail print anything but the first x-1 lines. tail -n +1 would print the whole file, tail -n +2 everything but the first line, etc.
GNU tail is much faster than sed. tail is also available on BSD and the -n +2 flag is consistent across both tools. Check the FreeBSD or OS X man pages for more.
The BSD version can be much slower than sed, though. I wonder how they managed that; tail should just read a file line by line while sed does pretty complex operations involving interpreting a script, applying regular expressions and the like.
Note: You may be tempted to use
# THIS WILL GIVE YOU AN EMPTY FILE!
tail -n +2 "$FILE" > "$FILE"
but this will give you an empty file. The reason is that the redirection (>) happens before tail is invoked by the shell:
Shell truncates file $FILE
Shell creates a new process for tail
Shell redirects stdout of the tail process to $FILE
tail reads from the now empty $FILE
If you want to remove the first line inside the file, you should use:
tail -n +2 "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"
The && will make sure that the file doesn't get overwritten when there is a problem.
You can use -i to update the file without using '>' operator. The following command will delete the first line from the file and save it to the file (uses a temp file behind the scenes).
sed -i '1d' filename
For those who are on SunOS which is non-GNU, the following code will help:
sed '1d' test.dat > tmp.dat
You can easily do this with:
cat filename | sed 1d > filename_without_first_line
on the command line; or to remove the first line of a file permanently, use the in-place mode of sed with the -i flag:
sed -i 1d <filename>
No, that's about as efficient as you're going to get. You could write a C program which could do the job a little faster (less startup time and processing arguments) but it will probably tend towards the same speed as sed as files get large (and I assume they're large if it's taking a minute).
But your question suffers from the same problem as so many others in that it pre-supposes the solution. If you were to tell us in detail what you're trying to do rather then how, we may be able to suggest a better option.
For example, if this is a file A that some other program B processes, one solution would be to not strip off the first line, but modify program B to process it differently.
Let's say all your programs append to this file A and program B currently reads and processes the first line before deleting it.
You could re-engineer program B so that it didn't try to delete the first line but maintains a persistent (probably file-based) offset into the file A so that, next time it runs, it could seek to that offset, process the line there, and update the offset.
Then, at a quiet time (midnight?), it could do special processing of file A to delete all lines currently processed and set the offset back to 0.
It will certainly be faster for a program to open and seek a file rather than open and rewrite. This discussion assumes you have control over program B, of course. I don't know if that's the case but there may be other possible solutions if you provide further information.
The sponge util avoids the need for juggling a temp file:
tail -n +2 "$FILE" | sponge "$FILE"
If you want to modify the file in place, you could always use the original ed instead of its streaming successor sed:
ed "$FILE" <<<$'1d\nwq\n'
The ed command was the original UNIX text editor, before there were even full-screen terminals, much less graphical workstations. The ex editor, best known as what you're using when typing at the colon prompt in vi, is an extended version of ed, so many of the same commands work. While ed is meant to be used interactively, it can also be used in batch mode by sending a string of commands to it, which is what this solution does.
The sequence <<<$'1d\nwq\n' takes advantage of modern shells' support for here-strings (<<<) and ANSI quotes ($'...') to feed input to the ed command consisting of two lines: 1d, which deletes line 1, and then wq, which writes the file back out to disk and then quits the editing session.
As Pax said, you probably aren't going to get any faster than this. The reason is that there are almost no filesystems that support truncating from the beginning of the file so this is going to be an O(n) operation where n is the size of the file. What you can do much faster though is overwrite the first line with the same number of bytes (maybe with spaces or a comment) which might work for you depending on exactly what you are trying to do (what is that by the way?).
You can edit the files in place: Just use perl's -i flag, like this:
perl -ni -e 'print unless $. == 1' filename.txt
This makes the first line disappear, as you ask. Perl will need to read and copy the entire file, but it arranges for the output to be saved under the name of the original file.
should show the lines except the first line :
cat textfile.txt | tail -n +2
Could use vim to do this:
vim -u NONE +'1d' +'wq!' /tmp/test.txt
This should be faster, since vim won't read whole file when process.
How about using csplit?
man csplit
csplit -k file 1 '{1}'
This one liner will do:
echo "$(tail -n +2 "$FILE")" > "$FILE"
It works, since tail is executed prior to echo and then the file is unlocked, hence no need for a temp file.
Since it sounds like I can't speed up the deletion, I think a good approach might be to process the file in batches like this:
While file1 not empty
file2 = head -n1000 file1
process file2
sed -i -e "1000d" file1
end
The drawback of this is that if the program gets killed in the middle (or if there's some bad sql in there - causing the "process" part to die or lock-up), there will be lines that are either skipped, or processed twice.
(file1 contains lines of sql code)
tail +2 path/to/your/file
works for me, no need to specify the -n flag. For reasons, see Aaron's answer.
You can use the sed command to delete arbitrary lines by line number
# create multi line txt file
echo """1. first
2. second
3. third""" > file.txt
deleting lines and printing to stdout
$ sed '1d' file.txt
2. second
3. third
$ sed '2d' file.txt
1. first
3. third
$ sed '3d' file.txt
1. first
2. second
# delete multi lines
$ sed '1,2d' file.txt
3. third
# delete the last line
sed '$d' file.txt
1. first
2. second
use the -i option to edit the file in-place
$ cat file.txt
1. first
2. second
3. third
$ sed -i '1d' file.txt
$cat file.txt
2. second
3. third
If what you are looking to do is recover after failure, you could just build up a file that has what you've done so far.
if [[ -f $tmpf ]] ; then
rm -f $tmpf
fi
cat $srcf |
while read line ; do
# process line
echo "$line" >> $tmpf
done
Based on 3 other answers, I came up with this syntax that works perfectly in my Mac OSx bash shell:
line=$(head -n1 list.txt && echo "$(tail -n +2 list.txt)" > list.txt)
Test case:
~> printf "Line #%2d\n" {1..3} > list.txt
~> cat list.txt
Line # 1
Line # 2
Line # 3
~> line=$(head -n1 list.txt && echo "$(tail -n +2 list.txt)" > list.txt)
~> echo $line
Line # 1
~> cat list.txt
Line # 2
Line # 3
Would using tail on N-1 lines and directing that into a file, followed by removing the old file, and renaming the new file to the old name do the job?
If i were doing this programatically, i would read through the file, and remember the file offset, after reading each line, so i could seek back to that position to read the file with one less line in it.