This question already has answers here:
Input and output redirection to the same file [duplicate]
(4 answers)
Closed 7 years ago.
I am facing this strange behaviour in a simple pipe:
me$ echo "AAA" > tmp.txt
me$ cat tmp.txt | sed 's/A/B/g' > tmp.txt
me$ cat tmp.txt
The result is an empty file and not the desired "BBB" inside the tmp.txt
It works though if I chose a different file for output. some ideas? thx in advance!
You can write this:
sed 's/A/B/g' tmp.txt > tmp2.txt
mv tmp2.txt tmp.txt
The first line writes the contents of the file, with the relevant string replacement, to a new file. The second line moves the new file to the location of the old file, overwriting the old file.
Why not prefer the 'in-place' edit to a cat or rename ?
sed 's/A/B/g' -i tmp.txt
To change a file in place, use sed -i:
$ echo "AAA" > tmp.txt
$ sed -i 's/A/B/g' tmp.txt
$ cat tmp.txt
BBB
The above uses GNU sed syntax. If you are using Mac OSX (BSD), use:
sed -i '' 's/A/B/g' tmp.txt
Discussion
From the question, consider this line of code:
cat tmp.txt | sed 's/A/B/g' > tmp.txt
cat tmp.txt attempts to read from tmp.txt. When the shell sees > tmp.txt, however, it truncates tmp.txt to an empty file in preparation for input. The result of something like this is not reliable.
sed -i by contrast was explicitly designed to handle this situation. It completely avoids the conflict.
If you like, sed -i can create a back-up of the original file. With GNU sed, use:
sed -i.bak 's/A/B/g' tmp.txt
With BSD (Mac OSX) sed, add a space:
sed -i .bak 's/A/B/g' tmp.txt
Related
So I have been using the following general syntax to be able to remove text on a line within a configuration file after the '=' sign:
sed -i 's|\(.*\txt_to_remove=\).*|\1|g' file.txt
Let's say I have a text file:
$ cat tmp.txt
datee='10'
alter='purple'
hr='green'
new_val='100'
and if I consecutively use the command I provided above to try and delete everything after '=':
sed -i 's|\(.*\datee=\).*|\1|g' tmp.txt
sed -i 's|\(.*\alter=\).*|\1|g' tmp.txt
sed -i 's|\(.*\hr=\).*|\1|g' tmp.txt
sed -i 's|\(.*\new_val=\).*|\1|g' tmp.txt
The results will be:
$ cat tmp.txt
datee=
alter='purple'
hr=
new_val='100'
The question is, why did the commands not work for the 'alter' or 'new_val', and what can be done to make sure that it always works for each individual instance of using the command?
Do them all at once with:
sed -i 's/^\(\(datee\|alter\|hr\|new_val\)=\).*/\1/g' tmp.txt
You have a backslash before the keyword that you want to replace. Backslash is the escape prefix, and \a and \n are escape sequences, so they don't match what's in the file.
Get rid of these backslashes.
sed -i 's|\(.*datee=\).*|\1|g' tmp.txt
sed -i 's|\(.*alter=\).*|\1|g' tmp.txt
sed -i 's|\(.*hr=\).*|\1|g' tmp.txt
sed -i 's|\(.*new_val=\).*|\1|g' tmp.txt
try this sed expression sed 's/=.*//'
I tried and got the desired results
cat tmp.txt | sed 's/=.*//'
datee
alter
hr
new_val
So, disclaimer: I am pretty new to using bash and zsh, so there is a chance the answer is really simple. Nonetheless. I checked previous postings and couldn't find anything. (edit: I have tried this in both bash and zsh shells- same problem.)
I have a directory with many files and am trying to remove the first line from each file.
So say the directory contains: file1.txt file2.txt file3.txt ... etc.
I am using the sed command (non-GNU):
sed -i -e "1d" *.txt
For some reason, this is only removing the first line of the first file. I thought that the *.txt would affect all files matching the pattern in directory. Strangely, it is creating the file duplicates with -e appended, but both the duplicate and original are the same.
I tried this with other commands (e.g. ls *.txt) and it works fine. Is there something about sed I am missing?
Thank you in advance.
Different versions of sed in differing operating systems support various parameters.
OpenBSD (5.4) sed
The -i flag is unavailable. You can use the following /bin/sh syntax:
for i in *.txt
do
f=`mktemp -p .`
sed -e "1d" "${i}" > "${f}" && mv -- "${f}" "${i}"
done
FreeBSD (11-CURRENT) sed
The -i flag requires an extension, even if it's empty. Thus must be written as sed -i "" -e "1d" *.txt
GNU sed
This looks to see if the argument following -i is another option (or possibly a command). If so, it assumes an in-place modification. If it appears to be a file extension such as ".bak", it will rename the original with the ".bak" and then modify it into the original file's name.
There might be other variations on other platforms, but those are the three I have at hand.
use it without -e !
for one file use:
sed -i '1d' filename
for all files use :
sed -i '1d' *.txt
or
files=/path/to/files/*.extension ; for var in $files ; do sed -i '1d' $var ; done
.for me i use ubuntu and debian based systems , this method is working for me 100% , but for other platformes i'm not sure , so this is other method :
replace first line with emty pattern , and remove empty lines , (double commands):
for files in $(ls /path/to/files/*.txt); do sed -i "s/$(head -1 "$files")//g" "$files" ; sed -i '/^$/d' "$files" ; done
Note: if your files contain splash '/' , then it will give error , so in this case sed command should look like this ( sed -i "s[$(head -1 "$files")[[g" )
hope that's what you're looking for :)
The issue here is that the line number isn't reset when sed opens a new file, so 1 only matches the first line of the first file.
One solution is to use a shell loop, calling sed once for each file. Gumnos' answer shows how to do this in the most widely compatible way, although if you have a version of sed supporting the -i flag, you could do this instead:
for i in *.txt; do
sed -i.bak '1d' "$i"
done
It is possible to avoid creating the backup file by passing an empty suffix but personally, I don't think it's such a bad thing. One day you'll be grateful for it!
It appears that you're not working with GNU tools but if you were, I would recommend using GNU awk for this task. The variable FNR is useful here, as it keeps track of the record number for each file individually, allowing you to do this:
gawk -i inplace 'FNR>1' *.txt
Using the inplace extension, this allows you to remove the first line from each of your files, by only printing the lines where FNR is greater than 1.
Testing it out:
$ seq 5 > file1
$ seq 5 > file2
$ gawk -i inplace 'FNR>1' file1 file2
$ cat file1
2
3
4
5
$ cat file2
2
3
4
5
The last argument you are passing to the Sed is the problem
try something like this.
var=(`find *txt`)
for file in "${var[#]}"
do
sed -i -e 1d $file
done
This did the trick for me.
I'm using Window 7.
sed -V
gives me
GNU sed version 3.02
I have a text file with:
foo bar
In command line if I write:
sed s/foo/tofu/ test.txt
I get printed on stdout;
tofu bar
However, I want to edit this file inplace, so using the -i option:
sed -i s/foo/tofu/ test.txt
gives me sed: invalid option -- i
What am I missing?
While I can't guarantee this will work because I'm not running it on Windows 7 but have you considered using redirection and then swapping?
Meaning something like this:
sed s/foo/tofu/ test.txt > tmp.txt && mv tmp.txt test.txt
While this is a very dirty way of doing things I think it might work. You can also try doing
sed s/foo/tofu/ test.txt > test.txt
but I feel that might lead to some errors because you're reading and writing on the same file.
It's the sed version. Try the one found here:
http://unxutils.sourceforge.net/UnxUpdates.zip
I recently wrote a script that does a sed command, to replace all the occurrences of "string1" with "string2" in a file named "test.txt".
It looks like this:
sed -i 's/string1/string2/g' test.txt
The catch is, "string1" does not necessarily exist in test.txt.
I notice after executing a bunch of these sed commands, I get a number of empty files, left behind in the directory, with names that look like this:
"sed4l4DpD"
Does anyone know why this might be, and how I can correct it?
-i is the suffix given to the new/output file. Also, you need -e for the command.
Here's how you use it:
sed -i '2' -e 's/string1/string2/g' test.txt
This will create a file called test.txt2 that is the backup of test.txt
To replace the file (instead of creating a new copy - called an "in-place" substitution), change the -i value to '' (ie blank):
sed -i '' -e 's/string1/string2/g' test.txt
EDIT II
Here's actual command line output from a Mac (Snow Leopard) that show that my modified answer (removed space from between the -i and the suffix) is correct.
NOTE: On a linux server, there must be no space between it -i and the suffix.
> echo "this is a test" > test.txt
> cat test.txt
this is a test
> sed -i '2' -e 's/a/a good/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a good test
> cat test.txt2
this is a test
> sed -i '' -e 's/a/a really/' test.txt
> ls test*
test.txt test.txt2
> cat test.txt
this is a really good test
I wasn't able to reproduce this with a quick test (using GNU sed 4.2.1) -- but strace did show sed creating a file called sedJd9Cuy and then renaming it to tmp (the file named on the command line).
It looks like something is going wrong after sed creates the temporary file and before it's able to rename it.
My best guess is that you've run out of room in the filesystem; you're able to create a new empty file, but unable to write to it.
What does df . say?
EDIT:
I still don't know what's causing the problem, but it shouldn't be too difficult to work around it.
Rather than
sed -i 's/string1/string2/g' test.txt
try something like this:
sed 's/string1/string2/g' test.txt > test.txt.$$ && mv -f test.txt.$$ test.txt
Something is going wrong with the way sed creates and then renames a text file to replace your original file. The above command uses sed as a simple input-output filter and creates and renames the temporary file separately.
So after much testing last night, it turns out that sed was creating these files when trying to operate on an empty string. The way i was getting the array of "$string1" arguments was through a grep command, which seems to be malformed. What I wanted from the grep was all lines containing something of the type "Text here '.'".
For example the string, "Text here 'ABC.DEF'" in a file, should have been caught by grep, then the ABC.DEF portion of the string, would be substituted by ABC_DEF. Unfortunately the grep I was using would catch lines of the type "Text here ''" (that is, nothing between the ''). When later on, the script attempted to perform a sed replacement using this empty string, the random file was created (probably because sed died).
Thanks for all your help in understanding how sed works.
Its better if you do it in this way:
cat large_file | sed 's/string1/string2/g' > file_filtred
I'd like edit a file with sed on OS X. I'm using the following command:
sed 's/oldword/newword/' file.txt
The output is sent to the terminal. file.txt is not modified. The changes are saved to file2.txt with this command:
sed 's/oldword/newword/' file1.txt > file2.txt
However I don't want another file. I just want to edit file1.txt. How can I do this?
I've tried the -i flag. This results in the following error:
sed: 1: "file1.txt": invalid command code f
You can use the -i flag correctly by providing it with a suffix to add to the backed-up file. Extending your example:
sed -i.bu 's/oldword/newword/' file1.txt
Will give you two files: one with the name file1.txt that contains the substitution, and one with the name file1.txt.bu that has the original content.
Mildly dangerous
If you want to destructively overwrite the original file, use something like:
sed -i '' 's/oldword/newword/' file1.txt
^ note the space
Because of the way the line gets parsed, a space is required between the option flag and its argument because the argument is zero-length.
Other than possibly trashing your original, I’m not aware of any further dangers of tricking sed this way. It should be noted, however, that if this invocation of sed is part of a script, The Unix Way™ would (IMHO) be to use sed non-destructively, test that it exited cleanly, and only then remove the extraneous file.
I've similar problem with MacOS
sed -i '' 's/oldword/newword/' file1.txt
doesn't works, but
sed -i"any_symbol" 's/oldword/newword/' file1.txt
works well.
The -i flag probably doesn't work for you, because you followed an example for GNU sed while macOS uses BSD sed and they have a slightly different syntax.
All the other answers tell you how to correct the syntax to work with BSD sed. The alternative is to install GNU sed on your macOS with:
brew install gsed
and then use it instead of the sed version shipped with macOS (note the g prefix), e.g:
gsed -i 's/oldword/newword/' file1.txt
If you want GNU sed commands to be always portable to your macOS, you could prepend "gnubin" directory to your path, by adding something like this to your .bashrc/.zshrc file (run brew info gsed to see what exactly you need to do):
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
and from then on the GNU sed becomes your default sed and you can simply run:
sed -i 's/oldword/newword/' file1.txt
sed -i -- "s/https/http/g" file.txt
You can use -i'' (--in-place) for sed as already suggested. See: The -i in-place argument, however note that -i option is non-standard FreeBSD extensions and may not be available on other operating systems. Secondly sed is a Stream EDitor, not a file editor.
Alternative way is to use built-in substitution in Vim Ex mode, like:
$ ex +%s/foo/bar/g -scwq file.txt
and for multiple-files:
$ ex +'bufdo!%s/foo/bar/g' -scxa *.*
To edit all files recursively you can use **/*.* if shell supports that (enable by shopt -s globstar).
Another way is to use gawk and its new "inplace" extension such as:
$ gawk -i inplace '{ gsub(/foo/, "bar") }; { print }' file1
This creates backup files. E.g. sed -i -e 's/hello/hello world/' testfile for me, creates a backup file, testfile-e, in the same dir.
You can use:
sed -i -e 's/<string-to-find>/<string-to-replace>/' <your-file-path>
Example:
sed -i -e 's/Hello/Bye/' file.txt
This works flawless in Mac.
If you need to substitute more than one different words:
sed -i '' -e 's/_tools/tools/' -e 's/_static/static/' test.txt