On my Mac, when I try to run:
sed -i 2d file.csv
from the answer to Unix script to remove the first line of a CSV file, I get the following error:
sed: 1: "file.csv": invalid command code f
What can I do if I would like to delete the first two lines of file.csv?
On a Mac, the BSD sed requires the suffix for the backup (but the suffix can be an empty string, '') — it is not optional as it is with GNU sed. Hence, your command is being interpreted as "backup the file with the suffix 2d and … oops, the script you gave is file.csv, but f isn't a sed command".
sed -i .bak -e 2d file.csv
This deletes the first line of data from the CSV file (leaving the heading line in place).
If you want to write the code so it works with both BSD and GNU sed, then you have to attach the suffix to the -i option (GNU sed requires the suffix attached to the -i option; that's how it identifies whether there's an optional suffix or not):
sed -i.bak -e 2d file.csv
Note that you can't use an empty suffix and have the command work with both BSD sed and GNU sed.
The -e isn't necessary in either command line, but I quite often use it. I also often quote the command in single quotes, though it isn't necessary here.
If you want to delete the first two data lines, use 2,3d as the command. If you want to delete the first two lines, use 1,2d.
If you don't want the backup, then you can either remove it after the sed command completes (easiest) or use the two stage or three stage dance:
sed 2d file.csv > file.csv.bak &&
mv file.csv.bak file.csv # Oops; there went the links
sed 2d file.csv > file.csv.bak &&
cp file.csv.bak file.csv
rm -f file.csv.bak
With these, you might need to add trap commands to clean up the intermediate .bak file if an interrupt or other signal terminates the script.
To quote from the Apple documentation for sed — which was originally quoted by Diego in an answer which he chose to delete, the -i option takes an argument which indicates the extension to use for backup copies.
-i extension
Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero-length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is exhausted, etc.
sed -i.bak '2,3d' filename.csv will delete lines 1 and 2 (assuming you have headers)
What this command does is edits the file being referenced while simultaneously creating a backup of the original file (hence the usage of .bak). The line deletion '2,3d' will delete the 2nd and 3rd line of the original file.
Related
I want to remove the first two characters of a column in a text file.
I am using the below but this is also truncating the headers.
sed -i 's/^..//' file1.txt
Below is my file:
FileName,Age
./Acct_Bal_Tgt.txt,7229
./IDQ_HB1.txt,5367
./IDQ_HB_LOGC.txt,5367
./IDQ_HB.txt,5367
./IGC_IDQ.txt,5448
./JobSchedule.txt,3851
I want the ./ to be removed from each line in the file name.
Transferring comments to an answer, as requested.
Modify your script to:
sed -e '2,$s/^..//' file1.txt
The 2,$ prefix limits the change to lines 2 to the end of the file, leaving line 1 unchanged.
An alternative is to remove . and / as the first two characters on a line:
sed -e 's%^[.]/%%' file1.txt
I tend to use -e to specify that the script option follows; it isn't necessary unless you split the script over several arguments (so it isn't necessary here where there's just one argument for the script). You could use \. instead of [.]; I'm allergic to backslashes (as you would be if you ever spent time working out whether you needed 8 or 16 consecutive backslashes to get the right result in a troff document).
Advice: Don't use the -i option until you've got your script working correctly. It overwrites your file with the incorrect output just as happily as it will with the correct output. Consequently, if you're asking about how to write a sed script on SO, it isn't safe to be using the -i option. Also note that the -i option is non-standard and behaves differently with different versions of sed (when it is supported at all). Specifically, on macOS, the BSD sed requires a suffix specified; if you don't want a backup, you have to use two arguments: -i ''.
Use this Perl one-liner:
perl -pe 's{^[.]/}{}' file1.txt > output.txt
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-p : Loop over the input one line at a time, assigning it to $_ by default. Add print $_ after each loop iteration.
s{^[.]/}{} : Replace a literal dot ([.]) followed by a slash ('/'), found at the beginning of the line (^), with nothing (delete them). This does not modify the header since it does not match the regex.
If you prefer to modify the file in-place, you can use this:
perl -i.bak -pe 's{^[.]/}{}' file1.txt
This creates the backup file file1.txt.bak.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches
perldoc perlrequick: Perl regular expressions quick start
I am on macOS High Sierra, and I've had a few issues with sed,
after spending a day on Google and this site, I've honestly tried everything I could think of and was suggested.
I have an example.txt file with 3 lines.
line1
line2
line3
The 3 lines can be anything, I do not know them upfront. (scenario)
And at some point I do know what the line is going to be.
All I wish to achieve is to use 'whatever' onliner that basically says:
in that file, replace line 2, with the new data.
sed -i "2s/.*/$NEW_DATA/" "$FILENAME"
This should work, but on macOS, this does not.
sed -ie "2s/.*/$NEW_DATA/" "$FILENAME"
Note the e? Someone on here suggested to add an e, and yes, that works.. but it means it adds the e behind the .txt. I end up with 2 files, example.txt and example.txte
sed -i'' "2s/.*/$NEW_DATA/" "$FILENAME"
Note the '' behind -i, and yes, without a space? This should work too.
It doesn't, it will complain that command c requires \
I could go on, but the request is quite simple:
on macOS, in a bash shell script, how do I simply replace a specified line of text in a .txt file, with a completely new line of text -- not knowing what the line of text was before?
If this can be a simple macOS one liner with awk or whatever, that's fine. It doesn't have to be sed. But when I search this site, it seems to be the recommended one to go with in this regards.
From the sed man page in macOS:
-i extension
Edit files in-place, saving backups with the specified extension.
If a zero-length extension is given, no backup will be saved.
Therefore this can be used to replace line 2 without keeping backup:
sed -i '' "2s/.*/$NEW_DATA/" testfile.txt
If the goal is just to replace contents of line 2 awk could also be used, for example:
awk 'NR==2{$0="your content"}1' testfile.txt > testfile.tmp && mv testfile.tmp testfile.txt
I have a lot of text files that are email templates. Many of them, for some reason, have the following line:
Best Regards,œ
That strange character at the end is what I am interested in removing from all of these files with a single command.
I tried:
for f in *
do
sed 's/"Best Regards,œ"/"Best Regards,"/g' $f | tee $f.t && mv $f.t $f
done
This ran through the process but did not actually remove the 'œ' character.
Please let me know what I am doing incorrectly so I can remove this character and maybe other non-alphanumeric characters using regex [:alnum:], perhaps.
I fixed the issue with removing the unwanted character with:
for f in *
do
sed 's/Best\ Regards\,\œ/Best\ Regards\,/g' $f | tee $f.t && mv $f.t $f
done
However, this still does not remove all of the non-alphanumeric characters from each line of each file. The other things I have tried either do not execute or remove the entire line.
I appreciate your help.
If ① you don't want to have to worry about Unicode, UTF-anything, LANG, etc, and ② you are confident that lines that start with the words "Best Regards," and ONLY those lines are the ones you want to affect, you can simply do this:
sed -i .bak '/^Best Regards,.*/s//Best Regards,/' *
Note that this processes all files in the current directory. If you want to do this in subdirectories, you could use find, with all its goodness. For example:
find /path/to/start/ -exec \
sed -i .bak '/^Best Regards,.*/s//Best Regards,/' {} \;
or if your shell is bash, you could use globstar:
shopt -s globstar
for f in **/*; do
sed -i .bak '/^Best Regards,.*/s//Best Regards,/' "$f"
done
Rather than using tee and mv, these solutions use sed's built-in "in-place" option, and creates a .bak file as a result. Consult the documentation for your implementation of sed to learn more about how to use the -i option -- it works a little differently with different seds.
This approach eliminates the need to search for that character in particular, so you won't need to worry about how it's being represented. Beware though, it will also eliminate any other text that follows the search string on the same line.
You don't need the loop. You can pass the results of the glob expression directly to sed and use the -i option for in place editing of files:
sed -i.bak 's/Best Regards,œ/Best Regards,/' *
-i.bak changes the input file in place and creates a backup file with the extension .bak.
Some implementations of sed, for example GNU sed even support -i without an argument other allow an empty string as argument for -i. In that case sed will not keep any backup files and simply change the original file.
With GNU sed:
sed -i 's/Best Regards,œ/Best Regards,/' *
# OR (BSD, MacOS)
sed -i '' 's/Best Regards,œ/Best Regards,/' *
I have file which contain following context like
abc...
include /home/user/file.txt'
some text
I need to remove include and also complete path after include.
I have used following command which remove include but did not remove path.
sed -i -r 's#include##g' 'filename'
I am also trying to understand above command but did not understand following thing ( copy paste from somewhere)
i - modify file change
r - read file
s- Need input
g - Need input
Try this,
$ sed '/^include /s/.*//g' file.txt
abc...
some text
It remove all the texts in a line which starts with include. s means substitute. so s/.*//g means replace all the texts with null.g means global. The substitution will be applied globally.
OR
$ sed '/^include /d' file.txt
abc...
some text
d means delete.
It deletes the line which starts with include. To save the changes made(inline edit), your commands should be
sed -i '/^include /s/.*//g' file.txt
sed -i '/^include /d' file.txt
I your case if you just want to delete the second line, you can use:
sed -i '2d' file
If you want to explore something about linux commands then man pages are there for you.
Just go to terminal and type:
man sed
as per your question, The above command without -i will show the file content on terminal by deleting the second line from the input file. However, the input file remains unchanged. To update the original file or to make the changes permanently in the source file, use the -i option.
-i[SUFFIX], --in-place[=SUFFIX] :
edit files in place (makes backup if extension supplied)
-r or --regexp-extended :
option is to use extended regular expressions in the script.
s/regexp/replacement/ :
Attempt to match regexp against the pattern space. If success‐
ful, replace that portion matched with replacement. The
replacement may contain the special character & to refer to that
portion of the pattern space which matched, and the special
escapes \1 through \9 to refer to the corresponding matching
sub-expressions in the regexp.
g G : Copy/append hold space to pattern space.
grep -v
This is not about learning sed, but as an alternative (and short) solution, there is:
grep -v '^include' filename_in
Or with output redirection:
grep -v '^include' filename_in > filename_out
-v option for grep inverts matching (hence printing non-matching lines).
For simple deletion that's what I'd use; if you have to modify your path after the include, stick with sed instead.
You can use awk to just delete the line:
awk '/^include/ {next}1' file
sed -i -r 's#include##g' 'filename'
-i: you directly modify the treated file, by default, sed read a file, modify the content via stdout (the original file stay the same).
-r: use of extended regular expression (and not reduce to POSIX limited one).This is not necessary in this case due to simple POSIX compliant action in action list (the s### string).
s#pattern#NewValue#: substitute in current line the pattern (Regular Expression) with "Newvalue" (that also use internal buffer or specific value). The traditionnal form is s/// but in this case, using / in path (pattern or new value) an alternate form is used to avoid to escape all / in pattern or new value
g: is an option of s### that specify change EVERY occurence and not the first (by default)
so here it replace ANY occurence of include by nothing (remove) directly into your file
As per the Avinash Raj solution you got what you want but you want some explaination about some parameter used in sed command
First one is
command: s for substitution
With the sed command the substitute command s changes all occurrences of the regular expression into a new value. A simple example is changing "my" in the "file1" to "yours" in the "file2" file:
sed s/my/yours/ file1 >file2
The character after the s is the delimiter. It is conventionally a slash, because this is what ed, more, and vi use. It can be anything you want, however. If you want to change a pathname that contains a slash - say /usr/local/bin to /common/bin - you could use the backslash to quote the slash:
sed 's/\/usr\/local\/bin/\/common\/bin/' <old >new
/g - Global replacement
Replace all matches, not just the first match.
If you tell it to change a word, it will only change the first occurrence of the word on a line. You may want to make the change on every word on the line instead of the first then add a g after the last delimiter and use the work-around:
Delete with d
Delete the pattern space; immediately start next cycle.
You can delete line by specifying the line number. like
sed '$d' filename.txt
It will remove last line of file
sed '2 d' file.txt
It will delete second line of file.
-i option
This option specifies that files are to be edited in-place. GNU sed does this by creating a temporary file and sending output to this file rather than to the standard output.
To modify file actully you can use -i option without it sed command repressent changes on stdout not actual file. You can take backup of original file before modification by using -i.bak option.
-r option
--regexp-extended
Use extended regular expressions rather than basic regular expressions. Extended regexps are those that egrep accepts; they can be clearer because they usually have less backslashes, but are a GNU extension and hence scripts that use them are not portable.
I have a series of text files that I want to convert to markdown. I want to remove any leading spaces and add a hash sign to the first line in every file. If I run this:
sed -i.bak '1s/ *\(.*\)/\#\1/g' *.md
It alters the first line of the first file and processes them all, leaving the rest of the files unchanged.
What am I missing that will search and replace something on the n-th line of multiple files?
Using bash on OSX 10.7
The problem is that sed by default treats any number of files as a single stream, and thus line-number offsets are relative to the start of the first file.
For GNU sed, you can use the -s (--separate) flag to modify this behavior:
sed -s -i.bak '1s/^ */#/' *.md
...or, with non-GNU sed (including the one on Mac OS X), you can loop over the files and invoke once per each:
for f in *.md; do sed -i.bak '1s/^ */#/' "$f"; done
Note that the regex is a bit simplified here -- no need to match parts of the line that you aren't going to change.
XARgs will do the trick for you:
http://en.wikipedia.org/wiki/Xargs
Remove the *.md from the end of your sed command, then use XArgs to gather your files one at a time and send them to your sed command as a single entity, sorry I don't have time to work it out for you but the wikiPedia article should show you what you need to know.
sed -rsi.bak '1s/^/#/;s/^[ \t]+//' *.md
You don't need g(lobally) at the end of the command(s), because you wan't to replace something at the begin of line, and not multiple times.
You use two commands, one to modify line 1 (1s...), seperated from the second command for the leading blanks (and tabs? :=\t) with a semicolon. To remove blanks in the first line, switch the order:
sed -rsi.bak 's/^[ \t]+//;1s/^/#/' *.md
Remove the \t if you don't need it. Then you don't need a group either:
sed -rsi.bak 's/^ +//;1s/^/#/' *.md
-r is a flag to signal special treatment of regular expressions. You don't need to mask the plus in that case.