I can remove the first line of csv file's starting with myfile and merge them using:
sed 1d myfile*.csv > myfile_merged.csv
I'd like to also remove the last line of the csv files.
I've tried:
sed 1d -i '$d' myfile*.csv > myfile_merged.csv
But get the error:
sed: can't read $d: No such file or directory
Problem is this command:
sed 1d -i '$d' myfile*.csv > myfile_merged.csv
You need not have an argument after -i (inline replacement) in sed otherwise it is treated as a SUFFIX to create a backup for inline replacement.
What you need is this gnu sed command:
sed -i '1d;$d' myfile*.csv
This will remove 1st and last line in each of the matched file and save it in place.
What you're probably trying to do is:
sed -e '1d' -e '$d' myfile*.csv > merged.csv
But this won't work, because it tells sed to remove the first and last line of ALL files, rather than EACH file. In other words, you'll strip the first line of the first file, and the last line of the last file ... and that's it.
To process each file individually, you probably need to process each file .. individually. :)
for f in myfile*.csv
sed -e '1d;$d' "$f"
done > merged.csv
Note that while this will run in bash, it's also POSIX compatible (both the shell and sed parts). And it does not care whether your input is CSV or any other format, as long as it can be parsed line by line using sed.
Related
How can I use a cat and sed to read data from a file and insert it into another file under known line?
For example I have a file named script1.txt that contains a few hundred lines, one of the line has the value "COMMANDS="commands"
If I wanted use sed to insert a line under it, simply I can use sed as the command bellow.
sed -i '/^COMMANDS=.*/a NEW LINE HERE' script1.txt
But if I want to insert a multi lines and these lines inside a file, and these line changes every a few hours.. how can i do that ?
I tried:
DATA=$(cat data.txt)
sed -i '/^COMMANDS=.*/a '$DATA'' script1.txt
I got the error bellow.
sed: -e expression #1, char 1: unknown command: `"'
Is there a way other than sed to insert the data from file under known line with no issues?
This might work for you (GNU sed):
sed -i '/^COMMANDS=/r dataFile' file
This will append the contents of the file dataFile after the line beginning COMMANDS= and update file
If the data you want to append is multi-line, you might want to replace newlines with \n.
#!/bin/sh
DATA="$(awk '{gsub(/[]\/$*.^&[]/, "\\\\&");printf (FNR>1)?"\\n%s":"%s",$0}END{print ""}' data.txt)"
sed -i -e '/^COMMANDS=.*/a\' -e "$DATA" script1.txt
Here the awk command escapes sed special characters (for basic regular expressions), then prints "%s" for the first line, and "\\n%s" for the others. A newline is printed at the end, but it's somewhat pointless as $() strips it anyway.
The sed command is almost the same but multiple expressions are used which is equivalent to a multi-line sed script (The a text sed alternative syntax can act weirdly with leading spaces/backslashes).
I've writted a sed script to replace all ^^ with NULL. It seems though that sed is only catching a pair, but not including the second in that pair as it continues to search.
echo "^^^^" | sed 's/\^\^/\^NULL\^/g'
produces
^NULL^^NULL^
when it should produce
^NULL^NULL^NULL^
Try with a loop to apply your command again to modified pattern space:
echo "^^^^" | sed ':a;s/\^\^/\^NULL\^/;t a;'
To edit a file in place on OSX, try the -i flag and multiline command:
sed -i '' ':a
s/\^\^/\^NULL\^/
t a' file
With GNU sed:
sed -i ':a;s/\^\^/\^NULL\^/;t a;' file
or simply redirect the command to a temporary file before renaming it:
sed ':a;s/\^\^/\^NULL\^/;t a;' file > tmp && mv tmp file
I really like SLePort solution, but since it is not working for you, you can try with (tested on Linux, not Mac):
echo "^^^^" | sed 's/\^\^/\^NULL\^/g; s//\^NULL\^/g'
It is doing the same as the former solution, but explicitly, not looping with tags.
You can omit the pattern in the second command and sed will use the previous pattern.
i am trying to remove some text from a file, i need to find the line number using a string and then remove from that line to the end of the file.
i tried using sed and outputting to a new file using a string as a search term but it does not keep the lines, it just prints the whole file on one line, which isnt helpful as the rest of the script relies on line numbers.
my code so far is:
file=$(sed -ie '/<div class="col-wrapper">/,$d' < file1.txt)
echo $file > file.txt
You can just use inline editing in sed:
sed -i '/<div class="col-wrapper">/,$d' file1.txt
to save changes to file1.txt
This might work for you (GNU sed):
sed -i '/<div class="col-wrapper">/Q' file1.txt
I need to delete the last 2 characters from a text file in shell script.
Any idea about how I can do it?
Delete the last two characters on the last line only with sed:
$ sed '$s/..$//' file
If you are happy with changes then use -i to store them back to the file:
$ sed -i '$s/..$//' file
If you wanted to delete the last two characters on every line it would be:
$ sed 's/..$//' file
Again use -i to store the changes back to the file:
$ sed -i 's/..$//' file
Suppose my file a.conf is as following
Include /1
Include /2
Include /3
I want to replace "Include /2" with a new line, I write the code in .sh file :
line="Include /2"
rep=""
sed -e "s/${line}/${rep}/g" /root/new_scripts/a.conf
But after running the sh file, It give me the following error
sed: -e expression #1, char 14: unknown option to `s'
If you are using a newer version of sed you can use -i to read from and write to the same file. Using -i you can specify a file extension so a backup will be made, incase something went wrong. Also you don't need to use the -e flag unless you are using multiple commands
sed -i.bak "s/${line}/${rep}/g" /root/new_scripts/a.conf
I have just noticed that as the variables you are using are quoted strings you may want to use single quotes around your sed expression. Also your string contains a forward slash, to avoid any errors you can use a different delimiter in your sed command (the delimiter doesn't need to be a slash):
sed -i.bak 's|${line}|${rep}|g' /root/new_scripts/a.conf
You have to write the changes to a new file and then, move the new file over the old one. Like this:
line="Include 2"
rep=""
sed -e "s/${line}/${rep}/g" /root/new_scripts/a.conf > /root/new_scripts/a.conf-new
mv /root/new_scripts/a.conf-new /root/new_scripts/a.conf
The redirection (> /root/new_scripts/a.conf) wipes the contents of the file before sed can see it.
You need to pass the -i option to sed to edit the file in-place:
sed -i "s/${line}/${rep}/g" /root/new_scripts/a.conf
You can also ask sed to create a backup of the original file:
sed -i.bak "s/${line}/${rep}/g" /root/new_scripts/a.conf
So, if you have to replace a substring in a file, you can use sed command like this, say we have a file as file.txt, so replacing a substring in it can be done like this
searchString="abc";
replaceString="def";
sed -i '' "s|$searchString|$replaceString|g" file.txt
This will all the occurrences of "abc" with "def" in file.txt. Also, this keeps a check for any / character present in the variables used, and with no backup file made.