This question already has answers here:
Using variable inside of sed -i (regex?) bash
(2 answers)
Why wont sed remove line from file?
(4 answers)
Closed 8 years ago.
I am trying to remove lines from a text file from a Bash Script using command sed.
Here is how this function works.
User enters record number
Program Searches for record number
Program deletes record
Here is my code:
r=$(grep -h "$record" student_records.txt|cut -d"," -f1) #find the record that needs to be deleted
echo $line -->> This proves that previous command works
sed -i '/^$r/d' student_records.txt -->> this does not work
Any ideas?
To remove a line containing $record from the file:
grep -v "$record" student_records.txt >delete.me && mv -f delete.me student_records.txt
In the above, $record is treated as a regular expression. This means, for example, that a period is a wildcard. If that is unwanted, add the -F option to grep to specify that $record is to be treated as a fixed string.
Comments
Consider these two line:
r=$(grep -h "$record" student_records.txt|cut -d"," -f1) #find the record that needs to be deleted
echo $line -->> This proves that previous command works
The first line defines a shell variable r. The second line prints the shell variable line, a variable which was unaffected by the previous command. Consequently, the second line is not a successful test on the first.
sed -i '/^$r/d' student_records.txt -->> this does not work
Obseve that the expression $r appears inside single-quotes. The shell does not alter any inside single quotes. Consequently, $r will remain a dollar sign followed by an r. Since a dollar sign matches the end of a line, this expression will match nothing. The following would work better:
sed -i "/^$r/d" student_records.txt
Unlike the grep command, however, the above sed command is potentially dangerous. It would be easy to construct a value of r that would cause sed to do surprising things. So, don't use this approach unless you trust the process by which you obtained r.
What if more than one line matches record?
If there is more than one line that matches record, then the following would generate an unterminated address regex error from sed:
r=$(grep -h "$record" student_records.txt|cut -d"," -f1)
sed -i "/^$r/d" student_records.txt
This error is an example of the surprising results that can happen when a shell variable is expanded into a sed command.
By contrast, this approach would remove all matching lines:
grep -v "$record" student_records.txt >delete.me && mv -f delete.me student_records.txt
Related
I am grabbing the contents of https://api.wordpress.org/secret-key/1.1/salt/ using curl and asigning it to a bash variable. I want to insert this block of text into a file at a specific line number - 10
#!/bin/bash
salts=$(curl -s https://api.wordpress.org/secret-key/1.1/salt/)
sed -i '10i '"$salts"'' myfile.php
however I keep getting the error
sed: -e expression #1, char 102: extra characters after command
I think there maybe carriage return chars in the returned payload using curl but i'm not sure. I have tried using tr to replace them with \n but am unsure how to use it in this situation.
I have looked at multiple existing questions but I cannot get them to work in my situation. I don't have to use sed for this.
using awk
awk -v "s=$salts" 'NR==10{print s} 1' file
you could also combine head and tail
echo -e "$( head -9 file )\n$salts\n$( tail -n +10 file )"
boh of them seem to work with your variable that contains a lot of special characters.
I am aware that the "deleting n rows" and "deleting column x" questions have both been answered individually before. My current problem is that I'm writing my first bash script, and am having trouble making that script work the way I want it to.
file0001.csv (there are several hundred files like these in one folder)
Data number of lines 540
No.,Profile,Unit
1,1027.84,µm
2,1027.92,µm
3,1028,µm
4,1028.81,µm
Desired output
1,1027.84
2,1027.92
3,1028
4,1028.81
I am able to use sed and cut individually but for some reason the following bash script doesn't take cut into account. It also gives me an error "sed: can't read ls: No such file or directory", yet sed is successful and the output is saved to the original files.
sem2csv.sh
for files in 'ls *.csv' #list of all .csv files
do
sed '1,2d' -i $files | cut -f '1-2' -d ','
done
Actual output:
1,1027.84,µm
2,1027.92,µm
3,1028,µm
4,1028.81,µm
I know there may be awk one-liners but I would really like to understand why this particular bash script isn't running as intended. What am I missing?
The -i option of sed modifies the file in place. Your pipeline to cut receives no input because sed -i produces no output. Without this option, sed would write the results to standard output, instead of back to the file, and then your pipeline would work; but then you would have to take care of writing the results back to the original file yourself.
Moreover, single quotes inhibit expansion -- you are "looping" over the single literal string ls *.csv. The fact that you are not quoting it properly then causes the string to be subject to wildcard expansion inside the loop. So after variable interpolation, your sed command expands to
sed -i 1,2d ls *.csv
and then the shell expands *.csv because it is not quoted. (You should have been receiving a warning that there is no file named ls in the current directory, too.) You probably attempted to copy an example which used backticks (ASCII 96) instead of single quotes (ASCII 39) -- the difference is quite significant.
Anyway, the ls is useless -- the proper idiom is
for files in *.csv; do
sed '1,2d' "$files" ... # the double quotes here are important
done
Mixing sed and cut is usually not a good idea because you can express anything cut can do in terms of a simple sed script. So your entire script could be
for f in *.csv; do
sed -i -e '1,2d' -e 's/,[^,]*$//' "$f"
done
which says to remove the last comma and everything after it. (If your sed does not like multiple -e options, try with a semicolon separator: sed -i '1,2d;s/,[^,]*$//' "$f")
You may use awk,
$ awk 'NR>2{sub(/,[^,]*$/,"",$0);print}' file
1,1027.84
2,1027.92
3,1028
4,1028.81
or
sed -i '1,2d;s/,[^,]*$//' file
1,2d; for deleting the first two lines.
s/,[^,]*$// removes the last comma part in remaining lines.
If I want to add content to the 10th line of a file, how do I do it?
This is what I came up with:
sed -ie '' "10s/^/new_content/g" file.txt
I keep getting the message "no such file or directory"
Also, if I want to replace 10 with N+1 and the new_content with a variable $VAR, would the format still be the same?
VAR= $(cat fileA.txt)
sed -ie '' "`expr $N +1`s/^/$VAR/g" fileB.txt
Thanks for your help!!!
Shell is fussy about spacing:
VAR= $(cat fileA.txt)
You do not want the space after the =. Actually, you don't want the assignment either, unless fileA.txt has only one line. You'd have to escape newlines with backslashes, etc, to get it to work, which is both hard and unnecessary.
Unless you're using an antique (non-POSIX) shell, use built-in arithmetic:
sed -i.bak -e "$(($N + 1))r fileA.txt" fileB.txt
The $(($N + 1)) has the shell evaluate the arithmetic expression. The r has sed read the file that is named.
[1addr]r file
Copy the contents of file to the standard output immediately before the next attempt to read a line of input. If file cannot be read for any reason, it is silently ignored and no error condition is set.
It turns out I needed to split -i and -e.
The final result that works was:
sed -i '' -e "10s/^/CONTENT/g" file.txt
Or to increase Ns by +1 each time it loops:
sed -i '' -e "$((N + 1))s/^/CONTENT/g" ~/dance/heatlist/prep.txt
I still have not figured out how to make CONTENT a variable $CONTENT
I'm trying to find a way using variables and sed to do a specific text substitution using a changing input file, but only if there is a value given to replace the existing string with. No value= do nothing (rather than remove the existing string).
Example
Substitute.csv contains 5 lines=
this-has-text
this-has-text
this-has-text
this-has-text
and file.text has one sentence=
"When trying this I want to be sure that text-this-has is left alone."
If I run the following command in a shell script
Text='text-this-has'
Change=`sed -n '3p' substitute.csv`
grep -rl $Text /home/username/file.txt | xargs sed -i "s|$Text|$Change|"
I end up with
"When trying this I want to be sure that is left alone."
But I'd like it to remain as
"When trying this I want to be sure that text-this-has is left alone."
Any way to tell sed "If I give you nothing new, do nothing"?
I apologize for the overthinking, bad habit. Essentially what I'd like to accomplish is if line 3 of the csv file has a value - replace $Text with $Change inline. If the line is empty, leave $Text as $Text.
Text='text-this-has'
Change=$(sed -n '3p' substitute.csv)
if [[ -n $Change ]]; then
grep -rl $Text /home/username/file.txt | xargs sed -i "s|$Text|$Change|"
fi
Just keep it simple and use awk:
awk -v t="$Text" -v c="$Change" 'c!=""{sub(t,c)} {print}' file
If you need inplace editing just use GNU awk with -i inplace.
Given your clarified requirement, this is probably what you actually want:
awk -v t="$Text" 'NR==FNR{if (NR==3) c=$0; next} c!=""{sub(t,c)} {print}' Substitute.csv file.txt
Testing whether $Change has a value before launching into the grep and sed is undoubtedly the most efficient bash solution, although I'm a bit skeptical about the duplication of grep and sed; it saves a temporary file in the case of files which don't contain the target string, but at the cost of an extra scan up to the match in the case of files which do contain it.
If you're looking for typing efficiency, though, the following might be interesting:
find . -name '*.txt' -exec sed -i "s|$Text|${Change:-&}|" {} \;
Which will recursively find all files whose names end with the extension .txt and execute the sed command on each one. ${Change:-&} means "the value of $Change if it exists and is non-empty, and otherwise an &"; & in the replacement of a sed s command means "the matched text", so s|foo|&| replaces every occurrence of foo with itself. That's an expensive no-op but if your time matters more than your cpu time, it might have been worth it.
I have redirected some string into one parameter for ex: ab=jyoti,priya, pranit
I have one file : Name.txt which contains -
jyoti
prathmesh
John
Kelvin
pranit
I want to delete the records from the Name.txt file which are contain in ab parameter.
Please suggest if this can be done ?
If ab is a shell variable, you can easily turn it into an extended regular expression, and use it with grep -E:
grep -E -x -v "${ab//,/|}" Name.txt
The string substitution ${ab//,/|} returns the value of $ab with every , substituted with a | which turns it into an extended regular expression, suitable for passing as an argument to grep -E.
The -v option says to remove matching lines.
The -x option specifies that the match needs to cover the whole input line, so that a short substring will not cause an entire longer line to be removed. Without it, ab=prat would cause pratmesh to be removed.
If you really require a sed solution, the transformation should be fairly trivial. grep -E -v -x 'aaa|bbb|ccc' is equivalent to sed '/^\(aaa\|bbb\|ccc)$/d' (with some dialects disliking the backslashes, and others requiring them).
To do an in-place edit (modify Name.txt without a temporary file), try this:
sed -i "/^\(${ab//,/\|}\)\$/d" Name.txt
This is not entirely robust against strings containing whitespace or other shell metacharacters, but if you just need
Try with
sed -e 's/\bjyoti\b//g;s/\bpriya\b//g' < Name.txt
(using \b assuming you need word boundaries)
this will do it:
for param in `echo $ab | sed -e 's/[ ]+//g' -e 's/,/ /g'` ; do res=`sed -e "s/$param//g" < name.txt`; echo $res > name.txt; done
echo $res