The codeline below adds two different fixed-strings to beginning, and end of a line.
sed -i 's/.*/somefixedtext,&,someotherfixedtext/' ${f}
somefixedtext,20,30,10,50,someotherfixedtext
I want to enhance the output by using a variable, to give me below, instead of whats above;
HELLO,somefixedtext,20,30,10,50,someotherfixedtext
Tried some variations but all failed;
VARIABLE="HELLO"
sed -i 's/.*/"$VARIABLE,somefixedtext,&,someotherfixedtext/' ${f}
How can I incorporate a string variable to beginning of the line, combined with a fixedstring, in Sed ?
Sed is a poor choice if the replacement string is from a variable, because it treats the variable as a sed command and not as literal string, so it will break if the variable contains special characters like / or & or similar.
Awk is better suited for this task, for example like:
$ cat file
20,30,10,50
$ var=hello
$ awk -F, -v prefix="$var" -v OFS=, '{print prefix, "sometext", $0, "somethingelse"}' file
hello,sometext,20,30,10,50,somethingelse
For inplace modification you will need a recent version of GNU awk, and the -i inplace arguments.
For what it's worth, the command would appear to work if you double quoted the variable, outside the single quotes, but it would be a bit buggy:
VARIABLE="HELLO"
sed -i 's/.*/'"$VARIABLE"',somefixedtext,&,someotherfixedtext/' ${f}
Related
I have a file file.txt and it has the lines below. I want the queuename to be converted to uppercase, like this: queuename=SP00245B
# Queue name
#
queuename=sp00245b
awk '$1 == "queuename" {$2 = toupper($2)}1' FS== OFS== input-file
Note that this will fail if there are 2 = in the line, and only the values between the first 2 = will be uppercased. If that's an issue, it's an easy fix (left as an exercise for the reader).
A simple Perl solution:
perl -i -pe 's/^\s*queuename=\K(.*)/\U$1/' file.txt
(Remove -i if you don't want to modify the file in place.)
With GNU sed:
sed -i 's/\(^[[:blank:]]*queuename=\)\(.*\)/\1\U\2/' file.txt
This uses two captures groups and the \U sequence to toggle uppercase substitution for the second group.
You can also use the sed conversion \U to convert the portions of the matched pattern with the substitution command to uppercase. To covert everything following the '=' sign you could use, e.g.
sed '/^queuename=/s/=.*$/\U&/' filename
To edit the file in-place, include the -i option, e.g.
sed -i '/^queuename=/s/=.*$/\U&/' filename
Example Use/Output
$ echo "queuename=sp00245b" | sed '/^queuename=/s/=.*$/\U&/'
queuename=SP00245B
I have a shell script that accepts a parameter that is comma delimited,
-s 1234,1244,1567
That is passed to a curl PUT json field. Json needs the values in a "1234","1244","1567" format.
Currently, I am passing the parameter with the quotes already in it:
-s "\"1234\",\"1244\",\"1567\"", which works, but the users are complaining that its too much typing and hard to do. So I'd like to just take a comma delimited list like I had at the top and programmatically stick the quotes in.
Basically, I want a parameter to be passed in as 1234,2345 and end up as a variable that is "1234","2345"
I've come to read that easiest approach here is to use sed, but I'm really not familiar with it and all of my efforts are failing.
You can do this in BASH:
$> arg='1234,1244,1567'
$> echo "\"${arg//,/\",\"}\""
"1234","1244","1567"
awk to the rescue!
$ awk -F, -v OFS='","' -v q='"' '{$1=$1; print q $0 q}' <<< "1234,1244,1567"
"1234","1244","1567"
or shorter with sed
$ sed -r 's/[^,]+/"&"/g' <<< "1234,1244,1567"
"1234","1244","1567"
translating this back to awk
$ awk '{print gensub(/([^,]+)/,"\"\\1\"","g")}' <<< "1234,1244,1567"
"1234","1244","1567"
you can use this:
echo QV=$(echo 1234,2345,56788 | sed -e 's/^/"/' -e 's/$/"/' -e 's/,/","/g')
result:
echo $QV
"1234","2345","56788"
just add double quotes at start, end, and replace commas with quote/comma/quote globally.
easy to do with sed
$ echo '1234,1244,1567' | sed 's/[0-9]*/"\0"/g'
"1234","1244","1567"
[0-9]* zero more consecutive digits, since * is greedy it will try to match as many as possible
"\0" double quote the matched pattern, entire match is by default saved in \0
g global flag, to replace all such patterns
In case, \0 isn't recognized in some sed versions, use & instead:
$ echo '1234,1244,1567' | sed 's/[0-9]*/"&"/g'
"1234","1244","1567"
Similar solution with perl
$ echo '1234,1244,1567' | perl -pe 's/\d+/"$&"/g'
"1234","1244","1567"
Note: Using * instead of + with perl will give
$ echo '1234,1244,1567' | perl -pe 's/\d*/"$&"/g'
"1234""","1244""","1567"""
""$
I think this difference between sed and perl is similar to this question: GNU sed, ^ and $ with | when first/last character matches
Using sed:
$ echo 1234,1244,1567 | sed 's/\([0-9]\+\)/\"\1\"/g'
"1234","1244","1567"
ie. replace all strings of numbers with the same strings of numbers quoted using backreferencing (\1).
My example text is,
AA BB CC
DDD
process.get('name1')
process.get('name2')
process.get('name3')
process.get('name4')
process.get('name5')
process.get('name6')
EEE
FFF
...
I want to search the string "process.get('name1')" first, if found then extract the lines from "process.get('name1')" to "process.get('name6')".
How do I extract the lines using sed?
This should work and... it uses sed as per OP request:
$ sed -n "/^process\.get('name1')$/,/^process\.get('name6')$/p" file
sed is for simple substitutions on individual lines, for anything more interesting you should be using awk:
$ awk -v beg="process.get('name1')" -v end="process.get('name6')" \
'index($0,beg){f=1} f; index($0,end){f=0}' file
process.get('name1')
process.get('name2')
process.get('name3')
process.get('name4')
process.get('name5')
process.get('name6')
Note that you could use a range in awk, just like you are forced to in sed:
awk -v beg="process.get('name1')" -v end="process.get('name6')" \
'index($0,beg),index($0,end)' file
and you could use regexps after escaping metachars in awk, just like you are forced to in sed:
awk "/process\.get\('name1'\)/,/process\.get\('name6'\)/" file
but the first awk version above using strings instead of regexps and a flag variable is simpler (in as much as you don't have to figure out which chars are/aren't RE metacharacters), more robust and more easily extensible in future.
It's important to note that sed CANNOT operate on strings, just regexps, so when you say "I want to search for a string" you should stop trying to force sed to behave as if it can do that.
Imagine your search strings are passed in to a script as positional parameters $1 and $2. With awk you'd just init the awk variables from them in the expected way:
awk -v beg="$1" -v end="$2" 'index($0,beg){f=1} f; index($0,end){f=0}' file
whereas with sed you'd have to do something like:
beg=$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<< "$1")
end=$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<< "$2")
sed -n "/^${beg}$/,/^${end}$/p" file
to deactivate any metacharacters present. See Is it possible to escape regex metacharacters reliably with sed for details on escaping RE metachars for sed.
Finally - as mentioned above you COULD use a range expression with strings in awk:
awk -v beg="$1" -v end="$2" 'index($0,beg),index($0,end)' file
but I personally have never found that useful, there's always some slight requirements change comes along to make me wish I'd started out using a flag. See Is a /start/,/end/ range expression ever useful in awk? for details on that
I have a bash script which checks for a string pattern in file and delete entire line i same file but somehow its not deleting the line and no throwing any error .same command from command prompt deletes from file .
#array has patterns
for k in "${patternarr[#]}
do
sed -i '/$k/d' file.txt
done
sed version is >4
when this loop completes i want all lines matching string pattern in array to be deleted from file.txt
when i run sed -i '/pataern/d file.txt from command prompt then it works fine but not inside bash
Thanks in advance
Here:
sed -i '/$k/d' file.txt
The sed script is singly-quoted, which prevents shell variable expansion. It will (probably) work with
sed -i "/$k/d" file.txt
I say "probably" because what it will do depends on the contents of $k, which is just substituted into the sed code and interpreted as such. If $k contains slashes, it will break. If it comes from an untrustworthy source, you open yourself up to code injection (particularly with GNU sed, which can be made to execute shell commands).
Consider k=^/ s/^/rm -Rf \//e; #.
It is generally a bad idea to substitute shell variables into sed code (or any other code). A better way would be with GNU awk:
awk -i inplace -v pattern="$k" '!($0 ~ pattern)' file.txt
Or to just use grep -v and a temporary file.
first of all, you got an unclosed double quote around ${patternarr[#]} in your for statement.
Then your problem is that you use single quotes in the sed argument, making your shell not evaluate the $k within the quotes:
% declare -a patternarr=(foo bar fu foobar)
% for k in ${patternarr[#]}; do echo sed -i '/$k/d' file.txt; done
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
if you replace them with double quotes, here it goes:
% for k in ${patternarr[#]}; do echo sed -i "/$k/d" file.txt; done
sed -i /foo/d file.txt
sed -i /bar/d file.txt
sed -i /fu/d file.txt
sed -i /foobar/d file.txt
Any time you write a loop in shell just to manipulate text you have the wrong approach. This is probably closer to what you really should be doing (no surrounding loop required):
awk -v ks="${patternarr[#]}" 'BEGIN{gsub(/ /,")|(",ks); ks="("ks")} $0 !~ ks' file.txt
but there may be even better approaches still (e.g. only checking 1 field instead of the whole line, or using word boundaries, or string comparison or....) if you show us some sample input and expected output.
You need to use double quotes to interpolate shell variables inside the sed command, like:
for k in ${patternarr[#]}; do
sed -i "/$k/d" file.txt
done
Basically I want to copy several lines of code from a template file to a script file.
Is it even possible to use sed to copy a string full of symbols that interact with the script?
I used these lines:
$SWAP='sudo cat /home/kaarel/template'
sed -i -e "s/#pointer/${SWAP}/" "script.sh"
The output is:
./line-adder.sh: line 11: =sudo cat /home/kaarel/template: No such file or directory
No, it is not possible to do this robustly with sed. Just use awk:
awk -v swap="$SWAP" '{sub(/#pointer/,swap)}1' script.sh > tmp && mv tmp script.sh
With recent versions of GNU awk there's a -i inplace flag for inplace editing if that's something you care about.
Good point about "&&". Here's the REALLY robust version that will work for absolutely any character in the search or replacement strings:
awk -v old="#pointer" -v new="$SWAP" 's=index($0,old){$0 = substr($0,1,s-1) new substr($0,s+length(old))} 1'
e.g.:
$ echo "abc" | awk -v old="b" -v new="m&&n" 's=index($0,old){$0 = substr($0,1,s-1) new substr($0,s+length(old))} 1'
am&&nc
There are two issues with the line:
$SWAP='sudo cat /home/kaarel/template'
The first is that, before executing the line, bash performs variable expansion and replaces $SWAP with the current value of SWAP. That is not what you wanted. You wanted bash to assign a value to SWAP.
The second issue is that the right-hand side is enclosed in single-quotes which protect the string from expansion. You didn't want to protect the string from expansion: you wanted to execute it. To execute it, you can use back-quotes which may look similar but act very differently.
Back-quotes, however, are an ancient form of asking for command execution. The more modern form is $(...) which eliminates some problems that back-quotes had.
Putting it all together, use:
SWAP=$(sudo cat /home/kaarel/template)
sed -i -e "s/#pointer/${SWAP}/" "script.sh"
Be aware, though, that the sed command may have problems if there are any sed-active characters in the template file.