Basically I want to copy several lines of code from a template file to a script file.
Is it even possible to use sed to copy a string full of symbols that interact with the script?
I used these lines:
$SWAP='sudo cat /home/kaarel/template'
sed -i -e "s/#pointer/${SWAP}/" "script.sh"
The output is:
./line-adder.sh: line 11: =sudo cat /home/kaarel/template: No such file or directory
No, it is not possible to do this robustly with sed. Just use awk:
awk -v swap="$SWAP" '{sub(/#pointer/,swap)}1' script.sh > tmp && mv tmp script.sh
With recent versions of GNU awk there's a -i inplace flag for inplace editing if that's something you care about.
Good point about "&&". Here's the REALLY robust version that will work for absolutely any character in the search or replacement strings:
awk -v old="#pointer" -v new="$SWAP" 's=index($0,old){$0 = substr($0,1,s-1) new substr($0,s+length(old))} 1'
e.g.:
$ echo "abc" | awk -v old="b" -v new="m&&n" 's=index($0,old){$0 = substr($0,1,s-1) new substr($0,s+length(old))} 1'
am&&nc
There are two issues with the line:
$SWAP='sudo cat /home/kaarel/template'
The first is that, before executing the line, bash performs variable expansion and replaces $SWAP with the current value of SWAP. That is not what you wanted. You wanted bash to assign a value to SWAP.
The second issue is that the right-hand side is enclosed in single-quotes which protect the string from expansion. You didn't want to protect the string from expansion: you wanted to execute it. To execute it, you can use back-quotes which may look similar but act very differently.
Back-quotes, however, are an ancient form of asking for command execution. The more modern form is $(...) which eliminates some problems that back-quotes had.
Putting it all together, use:
SWAP=$(sudo cat /home/kaarel/template)
sed -i -e "s/#pointer/${SWAP}/" "script.sh"
Be aware, though, that the sed command may have problems if there are any sed-active characters in the template file.
Related
The codeline below adds two different fixed-strings to beginning, and end of a line.
sed -i 's/.*/somefixedtext,&,someotherfixedtext/' ${f}
somefixedtext,20,30,10,50,someotherfixedtext
I want to enhance the output by using a variable, to give me below, instead of whats above;
HELLO,somefixedtext,20,30,10,50,someotherfixedtext
Tried some variations but all failed;
VARIABLE="HELLO"
sed -i 's/.*/"$VARIABLE,somefixedtext,&,someotherfixedtext/' ${f}
How can I incorporate a string variable to beginning of the line, combined with a fixedstring, in Sed ?
Sed is a poor choice if the replacement string is from a variable, because it treats the variable as a sed command and not as literal string, so it will break if the variable contains special characters like / or & or similar.
Awk is better suited for this task, for example like:
$ cat file
20,30,10,50
$ var=hello
$ awk -F, -v prefix="$var" -v OFS=, '{print prefix, "sometext", $0, "somethingelse"}' file
hello,sometext,20,30,10,50,somethingelse
For inplace modification you will need a recent version of GNU awk, and the -i inplace arguments.
For what it's worth, the command would appear to work if you double quoted the variable, outside the single quotes, but it would be a bit buggy:
VARIABLE="HELLO"
sed -i 's/.*/'"$VARIABLE"',somefixedtext,&,someotherfixedtext/' ${f}
I have a folder with about 2 million files in it. I need to run the following commands:
sed -i 's/<title>/<item><title>/g;s/rel="nofollow"//g;s/<\/a> •/]]><\/wp:meta_value><\/wp:postmeta><content:encoded><![CDATA[/g;s/By <a href="http:\/\/www.website.com\/authors.*itemprop="author">/<wp:postmeta><wp:meta_key><![CDATA[custom_author]]><\/wp:meta_key><wp:meta_value><![CDATA[/g' /home/testing/*
sed -i '$a]]></content:encoded><wp:status><![CDATA[draft]]></wp:status><wp:post_type><![CDATA[post]]></wp:post_type><dc:creator><![CDATA[Database]]></dc:creator></item>\' /home/testing/*
awk -i inplace 1 ORS=' ' /home/testing/*
The problem I'm having is that when I run the first command, it cycles through all 2 million files, then I move on to the second command and so on. The problem is that I'm basically having to open files 6 million times in total.
I'd prefer that when each file is opened, all 3 commands are run on it and then it moves on to the next. Hopefully that makes sense.
You can do everything in one awk command as something like:
awk -i inplace -v ORS=' ' '{
gsub(/<title>/,"<item><title>")
gsub(/rel="nofollow"/,"")
gsub(/<\/a> •/,"]]><\/wp:meta_value><\/wp:postmeta><content:encoded><![CDATA[")
gsub(/By <a href="http:\/\/www.website.com\/authors.*itemprop="author">/,"<wp:postmeta><wp:meta_key><![CDATA[custom_author]]><\/wp:meta_key><wp:meta_value><![CDATA[")
print $0 "]]></content:encoded><wp:status><![CDATA[draft]]></wp:status><wp:post_type><![CDATA[post]]></wp:post_type><dc:creator><![CDATA[Database]]></dc:creator></item>"
}' /home/testing/*
but that doesn't mean it's necessarily the best way to do what you want.
The above relies on my correctly interpreting what your commands are doing and is obviously untested since you didn't provide any sample input and expected output. It also still relies on GNU awk for -i inplace like your original script did.
Assuming that your files are small enough for a single file to fit into memory as a whole (and assuming GNU sed, which your use of -i without an option-argument implies):
sed -i -e ':a;$!{N;ba}; s/.../.../g; ...; $a...' -e 's/\n/ /g' /home/testing/*
s/.../.../g; ...; and $a... in the command above represent your actual substitution and append commands.
:a;$!{N;ba}; reads each input file as a whole, and then performs the desired substitutions, appending, and replacement of all newlines with a single space each.[1]
This allows you to make do with a single sed command per input file.
[1] Your awk 1 ORS=' ' command actually creates output with a trailing space instead of a newline. By contrast, 's/\n/ /g' applied to the whole input file will only place a space between lines, and terminate the overall file with a newline (assuming the input file ended in one).
I am aware that the "deleting n rows" and "deleting column x" questions have both been answered individually before. My current problem is that I'm writing my first bash script, and am having trouble making that script work the way I want it to.
file0001.csv (there are several hundred files like these in one folder)
Data number of lines 540
No.,Profile,Unit
1,1027.84,µm
2,1027.92,µm
3,1028,µm
4,1028.81,µm
Desired output
1,1027.84
2,1027.92
3,1028
4,1028.81
I am able to use sed and cut individually but for some reason the following bash script doesn't take cut into account. It also gives me an error "sed: can't read ls: No such file or directory", yet sed is successful and the output is saved to the original files.
sem2csv.sh
for files in 'ls *.csv' #list of all .csv files
do
sed '1,2d' -i $files | cut -f '1-2' -d ','
done
Actual output:
1,1027.84,µm
2,1027.92,µm
3,1028,µm
4,1028.81,µm
I know there may be awk one-liners but I would really like to understand why this particular bash script isn't running as intended. What am I missing?
The -i option of sed modifies the file in place. Your pipeline to cut receives no input because sed -i produces no output. Without this option, sed would write the results to standard output, instead of back to the file, and then your pipeline would work; but then you would have to take care of writing the results back to the original file yourself.
Moreover, single quotes inhibit expansion -- you are "looping" over the single literal string ls *.csv. The fact that you are not quoting it properly then causes the string to be subject to wildcard expansion inside the loop. So after variable interpolation, your sed command expands to
sed -i 1,2d ls *.csv
and then the shell expands *.csv because it is not quoted. (You should have been receiving a warning that there is no file named ls in the current directory, too.) You probably attempted to copy an example which used backticks (ASCII 96) instead of single quotes (ASCII 39) -- the difference is quite significant.
Anyway, the ls is useless -- the proper idiom is
for files in *.csv; do
sed '1,2d' "$files" ... # the double quotes here are important
done
Mixing sed and cut is usually not a good idea because you can express anything cut can do in terms of a simple sed script. So your entire script could be
for f in *.csv; do
sed -i -e '1,2d' -e 's/,[^,]*$//' "$f"
done
which says to remove the last comma and everything after it. (If your sed does not like multiple -e options, try with a semicolon separator: sed -i '1,2d;s/,[^,]*$//' "$f")
You may use awk,
$ awk 'NR>2{sub(/,[^,]*$/,"",$0);print}' file
1,1027.84
2,1027.92
3,1028
4,1028.81
or
sed -i '1,2d;s/,[^,]*$//' file
1,2d; for deleting the first two lines.
s/,[^,]*$// removes the last comma part in remaining lines.
I have file called "text_file1.txt" and the content in the file is
"subject= /C=US/O=AAA/OU=QA/OU=12345/OU=TESTAPP/"
Now what i want to achieve is to the content to be like below:
"subject= /C=US/O=AAA/$$$QA/###12345/###TESTAPP/"
when i execute the below piece of code:
#! /bin/ksh
OU1="QA"
OU2=12345
OU3="TESTAPP"
`sed -i "s/OU=$OU1/$$$\${OU1}/g" text_file1.txt`
`sed -i "s/OU=$OU2/###\${OU2}/g" text_file1.txt`
`sed -i "s/OU=$OU3/###\${OU3}/g" text_file1.txt`
content=`cat text_file1.txt`
echo "content:$content"
i get the output like this:
content:subject= /C=US/O=Wells Fargo/2865528655{OU1}/###12345/###TESTAPP/CN=03032015_CUST_2131_Unix_CLBLABB34C02.wellsfargo.com
only this command "sed -i "s/OU=$OU1/$$$\${OU1}/g" text_file1.txt" is not working as expected.Can anyone please suggest some idea on this?
Thanks in advance.
Two things play into this:
You have to escape $ (i.e., use \$) in doubly-quoted shell strings if you want a literal $, and
\ does not retain its literal meaning when it comes before a $ inside backticks (that is to say, inside backticks, \$ becomes just $).
When you write
`sed -i "s/OU=$OU1/$$$\${OU1}/g" text_file1.txt`
because the command is in backticks, you spawn a subshell with the command
sed -i "s/OU=$OU1/$$$${OU1}/g" text_file1.txt
Since $$$$ is inside a doubly-quoted string, variable expansion takes place, and it is expanded as two occurrences of $$ (the process ID of the shell that's doing the expansion). This means that the code sed sees is ultimately
s/OU=QA/1234512345{OU1}/g
...if the process ID of the spawned subshell is 12345.
In this particular case, you don't need the command substitution (the backticks), so you could write
sed -i "s/OU=$OU1/\$\$\$${OU1}/g" text_file1.txt
However, using shell variables in sed code is always a problem. Consider, if you will, what would happen if OU1 had the value /; e rm -Rf * # (hint: GNU sed has an e instruction that runs shell commands). For this reason, I would always prefer awk to do substitutions that involve shell variables:
cp text_file1.txt text_file1.txt~
awk -v OU1="$OU1" '{ gsub("OU=" OU1, "$$$" OU1) } 1' text_file1.txt~ > text_file1.txt
This avoids code injection problems by not treating OU1 as code.
If you have GNU awk 4.1 or later,
awk -v OU1="$OU1" -i inplace '{ gsub("OU=" OU1, "$$$" OU1) } 1' text_file1.txt
can do the whole thing without a (visible) temporary file.
Does this help as a start?
echo ''
OU1="QA"
echo "subject= /C=US/O=AAA/OU=${OU1}/OU=12345/OU=TESTAPP/" \
| sed -e "s|/OU=${OU1}/|/OU=\$\$\$${OU1}/|g"
The result is:
subject= /C=US/O=AAA/OU=$$$QA/OU=12345/OU=TESTAPP/
(You are mixing up the use of $ signs .)
You must be careful when putting $ inside double quotes.
sed -i "s/OU=$OU1/"'$$$'"${OU1}/g" text_file1.txt
Example:
$ OU1="QA"
$ echo 'OU=QA' | sed "s/OU=$OU1/"'$$$'"${OU1}/g"
$$$QA
I have a bash script which checks for a string pattern in file and delete entire line i same file but somehow its not deleting the line and no throwing any error .same command from command prompt deletes from file .
#array has patterns
for k in "${patternarr[#]}
do
sed -i '/$k/d' file.txt
done
sed version is >4
when this loop completes i want all lines matching string pattern in array to be deleted from file.txt
when i run sed -i '/pataern/d file.txt from command prompt then it works fine but not inside bash
Thanks in advance
Here:
sed -i '/$k/d' file.txt
The sed script is singly-quoted, which prevents shell variable expansion. It will (probably) work with
sed -i "/$k/d" file.txt
I say "probably" because what it will do depends on the contents of $k, which is just substituted into the sed code and interpreted as such. If $k contains slashes, it will break. If it comes from an untrustworthy source, you open yourself up to code injection (particularly with GNU sed, which can be made to execute shell commands).
Consider k=^/ s/^/rm -Rf \//e; #.
It is generally a bad idea to substitute shell variables into sed code (or any other code). A better way would be with GNU awk:
awk -i inplace -v pattern="$k" '!($0 ~ pattern)' file.txt
Or to just use grep -v and a temporary file.
first of all, you got an unclosed double quote around ${patternarr[#]} in your for statement.
Then your problem is that you use single quotes in the sed argument, making your shell not evaluate the $k within the quotes:
% declare -a patternarr=(foo bar fu foobar)
% for k in ${patternarr[#]}; do echo sed -i '/$k/d' file.txt; done
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
if you replace them with double quotes, here it goes:
% for k in ${patternarr[#]}; do echo sed -i "/$k/d" file.txt; done
sed -i /foo/d file.txt
sed -i /bar/d file.txt
sed -i /fu/d file.txt
sed -i /foobar/d file.txt
Any time you write a loop in shell just to manipulate text you have the wrong approach. This is probably closer to what you really should be doing (no surrounding loop required):
awk -v ks="${patternarr[#]}" 'BEGIN{gsub(/ /,")|(",ks); ks="("ks")} $0 !~ ks' file.txt
but there may be even better approaches still (e.g. only checking 1 field instead of the whole line, or using word boundaries, or string comparison or....) if you show us some sample input and expected output.
You need to use double quotes to interpolate shell variables inside the sed command, like:
for k in ${patternarr[#]}; do
sed -i "/$k/d" file.txt
done