I have a list of substitutions that I would like to perform with sed. Instead of combining the substitutions into a single sed command, I would like to perform each substitution in an iteration of a bash loop. For example:
cat ${input} |
for subst in "${substlist}"; do
sed 's/'${subst}'/modified_'${subst}'/g'
done > ${output}
I would expect that each iteration modifies the entire stream but I'm only seeing that the first iteration gets the input.
Is this pattern possible in bash?
Create an array of -e options to pass to sed.
filters=()
for subst in ${substlist}; do
filters+=(-e "s/$subst/modified_$subst/")
done
sed "${filters[#]}" "$input" > "$output"
(The question of iterating over an unquoted parameter expansion and dynamically creating each sed filter is beyond the scope of this answer.)
Here is one way to do it as a single stream. Building up the sed arguments from ${substlist} and calling sed once:
#!/bin/sh
cat ${input} |
sed `for subst in ${substlist}; do
echo " -e s/${subst}/modified_${subst}/g "
done` > ${output}
Depending on what is in ${substlist} you may need to do additional escaping.
Copy the input file to the output file, then perform the sed substitutions using the -i option to keep overwriting that file.
cp "$input" "$output"
for subst in $substlist
do
sed -i "s/$subst/modified_$subst/g" "$output"
done
With BSD/OSX sed that needs to be:
sed -i '' "s/$subst/modified_$subst/g" "$output"
Related
I have a list like this:
> echo $candidates
ENV-NONPROD-SANDBOX
ENV-NONPROD-SANDBOX-SECRETS
ENV-NONPROD-DEMO
ENV-NONPROD-DEMO-SECRETS
ENV-PROD-EU
ENV-PROD-EU-SECRETS
ENV-PROD-US
ENV-PROD-US-SECRETS
I also have a dynamically created list of expressions which I want to apply as filters (AND) to narrow that list to possible candidates:
$ filters=('/-SECRETS/!d' '/-NONPROD/!d') # static exanmple
Then I concatenate this and try to apply, but that does not work:
$ filterParam=$(printf "-e '%s' " "${filters[#]}")
$ echo $filterParam
-e "/-SECRETS/!d" -e "/-NONPROD/!d"
$ echo "$candidates" | sed $filterParam
sed: 1: " '/-SECRETS/\!d' ...": invalid command code '
The strange thing: If I execute it manually, it works!
> echo "$candidates" | sed -e "/-SECRETS/!d" -e "/-NONPROD/!d"
ENV-NONPROD-SANDBOX-SECRETS
ENV-NONPROD-DEMO-SECRETS
I execute this on macOS and zsh 5.8.1 (x86_64-apple-darwin21.0)
filterParam=$(printf "-e '%s' "
No, you can't store command line arguments in variables. Read https://mywiki.wooledge.org/BashFAQ/050 .
You can use bash arrays, which you already use to store filters, so just use them:
sedargs=()
for i in "${filters[#]}"; do
sedargs+=(-e "$i")
done
sed "${sedargs[#]}"
But sed is sed, just join array elements with a newline or a semicolon, which delimits sed expressions:
sed "$(printf "%s\n" "${filters[#]}")"
When you do a
sed $filterParam
in zsh, sed is invoked with one single argument, which is the content of the variable filterParam. sed does not know how to handle this.
If you would type the parameters explicitly, i.e.
sed -e "/-SECRETS/!d" -e "/-NONPROD/!d"
sed is invoked with four arguments, and this is what sed understands.
In bash, in the command
sed $filterParam
the value of filterParam would be split at the spaces and each "word" would be passed as a separate argument. In your concrete setting, this would make have sed receive 4 arguments.
I am reading some configs from a text file and storing them into a variable. THey are pipe separated like this:
a,b|c,d|e,f|g,h
I want to pass these into a sed command to substitute the left value from the right value in a text file and output that to another file.
I have:
#!/bin/bash
test="a,b|c,d|e,f|j,l"
arr=(`echo $test | sed 's/|/\n/g'`)
sub=""
for i in "${arr[#]}"
do
echo "$i"
array=(`echo $i | sed 's/,/\n/g'`)
sub+="s/${array[0]}/${array[1]}/g; "
done
sed $sub ${FILE} > output.json
When I do this it runs a sed command for each element in arr. So after the first sed it says:
No such file or directory
Now if I don't pass in $sub as a variable and just put the entire string in it does it in one sed command and it works:
sed "s/a/b/g; s/c/d/g; s/e/f/g; s/g/h/g;" ${FILE} > output.json
How can I pass in my matchers as a variable like in my first example, but get it to do it all in one sed command like the second example?
How can I pass in my matchers as a variable like in my first example, but get it to do it all in one sed command like the second example?
Shellcheck will hint you:
Line 12:
sed $sub ${FILE} > output.json
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
Did you mean: (apply this, apply all SC2086)
sed "$sub" "${FILE}" > output.json
Because there is a space after ; the pattern is word splitted. Quote it to prevent expansion. When $sub is not quoted, the s/c/d/g; is parsed as a filename, so sed exits with no such file or directory.
I have these files
NotRequired.txt (having lines which need to be remove)
Need2CleanSED.txt (big file , need to clean)
Need2CleanGRP.txt (big file , need to clean)
content:
more NotRequired.txt
[abc-xyz_pqr-pe2_123]
[lon-abc-tkt_1202]
[wat-7600-1_414]
[indo-pak_isu-5_761]
I am reading above file and want to remove lines from Need2Clean???.txt, trying via SED and GREP but no success.
myFile="NotRequired.txt"
while IFS= read -r HKline
do
sed -i '/$HKline/d' Need2CleanSED.txt
done < "$myFile"
myFile="NotRequired.txt"
while IFS= read -r HKline
do
grep -vE \"$HKline\" Need2CleanGRP.txt > Need2CleanGRP.txt
done < "$myFile"
Looks as if the Variable and characters [] making some problem.
What you're doing is extremely inefficient and error prone. Just do this:
grep -vF -f NotRequired.txt Need2CleanGRP.txt > tmp &&
mv tmp Need2CleanGRP.txt
Thanks to grep -F the above treats each line of NotRequired.txt as a string rather than a regexp so you don't have to worry about escaping RE metachars like [ and you don't need to wrap it in a shell loop - that one command will remove all undesirable lines in one execution of grep.
Never do command file > file btw as the shell might decide to execute the > file first and so empty file before command gets a chance to read it! Always do command file > tmp && mv tmp file instead.
Your assumption is correct. The [...] construct looks for any characters in that set, so you have to preface ("escape") them with \. The easiest way is to do that in your original file:
sed -i -e 's:\[:\\[:' -e 's:\]:\\]:' "${myFile}"
If you don't like that, you can probably put the sed command in where you're directing the file in:
done < replace.txt|sed -e 's:\[:\\[:' -e 's:\]:\\]:'
Finally, you can use sed on each HKline variable:
HKline=$( echo $HKline | sed -e 's:\[:\\[:' -e 's:\]:\\]:' )
try gnu sed:
sed -Ez 's/\n/\|/g;s!\[!\\[!g;s!\]!\\]!g; s!(.*).!/\1/d!' NotRequired.txt| sed -Ef - Need2CleanSED.txt
Two sed process are chained into one by shell pipe
NotRequired.txt is 'slurped' by sed -z all at once and substituted its \n and [ meta-char with | and \[ respectively of which the 2nd process uses it as regex script for the input file, ie. Need2CleanSED.txt. 1st process output;
/\[abc-xyz_pqr-pe2_123\]|\[lon-abc-tkt_1202\]|\[wat-7600-1_414\]|\[indo-pak_isu-5_761\]/d
add -u ie. unbuffered, option to evade from batch process, sort of direct i/o
I have a bash script which checks for a string pattern in file and delete entire line i same file but somehow its not deleting the line and no throwing any error .same command from command prompt deletes from file .
#array has patterns
for k in "${patternarr[#]}
do
sed -i '/$k/d' file.txt
done
sed version is >4
when this loop completes i want all lines matching string pattern in array to be deleted from file.txt
when i run sed -i '/pataern/d file.txt from command prompt then it works fine but not inside bash
Thanks in advance
Here:
sed -i '/$k/d' file.txt
The sed script is singly-quoted, which prevents shell variable expansion. It will (probably) work with
sed -i "/$k/d" file.txt
I say "probably" because what it will do depends on the contents of $k, which is just substituted into the sed code and interpreted as such. If $k contains slashes, it will break. If it comes from an untrustworthy source, you open yourself up to code injection (particularly with GNU sed, which can be made to execute shell commands).
Consider k=^/ s/^/rm -Rf \//e; #.
It is generally a bad idea to substitute shell variables into sed code (or any other code). A better way would be with GNU awk:
awk -i inplace -v pattern="$k" '!($0 ~ pattern)' file.txt
Or to just use grep -v and a temporary file.
first of all, you got an unclosed double quote around ${patternarr[#]} in your for statement.
Then your problem is that you use single quotes in the sed argument, making your shell not evaluate the $k within the quotes:
% declare -a patternarr=(foo bar fu foobar)
% for k in ${patternarr[#]}; do echo sed -i '/$k/d' file.txt; done
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
if you replace them with double quotes, here it goes:
% for k in ${patternarr[#]}; do echo sed -i "/$k/d" file.txt; done
sed -i /foo/d file.txt
sed -i /bar/d file.txt
sed -i /fu/d file.txt
sed -i /foobar/d file.txt
Any time you write a loop in shell just to manipulate text you have the wrong approach. This is probably closer to what you really should be doing (no surrounding loop required):
awk -v ks="${patternarr[#]}" 'BEGIN{gsub(/ /,")|(",ks); ks="("ks")} $0 !~ ks' file.txt
but there may be even better approaches still (e.g. only checking 1 field instead of the whole line, or using word boundaries, or string comparison or....) if you show us some sample input and expected output.
You need to use double quotes to interpolate shell variables inside the sed command, like:
for k in ${patternarr[#]}; do
sed -i "/$k/d" file.txt
done
I have a large number of words in a text file to replace.
This script is working up until the sed command where I get:
sed: 1: "*.js": invalid command code *
PS... Bash isn't one of my strong points - this doesn't need to be pretty or efficient
cd '/Users/xxxxxx/Sites/xxxxxx'
echo `pwd`;
for line in `cat myFile.txt`
do
export IFS=":"
i=0
list=()
for word in $line; do
list[$i]=$word
i=$[i+1]
done
echo ${list[0]}
echo ${list[1]}
sed -i "s/{$list[0]}/{$list[1]}/g" *.js
done
You're running BSD sed (under OS X), therefore the -i flag requires an argument specifying what you want the suffix to be.
Also, no files match the glob *.js.
This looks like a simple typo:
sed -i "s/{$list[0]}/{$list[1]}/g" *.js
Should be:
sed -i "s/${list[0]}/${list[1]}/g" *.js
(just like the echo lines above)
So myFile.txt contains a list of from:to substitutions, and you are looping over each of those. Why don't you create a sed script from this file instead?
cd '/Users/xxxxxx/Sites/xxxxxx'
sed -e 's/^/s:/' -e 's/$/:/' myFile.txt |
# Output from first sed script is a sed script!
# It contains substitutions like this:
# s:from:to:
# s:other:substitute:
sed -f - -i~ *.js
Your sed might not like the -f - which means sed should read its script from standard input. If that is the case, perhaps you can create a temporary script like this instead;
sed -e 's/^/s:/' -e 's/$/:/' myFile.txt >script.sed
sed -f script.sed -i~ *.js
Another approach, if you don't feel very confident with sed and think you are going to forget in a week what the meaning of that voodoo symbols is, could be using IFS in a more efficient way:
IFS=":"
cat myFile.txt | while read PATTERN REPLACEMENT # You feed the while loop with stdout lines and read fields separated by ":"
do
sed -i "s/${PATTERN}/${REPLACEMENT}/g"
done
The only pitfall I can see (it may be more) is that if whether PATTERN or REPLACEMENT contain a slash (/) they are going to destroy your sed expression.
You can change the sed separator with a non-printable character and you should be safe.
Anyway, if you know whats on your myFile.txt you can just use any.