I have a shell script with up to 5 parameters. There is files with placeholders. I would like to use sed script file depending of the one variable. The problem is that when I have variables defined in the sed script - values of those variables are not put in placeholders.
#!/bin/bash
A=$1
B=$2
echo "Some string with _PH1_ place holders _PH2_"|sed -i -f script1.sed >> file.txt
one of the sed scripts file
#Sed script 1
s/_PH1_/${A}/g
s/_PH2_/${B}/g
If your sed script is that short, you might as well inline it in bash:
#!/bin/bash
A=$1
B=$2
#your sed commands but still part of the bash script:
sed -i -e "s/_PH1_/${A}/g" file.txt
sed -i -e "/_PH2_/${B}/g" file.txt
You need to create the sed script as part of your bash script so the variable substitution takes place:
#!/bin/bash
A=$1
B=$2
cat >> script1.sed << EOF
s/_PH1_/${A}/g
s/_PH2_/${B}/g
EOF
echo "Some string with _PH1_ place holders _PH2_"|sed -i -f script1.sed >> file.txt
$ cat tst.sh
#!/bin/bash
set -a
A="$1"
B="$2"
sedScript=$(mktemp)
printf 'sed "\n' >> "$sedScript"
cat script1.sed >> "$sedScript"
printf '"\n' >> "$sedScript"
echo "Some string with _PH1_ place holders _PH2_" | "$sedScript"
rm -f "$sedScript"
$ ./tst.sh foo bar
Some string with foo place holders bar
Related
#!/bin/bash
{ cat sample.txt; echo; } | while read -r -a A_Name; do
if [ ! -z "${A_Name[0]}" ]; then
echo " ${A_Name[0]%.isx} "
fi
done
I am trying to display contents of a text file (which includes .isx files) using while loop but when i try to eliminate extension with %, it doesnt work.
Output
.isx is appearing for first two values:
./test.sh
abc.isx
def.isx
ghi
Input
sample.txt file:
abc.isx
def.isx
ghi.isx
Please, assist. Thank you.
why so complicated?
#!/bin/bash
cat sample.txt | while read line; do
echo "${line%.isx}"
done
or with sed
sed "s/\.isx//" sample.txt >output.txt
or with sed and inplace replacement
sed -i "s/\.isx//" sample.txt
Sample data in a stackoverflow.csv file I have:
foo
foo
foo
foo
bar
bar
bar
bar
baz
baz
baz
baz
I know with sed -n /foo/p stackoverflow.csv > foo.csv
I'll get all records matching foo directed to that file, but I don't want to specify the matching pattern on the cli, I'd rather put it in a script and have all records (foo, bar and baz) sent to their own file.
Basically this in a script:
sed -n /foo/p stackoverflow.csv > foo.csv
sed -n /bar/p stackoverflow.csv > bar.csv
sed -n /baz/p stackoverflow.csv > baz.csv
Like this:
#!/bin/sh
sleep 2
sed -n /foo/p > foo.csv
sed -n /bar/p > bar.csv
sed -n /baz/p > baz.csv
This creates the files but they're all empty.
Also, if the script were to have just one print statement, it works.
Any input?
You missed the input filename
#!/bin/sh
sleep 2
sed -n /foo/p stackoverflow.csv > foo.csv
sed -n /bar/p stackoverflow.csv > bar.csv
sed -n /baz/p stackoverflow.csv > baz.csv
You can provide input file as an argument to your script, in that case change the script to read an argument and pass it to the sed command.
Run the script like this: ./script.sh input_filename, where you can specify different input files as argument.
#!/bin/sh
file=${1}
sed -n /foo/p ${file} > foo.csv
sed -n /bar/p ${file} > bar.csv
sed -n /baz/p ${file} > baz.csv
Here's a solution with grep in a loop where you don't need to provideeach term in advance. Assuming a bash shell:
for i in $(sort stackoverflow.csv | uniq); do grep $i testfile > $i; done
The w command in sed allows you to write to a named file.
sed -n -e '/foo/wfoo.csv' -e '/bar/wbar.csv' \
-e '/baz/wbaz.csv' stackoverflow.csv
Not all sed dialects accept multiple -e arguments; a multi-line string with the commands separated by newlines may be the most portable solution.
sed -n '/foo/wfoo.csv
/bar/wbar.csv
/baz/wbaz.csv' stackoverflow.csv
This examines the input file just once, and writes out the matching lines as it proceeds through the file.
To create a script which accepts one or more file names as arguments, replace the hard-coded file name with "$#".
#!/bin/sh
sed -n ... "$#"
I need to substitute a unique string in a json file: {FILES} by a bash variable that contains thousands of paths: ${FILES}
sed -i "s|{FILES}|$FILES|" ./myFile.json
What would be the most elegant way to achieve that ? The content of ${FILES} is a result of an "aws s3" command. The content would look like :
FILES="/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."
I can't think of a solution where xargs would help me.
The safest way is probably to let Bash itself expand the variable. You can create a Bash script containing a here document with the full contents of myFile.json, with the placeholder {FILES} replaced by a reference to the variable $FILES (not the contents itself). Execution of this script would generate the output you seek.
For example, if myFile.json would contain:
{foo: 1, bar: "{FILES}"}
then the script should be:
#!/bin/bash
cat << EOF
{foo: 1, bar: "$FILES"}
EOF
You can generate the script with a single sed command:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json
Notice sed is doing two replacements; the first one (s/\$/\\$/g) to escape any dollar signs that might occur within the JSON data (replace every $ by \$). The second replaces {FILES} by $FILES; the literal text $FILES, not the contents of the variable.
Now we can combine everything into a single Bash one-liner that generates the script and immediately executes it by piping it to Bash:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | /bin/bash
Or even better, execute the script without spawning a subshell (useful if $FILES is set without export):
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | source /dev/stdin
Output:
{foo: 1, bar: "/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."}
Maybe perl would have fewer limitations?
perl -pi -e "s#{FILES}#${FILES}#" ./myFile.json
It's a little gross, but you can do it all within shell...
while read l
do
if ! echo "$l" | grep -q '{DATA}'
then
echo "$l"
else
echo "$l" | sed 's/{DATA}.*$//'
echo "$FILES"
echo "$l" | sed 's/^.*{DATA}//'
fi
done <./myfile.json >newfile.json
#mv newfile.json myfile.json
Obviously I'd leave the final line commented until you were confident it worked...
Maybe just don't do it? Can you just :
echo "var f = " > myFile2.json
echo $FILES >> myFile2.json
And reference myFile2.json from within your other json file? (You should put the global f variable into a namespace if this works for you.)
Instead of putting all those variables in an environment variable, put them in a file. Then read that file in perl:
foo.pl:
open X, "$ARGV[0]" or die "couldn't open";
shift;
$foo = <X>;
while (<>) {
s/world/$foo/;
print;
}
Command to run:
aws s3 ... >/tmp/myfile.$$
perl foo.pl /tmp/myfile.$$ <myFile.json >newFile.json
Hopefully that will bypass the limitations of the environment variable space and the argument length by pulling all the processing within perl itself.
I have a script named "test.sh"
#!/bin/bash
echo "sed -i '/$1/c\$2' $3"
Running as: ./test.sh "This line is removed." "some text here" out.txt . I want this to be printed exactly:
sed -i '/This line is removed./c\some text here' out.txt
Output is printing as:
sed -i '/This line is removed./c$2' out.txt
The problem is there with \$ it is printing $ not allowing content of $2. How to do this?
Use this line in your script with double slash:
#!/bin/bash
echo "sed -i '/$1/c\\$2' $3"
try this
echo "sed -i '/$1/c\\$2' $3"
my question seems to be general, but i can't find any answers.
In sed command, how can you replace the substitution pattern by a value returned by a simple bash function.
For instance, I created the following function :
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
and the folowing sed command :
myCatFile=`sed -e "s/[0-3][0-9]\/[0-1][0-9]\/[0-9][0-9]/& parseDates &\}/p" myfile`
I found that the caracter '&' represents the current pattern found, i'd like it to be passed to my bash function and the whole pattern to be substituted by the pattern found +dateParsed.
Does anybody have an idea ?
Thanks
you can use the "e" option in sed command like this:
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/e" <<END
ni
END
you can see the result without "e":
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/" <<END
ni
END
Agree with Glenn Jackman.
If you want to use bash function in sed, something like this :
sed -rn 's/^([[:digit:].]+)/`date -d #&`/p' file |
while read -r line; do
eval echo "$line"
done
My file here begins with a unix timestamp (e.g. 1362407133.936).
Bash function inside sed (maybe for other purposes):
multi_stdin(){ #Makes function accepet variable or stdin (via pipe)
[[ -n "$1" ]] && echo "$*" || cat -
}
sans_accent(){
multi_stdin "$#" | sed '
y/àáâãäåèéêëìíîïòóôõöùúûü/aaaaaaeeeeiiiiooooouuuu/
y/ÀÁÂÃÄÅÈÉÊËÌÍÎÏÒÓÔÕÖÙÚÛÜ/AAAAAAEEEEIIIIOOOOOUUUU/
y/çÇñÑߢÐð£Øø§µÝý¥¹²³ªº/cCnNBcDdLOoSuYyY123ao/
'
}
eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | sans_accent#p')
or
eval $(echo "Rogério Madureira" | sed -n 's#.*#sans_accent &#p')
Rogerio
And if you need to keep the output into a variable:
VAR=$( eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | desacentua#p') )
echo "$VAR"
do it step by step. (also you could use an alternate delimiter , such as "|" instead of "/"
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
value=$(parseDates)
sed -n "s|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& $value &|p" myfile
Note the use of double quotes instead of single quotes, so that $value can be interpolated
I'd like to know if there's a way to do this too. However, for this particular problem you don't need it. If you surround the different components of the date with ()s, you can back reference them with \1 \2 etc and reformat however you want.
For instance, let's reverse 03/04/1973:
echo 03/04/1973 | sed -e 's/\([0-9][0-9]\)\/\([0-9][0-9]\)\/\([0-9][0-9][0-9][0-9]\)/\3\/\2\/\1/g'
sed -e 's#[0-3][0-9]/[0-1][0-9]/[0-9][0-9]#& $(parseDates &)#' myfile |
while read -r line; do
eval echo "$line"
done
You can glue together a sed-command by ending a single-quoted section, and reopening it again.
sed -n 's|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& '$(parseDates)' &|p' datefile
However, in contrast to other examples, a function in bash can't return strings, only put them out:
function parseDates(){
# Some process here with $1 (the pattern found)
echo dateParsed
}