I need to output some text as bash script, but in a script. I use cat for this, but it has one drawback. It interprets variables and stuff during it is being written. I do want to prevent this.
How to do that without quoting all varibles (my script is failrly long)? Example
cat >/tmp/script << EOF
$HOSTNAME
# lots of other stuff I do NOT want to escape like \$VARIABLE
# ...
EOF
cat /tmp/script
myhostname.mylan
I want:
cat /tmp/script
$HOSTNAME
Edit: Please note my script (here only $HOSTNAME) is very long, I dont want to change it all. Also single quoting does not work with <<
cat >/tmp/script '<< EOF
$HOSTNAME
EOF'
File not found: EOF'
What's the trick? Thanks.
If you want everything quoted:
cat << 'EOF'
stuff here with $signs is OK
as are `backquotes`
EOF
See the section on "here documents" in the manual.
Escape the $:
cat >/tmp/script << EOF
\$HOSTNAME
EOF
Use sed:
sed -n '20,30p' "$0"
to print line 20 to 30, SSCE:
#!/bin/bash
cat >/dev/null << EOF
3
4 $HOSTNAME
5 ls
6 $(ls -l)
7
8 echo 'foo
9 bar'
10
11 echo "Foo
12 $((4+4)) Bar"
EOF
sed -n '3,12p' "$0"
echo "fine?"
working with head/tail should work too.
You will have to adjust the numbers, if you work on it and insert or delete lines.
Related
I have to write to a file with some bash script. To substitute the variables we can use simply EOF and we escape the substitution with \. But, I want to escape everything, so, I can use 'EOF' and also want to substitute one variable, then how?.
cat > myfile <<'EOF'
$a
$b
$c
$d
$e
$f
$g
.....
.....
$multiple lines like this
EOF
I want to substitute only one variable let $c with it's value. How can I do in this case?. I can't use \ without quoting EOF escaping all the lines as there are many lines.
I just want to escape all the variable substitution('EOF') but want to substitute one variable with its value(How?).
To avoid escaping the many variables but still substitute for one of them, try:
$ cat script
sed 's/$c/3/' >myfile <<'EOF'
$a
$b
$c
$multiple lines like this
EOF
Let's run the script and examine the output file:
$ bash script
$ cat myfile
$a
$b
3
$multiple lines like this
Alternative
This version allows for a variable $c and, thus, may be more flexible:
$ cat script
c=New
sed "s/\$c/$c/" >myfile <<'EOF'
$a
$b
$c
$multiple lines like this
EOF
Execution of this results in:
$ bash script
$ cat myfile
$a
$b
New
$multiple lines like this
I need to substitute a unique string in a json file: {FILES} by a bash variable that contains thousands of paths: ${FILES}
sed -i "s|{FILES}|$FILES|" ./myFile.json
What would be the most elegant way to achieve that ? The content of ${FILES} is a result of an "aws s3" command. The content would look like :
FILES="/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."
I can't think of a solution where xargs would help me.
The safest way is probably to let Bash itself expand the variable. You can create a Bash script containing a here document with the full contents of myFile.json, with the placeholder {FILES} replaced by a reference to the variable $FILES (not the contents itself). Execution of this script would generate the output you seek.
For example, if myFile.json would contain:
{foo: 1, bar: "{FILES}"}
then the script should be:
#!/bin/bash
cat << EOF
{foo: 1, bar: "$FILES"}
EOF
You can generate the script with a single sed command:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json
Notice sed is doing two replacements; the first one (s/\$/\\$/g) to escape any dollar signs that might occur within the JSON data (replace every $ by \$). The second replaces {FILES} by $FILES; the literal text $FILES, not the contents of the variable.
Now we can combine everything into a single Bash one-liner that generates the script and immediately executes it by piping it to Bash:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | /bin/bash
Or even better, execute the script without spawning a subshell (useful if $FILES is set without export):
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | source /dev/stdin
Output:
{foo: 1, bar: "/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."}
Maybe perl would have fewer limitations?
perl -pi -e "s#{FILES}#${FILES}#" ./myFile.json
It's a little gross, but you can do it all within shell...
while read l
do
if ! echo "$l" | grep -q '{DATA}'
then
echo "$l"
else
echo "$l" | sed 's/{DATA}.*$//'
echo "$FILES"
echo "$l" | sed 's/^.*{DATA}//'
fi
done <./myfile.json >newfile.json
#mv newfile.json myfile.json
Obviously I'd leave the final line commented until you were confident it worked...
Maybe just don't do it? Can you just :
echo "var f = " > myFile2.json
echo $FILES >> myFile2.json
And reference myFile2.json from within your other json file? (You should put the global f variable into a namespace if this works for you.)
Instead of putting all those variables in an environment variable, put them in a file. Then read that file in perl:
foo.pl:
open X, "$ARGV[0]" or die "couldn't open";
shift;
$foo = <X>;
while (<>) {
s/world/$foo/;
print;
}
Command to run:
aws s3 ... >/tmp/myfile.$$
perl foo.pl /tmp/myfile.$$ <myFile.json >newFile.json
Hopefully that will bypass the limitations of the environment variable space and the argument length by pulling all the processing within perl itself.
I have a file template.txt and its content is below:
param1=${1}
param2=${2}
param3=${3}
I want to replace ${1},{2},${3}...${n} string values by elements of scriptParams variable.
The below code, only replaces first line.
scrpitParams="test1,test2,test3"
cat template.txt | for param in ${scriptParams} ; do i=$((++i)) ; sed -e "s/\${$i}/$param/" ; done
RESULT:
param1=test1
param2=${2}
param3=${3}
EXPECTED:
param1=test1
param2=test2
param3=test3
Note: I don't want to save replaced file, want to use its replaced value.
If you intend to use an array, use a real array. sed is not needed either:
$ cat template
param1=${1}
param2=${2}
param3=${3}
$ scriptParams=("test one" "test two" "test three")
$ while read -r l; do for((i=1;i<=${#scriptParams[#]};i++)); do l=${l//\$\{$i\}/${scriptParams[i-1]}}; done; echo "$l"; done < template
param1=test one
param2=test two
param3=test three
Learn to debug:
cat template.txt | for param in ${scriptParams} ; do i=$((++i)) ; echo $i - $param; done
1 - test1,test2,test3
Oops..
scriptParams="test1 test2 test3"
cat template.txt | for param in ${scriptParams} ; do i=$((++i)) ; echo $i - $param; done
1 - test1
2 - test2
3 - test3
Ok, looks better...
cat template.txt | for param in ${scriptParams} ; do i=$((++i)) ; sed -e "s/\${$i}/$param/" ; done
param1=test1
param2=${2}
param3=${3}
Ooops... so what's the problem? Well, the first sed command "eats" all the input. You haven't built a pipeline, where one sed command feeding the next... You have three seds trying to read the same input. Obviously the first one processed the whole input.
Ok, let's take a different approach, let's create the arguments for a single sed command (note: the "" is there to force echo not to interpret -e as an command line switch).
sedargs=$(for param in ${scriptParams} ; do i=$((++i)); echo "" -e "s/\${$i}/$param/"; done)
cat template.txt | sed $sedargs
param1=test1
param2=test2
param3=test3
That's it. Note that this isn't perfect, you can have all sort of problems if the replace texts are complex (e.g.: contain space).
Let me think how to do this in a better way... (well, the obvious solution which comes to mind is not to use a shell script for this task...)
Update:
If you want to build a proper pipeline, here are some solutions: How to make a pipe loop in bash
You can do that with just bash alone:
#!/bin/bash
scriptParams=("test1" "test2" "test3") ## Better store it as arrays.
while read -r line; do
for i in in "${!scriptParams[#]}"; do ## Indices of array scriptParams would be populated to i starting at 0.
line=${line/"\${$((i + 1))}"/"${scriptParams[i]}"} ## ${var/p/r} replaces patterns (p) with r in the contents of var. Here we also add 1 to the index to fit with the targets.
done
echo "<br>$line</br>"
done < template.txt
Save it in a script and run bash script.sh to get an output like this:
<br>param1=test1</br>
<br>param2=test2</br>
<br>param3=test3</br>
I have this multi-line string (quotes included):
abc'asdf"
$(dont-execute-this)
foo"bar"''
How would I assign it to a variable using a heredoc in Bash?
I need to preserve newlines.
I don't want to escape the characters in the string, that would be annoying...
You can avoid a useless use of cat and handle mismatched quotes better with this:
$ read -r -d '' VAR <<'EOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
EOF
If you don't quote the variable when you echo it, newlines are lost. Quoting it preserves them:
$ echo "$VAR"
abc'asdf"
$(dont-execute-this)
foo"bar"''
If you want to use indentation for readability in the source code, use a dash after the less-thans. The indentation must be done using only tabs (no spaces).
$ read -r -d '' VAR <<-'EOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
EOF
$ echo "$VAR"
abc'asdf"
$(dont-execute-this)
foo"bar"''
If, instead, you want to preserve the tabs in the contents of the resulting variable, you need to remove tab from IFS. The terminal marker for the here doc (EOF) must not be indented.
$ IFS='' read -r -d '' VAR <<'EOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
EOF
$ echo "$VAR"
abc'asdf"
$(dont-execute-this)
foo"bar"''
Tabs can be inserted at the command line by pressing Ctrl-V Tab. If you are using an editor, depending on which one, that may also work or you may have to turn off the feature that automatically converts tabs to spaces.
Use $() to assign the output of cat to your variable like this:
VAR=$(cat <<'END_HEREDOC'
abc'asdf"
$(dont-execute-this)
foo"bar"''
END_HEREDOC
)
# this will echo variable with new lines intact
echo "$VAR"
# this will echo variable without new lines (changed to space character)
echo $VAR
Making sure to delimit starting END_HEREDOC with single-quotes.
Note that ending heredoc delimiter END_HEREDOC must be alone on the line (hence ending parenthesis is on the next line).
Thanks to #ephemient for the answer.
this is variation of Dennis method, looks more elegant in the scripts.
function definition:
define(){ IFS='\n' read -r -d '' ${1} || true; }
usage:
define VAR <<'EOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
EOF
echo "$VAR"
enjoy
p.s. made a 'read loop' version for shells that do not support read -d. should work with set -eu and unpaired backticks, but not tested very well:
define(){ o=; while IFS="\n" read -r a; do o="$o$a"'
'; done; eval "$1=\$o"; }
VAR=<<END
abc
END
doesn't work because you are redirecting stdin to something that doesn't care about it, namely the assignment
export A=`cat <<END
sdfsdf
sdfsdf
sdfsfds
END
` ; echo $A
works, but there's a back-tic in there that may stop you from using this. Also, you should really avoid using backticks, it's better to use the command substitution notation $(..).
export A=$(cat <<END
sdfsdf
sdfsdf
sdfsfds
END
) ; echo $A
There is still no solution that preserves newlines.
This is not true - you're probably just being misled by the behaviour of echo:
echo $VAR # strips newlines
echo "$VAR" # preserves newlines
Branching off Neil's answer, you often don't need a var at all, you can use a function in much the same way as a variable and it's much easier to read than the inline or read-based solutions.
$ complex_message() {
cat <<'EOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
EOF
}
$ echo "This is a $(complex_message)"
This is a abc'asdf"
$(dont-execute-this)
foo"bar"''
An array is a variable, so in that case mapfile will work
mapfile y <<'z'
abc'asdf"
$(dont-execute-this)
foo"bar"''
z
Then you can print like this
printf %s "${y[#]}"
I can't believe I'm the first to post this.
#Erman and #Zombo are close, but mapfile doesn't just read arrays...
Consider this:
#!/bin/bash
mapfile -d '' EXAMPLE << 'EOF'
Hello
こんにちは
今晩は
小夜なら
EOF
echo -n "$EXAMPLE"
Yielding:
Hello
こんにちは
今晩は
小夜なら
$'' is the delimiter given to mapfile, it will never occur, it means "not delimited".
So there's no need for a useless use of cat and no need to incur the penalty of recombining arrays.
Furthermore, you get this benefit:
$ echo $EXAMPLE
Hello こんにちは 今晩は 小夜なら
Which you do not receive with #Zombo's method:
mapfile y <<'z'
abc'asdf"
$(dont-execute-this)
foo"bar"''
z
echo $y
abc'asdf"
Bonus
If you run it through head -c -1 you can also get rid of that last newline in a way that won't be non-performant:
unset EXAMPLE
mapfile -d '' EXAMPLE < <(head -c -1 << EOF
Hello
こんにちは
今晩は
小夜なら
EOF
)
printf "%q" "$EXAMPLE"
$'Hello\nこんにちは\n今晩は\n小夜なら'
assign a heredoc value to a variable
VAR="$(cat <<'VAREOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
VAREOF
)"
used as an argument of a command
echo "$(cat <<'SQLEOF'
xxx''xxx'xxx'xx 123123 123123
abc'asdf"
$(dont-execute-this)
foo"bar"''
SQLEOF
)"
Thanks to dimo414's answer, this shows how his great solution works, and shows that you can have quotes and variables in the text easily as well:
example output
$ ./test.sh
The text from the example function is:
Welcome dev: Would you "like" to know how many 'files' there are in /tmp?
There are " 38" files in /tmp, according to the "wc" command
test.sh
#!/bin/bash
function text1()
{
COUNT=$(\ls /tmp | wc -l)
cat <<EOF
$1 Would you "like" to know how many 'files' there are in /tmp?
There are "$COUNT" files in /tmp, according to the "wc" command
EOF
}
function main()
{
OUT=$(text1 "Welcome dev:")
echo "The text from the example function is: $OUT"
}
main
I found myself having to read a string with NULL in it, so here is a solution that will read anything you throw at it. Although if you actually are dealing with NULL, you will need to deal with that at the hex level.
$ cat > read.dd.sh
read.dd() {
buf=
while read; do
buf+=$REPLY
done < <( dd bs=1 2>/dev/null | xxd -p )
printf -v REPLY '%b' $( sed 's/../ \\\x&/g' <<< $buf )
}
Proof:
$ . read.dd.sh
$ read.dd < read.dd.sh
$ echo -n "$REPLY" > read.dd.sh.copy
$ diff read.dd.sh read.dd.sh.copy || echo "File are different"
$
HEREDOC example (with ^J, ^M, ^I):
$ read.dd <<'HEREDOC'
> (TAB)
> (SPACES)
(^J)^M(^M)
> DONE
>
> HEREDOC
$ declare -p REPLY
declare -- REPLY=" (TAB)
(SPACES)
(^M)
DONE
"
$ declare -p REPLY | xxd
0000000: 6465 636c 6172 6520 2d2d 2052 4550 4c59 declare -- REPLY
0000010: 3d22 0928 5441 4229 0a20 2020 2020 2028 =".(TAB). (
0000020: 5350 4143 4553 290a 285e 4a29 0d28 5e4d SPACES).(^J).(^M
0000030: 290a 444f 4e45 0a0a 220a ).DONE
Here's a way to do it that is (imho) quite elegant and avoids a UUOC:
VAR=$(sed -e 's/[ ]*\| //g' -e '1d;$d' <<'--------------------'
|
| <!DOCTYPE html>
| <html>
| <head>
| <script src='script.js'></script>
| </head>
| <body>
| <span id='hello-world'></span>
| </body>
| </html>
|
--------------------
)
The '|' characters define the margin, and only the whitespace to the right of the margin is respected in the printed string. The '1d;$d' strips the first and last line, which are just added as a top and bottom margin around the content. Everything can be indented to whatever level you like, except the HEREDOC delimiter, which in this case is just a bunch of hyphens.
echo "$VAR"
# prints
<!DOCTYPE html>
<html>
<head>
<script src='script.js'></script>
</head>
<body>
<span id='hello-world'></span>
</body>
</html>
$TEST="ok"
read MYTEXT <<EOT
this bash trick
should preserve
newlines $TEST
long live perl
EOT
echo -e $MYTEXT
I have little problem with specifying my variable. I have a file with normal text and somewhere in it there are brackets [ ] (only 1 pair of brackets in whole file), and some text between them. I need to capture the text within these brackets in a shell (bash) variable. How can I do that, please?
Bash/sed:
VARIABLE=$(tr -d '\n' filename | sed -n -e '/\[[^]]/s/^[^[]*\[\([^]]*\)].*$/\1/p')
If that is unreadable, here's a bit of an explanation:
VARIABLE=`subexpression` Assigns the variable VARIABLE to the output of the subexpression.
tr -d '\n' filename Reads filename, deletes newline characters, and prints the result to sed's input
sed -n -e 'command' Executes the sed command without printing any lines
/\[[^]]/ Execute the command only on lines which contain [some text]
s/ Substitute
^[^[]* Match any non-[ text
\[ Match [
\([^]]*\) Match any non-] text into group 1
] Match ]
.*$ Match any text
/\1/ Replaces the line with group 1
p Prints the line
May I point out that while most of the suggested solutions might work, there is absolutely no reason why you should fork another shell, and spawn several processes to do such a simple task.
The shell provides you with all the tools you need:
$ var='foo[bar] pinch'
$ var=${var#*[}; var=${var%%]*}
$ echo "$var"
bar
See: http://mywiki.wooledge.org/BashFAQ/073
Sed is not necessary:
var=`egrep -o '\[.*\]' FILENAME | tr -d ][`
But it's only works with single line matches.
Using Bash builtin regex matching seems like yet another way of doing it:
var='foo[bar] pinch'
[[ "$var" =~ [^\]\[]*\[([^\[]*)\].* ]] # Bash 3.0
var="${BASH_REMATCH[1]}"
echo "$var"
Assuming you are asking about bash variable:
$ export YOUR_VAR=$(perl -ne'print $1 if /\[(.*?)\]/' your_file.txt)
The above works if brackets are on the same line.
What about:
shell_variable=$(sed -ne '/\[/,/\]/{s/^.*\[//;s/\].*//;p;}' $file)
Worked for me on Solaris 10 under Korn shell; should work with Bash too. Replace '$(...)' with back-ticks in Bourne shell.
Edit: worked when given [ on one line and ] on another. For the single line case as well, use:
shell_variable=$(sed -n -e '/\[[^]]*$/,/\]/{s/^.*\[//;s/\].*//;p;}' \
-e '/\[.*\]/s/^.*\[\([^]]*\)\].*$/\1/p' $file)
The first '-e' deals with the multi-line spread; the second '-e' deals with the single-line case. The first '-e' says:
From the line containing an open bracket [ not followed by a close bracket ] on the same line
Until the line containing close bracket ],
substitute anything up to and including the open bracket with an empty string,
substitute anything from the close bracket onwards with an empty string, and
print the result
The second '-e' says:
For any line containing both open bracket and close bracket
Substitute the pattern consisting of 'characters up to and including open bracket', 'characters up to but excluding close bracket' (and remember this), 'stuff from close bracket onwards' with the remembered characters in the middle, and
print the result
For the multi-line case:
$ file=xxx
$ cat xxx
sdsajdlajsdl
asdajsdkjsaldjsal
sdasdsad [aaaa
bbbbbbb
cccc] asdjsalkdjsaldjlsaj
asdjsalkdjlksjdlaj
asdasjdlkjsaldja
$ shell_variable=$(sed -n -e '/\[[^]]*$/,/\]/{s/^.*\[//;s/\].*//;p;}' \
-e '/\[.*\]/s/^.*\[\([^]]*\)\].*$/\1/p' $file)
$ echo $shell_variable
aaaa bbbbbbb cccc
$
And for the single-line case:
$ cat xxx
sdsajdlajsdl
asdajsdkjsaldjsal
sdasdsad [aaaa bbbbbbb cccc] asdjsalkdjsaldjlsaj
asdjsalkdjlksjdlaj
asdasjdlkjsaldja
$
$ shell_variable=$(sed -n -e '/\[[^]]*$/,/\]/{s/^.*\[//;s/\].*//;p;}' \
-e '/\[.*\]/s/^.*\[\([^]]*\)\].*$/\1/p' $file)
$ echo $shell_variable
aaaa bbbbbbb cccc
$
Somewhere about here, it becomes simpler to do the whole job in Perl, slurping the file and editing the result string in two multi-line substitute operations.
var=`grep -e '\[.*\]' test.txt | sed -e 's/.*\[\(.*\)\].*/\1/' infile.txt`
Thanks to everyone, i used Strager's version and works perfectly, thanks alot once again...
var=`grep -e '\[.*\]' test.txt | sed -e 's/.*\[\(.*\)\].*/\1/' infile.txt`
Backslashes (BSL) got munched up ... :
var='foo[bar] pinch'
[[ "$var" =~ [^\]\[]*\[([^\[]*)\].* ]] # Bash 3.0
# Just in case ...:
[[ "$var" =~ [^BSL]BSL[]*BSL[([^BSL[]*)BSL].* ]] # Bash 3.0
var="${BASH_REMATCH[1]}"
echo "$var"
2 simple steps to extract the text.
split var at [ and get the right part
split var at ] and get the left part
cb0$ var='foo[bar] pinch'
cb0$ var=${var#*[}
cb0$ var=${var%]*} && echo $var
bar