I have to write to a file with some bash script. To substitute the variables we can use simply EOF and we escape the substitution with \. But, I want to escape everything, so, I can use 'EOF' and also want to substitute one variable, then how?.
cat > myfile <<'EOF'
$a
$b
$c
$d
$e
$f
$g
.....
.....
$multiple lines like this
EOF
I want to substitute only one variable let $c with it's value. How can I do in this case?. I can't use \ without quoting EOF escaping all the lines as there are many lines.
I just want to escape all the variable substitution('EOF') but want to substitute one variable with its value(How?).
To avoid escaping the many variables but still substitute for one of them, try:
$ cat script
sed 's/$c/3/' >myfile <<'EOF'
$a
$b
$c
$multiple lines like this
EOF
Let's run the script and examine the output file:
$ bash script
$ cat myfile
$a
$b
3
$multiple lines like this
Alternative
This version allows for a variable $c and, thus, may be more flexible:
$ cat script
c=New
sed "s/\$c/$c/" >myfile <<'EOF'
$a
$b
$c
$multiple lines like this
EOF
Execution of this results in:
$ bash script
$ cat myfile
$a
$b
New
$multiple lines like this
Related
Is it possible to expand the matched string in a sed command? I want to substitute variable names in a file with their values, this is my script at the moment:
#!/bin/bash
echo "Running the build script..."
VAR1="2005648"
VAR2="7445aa"
SERVER_NAME=$(hostname)
TIMESTAMP=$(date +%m-%d-%Y)
sed -i "s/{[A-Z_][A-Z_]*}/$&/g" my_file.txt #variable names in the file are written between { }
and this is a snapshot of my_file.txt:
Building finished at {TIMESTAMP}
{VAR1}:{VAR2}
On: {SERVER_NAME}
current working directory: {PWD}
But it doesn't work. Instead of substituting the variable name with it's value, It inserts a dollar sign right before the curly bracket.
How do I resolve this?
You could use envsubst to substitute environment variables, otherwise you would need a bunch of sed commands to replace everything.
Change your template file to:
Building finished at ${TIMESTAMP}
${VAR1}:${VAR2}
On: ${SERVER_NAME}
current working directory: ${PWD}
And the script to:
#!/bin/bash
echo "Running the build script..."
export VAR1="2005648"
export VAR2="7445aa"
export SERVER_NAME=$(hostname)
export TIMESTAMP=$(date +%m-%d-%Y)
# only replace the defined variables
envsubst '$VAR1 $VAR2 $SERVER_NAME $TIMESTAMP' < my_file.txt > newfile
# replace all environment variables ($USER, $HOME, $HOSTNAME, etc.)
#envsubst < my_file.txt > newfile.txt > newfile
The script replaces environment variables $VAR1, $VAR2, $SERVER_NAME and $TIMESTAMP in my_file.txt and saves the output to newfile.
You can see that ${PWD} doesn't get replaced, because I forgot to add it to the list.
In the second commented example all environment variables are replaced and non-existing variables are replaced by an empty string.
You can use the $VARNAME or ${VARNAME} syntax in the template.
I'd actually do it in a single pass this way using an awk that supports ENVIRON[], e.g. any POSIX awk:
$ cat tst.sh
#!/bin/env bash
echo "Running the build script..."
VAR1=2005648 \
VAR2=7445aa \
SERVER_NAME=$(hostname) \
TIMESTAMP=$(date +%m-%d-%Y) \
awk '
{
while ( match($0,/{[[:alnum:]_]+}/) ) {
printf "%s", substr($0,1,RSTART-1) ENVIRON[substr($0,RSTART+1,RLENGTH-2)]
$0 = substr($0,RSTART+RLENGTH)
}
print
}
' file
$ ./tst.sh
Running the build script...
Building finished at 04-14-2020
2005648:7445aa
On: MyLogin
current working directory: /home/MyLogin
but if you really want to do multiple passes calling sed inside a shell loop then ${!variable} is your friend, here's a start:
$ cat tst.sh
#!/bin/env bash
VAR1='2005648'
VAR2='7445aa'
SERVER_NAME='foo'
for var in VAR1 VAR2 SERVER_NAME; do
echo "var, $var, ${!var}"
done
$ ./tst.sh
var, VAR1, 2005648
var, VAR2, 7445aa
var, SERVER_NAME, foo
.
$ VAR1='stuff'
$ var='VAR1'; echo 'foo {VAR1} bar' | sed "s/{$var}/${!var}/"
foo stuff bar
The awk script is robust but YMMV using sed depending on the contents of the variables, e.g. it'd fail if they contain & or / or \1 or .... ENVIRON[] only has access to shell variables set on the awk command line or exported, hence the escape at the end of each line that sets a shell variable so it's part of the awk command line.
You can try this.
#!/usr/bin/env bash
echo "Running the build script..."
VAR1="2005648"
VAR2="7445aa"
SERVER_NAME=$(hostname)
TIMESTAMP=$(date +%m-%d-%Y)
sed "s|{TIMESTAMP}|$TIMESTAMP|;s|{VAR1}|$VAR1|;s|{VAR2}|$VAR2|;s|{SERVER_NAME}|$SERVER_NAME|;s|{PWD}|$PWD|" file.txt
Just add {} in the variables e.g. {$TIMESTAMP} and so on, if you really need it.
That should work unless there is something more that is not included in the question above.
I have been working in bash, and need to create a string argument. bash is a newish for me, to the point that I dont know how to build a string in bash from a list.
// foo.txt is a list of abs file names.
/foo/bar/a.txt
/foo/bar/b.txt
/delta/test/b.txt
should turn into: a.txt,b.txt,b.txt
OR: /foo/bar/a.txt,/foo/bar/b.txt,/delta/test/b.txt
code
s = ""
for file in $(cat foo.txt);
do
#what goes here? s += $file ?
done
myShellScript --script $s
I figure there was an easy way to do this.
with for loop:
for file in $(cat foo.txt);do echo -n "$file",;done|sed 's/,$/\n/g'
with tr:
cat foo.txt|tr '\n' ','|sed 's/,$/\n/g'
only sed:
sed ':a;N;$!ba;s/\n/,/g' foo.txt
This seems to work:
#!/bin/bash
input="foo.txt"
while IFS= read -r var
do
basename $var >> tmp
done < "$input"
paste -d, -s tmp > result.txt
output: a.txt,b.txt,b.txt
basename gets you the file names you need and paste will put them in the order you seem to need.
The input field separator can be used with set to create split/join functionality:
# split the lines of foo.txt into positional parameters
IFS=$'\n'
set $(< foo.txt)
# join with commas
IFS=,
echo "$*"
For just the file names, add some sed:
IFS=$'\n'; set $(sed 's|.*/||' foo.txt); IFS=,; echo "$*"
I need to substitute a unique string in a json file: {FILES} by a bash variable that contains thousands of paths: ${FILES}
sed -i "s|{FILES}|$FILES|" ./myFile.json
What would be the most elegant way to achieve that ? The content of ${FILES} is a result of an "aws s3" command. The content would look like :
FILES="/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."
I can't think of a solution where xargs would help me.
The safest way is probably to let Bash itself expand the variable. You can create a Bash script containing a here document with the full contents of myFile.json, with the placeholder {FILES} replaced by a reference to the variable $FILES (not the contents itself). Execution of this script would generate the output you seek.
For example, if myFile.json would contain:
{foo: 1, bar: "{FILES}"}
then the script should be:
#!/bin/bash
cat << EOF
{foo: 1, bar: "$FILES"}
EOF
You can generate the script with a single sed command:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json
Notice sed is doing two replacements; the first one (s/\$/\\$/g) to escape any dollar signs that might occur within the JSON data (replace every $ by \$). The second replaces {FILES} by $FILES; the literal text $FILES, not the contents of the variable.
Now we can combine everything into a single Bash one-liner that generates the script and immediately executes it by piping it to Bash:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | /bin/bash
Or even better, execute the script without spawning a subshell (useful if $FILES is set without export):
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | source /dev/stdin
Output:
{foo: 1, bar: "/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."}
Maybe perl would have fewer limitations?
perl -pi -e "s#{FILES}#${FILES}#" ./myFile.json
It's a little gross, but you can do it all within shell...
while read l
do
if ! echo "$l" | grep -q '{DATA}'
then
echo "$l"
else
echo "$l" | sed 's/{DATA}.*$//'
echo "$FILES"
echo "$l" | sed 's/^.*{DATA}//'
fi
done <./myfile.json >newfile.json
#mv newfile.json myfile.json
Obviously I'd leave the final line commented until you were confident it worked...
Maybe just don't do it? Can you just :
echo "var f = " > myFile2.json
echo $FILES >> myFile2.json
And reference myFile2.json from within your other json file? (You should put the global f variable into a namespace if this works for you.)
Instead of putting all those variables in an environment variable, put them in a file. Then read that file in perl:
foo.pl:
open X, "$ARGV[0]" or die "couldn't open";
shift;
$foo = <X>;
while (<>) {
s/world/$foo/;
print;
}
Command to run:
aws s3 ... >/tmp/myfile.$$
perl foo.pl /tmp/myfile.$$ <myFile.json >newFile.json
Hopefully that will bypass the limitations of the environment variable space and the argument length by pulling all the processing within perl itself.
I'm trying to write an script that saves each line from "test" file in a variable line1, line2, and so on..
x=$(cat test | wc -l) #here i have how many lines
i="1"
while [ "$i" -lt "$x" ]
do
line$i=$(sed '$iq;d' test) #i try to get the number $i line one by one
i=$[$i+1]
done
Could you please help me?
Thanks!
To read each line of the file test into an array called lines, use:
mapfile -t lines <test
Example
Consider this file:
$ cat test
dogs and cats
lions and tigers
bears
Execute this statement:
$ mapfile -t lines <test
We can now see the value of lines using declare -p:
$ declare -p lines
declare -a lines2='([0]="dogs and cats" [1]="lions and tigers" [2]="bears")'
Each line of file test is now an element of array lines and we can access them by number:
$ echo "${lines[0]}"
dogs and cats
lines$i should be lines[$i] (or google eval) but chances are you're going about this completely wrong and you should be writing it in awk, e.g.:
awk '{lines[NR] = $0} END{print NR; for (i=1;i<=NR;i++) print i, lines[i]}' test
Let's take advantage of available tooling to see what's wrong:
$ shellcheck linereader
In linereader line 1:
x=$(cat test | wc -l) #here i have how many lines
^-- SC2148: Shebang (#!) missing. Assuming Bash.
^-- SC2002: Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.
In linereader line 6:
line$i=$(sed '$iq;d' test) #i try to get the number $i line one by one
^-- SC1067: For indirection, use (associative) arrays or 'read "var$n" <<< "value"'
^-- SC2034: line appears unused. Verify it or export it.
^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.
In linereader line 7:
i=$[$i+1]
^-- SC2007: Use $((..)) instead of deprecated $[..]
That's a good start:
#!/bin/bash
# 1. Shebang added
# 2. Redirection for reading files as suggested (stylistic)
x=$(wc -l < test) #here i have how many lines
i="1"
while [ "$i" -lt "$x" ]
do
# 3. Using arrays as suggested
# 4. Using double quotes for sed as suggested
# Additionally, use ${i} so we reference $i and not $iq
line[$i]=$(sed "${i}q;d" test) #i try to get the number $i line one by one
# 5. Use $((..)) as suggested (stylistic)
i=$((i+1))
done
# Print a line to show that it works:
echo "Line #2 is ${line[2]}"
And now let's try:
$ cat test
foo
bar
baz
$ ./linereader
Line #2 is bar
We could also have done this more easily (and in O(n) time) with mapfile:
#!/bin/bash
mapfile line < test
echo "Line #2 (zero-based index 1) is: ${line[1]}"
I have little problem with specifying my variable. I have a file with normal text and somewhere in it there are brackets [ ] (only 1 pair of brackets in whole file), and some text between them. I need to capture the text within these brackets in a shell (bash) variable. How can I do that, please?
Bash/sed:
VARIABLE=$(tr -d '\n' filename | sed -n -e '/\[[^]]/s/^[^[]*\[\([^]]*\)].*$/\1/p')
If that is unreadable, here's a bit of an explanation:
VARIABLE=`subexpression` Assigns the variable VARIABLE to the output of the subexpression.
tr -d '\n' filename Reads filename, deletes newline characters, and prints the result to sed's input
sed -n -e 'command' Executes the sed command without printing any lines
/\[[^]]/ Execute the command only on lines which contain [some text]
s/ Substitute
^[^[]* Match any non-[ text
\[ Match [
\([^]]*\) Match any non-] text into group 1
] Match ]
.*$ Match any text
/\1/ Replaces the line with group 1
p Prints the line
May I point out that while most of the suggested solutions might work, there is absolutely no reason why you should fork another shell, and spawn several processes to do such a simple task.
The shell provides you with all the tools you need:
$ var='foo[bar] pinch'
$ var=${var#*[}; var=${var%%]*}
$ echo "$var"
bar
See: http://mywiki.wooledge.org/BashFAQ/073
Sed is not necessary:
var=`egrep -o '\[.*\]' FILENAME | tr -d ][`
But it's only works with single line matches.
Using Bash builtin regex matching seems like yet another way of doing it:
var='foo[bar] pinch'
[[ "$var" =~ [^\]\[]*\[([^\[]*)\].* ]] # Bash 3.0
var="${BASH_REMATCH[1]}"
echo "$var"
Assuming you are asking about bash variable:
$ export YOUR_VAR=$(perl -ne'print $1 if /\[(.*?)\]/' your_file.txt)
The above works if brackets are on the same line.
What about:
shell_variable=$(sed -ne '/\[/,/\]/{s/^.*\[//;s/\].*//;p;}' $file)
Worked for me on Solaris 10 under Korn shell; should work with Bash too. Replace '$(...)' with back-ticks in Bourne shell.
Edit: worked when given [ on one line and ] on another. For the single line case as well, use:
shell_variable=$(sed -n -e '/\[[^]]*$/,/\]/{s/^.*\[//;s/\].*//;p;}' \
-e '/\[.*\]/s/^.*\[\([^]]*\)\].*$/\1/p' $file)
The first '-e' deals with the multi-line spread; the second '-e' deals with the single-line case. The first '-e' says:
From the line containing an open bracket [ not followed by a close bracket ] on the same line
Until the line containing close bracket ],
substitute anything up to and including the open bracket with an empty string,
substitute anything from the close bracket onwards with an empty string, and
print the result
The second '-e' says:
For any line containing both open bracket and close bracket
Substitute the pattern consisting of 'characters up to and including open bracket', 'characters up to but excluding close bracket' (and remember this), 'stuff from close bracket onwards' with the remembered characters in the middle, and
print the result
For the multi-line case:
$ file=xxx
$ cat xxx
sdsajdlajsdl
asdajsdkjsaldjsal
sdasdsad [aaaa
bbbbbbb
cccc] asdjsalkdjsaldjlsaj
asdjsalkdjlksjdlaj
asdasjdlkjsaldja
$ shell_variable=$(sed -n -e '/\[[^]]*$/,/\]/{s/^.*\[//;s/\].*//;p;}' \
-e '/\[.*\]/s/^.*\[\([^]]*\)\].*$/\1/p' $file)
$ echo $shell_variable
aaaa bbbbbbb cccc
$
And for the single-line case:
$ cat xxx
sdsajdlajsdl
asdajsdkjsaldjsal
sdasdsad [aaaa bbbbbbb cccc] asdjsalkdjsaldjlsaj
asdjsalkdjlksjdlaj
asdasjdlkjsaldja
$
$ shell_variable=$(sed -n -e '/\[[^]]*$/,/\]/{s/^.*\[//;s/\].*//;p;}' \
-e '/\[.*\]/s/^.*\[\([^]]*\)\].*$/\1/p' $file)
$ echo $shell_variable
aaaa bbbbbbb cccc
$
Somewhere about here, it becomes simpler to do the whole job in Perl, slurping the file and editing the result string in two multi-line substitute operations.
var=`grep -e '\[.*\]' test.txt | sed -e 's/.*\[\(.*\)\].*/\1/' infile.txt`
Thanks to everyone, i used Strager's version and works perfectly, thanks alot once again...
var=`grep -e '\[.*\]' test.txt | sed -e 's/.*\[\(.*\)\].*/\1/' infile.txt`
Backslashes (BSL) got munched up ... :
var='foo[bar] pinch'
[[ "$var" =~ [^\]\[]*\[([^\[]*)\].* ]] # Bash 3.0
# Just in case ...:
[[ "$var" =~ [^BSL]BSL[]*BSL[([^BSL[]*)BSL].* ]] # Bash 3.0
var="${BASH_REMATCH[1]}"
echo "$var"
2 simple steps to extract the text.
split var at [ and get the right part
split var at ] and get the left part
cb0$ var='foo[bar] pinch'
cb0$ var=${var#*[}
cb0$ var=${var%]*} && echo $var
bar