Assume a file file with multiple lines.
$ cat file
foo
bar
baz
Assume further that I wish to loop through each line with a while-loop.
$ while IFS= read -r line; do
$ echo $line
$ # do stuff
$ done < file
foo
bar
baz
Finally, please assume that I wish to pass lines stored in a variable rather than lines stored in a file. How can I loop through lines that are saved as a variable without receiving the below error?
$ MY_VAR=$(cat file)
$ while IFS= read -r line; do
$ echo $line
$ # do stuff
$ done < $(echo "$MY_VAR")
bash: $(echo "$MY_VAR"): ambiguous redirect
You have several options:
A herestring (note that this is a non-POSIX extension): done <<<"$MY_VAR"
A heredoc (POSIX-compliant, will work with /bin/sh):
done <<EOF
$MY_VAR
EOF
A process substitution (also a non-POSIX extension, but using printf rather than echo makes it more predictable across shells that support it; see the APPLICATION USAGE note in the POSIX spec for echo): done < <(printf '%s\n' "$MY_VAR")
Note that the first two options will (in bash) create a temporary file on disk with the variable's contents, whereas the last one uses a FIFO.
< needs to be followed by a filename. You can use a here-string:
done <<< "$MY_VAR"
or process substitution:
done < <(echo "$MY_VAR")
I am trying to use the following pattern inside a shell script eg. I want to use the following:
grep ^"#" $fn | grep -v NM$ > $op
but inside bash script. The problem is normally bash considers everything after the "#" as a comment. If i use
grep ^"\#" $fn
I think it changes the meaning. I am a newbie.
Any help will be appreciated.
There are multiple possibilities:
grep '^#' $fn
or
grep "^#" $fn
or
grep ^\# $fn
Here is a little example which I saved as foo.sh
echo "#"
echo '#' # this is a comment
echo '\#'
when run as sh foo.sh this is what is printed out
#
#
\#
I have a loop in my script that will append a list of email address's to a file "$CRN". If this script is executed again, it will append to this old list. I want it to overwrite with the new list rather then appending to the old list. I can submit my whole script if needed. I know I could test if "$CRN" exists then remove file, but I'm interested in some other suggestions? Thanks.
for arg in "$#"; do
if ls /students | grep -q "$arg"; then
echo "${arg}#mail.ccsf.edu">>$CRN
((students++))
elif ls /users | grep -q "$arg$"; then
echo "${arg}#ccsf.edu">>$CRN
((faculty++))
fi
Better do this :
CRN="/path/to/file"
:> "$CRN"
for arg; do
if printf '%s\n' /students/* | grep -q "$arg"; then
echo "${arg}#mail.ccsf.edu" >> "$CRN"
((students++))
elif printf '%s\n'/users/* | grep -q "${arg}$"; then
echo "${arg}#ccsf.edu" >> "$CRN"
((faculty++))
fi
done
don't parse ls output ! use bash glob instead. ls is a tool for interactively looking at file information. Its output is formatted for humans and will cause bugs in scripts. Use globs or find instead. Understand why: http://mywiki.wooledge.org/ParsingLs
"Double quote" every expansion, and anything that could contain a special character, eg. "$var", "$#", "${array[#]}", "$(command)". See http://mywiki.wooledge.org/Quotes http://mywiki.wooledge.org/Arguments and http://wiki.bash-hackers.org/syntax/words
take care to false positives like arg=foo and glob : foobar, that will match. You need grep -qw then if you want word boundaries. UP2U
Consider a ASCII text file (lets say it contains code of a non-shell scripting language):
Text_File.msh:
spool on to '$LOG_FILE_PATH/logfile.log';
login 'username' 'password';
....
Now if this were a shell script I could run it as $ sh Text_File.msh and the shell would automatically expand the variables.
What I want to do is have shell expand these variables and then create a new file as Text_File_expanded.msh as follows:
Text_File_expanded.msh:
spool on to '/expanded/path/of/the/log/file/../logfile.log';
login 'username' 'password';
....
Consider:
$ a=123
$ echo "$a"
123
So technically this should do the trick:
$ echo "`cat Text_File.msh`" > Text_File_expanded.msh
...but it doesn't work as expected and the output-file while is identical to the source.
So I am unsure how to achieve this.. My goal is make it easier to maintain the directory paths embedded within my non-shell scripts. These scripts cannot contain any UNIX code as it is not compiled by the UNIX shell.
This question has been asked in another thread, and this is the best answer IMO:
export LOG_FILE_PATH=/expanded/path/of/the/log/file/../logfile.log
cat Text_File.msh | envsubst > Text_File_expanded.msh
if on Mac, install gettext first: brew install gettext
see:
Forcing bash to expand variables in a string loaded from a file
This solution is not elegant, but it works. Create a script call shell_expansion.sh:
echo 'cat <<END_OF_TEXT' > temp.sh
cat "$1" >> temp.sh
echo 'END_OF_TEXT' >> temp.sh
bash temp.sh >> "$2"
rm temp.sh
You can then invoke this script as followed:
bash shell_expansion.sh Text_File.msh Text_File_expanded.msh
If you want it in one line (I'm not a bash expert so there may be caveats to this but it works everywhere I've tried it):
when test.txt contains
${line1}
${line2}
then:
>line1=fark
>line2=fork
>value=$(eval "echo \"$(cat test.txt)\"")
>echo "$value"
line1 says fark
line2 says fork
Obviously if you just want to print it you can take out the extra value=$() and echo "$value".
If a Perl solution is ok for you:
Sample file:
$ cat file.sh
spool on to '$HOME/logfile.log';
login 'username' 'password';
Solution:
$ perl -pe 's/\$(\w+)/$ENV{$1}/g' file.sh
spool on to '/home/user/logfile.log';
login 'username' 'password';
One limitation of the above answers is that they both require the variables to be exported to the environment. Here's what i came up with that would allow the variables to be local to the current shell script:
#!/bin/sh
FOO=bar;
FILE=`mktemp`; # Let the shell create a temporary file
trap 'rm -f $FILE' 0 1 2 3 15; # Clean up the temporary file
(
echo 'cat <<END_OF_TEXT'
cat "$#"
echo 'END_OF_TEXT'
) > $FILE
. $FILE
The above example allows the variable $FOO to be substituted in the files named on the command line. I'm sure it can be improved, but this works for me so far.
Thanks to both previous answers for their ideas!
If the variables you want to translate are known and limited in number, you can always do the translation yourself:
sed "s/\$LOG_FILE_PATH/$LOG_FILE_PATH/g" input > output
And also assuming the variable itself is already known
This solution allows you to keep the same formatting in the ouput file
Copy and paste the following lines in your script
cat $1 | while read line
do
eval $line
echo $line
eval echo $line
done | uniq | grep -v '\$'
this will read the file passed as argument line by line, and then process to try and print each line twice:
- once without substitution
- once with substitution of the variables.
then remove the duplicate lines
then remove the lines containing visible variables ($)
Yes eval should be used carefully, but it provided me this simple oneliner for my problem. Below is an example using your filename:
eval "echo \"$(<Text_File.msh)\""
I use printf instead of echo for my own purposes, but that should do the trick. Thank you abyss.7 providing the link that solve my problem. Hope it helps.
Create an ascii file test.txt with the following content:
Try to replace this ${myTestVariable1}
bla bla
....
Now create a file “sub.sed” containing variable names, eg
's,${myTestVariable1},'"${myTestVariable1}"',g;
s,${myTestVariable2},'"${myTestVariable2}"',g;
s,${myTestVariable3},'"${myTestVariable3}"',g;
s,${myTestVariable4},'"${myTestVariable4}"',g'
Open a terminal move to the folder containing test.txt and sub.sed.
Define the value of the varible to be replaced
myTestVariable1=SomeNewText
Now call sed to replace that variable
sed "$(eval echo $(cat sub.sed))" test.txt > test2.txt
The output will be
$cat test2.txt
Try to replace this SomeNewText
bla bla
....
#logfiles.list:
$EAMSROOT/var/log/LinuxOSAgent.log
$EAMSROOT/var/log/PanacesServer.log
$EAMSROOT/var/log/PanacesStrutsGUI.log
#My Program:
cat logfiles.list | while read line
do
eval Eline=$line
echo $Eline
done
I have a loop in a bash file to show me all of the files in a directory, each as its own variable. I need to take that variable (filename) and parse out only a section of it.
Example:
92378478234978ehbWHATIWANT#98712398712398723
Now, assuming "ehb" and the pound symbol never change, how can I just capture WHATIWANT into its own variable?
So far I have:
#!/bin/bash
for FILENAME in `dir -d *` ; do
done
You can use sed to edit out the parts you don't want.
want=$(echo "$FILENAME" | sed -e 's/.*ehb\(.*\)#.*/\1/')
Or you can use Bash's parameter expansion to strip out the tail and head.
want=${FILENAME%#*}; want=${want#*ehb}
One possibility:
for i in '92378478234978ehbWHATIWANT#98712398712398723' ; do
j=$(echo $i | sed -e 's/^.*ehb//' -e 's/#.*$//')
echo $j
done
produces:
WHATIWANT
using only the bash shell, no need external tools
$ string=92378478234978ehbWHATIWANT#98712398712398723
$ echo ${string#*ehb}
WHATIWANT#98712398712398723
$ string=${string#*ehb}
$ echo ${string%#*}
WHATIWANT