In general I don't understand how to make most commands in a UNIX shell script do loop work the same as they work directly from the command line (using bash).
As a simple test, a script called looping.sh to execute an SQL script (what's in filelist.txt doesn't matter in this case):
for i in $(cat filelist.txt)
do $(sqlplus DB_USER/password#abc #test.sql)
done
results in
looping.sh: line 2: SQL*Plus:: command not found
for each line in filelist.txt. Other variations on the 2nd line don't work, like putting it in quotes etc.
Or, if filelist.txt has names of other sh scripts, let's say a single line in this case called_file1.sh and I want to execute it
for i in $(cat filelist.txt)
do exec $i
done
results in
: not found line 2: exec: called_file1.sh
The files are all in the same folder. I tried variations for the second line like /bin/sh $i, putting it in quotes and so on. What's the magic way to execute a command in the do loop?
$(...) takes the contents and runs it as a command and then returns the output from the command.
So when you write:
for i in $(cat filelist.txt)
do $(sqlplus DB_USER/password#abc #test.sql)
done
what the shell does when it hits the body of the loop is run sqlplus DB_USER/password#abc #test.sql and then it takes the output from that command (whatever it may be) and replaces the $(...) bit with it. So you end up with (not exactly since it happens again every loop but for sake of illustration) a loop that looks like this:
for i in $(cat filelist.txt)
do <output of 'sqlplus DB_USER/password#abc #test.sql' command>
done
and if that output isn't a valid shell command you are going to get an error.
The solution there is to not do that. You don't want the wrapping $() there at all.
for i in $(cat filelist.txt)
do sqlplus DB_USER/password#abc #test.sql
done
In your second example:
for i in $(cat filelist.txt)
do exec $i
done
you are telling the shell that the filename in $i is something that it should try to execute like a binary or executable shell script.
In your case two things are happening here. The filename in $i can't be found and (and this is harder to notice) the filename in $i contains a carriage-return at the end (probably a DOS line-ending file). (That's why the error message is a bit more confused then normal.) (I actually wonder about that since I wouldn't have expected that from an unquoted $i but from a quoted "$i" but I might just be wrong about that.)
So, for this case, you need to both strip the carriage-returns from the file (see point 1 of the "Before asking about problematic code" section of the bash tag info wiki for more about this) and then you need to make sure that filename is an executable script and you have the correct path to it.
Oh, also, exec never returns so that loop will only execute one file ever.
If you want multiple executions then drop exec.
That all being said you Don't Read Lines With For. See Bash FAQ 001 for how to correctly (and safely) read lines from a file.
Related
I have a simple Bash script:
#!/usr/bin/env bash
read X
echo "X=$X"
When I execute it with ./myscript.sh it works. But when I execute it with cat myscript.sh | bash it actually puts echo "X=$X" into $X.
So this script prints Hello World executed with cat myscript.sh | bash:
#!/usr/bin/env bash
read X
hello world
echo "$X"
What's the benefit of executing a script with cat myscript.sh | bash? Why doesn't do it the same things as if I execute it with ./myscript.sh?
How can I avoid Bash to execute line by line but execute all lines after the STDIN reached the end?
Instead of just running
read X
...instead replace it with...
read X </dev/tty || {
X="some default because we can't read from the TTY here"
}
...if you want to read from the console. Of course, this only works if you have a /dev/tty, but if you wanted to do something robust, you wouldn't be piping from curl into a shell. :)
Another alternative, of course, is to pass in your value of X on the command line.
curl https://some.place/with-untrusted-code-only-idiots-will-run-without-reading \
| bash -s "value of X here"
...and refer to "$1" in your script when you want X.
(By the way, I sure hope you're at least using SSL for this, rather than advising people to run code they download over plain HTTP with no out-of-band validation step. Lots of people do it, sure, but that's making sites they download from -- like rvm.io -- big targets. Big, easy-to-man-in-the-middle-or-DNS-hijack targets).
When you cat a script to bash the code to execute is coming from standard input.
Where does read read from? That's right also standard input. This is why you can cat input to programs that take standard input (like sed, awk, etc.).
So you are not running "a script" per-se when you do this. You are running a series of input lines.
Where would you like read to read data from in this setup?
You can manually do that (if you can define such a place). Alternatively you can stop running your script like this.
I have a text file of roughly 900 cURLs to run. They are pretty hairy, with tons of quotes, apostrophes and other special characters.
To run them I have been trying to create a bash script to loop through the list:
#!/bin/sh
OLDIFS=$IFS
IFS="&&&"
echo "getting started"
cat staging_curl_script|while read line
do
$line
done
echo "done"
Unfortunately I have had an unusual issue. commands that run fine in the command prompt are returning the "file name too long" error. I echoed out these commands from the script and compared them to the manually run command, and they are identical.
Any idea why I am seeing different results?
silly mistake here, needed bash -c "$line"
I'm trying to install RVM. There is a magical command line:
bash < <(curl -s https://rvm.io/install/rvm)
I know what bash and curl are. I know the first < is the I/O redirection. But what does <() syntax mean?
What's the difference between this command and
bash < `curl -s https://rvm.io/install/rvm`
?(the latter command doesn't work)
This is bash's process substitution.
The expression <(list) gets replaced by a file name, either a named FIFO or an entry under /dev/fd. So to actually redirect input from it, you have to use < <(list).
[edit; forgot to answer your second question]
The backticks are called "command substitution". Unlike process substitution, it is part of the POSIX shell specification (i.e., not a bash extension). The shell runs the command in the backticks and substitutes its output on the command line. So this would make sense:
cat < `echo /etc/termcap`
But this would not:
cat < `cat /etc/termcap`
The latter is similar to your example; it tries to use the (complex) output of a command as a file name from which to redirect standard input.
The others have already answered your question very nicely. I'll just add an example to build on them... 99% of the time when I personally use <(), it's to diff the output of two different commands in one shot. For instance,
diff <( some_command ) <( some_other_command )
The syntax for io redirection is
process < file
Hence you need whatever appears after the io redirect to be a filename.
The backtick expansion literally puts the results of the command into the command line. Thus,
`curl -s https://rvm.io/install/rvm`
expands to something like
#!/usr/bin/env bash ...
and the shell would be confused because it would see
bash < #...
instead of a filename.
the <() operator is process substitution, which spawns a new process to run the command within the (..). A new file or pipe is created that will capture the result. The fact that the arrow is pointing left <() instead of >() means that the output from the inner process will be written to the file, which can be read by the process.
In your case, bash < <(...) will be seen as something like bash < /dev/fd/100
If you actually want to see what is going on, run
echo <(curl -s https://rvm.io/install/rvm)
It is called Process Substitution.
I'm trying to read commands from a text file and execute each line from a bash script.
#!/bin/bash
while read line; do
$line
done < "commands.txt"
In some cases, if $line contains commands that are meant to run in background, eg command 2>&1 & they will not start in background, and will run in the current script context.
Any ideea why?
if all your commands are inside "commands.txt", essentially, you can call it a shell script. That's why you can either source it, or run it like normal, ie chmod u+x , then you can execute it using sh commands.txt
I don't have anything to add to ghostdog74's answer about the right way to do this, but I can cover why it's failing: The shell parses I/O redirections, backgrounding, and a bunch of other things before it does variable expansion, so by the time $line is replaced by command 2>&1 & it's too late to recognize 2>&1 and & as anything other than parameters to command.
You could improve this by using eval "$line" but even there you'll run into problems with multiline commands (e.g. while loops, if blocks, etc). The source and sh approaches don't have this problem.
I have a bash script that sources contents from another file. The contents of the other file are commands I would like to execute and compare the return value. Some of the commands are have multiple commands separated by either a semicolon (;) or by ampersands (&&) and I can't seem to make this work. To work on this, I created some test scripts as shown:
test.conf is the file being sourced by test
Example-1 (this works), My output is 2 seconds in difference
test.conf
CMD[1]="date"
test.sh
. test.conf
i=2
echo "$(${CMD[$i]})"
sleep 2
echo "$(${CMD[$i]})"
Example-2 (this does not work)
test.conf (same script as above)
CMD[1]="date;date"
Example-3 (tried this, it does not work either)
test.conf (same script as above)
CMD[1]="date && date"
I don't want my variable, CMD, to be inside tick marks because then, the commands would be executed at time of invocation of the source and I see no way of re-evaluating the variable.
This script essentially calls CMD on pass-1 to check something, if on pass-1 I get a false reading, I do some work in the script to correct the false reading and re-execute & re-evaluate the output of CMD; pass-2.
Here is an example. Here I'm checking to see if SSHD is running. If it's not running when I evaluate CMD[1] on pass-1, I will start it and re-evaluate CMD[1] again.
test.conf
CMD[1]=`pgrep -u root -d , sshd 1>/dev/null; echo $?`
So if I modify this for my test script, then test.conf becomes:
NOTE: Tick marks are not showing up but it's the key below the ~ mark on my keyboard.
CMD[1]=`date;date` or `date && date`
My script looks like this (to handle the tick marks)
. test.conf
i=2
echo "${CMD[$i]}"
sleep 2
echo "${CMD[$i]}"
I get the same date/time printed twice despite the 2 second delay. As such, CMD is not getting re-evaluate.
First of all, you should never use backticks unless you need to be compatible with an old shell that doesn't support $() - and only then.
Secondly, I don't understand why you're setting CMD[1] but then calling CMD[$i] with i set to 2.
Anyway, this is one way (and it's similar to part of Barry's answer):
CMD[1]='$(date;date)' # no backticks (remember - they carry Lime disease)
eval echo "${CMD[1]}" # or $i instead of 1
From the couple of lines of your question, I would have expected some approach like this:
#!/bin/bash
while read -r line; do
# munge $line
if eval "$line"; then
# success
else
# fail
fi
done
Where you have backticks in the source, you'll have to escape them to avoid evaluating them too early. Also, backticks aren't the only way to evaluate code - there is eval, as shown above. Maybe it's eval that you were looking for?
For example, this line:
CMD[1]=`pgrep -u root -d , sshd 1>/dev/null; echo $?`
Ought probably look more like this:
CMD[1]='`pgrep -u root -d , sshd 1>/dev/null; echo $?`'