Shell script to check if file exists - shell

I'm trying to write a simple script that will tell me if a filename exist in $Temp that starts with the string "Test".
For example, I have these files
Test1989.txt
Test1990.txt
Test1991.txt
Then I just want to echo that a file was found.
For example, something like this:
file="home/edward/bank1/fiche/Test*"
if test -s "$file"
then
echo "found one"
else
echo "found none"
fi
But this doesn't work.

One approach:
(
shopt -s nullglob
files=(/home/edward/bank1/fiche/Test*)
if [[ "${#files[#]}" -gt 0 ]] ; then
echo found one
else
echo found none
fi
)
Explanation:
shopt -s nullglob will cause /home/edward/bank1/fiche/Test* to expand to nothing if no file matches that pattern. (Without it, it will be left intact.)
( ... ) sets up a subshell, preventing shopt -s nullglob from "escaping".
files=(/home/edward/bank1/fiche/Test*) puts the file-list in an array named files. (Note that this is within the subshell only; files will not be accessible after the subshell exits.)
"${#files[#]}" is the number of elements in this array.
Edited to address subsequent question ("What if i also need to check that these files have data in them and are not zero byte files"):
For this version, we need to use -s (as you did in your question), which also tests for the file's existence, so there's no point using shopt -s nullglob anymore: if no file matches the pattern, then -s on the pattern will be false. So, we can write:
(
found_nonempty=''
for file in /home/edward/bank1/fiche/Test* ; do
if [[ -s "$file" ]] ; then
found_nonempty=1
fi
done
if [[ "$found_nonempty" ]] ; then
echo found one
else
echo found none
fi
)
(Here the ( ... ) is to prevent file and found_file from "escaping".)

You have to understand how Unix interprets your input.
The standard Unix shell interpolates environment variables, and what are called globs before it passes the parameters to your program. This is a bit different from Windows which makes the program interpret the expansion.
Try this:
$ echo *
This will echo all the files and directories in your current directory. Before the echo command acts, the shell interpolates the * and expands it, then passes that expanded parameter back to your command. You can see it in action by doing this:
$ set -xv
$ echo *
$ set +xv
The set -xv turns on xtrace and verbose. Verbose echoes the command as entered, and xtrace echos the command that will be executed (that is, after the shell expansion).
Now try this:
$ echo "*"
Note that putting something inside quotes hides the glob expression from the shell, and the shell cannot expand it. Try this:
$ foo="this is the value of foo"
$ echo $foo
$ echo "$foo"
$ echo '$foo'
Note that the shell can still expand environment variables inside double quotes, but not in single quotes.
Now let's look at your statement:
file="home/edward/bank1/fiche/Test*"
The double quotes prevent the shell from expanding the glob expression, so file is equal to the literal home/edward/bank1/finche/Test*. Therefore, you need to do this:
file=/home/edward/bank1/fiche/Test*
The lack of quotes (and the introductory slash which is important!) will now make file equal to all files that match that expression. (There might be more than one!). If there are no files, depending upon the shell, and its settings, the shell may simply set file to that literal string anyway.
You certainly have the right idea:
file=/home/edward/bank1/fiche/Test*
if test -s $file
then
echo "found one"
else
echo "found none"
fi
However, you still might get found none returned if there is more than one file. Instead, you might get an error in your test command because there are too many parameters.
One way to get around this might be:
if ls /home/edward/bank1/finche/Test* > /dev/null 2>&1
then
echo "There is at least one match (maybe more)!"
else
echo "No files found"
fi
In this case, I'm taking advantage of the exit code of the ls command. If ls finds one file it can access, it returns a zero exit code. If it can't find one matching file, it returns a non-zero exit code. The if command merely executes a command, and then if the command returns a zero, it assumes the if statement as true and executes the if clause. If the command returns a non-zero value, the if statement is assumed to be false, and the else clause (if one is available) is executed.
The test command works in a similar fashion. If the test is true, the test command returns a zero. Otherwise, the test command returns a non-zero value. This works great with the if command. In fact, there's an alias to the test command. Try this:
$ ls -li /bin/test /bin/[
The i prints out the inode. The inode is the real ID of the file. Files with the same ID are the same file. You can see that /bin/test and /bin/[ are the same command. This makes the following two commands the same:
if test -s $file
then
echo "The file exists"
fi
if [ -s $file ]
then
echo "The file exists"
fi

You can do it in one line:
ls /home/edward/bank1/fiche/Test* >/dev/null 2>&1 && echo "found one" || echo "found none"
To understand what it does you have to decompose the command and have a basic awareness of boolean logic.
Directly from bash man page:
[...]
expression1 && expression2
True if both expression1 and expression2 are true.
expression1 || expression2
True if either expression1 or expression2 is true.
[...]
In the shell (and in general in unix world), the boolean true is a program that exits with status 0.
ls tries to list the pattern, if it succeed (meaning the pattern exists) it exits with status 0, 2 otherwise (have a look at ls man page for details).
In our case there are actually 3 expressions, for the sake of clarity I will put parenthesis, although they are not needed because && has precedence on ||:
(expression1 && expression2) || expression3
so if expression1 is true (ie: ls found the pattern) it evaluates expression2 (which is just an echo and will exit with status 0). In this case expression3 is never evaluate because what's on the left site of || is already true and it would be a waste of resources trying to evaluate what's on the right.
Otherwise, if expression1 is false, expression2 is not evaluated but in this case expression3 is.

for entry in "/home/loc/etc/"/*
do
if [ -s /home/loc/etc/$entry ]
then
echo "$entry File is available"
else
echo "$entry File is not available"
fi
done
Hope it helps

The following script will help u to go to a process if that script exist in a specified variable,
cat > waitfor.csh
#!/bin/csh
while !( -e $1 )
sleep 10m
end
ctrl+D
here -e is for working with files,
$1 is a shell variable,
sleep for 10 minutes
u can execute the script by ./waitfor.csh ./temp ; echo "the file exits"

One liner to check file exist or not -
awk 'BEGIN {print getline < "file.txt" < 0 ? "File does not exist" : "File Exists"}'

Wildcards aren't expanded inside quoted strings. And when wildcard is expanded, it's returned unchanged if there are no matches, it doesn't expand into an empty string. Try:
output="$(ls home/edward/bank1/fiche/Test* 2>/dev/null)"
if [ -n "$output" ]
then echo "Found one"
else echo "Found none"
fi
If the wildcard expanded to filenames, ls will list them on stdout; otherwise it will print an error on stderr, and nothing on stdout. The contents of stdout are assigned to output.
if [ -n "$output" ] tests whether $output contains anything.
Another way to write this would be:
if [ $(ls home/edward/bank1/fiche/Test* 2>/dev/null | wc -l) -gt 0 ]

Related

Bash script doesn't continue when condition fulfilled

To check the validity of lines in a file I'm using a condition which is met when egrep -v does NOT return an empty result. When there are invalid lines, then this works fine (i.e. the conditional block is executed), but when every line is valid then the script ends without further processing.
Script:
INVALID_HOSTS=$(egrep -v ${IP_REGEX} hosts)
if [[ ! -z "${INVALID_HOSTS}" ]]; then
echo "Invalid hosts:"
for entry in ${INVALID_HOSTS}
do echo ${entry}
done
exit_with_error_msg "hosts file contains invalid hosts (Pattern must be: \"\d+.\d+.\d+.\d+:\d+\"), exiting"
else
echo "all cool"
fi
echo "after if-else"
So when there are no invalid lines then neither the echo "all cool" nor echo "after if-else" get executed. The script just stops and returns to the shell.
When set -x is enabled, then it prints:
++ egrep -v '^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?):([1-9]|[1-5]?[0-9]{2,4}|6[1-4][0-9]{3}|65[1-4][0-9]{2}|655[1-2][0-9]|6553[1-5])$' hosts
+ INVALID_HOSTS=
Playing around with it I'm sure that it's about the if [[ ! -z "${INVALID_HOSTS}" ]]; then, but my bash wizardry is not strong enough to overcome this magical barrier.
Thanks for any help!
This is a bit long for a comment. I'll start it as an answer and we can work our way through further details or I can scrap it entirely if not helpful. I'll make some assumptions and let us see if it hits the spot.
For starters, you do indeed use the value further, so command expansion into a variable is not entirely useless, but otherwise it's much easier to determine match (or lack thereof) of grep through it's return value. If anything matched (output would be non-empty), it returns (shell true) value of 0, otherwise it returns false (in this case 1). Not to mention the ! -z test notation should really be -n if used at all.
And this is where I'd start assuming a bit. I suspect this is not your entire script and you have errexit option turned on in that shell session (or through rc file in general). Either by means of set -o errexit or set -e or running bash with -e option. Since grep not matching anything returns as failed, your shell (script execution) would terminate after having encountered a failing command.
Observe the difference between:
$ bash -ec 'grep "BOGUS" /etc/fstab ; echo "$?"'
$ bash -c 'grep "BOGUS" /etc/fstab ; echo "$?"'
1
With errexit, bash terminates after grep has "failed" and we never even reach the echo.
Since the assumption has proven to be correct, small extension. If errexit is what you want, you'd need to either change the option value before/after a command you want to be able to fail (return non-zero value) without affecting your script:
set +o errexit
grep THIS_COULD_NOT_MATCH...
set -o errexit
Or you can ignore return value of individual commands by ensuring their success:
grep THIS_COULD_NOT_MATCH... || true
You can also still use potentially "failing" commands safely in conditionals (such as if) without terminating your shell.

meaning of read and -d and $'\0' in bash [duplicate]

I do not understand this line of shell script. Doesn't the while statement need a 'test' or [ ] or [[ ]] expression that will set $? to 1 or 0? How does
while IFS= read -r -d $'\0'; do ...; done
do that? Any help understanding the syntax here is greatly appreciated.
In Bash, varname=value command runs command with the environment variable varname set to value (and all other environment variables inherited normally). So IFS= read -r -d $'\0' runs the command read -r -d $'\0' with the environment variable IFS set to the empty string (meaning no field separators).
Since read returns success (i.e., sets $? to 0) whenever it successfully reads input and doesn't encounter end-of-file, the overall effect is to loop over a set of NUL-separated records (saved in the variable REPLY).
Doesn't the while statement need a 'test' or [ ] or [[ ]] expression that will set $? to 1 or 0?
test and [ ... ] and [[ ... ]] aren't actually expressions, but commands. In Bash, every command returns either success (setting $? to 0) or failure (setting $? to a non-zero value, often 1).
(By the way, as nosid notes in a comment above, -d $'\0' is equivalent to -d ''. Bash variables are internally represented as C-style/NUL-terminated strings, so you can't really include a NUL in a string; e.g., echo $'a\0b' just prints a.)
I do not understand this line of shell script. Doesn't the while statement need a 'test' or [ ] or [[ ]] expression that will set $? to 1 or 0? How does [the statement] do that?
Everyone is answering about the -d and IFS= parameter, but no one answered your question about test and [[..]].
All standard Unix commands will return a zero value on success and a non-zero value on exit. The while and if statements use that value to determine whether or not to continue to loop:
Let's say you do the following:
$ touch foo.txt #Create a "foo.txt" file
$ ls foo.txt
foo.txt
$ echo $?
0
$
Notice that the ls command returned a zero on exit. Now try this:
$ rm foo.txt
$ ls foo.txt
ls: foo.txt: No such file or directory
$ echo $?
1
$
If foo.txt doesn't exist, the ls command returns a 1 on exit.
The if and while commands operate on that.
Go into another terminal window and create a foo.txt file. Then return to the original window and try this:
$ while ls foo.txt > /dev/null
> do
> echo "The file foo.txt exists"
> sleep 2
> done
The file foo.txt exists
The file foo.txt exists
The file foo.txt exists
...
Every two seconds, the while will loop through checking for the existence of foo.txt and print out that statement. Now, go back to the second terminal window and delete foo.txt. If you go back to the first terminal window, you'll see that the while loop ended.
You can do the same with the if:
$ if ls foo.txt > /dev/null
> then
> echo "foo.txt exists"
> else
> echo "Whoops, it's gone!"
> fi
This if statement will print either foo.txt exists or Whoops, it's gone! depending whether or not a file called foo.txt exists in the directory.
Notice that if and while work upon the return exit value of the command. There's no need for a test at all.
So why the test, [...], and [[...]] syntax?
Let's say you want to find out whether $foo is equal to $bar. That's where test comes into play. Try this:
$ foo=42
$ bar=42
$ test "$foo" -eq "$bar"
$ echo $?
0
$ bar=0
$ test "$foo" -eq "$bar"
$ echo $?
1
All the test command does is return a zero on true tests and a non-zero value on false tests. This allows you to use the test command in if and while statements.
$ if test $foo -eq $bar
> then
> echo "foo is equal to bar"
> else
> echo "foo and bar are different"
> fi
foo and bar are different
$
So what's [...]?
$ ls -il /bin/test /bin/[
10958 -rwxr-xr-x 2 root wheel 18576 May 28 22:27 /bin/[
10958 -rwxr-xr-x 2 root wheel 18576 May 28 22:27 /bin/test
That first column is the inode. If two files have the same inode, they are hard linked and refer to the same file. Notice that they're both 18,576 bytes on my system.
The [ is just another way to do the test command itself. Both of these lines are the same:
if test $foo -eq $bar
if [ $foo -eq $bar ]
The [ was created to make shell scripts look a wee bit nicer. However, underneath, it's running a test command which will return either a zero or non-zero value that if and while can use.
By the way, in Kornshell and BASH, both [ and test are builtin to the shell. But, they do reference the same builtin command.
The [[...]] is a special version of the test that first appeared in Kornshell. It allows for the use of pattern matches and it's a bit faster since it doesn't spawn another process to run the test command.
Code explanations :
IFS= set IFS to null (input fiels separator)
read command that reads the current line with the following options :
-r : do not allow backslashes to escape any characters
-d continue until the first character of DELIM is read, rather than newline
$'\0' zero byte (the ASCII NUL character)
See
help read
No. The test command and [[ ...]] expression are merely constructs that supply an exit status (the only thing while cares about) that reflects the truth of the conditional expression it contains.
For example, the exit status of [[ $foo = bar ]] is 0 (success) if $foo has the value bar, and a non-zero exist status (specifically 1) otherwise.
A while loop executes its body as long as the exit status of the command following the while keyword is 0. In this code, read has a 0 exit status as long as it can read something from its input.

while IFS= read -r -d $'\0' file ... explanation

I do not understand this line of shell script. Doesn't the while statement need a 'test' or [ ] or [[ ]] expression that will set $? to 1 or 0? How does
while IFS= read -r -d $'\0'; do ...; done
do that? Any help understanding the syntax here is greatly appreciated.
In Bash, varname=value command runs command with the environment variable varname set to value (and all other environment variables inherited normally). So IFS= read -r -d $'\0' runs the command read -r -d $'\0' with the environment variable IFS set to the empty string (meaning no field separators).
Since read returns success (i.e., sets $? to 0) whenever it successfully reads input and doesn't encounter end-of-file, the overall effect is to loop over a set of NUL-separated records (saved in the variable REPLY).
Doesn't the while statement need a 'test' or [ ] or [[ ]] expression that will set $? to 1 or 0?
test and [ ... ] and [[ ... ]] aren't actually expressions, but commands. In Bash, every command returns either success (setting $? to 0) or failure (setting $? to a non-zero value, often 1).
(By the way, as nosid notes in a comment above, -d $'\0' is equivalent to -d ''. Bash variables are internally represented as C-style/NUL-terminated strings, so you can't really include a NUL in a string; e.g., echo $'a\0b' just prints a.)
I do not understand this line of shell script. Doesn't the while statement need a 'test' or [ ] or [[ ]] expression that will set $? to 1 or 0? How does [the statement] do that?
Everyone is answering about the -d and IFS= parameter, but no one answered your question about test and [[..]].
All standard Unix commands will return a zero value on success and a non-zero value on exit. The while and if statements use that value to determine whether or not to continue to loop:
Let's say you do the following:
$ touch foo.txt #Create a "foo.txt" file
$ ls foo.txt
foo.txt
$ echo $?
0
$
Notice that the ls command returned a zero on exit. Now try this:
$ rm foo.txt
$ ls foo.txt
ls: foo.txt: No such file or directory
$ echo $?
1
$
If foo.txt doesn't exist, the ls command returns a 1 on exit.
The if and while commands operate on that.
Go into another terminal window and create a foo.txt file. Then return to the original window and try this:
$ while ls foo.txt > /dev/null
> do
> echo "The file foo.txt exists"
> sleep 2
> done
The file foo.txt exists
The file foo.txt exists
The file foo.txt exists
...
Every two seconds, the while will loop through checking for the existence of foo.txt and print out that statement. Now, go back to the second terminal window and delete foo.txt. If you go back to the first terminal window, you'll see that the while loop ended.
You can do the same with the if:
$ if ls foo.txt > /dev/null
> then
> echo "foo.txt exists"
> else
> echo "Whoops, it's gone!"
> fi
This if statement will print either foo.txt exists or Whoops, it's gone! depending whether or not a file called foo.txt exists in the directory.
Notice that if and while work upon the return exit value of the command. There's no need for a test at all.
So why the test, [...], and [[...]] syntax?
Let's say you want to find out whether $foo is equal to $bar. That's where test comes into play. Try this:
$ foo=42
$ bar=42
$ test "$foo" -eq "$bar"
$ echo $?
0
$ bar=0
$ test "$foo" -eq "$bar"
$ echo $?
1
All the test command does is return a zero on true tests and a non-zero value on false tests. This allows you to use the test command in if and while statements.
$ if test $foo -eq $bar
> then
> echo "foo is equal to bar"
> else
> echo "foo and bar are different"
> fi
foo and bar are different
$
So what's [...]?
$ ls -il /bin/test /bin/[
10958 -rwxr-xr-x 2 root wheel 18576 May 28 22:27 /bin/[
10958 -rwxr-xr-x 2 root wheel 18576 May 28 22:27 /bin/test
That first column is the inode. If two files have the same inode, they are hard linked and refer to the same file. Notice that they're both 18,576 bytes on my system.
The [ is just another way to do the test command itself. Both of these lines are the same:
if test $foo -eq $bar
if [ $foo -eq $bar ]
The [ was created to make shell scripts look a wee bit nicer. However, underneath, it's running a test command which will return either a zero or non-zero value that if and while can use.
By the way, in Kornshell and BASH, both [ and test are builtin to the shell. But, they do reference the same builtin command.
The [[...]] is a special version of the test that first appeared in Kornshell. It allows for the use of pattern matches and it's a bit faster since it doesn't spawn another process to run the test command.
Code explanations :
IFS= set IFS to null (input fiels separator)
read command that reads the current line with the following options :
-r : do not allow backslashes to escape any characters
-d continue until the first character of DELIM is read, rather than newline
$'\0' zero byte (the ASCII NUL character)
See
help read
No. The test command and [[ ...]] expression are merely constructs that supply an exit status (the only thing while cares about) that reflects the truth of the conditional expression it contains.
For example, the exit status of [[ $foo = bar ]] is 0 (success) if $foo has the value bar, and a non-zero exist status (specifically 1) otherwise.
A while loop executes its body as long as the exit status of the command following the while keyword is 0. In this code, read has a 0 exit status as long as it can read something from its input.

Bash Scripting: greping the parameter

Here is my problem. I have script and I want to make sure that the parameter that is entered when the script is called matches a variable name inside the script.
For example:
./valid foo <- being the script call
#!/bin/bash
PARAM=$1
VAR=/foo/
if grep -c $PARAM == $VAR
then
echo yes
fi
echo no
I am having the worst time using grep, I'm not sure how to use it properly inside of a script and after scouring the internet I think I need some specific feedback on my problem.
Thanks,
EA
This is not robust, but you can do:
if echo "$VAR" | grep -q "$PARAM"; then
It is probably better to simply do:
if test "$VAR" = "$PARAM"; then
If you are trying to match a regex, bash allows:
if [[ "$VAR" =~ "$PARAM" ]]; then
to match the fixed string $VAR against the regex $PARAM. If $VAR is the regex, you should reverse the order of the arguments. (That is, [[ "$PARAM" =~ "$VAR ]].)
You could search inside your script, since the declaration is the name followed by an equal sign:
if egrep "^\s*$PARAM=" $0
then
echo yes
else
echo no
fi
to list variables use
set -o posix ; set
the posix thingie prevents listing of functions.
In order to isolate parameters local to script, run it from the shell and store the result,
then run it from your script and compare output
(set -o posix ; set) >/tmp/variables.before
(set -o posix ; set) >/tmp/variables.after

shell script working fine on one server but not on another

the following script is working fine on one server but on the other it gives an error
#!/bin/bash
processLine(){
line="$#" # get the complete first line which is the complete script path
name_of_file=$(basename "$line" ".php") # seperate from the path the name of file excluding extension
ps aux | grep -v grep | grep -q "$line" || ( nohup php -f "$line" > /var/log/iphorex/$name_of_file.log & )
}
FILE=""
if [ "$1" == "" ]; then
FILE="/var/www/iphorex/live/infi_script.txt"
else
FILE="$1"
# make sure file exist and readable
if [ ! -f $FILE ]; then
echo "$FILE : does not exists. Script will terminate now."
exit 1
elif [ ! -r $FILE ]; then
echo "$FILE: can not be read. Script will terminate now."
exit 2
fi
fi
# read $FILE using the file descriptors
# $ifs is a shell variable. Varies from version to version. known as internal file seperator.
# Set loop separator to end of line
BACKUPIFS=$IFS
#use a temp. variable such that $ifs can be restored later.
IFS=$(echo -en "\n")
exec 3<&0
exec 0<"$FILE"
while read -r line
do
# use $line variable to process line in processLine() function
processLine $line
done
exec 0<&3
# restore $IFS which was used to determine what the field separators are
IFS=$BAKCUPIFS
exit 0
i am just trying to read a file containing path of various scripts and then checking whether those scripts are already running and if not running them. The file /var/www/iphorex/live/infi_script.txt is definitely present. I get the following error on my amazon server-
[: 24: unexpected operator
infinity.sh: 32: cannot open : No such file
Thanks for your helps in advance.
You should just initialize file with
FILE=${1:-/var/www/iphorex/live/infi_script.txt}
and then skip the existence check. If the file
does not exist or is not readable, the exec 0< will
fail with a reasonable error message (there's no point
in you trying to guess what the error message will be,
just let the shell report the error.)
I think the problem is that the shell on the failing server
does not like "==" in the equality test. (Many implementations
of test only accept one '=', but I thought even older bash
had a builtin that accepted two '==' so I might be way off base.)
I would simply eliminate your lines from FILE="" down to
the end of the existence check and replace them with the
assignment above, letting the shell's standard default
mechanism work for you.
Note that if you do eliminate the existence check, you'll want
to either add
set -e
near the top of the script, or add a check on the exec:
exec 0<"$FILE" || exit 1
so that the script does not continue if the file is not usable.
For bash (and ksh and others), you want [[ "$x" == "$y" ]] with double brackets. That uses the built-in expression handling. A single bracket calls out to the test executable which is probably barfing on the ==.
Also, you can use [[ -z "$x" ]] to test for zero-length strings, instead of comparing to the empty string. See "CONDITIONAL EXPRESSIONS" in your bash manual.

Resources