I am new to bash scripting and I have to create this script that takes 3 directories as arguments and copies in the third one all the files in the first one that are NOT in the second one.
I did it like this:
#!/bin/bash
if [ -d $1 && -d $2 && -d $3 ]; then
for FILE in [ ls $1 ]; do
if ! [ find $2 -name $FILE ]; then
cp $FILE $3
done
else echo "Error: one or more directories are not present"
fi
The error I get when I try to execute it is: "line 7: syntax error near unexpected token `done' "
I don't really know how to make it work!
Also even if I'm using #!/bin/bash I still have to explicitly call bash when trying to execute, otherwise it says that executing is not permitted, anybody knows why?
Thanks in advance :)
Couple of suggestions :
No harm double quoting variables
cp "$FILE" "$3" # prevents wordsplitting, helps you filenames with spaces
for statement fails for the fundamental reason -bad syntax- it should've been:
for FILE in ls "$1";
But then, never parse ls output. Check [ this ].
for FILE in ls "$1"; #drastic
Instead of the for-loop in step2 use a find-while-read combination:
find "$1" -type f -print0 | while read -rd'' filename #-type f for files
do
#something with $filename
done
Use lowercase variable names for your script as uppercase variables are reserved for the system. Check [this].
Use tools like [ shellcheck ] to improve script quality.
Edit
Since you have mentioned the input directories contain only files, my alternative approach would be
[[ -d "$1" && -d "$2" && -d "$3" ]] && for filename in "$1"/*
do
[ ! -e "$2/${filename##*/}" ] && cp "$filename" "$3"
done
If you are baffled by ${filename##*/} check [ shell parameter expansion ].
Sidenote: In linux, although discouraged it not uncommon to have non-standard filenames like file name.
Courtesy: #chepner & #mklement0 for their comments that greatly improved this answer :)
Your script:
if ...; then
for ...; do
if ...; then
...
done
else
...
fi
Fixed structure:
if ...; then
for ...; do
if ...; then
...
fi # <-- missing
done
else
...
fi
If you want the script executable, then make it so:
$ chmod +x script.sh
Notice that you also have other problems in you script. It is better written as
dir1="$1"
dir2="$2"
dir3="$3"
for f in "$dir1"/*; do
if [ ! -f "$dir2/$(basename "$f")" ]; then
cp "$f" "$dir3"
fi
done
this is not totally correct:
for FILE in $(ls $1); do
< whatever you do here >
done
There is a big problem with that loop if in that folder there is a filename like this: 'I am a filename with spaces.txt'.
Instead of that loop try this:
for FILE in "$1"/*; do
echo "$FILE"
done
Also you have to close every if statement with fi.
Another thing, if you are using BASH ( #!/usr/bin/env bash ), it is highly recommended to use double brackets in your test conditions:
if [[ test ]]; then
...
fi
For example:
$ a='foo bar'
$ if [[ $a == 'foo bar' ]]; then
> echo "it's ok"
> fi
it's ok
However, this:
$ if [ $a == 'foo bar' ]; then
> echo "it's ok";
> fi
bash: [: too many arguments
You've forgot fi after the innermost if.
Additionally, neither square brackets nor find do work this way. This one does what your script (as it is now) is intended to on my PC:
#!/bin/bash
if [[ -d "$1" && -d "$2" && -d "$3" ]] ; then
ls -1 "$1" | while read FILE ; do
ls "$2/$FILE" >/dev/null 2>&1 || cp "$1/$FILE" "$3"
done
else echo "Error: one or more directories are not present"
fi
Note that after a single run, when $2 and $3 refer to different directories, those files are still not present in $2, so next time you run the script they will be copied once more despite they already are present in $3.
Related
I'm trying to iterate over a folder, running a grep on each file, and putting them into separate files, tagged with a .res extension. Here's what I have so far....
#!/bin/bash
directory=$(pwd)
searchterms="searchterms.txt"
extension=".end"
usage() {
echo "usage: fmat [[[-f file ] [-d directory ] [-e ext]] | [-h]]"
echo " file - text file containing a return-delimited list of materials"
echo " directory - directory to process"
echo " ext - file extension of files to process"
echo ""
}
while [ "$1" != "" ]; do
case $1 in
-d | --directory ) shift
directory=$1
;;
-f | --file ) shift
searchterms=$1
;;
-e | --extension ) shift
extension=$1
;;
-h | --help ) usage
exit
;;
* ) usage
exit 1
esac
shift
done
if [ ! -d "$directory" ]; then
echo "Sorry, the directory '$directory' does not exist"
exit 1
fi
if [ ! -f "$searchterms" ]; then
echo "Sorry, the searchterms file '$searchterms' does not exist"
exit 1
fi
echo "Searching '$directory' ..."
for file in "${directory}/*"; do
printf "File: %s\n" ${file}
[ -e "$file" ] || continue
printf "%s\n" ${file}
if [ ${file: -3} == ${extension} ]; then
printf "%s will be processed\n" ${file}
#
# lots of processing here
#
fi
done
I know that it's down to my poor understanding of of globbing... but I can't get the test on the extension to work.
Essentially, I want to be able to specify a source directory, a file with search terms, and an extension to search for.
NOW, I realise there may be quicker ways to do this, e.g.
grep -f searchterms.txt *.end > allchanges.end.res
but I may have other processing I need to do to the files, and I want to save them into separate files: so bing.end, bong.end, would be grep'ed into bing.end.res, bong.end.res .
Please let me know, just how stupid I'm being ;-)
Just for completeness sake, here's the last part, working, thanks to #chepner and #Gordon Davisson :
echo "Searching '$directory' ..."
for file in "${directory}"/*; do
[ -e "$file" ] || continue
# show which files will be processed
if [[ $file = *.${extension#.} ]]; then
printf "Processing %s \n" "$file"
head -n 1 "${file}" > "${file}.res"
grep -f $searchterms "${file}" >> "${file}.res"
fi
done
You just need to leave the * out of the quotes, so that it isn't treated as a literal *:
for file in "${directory}"/*; do
Unlike most languages, the quotes don't define a string (as everything in bash is already a string: it's the only data type). They simply escape each character inside the quotes. "foo" is exactly the same as \f\o\o, which (because escaping most characters doesn't really have any effect) is the same as foo. Quoted or not, all characters not separated by word-splitting characters are part of the same word.
http://shellcheck.net will catch this, although not with the most useful error message. (It will also catch the other parameter expansions that you did not quote but should.)
I have a placeholder file and would like to override it with the output of a command only if the output is not zero length. I guess I could do FOO="$(command)" then [-z $FOO]. Is there a better way?
There are different ways, but I don't know about "better". You could block on a read and only set up the redirection once some data come through:
cmd | { read j && { echo "$j"; cat; } > placeholder; }
(Note, if your command generates output but no newlines, this will ignore the data.)
If you don't need the output of the command, you can run: [ -z "$(command)" ] directly. For example, the following prints "empty":
#!/bin/sh
if [ -z "$(echo -n)" ]; then
echo "empty"
fi
Your example with -z will work, but you can also check if a variable is non-empty just with
[ "$var" ]
So, a simple solution could look like this:
#!/bin/sh
output="$( command )"
[ "$output" ] && echo "$output" > your_file.txt
If you are going to do this type of thing a lot, better make a function:
write_if_non_zero(){
local msg=$1
local file=$2
if [[ ! -z "$msg" ]]; then
echo "$msg" > "$file"
fi
}
Then
write_if_non_zero "$FOO" "$FILE"
I have a script that must be able to accept both by files and stdin on the first argument. Then if more or less than 1 arguments, reject them
The goal that I'm trying to accomplish is able to accpet using this format
./myscript myfile
AND
./myscript < myfile
What I have so far is
if [ "$#" -eq 1 ]; then #check argument
if [ -t 0 ]; then #check whether input from keyboard (read from github)
VAR=${1:-/dev/stdin} #get value to VAR
#then do stuff here!!
else #if not input from keyboard
VAR=$1
if [ ! -f "$VAR" ]; then #check whether file readable
echo "ERROR!"
else
#do stuff heree!!!
fi
fi
fi
The PROBLEM is when I tried to say
./myscript < myfile
it prints
ERROR!
I dont know whether this is the correct way to do this, I really appreciate for suggestion or the correct code for my problem. Thank you
#!/bin/bash
# if nothing passed in command line pass "/dev/stdin" to myself
# so all below code can be made branch-free
[[ ${#} -gt 0 ]] || set -- /dev/stdin
# loop through the command line arguments, treating them as file names
for f in "$#"; do
echo $f
[[ -r $f ]] && while read line; do echo 'echo:' $line; done < $f
done
Examples:
$ args.sh < input.txt
$ args.sh input.txt
$ cat input.txt | args.sh
I have to write a bash script which will count all the commands in a text file. Arguments to a script are -p, -n num, and a file. This means that commands like:
script.sh -n 3 -p file.txt
script -p -n 3 file.txt
and similar are all legit.
However, I have to echo an error for any commands that are not similar to this: script.sh -n -k file.txt for example.
Here is a link to my code.
I managed to make it work, but it is way too long and redundant. Is there a way I can do this in a short way?
You may want to have a look at one of the following standard commands:
getopts is a Bash builtin. It is newer and simple to use, but does not support long options (--option).
getopt is an external program which may involve a little more glue code. There are different implementations. getopt usually supports long options.
This is a small getopts example (modified one of the examples from this external site):
#!/bin/bash
flag=off
dir=
# iterate over each option with getopts:
while getopts fd: opt
do
case "$opt" in
f) flag=on;;
d) dir="$OPTARG";;
*) echo >&2 "usage: $0 [-f] [-d directory] [file ...]"
exit 1;;
esac
done
# remove all positional pararmeters we already
# handled from the command line:
shift $(( expr $OPTIND - 1 ))
# main part of your program, remaining arguments are now in
# $# resp. $0, $1, ...
I'd like to suggest another snippet that is a lot simpler to read than yours, because it exactly depicts the only two valid cases you specified in your comment:
If I want to "call" my script it has to look like this: script.sh -n +number -p file.txt. file.txt must be the last argument, however, -n and -p can be switched.
So the cases are ($0 to $4):
script.sh -n +number -p file.txt
script.sh -p -n +number file.txt
It uses only if and Bash's logical operators:
#!/bin/bash
if ! { [[ "$1" = "-n" ]] && [[ "$2" =~ ^-[0-9]+$ ]] && [[ "$3" = "-p" ]] && [[ "$4" =~ ".txt"$ ]] ; } &&
! { [[ "$2" = "-n" ]] && [[ "$3" =~ ^-[0-9]+$ ]] && [[ "$1" = "-p" ]] && [[ "$4" =~ ".txt"$ ]] ; }
then
echo "Error" && exit 1
fi
Notes:
The group ({, }) syntax expects a ; at the end of its list.
You have to use a regex to check for *.txt
The number regex you gave will require the number to start with a -, while in your specification you say +.
I need a script to keep polling "receive_dir" directory till "stopfile" get written in the directory.
This has to run despite empty directory.
So far i have this but fails if receive_dir is empty with no files with "unary operator expected". Help !!
#!/usr/bin/ksh
until [ $i = stopfile ]
do
for i in `ls receive_dir`; do
time=$(date +%m-%d-%Y-%H:%M:%S)
echo $time
echo $i;
done
done
This will do what you ask for (loop until the stop file exist). I added a "sleep 1" to lower resource usage. It's also good practice to use "#!/usr/bin/env ksh" as shebang.
#!/usr/bin/env ksh
until [ -e receive_dir/stopfile ]
do
time=$(date +%m-%d-%Y-%H:%M:%S)
echo $time
sleep 1
done
If you have empty dir, the
until [ $i = stopfile ]
is evaluated as
until [ = stopfile ]
what is ofcourse syntax error.
One comment: Never parse output from ls.
#!/bin/bash
do_something() {
echo $(date +%m-%d-%Y-%H:%M:%S) "$1"
}
dir="."
until [[ -f "$dir/stopfile" ]]
do
find "$dir" -print0 | while IFS= read -r -d '' filename
do
do_something "$filename"
done
done
or (much slower)
do_something() {
echo $(date +%m-%d-%Y-%H:%M:%S) "$1"
}
export -f do_something
dir="."
until [[ -f "$dir/stopfile" ]]
do
find "$dir" -exec bash -c 'do_something "{}"' \;
done
You're evaluating nothing, and the 'test' isn't able to evaluate it.
~> [ $empty_var = stopfile ]
-bash: [: =: unary operator expected
First, don't parse ls:
http://mywiki.wooledge.org/BashPitfalls#for_i_in_.24.28ls_.2A.mp3.29
EDIT: Part of the issue is your loop is actually doing the test, try something like this (assuming receive_dir is relative):
#user000001 is right; my original find example would suffer the same issue, so this:
for i in receive_dir/*
do
time=$(date +%m-%d-%Y-%H:%M:%S)
echo $time
echo $i
[ $i = stopfile ] && break
done
EDIT: Adding in another example based on your comment:
How about this...
FOUND="N"
while [ "${FOUND}" = "N" ]
do
for i in receive_dir/*
do
time=$(date +%m-%d-%Y-%H:%M:%S)
echo $time
echo $i
[ "$i" = stopfile ] && FOUND="Y"
done
sleep 60
done
Another option is to use inotifywait to monitor the status of the directory. For example, this script will run the loop until the file "stopfile" is touched.
until inotifywait "receive_dir" | grep "stopfile"
do
echo "running"
done
echo "done"
The advantage is that these is no busy loop, and that you don't have to repeatedly call the (potentially expensive) find command