I have a script, /home/user/me/my_script.sh that is supposed to loop over multiple directories and process files. My current working directory is /home/user/me. A call to ls -R yields:
./projects:
dir1 dir2 dir3
./projects/dir1:
image1.ntf points2.csv image1.img image1.hdr
./projects/dir2:
image2.ntf points2.csv image2.img image2.hdr
./projects/dir3:
image3.ntf points3.csv image3.img image3.hdr
I have this script:
#! /bin/bash -f
for $dir in $1*
do
echo $dir
set cmd = `/home/tools/tool.sh -i $dir/*.ntf -flag1 -flag2 -flag3 opt3`
$cmd
done
This is how it is run (from cwd /home/user/me) and the result:
bash-4.1$ ./myscript.sh projects/
projects/*
bash-4.1$
This is not the expected output.The expected output is:
bash-4.1$ ./myscript.sh projects/
projects/dir1
[output from tool.sh]
projects/dir2
[output from tool.sh]
projects/dir3
[output from tool.sh]
bash-4.1$
What should happen is the script should go into the first directory, find the *.ntf file and pass it to tool.sh. At that point I would start seeing output from that tool. I have run the tool on a single file:
bash-4.1$ /home/tools/tool.sh -i /home/user/me/projects/dir1/image1.ntf -flag1 -flag2 -flag3 opt3
[expected output from tool. lengthy.]
bash-4.1$
I have tried syntax found here: How to loop over directories in Linux? and here: Looping over directories in Bash
for $dir in /$1*/
do ...
Result:
bash-4.1$ ./myscript.sh projects/
/projects/*/
And:
for $dir in $1/*
do ...
Result:
bash-4.1$ ./myscript.sh projects
projects/*
I'm not sure how many other iterations of wildcard and slash I can come up with. What is the correct syntax?
First, you should remove flag -f in your shebang, because it utterly means:
$ man bash
[…]
-f Disable pathname expansion.
Second, there are some typical bug patterns: spaces missing around variables (write "$dir" to cope with directory names containing spaces), there is a spurious $ in your for line (write for dir in "$1"*) instead, the set line is incorrect (set is a shell builtin only used to change the configuration of the shell, e.g., set -x), according to your answer to #ghoti's question it seems that the $cmd line is unnecessary. Also, the backquotes syntax is deprecated and could have been replaced with cmd=$(/home/tools/tool.sh -i "$dir"/*.ntf -flag1 -flag2 -flag3 opt3).
This would lead to the following script:
#!/bin/bash
for dir in "$1"*
do
[[ -d "$dir" ]] || continue # only consider existing folders
printf "%s=%q\n" dir "$dir"
/home/tools/tool.sh -i "$dir"/*.ntf -flag1 -flag2 -flag3 opt3
done
As an aside, I would recommend to always run the ShellCheck static analyzer on your Bash scripts, in order to detect typical bugs and have feedback w.r.t. good practices. If you have a Linux distribution, it should be installable with your standard package manager.
Related
In shell scripts, our corporate coding standard requires using...
set -x
command
set +x
...for logging, rather than...
echo "doing command"
command
However, when a wildcard is part of the command, this can produce very verbose output.
For example...
for i in {1..10}; do touch $i.foo; done; # create 10 foo files
set -x # log command execution (stdout to be redirected to log file)
rm *.foo # delete foo files
set +x # end logging
...produces the output...
rm 10.foo 1.foo 2.foo 3.foo 4.foo 5.foo 6.foo 7.foo 8.foo 9.foo
Okay for 10 files, but not so great for 10,000.
The desired output is...
rm *.foo
My first thought was to put *.foo in quotes...
rm "*.foo"
However, that gives the error...
rm: cannot remove ‘*.foo’: No such file or directory
Is there a way, using set -x, to echo the command without expanding the wildcard?
For many cases, where simple '-x' or '-v' do not work (as per comments above), and staying within your coding standard (no separate echo), consider:
VAR=/tmp/123
$SHELL -cv "ls $VAR/*"
which will execute the command, but will log the command WITH variable substitution, command substitution, but WITHOUT wild-card substitution.
I have an rsync command in my csh script like this:
#! /bin/csh -f
set source_dir = "blahDir/blahBlahDir"
set dest_dir = "foo/anotherFoo"
rsync -av --exclude=*.csv ${source_dir} ${dest_dir}
When I run this I get the following error:
rsync: No match.
If I remove the --exclude option it works. I wrote the equivalent script in bash and that works as expected
#/bin/bash -f
source_dir="blahDir/blahBlahDir"
dest_dir="foo/anotherFoo"
rsync -av --exclude=*.csv ${source_dir} ${dest_dir}
The problem is that this has to be done in csh only. Any ideas on how I can get his to work?
It's because csh is trying to expand --exclude=*.csv into a filename, and complaining because it cannot find a file matching that pattern.
You can get around this by enclosing the option in quotes:
rsynv -rv '--exclude=*.csv' ...
or escaping the asterisk:
rsynv -rv --exclude=\*.csv ...
This is a consequence of the way csh and bash differ in their default treatment of arguments with wildcards that don't match a file. csh will complain while bash will simply leave it alone.
You may think bash has chosen the better way but that's not necessarily so, as shown in the following transcript where you have a file matching the argument:
pax> touch -- '--file=xyzzy.csv' ; ls -- *.csv
--file=xyzzy.csv
pax> echo --file=*.csv
--file=xyzzy.csv
You can see there that the bash shell expands the argument rather than giving it to the program as is. Both sides have their pros and cons.
I write up a little shell script in bash that allows me to execute commands in sub-directories. Here is the script
bat.sh:
#!/bin/sh
for d in */; do
echo "Executing \"$#\" in $d"
cd $d
`$#`
cd ..
done
With my following directory structures
/home/user
--a/
----x.txt
----y.txt
--b/
----u.txt
----v.txt
I expect the following command to list out the content of directories a and b when it is executed in the home directory
bat.sh ls
The result is
Executing "ls" in a/
/home/user/bin/bat.sh: line 6: x.txt: command not found
Executing "ls" in b/
/home/user/bin/bat.sh: line 6: u.txt: command not found
Any idea on what is going wrong here?
You don't want the back quotes; you want double quotes.
#!/bin/sh
for d in */
do
echo "Executing \"$*\" in $d"
(cd "$d" && "$#")
done
You are trying to execute the output of the command you pass, whereas you simply want to execute the command.
The use of an explicit subshell (the ( … ) notation) may avoid some problems with symlinks that jump to other directories. It is, in my (perhaps archaic) view, a safer way to switch directories for the purposes of executing commands.
I have the following simple bash script:
#!/bin/bash -fx
ls *sh
The problem is that bash add a quote to the pattern and I get wrong output.
+ ls '*sh'
ls: cannot access *sh: No such file or directory
How can I change this behavior?
The output of ls *sh from the terminal is:
$ls *sh
a.bash a.sh b.sh
I tried to add quotes according to this post - "Bash variable containing file wildcard"
without success
That because you're disabling pathname expansion with the -f option.
#!/bin/bash -fx
From man:
-f
Disable filename expansion (globbing).
#!/bin/sh
files = 'ls /myDir/myDir2/myDir3/'
for file in $files do
echo $file
java myProg $file /another/directory/
done
What i'm trying to do is iterate through every file name under /myDir/myDir2/myDir3/, then use that file name as the first argument in calling a java program (second argument is "/another/directory")
When I run this script: . myScript.sh
I get this error:
-bash: files: command not found
What did I do wrong in my script? Thanks!
Per Neeaj's answer, strip off the whitespace from files =.
Better yet, use:
#!/bin/sh -f
dir=/myDir/MyDir2/MyDir3
for path in $dir/*; do
file=$(basename $path)
echo "$file"
java myProg "$file" arg2 arg3
done
Bash is perfectly capable of expanding the * wildcard itself, without spawning a copy of ls to do the job for it!
EDIT: changed to call basename rather than echo to meet OP's (previously unstated) requirement that the path echoed be relative and not absolute. If the cwd doesn't matter, then even better I'd go for:
#!/bin/sh -f
cd /myDir/MyDir2/MyDir3
for file in *; do
echo "$file"
java myProg "$file" arg2 arg3
done
and avoid the calls to basename altogether.
strip off the whitespace in and after files = as files=RHS of assignment
Remove the space surrounding the '=' : change
files = 'ls /myDir/myDir2/myDir3/'
into:
files='ls /myDir/myDir2/myDir3/'
and move the 'do' statement to its own line:
for file in $files
do
....
quote your variables and no need to use ls.
#!/bin/sh
for file in /myDir/myDir2/*
do
java myProg "$file" /another/directory/
done