I'm trying to print all directories/subdirectories from a given start directory.
for i in $(ls -A -R -p); do
if [ -d "$i" ]; then
printf "%s/%s \n" "$PWD" "$i"
fi
done;
This script returns all of the directories found in the . directory and all of the files in that directory, but for some reason the test fails for subdirectories. All of the directories end up in $i and the output looks exactly the same.
Let's say I have the following structure:
foo/bar/test
echo $i prints
foo/
bar/
test/
While the contents of the folders are listed like this:
./foo:
file1
file2
./bar:
file1
file2
However the test statement just prints:
PWD/TO/THIS/DIRECTORY/foo
For some reason it returns true for the first level directories, but false for all of the subdirectories.
(ls is probably not a good way of doing this and I would be glad for a find statement that solves all of my issues, but first I want to know why this script doesn't work the way you'd think.)
As pointed out in the comments, the issue is that the directory names include a :, so -d is false.
I guess that this command gives you the output you want (although it requires Bash):
# enable globstar for **
# disabled in non-interactive shell (e.g. a script)
shopt -s globstar
# print each path ending in a / (all directories)
# ** expands recursively
printf '%s\n' **/*/
The standard way would either to do the recursion yourself, or to use find:
find . -type d
Consider your output:
dir1:
dir1a
Now, the following will be true:
[ -d dir1/dir1a ]
but that's not what your code does; instead, it runs:
[ -d dir1a ]
To avoid this, don't attempt to parse ls; if you want to implement recursion in baseline POSIX sh, do it yourself:
callForEachEntry() {
# because calling this without any command provided would try to execute all found files
# as commands, checking for safe/correct invocation is essential.
if [ "$#" -lt 2 ]; then
echo "Usage: callForEachEntry starting-directory command-name [arg1 arg2...]" >&2
echo " ...calls command-name once for each file recursively found" >&2
return 1
fi
# try to declare variables local, swallow/hide error messages if this fails; code is
# defensively written to avoid breaking if recursing changes either, but may be faulty if
# the command passed as an argument modifies "dir" or "entry" variables.
local dir entry 2>/dev/null ||: "not strict POSIX, but available in dash"
dir=$1; shift
for entry in "$dir"/*; do
# skip if the glob matched nothing
[ -e "$entry" ] || [ -L "$entry" ] || continue
# invoke user-provided callback for the entry we found
"$#" "$entry"
# recurse last for if on a baseline platform where the "local" above failed.
if [ -d "$entry" ]; then
callForEachEntry "$entry" "$#"
fi
done
}
# call printf '%s\n' for each file we recursively find; replace this with the code you
# actually want to call, wrapped in a function if appropriate.
callForEachEntry "$PWD" printf '%s\n'
find can also be used safely, but not as a drop-in replacement for the way ls was used in the original code -- for dir in $(find . -type d) is just as buggy. Instead, see the "Complex Actions" and "Actions In Bulk" section of Using Find.
I got a recursive script which iterates a list of names, some of which are files and some are directories.
If it's a (non-empty) directory, I should call the script again with all of the files in the directory and check if they are legal.
The part of the code making the recursive call:
if [[ -d $var ]] ; then
if [ "$(ls -A $var)" ]; then
./validate `ls $var`
fi
fi
The part of code checking if the files are legal:
if [[ -f $var ]]; then
some code
fi
But, after making the recursive calls, I can no longer check any of the files inside that directory, because they are not in the same directory as the main script, the -f $var if cannot see them.
Any suggestion how can I still see them and use them?
Why not use find? Simple and easy solution to the problem.
Always quote variables, you never known when you will find a file or directory name with spaces
shopt -s nullglob
if [[ -d "$path" ]] ; then
contents=( "$path"/* )
if (( ${#contents[#]} > 0 )); then
"$0" "${contents[#]}"
fi
fi
you're re-inventing find
of course, var is a lousy variable name
if you're recursively calling the script, you don't need to hard-code the script name.
you should consider putting the logic into a function in the script, and the function can recursively call itself, instead of having to spawn an new process to invoke the shell script each time. If you do this, use $FUNCNAME instead of "$0"
A few people have mentioned how find might solve this problem, I just wanted to show how that might be done:
find /yourdirectory -type f -exec ./validate {} +;
This will find all regular files in yourdirectory and recursively in all its sub-directories, and return their paths as arguments to ./validate. The {} is expanded to the paths of the files that find locates within yourdirectory. The + at the end means that each call to validate will be on a large number of files, instead of calling it individually on each file (wherein the + is replaced with a \), this provides a huge speedup sometimes.
One option is to change directory (carefully) into the sub-directory:
if [[ -d "$var" ]] ; then
if [ "$(ls -A $var)" ]; then
(cd "$var"; exec ./validate $(ls))
fi
fi
The outer parentheses start a new shell so the cd command does not affect the main shell. The exec replaces the original shell with (a new copy of) the validate script. Using $(...) instead of back-ticks is sensible. In general, it is sensible to enclose variable names in double quotes when they refer to file names that might contain spaces (but see below). The $(ls) will list the files in the directory.
Heaven help you with the ls commands if any file names or directory names contain spaces; you should probably be using * glob expansion instead. Note that a directory containing a single file with a name such as -n would trigger a syntax error in your script.
Corrigendum
As Jens noted in a comment, the location of the shell script (validate) has to be adjusted as you descend the directory hierarchy. The simplest mechanism is to have the script on your PATH, so you can write exec validate or even exec $0 instead of exec ./validate. Failing that, you need to adjust the value of $0 — assuming your shell leaves $0 as a relative path and doesn't mess around with converting it to an absolute path. So, a revised version of the code fragment might be:
# For validate on PATH or absolute name in $0
if [[ -d "$var" ]] ; then
if [ "$(ls -A $var)" ]; then
(cd "$var"; exec $0 $(ls))
fi
fi
or:
# For validate not on PATH and relative name in $0
if [[ -d "$var" ]] ; then
if [ "$(ls -A $var)" ]; then
(cd "$var"; exec ../$0 $(ls))
fi
fi
I made this script that should receive one or more parameter, and those parameter are all directories, and it has to browse those directories (one by one) and do some operations.
The operations work fine if the parameter is 1 (only one directory),
How should I modify my script to make it works if more than 1 parameter is passed
Example if I want it to do the same operations in 2 or 3 directories at the same time?
Thanks
#!/bin/sh
cd $1
for file in ./* # */
do
if [[ -d $file ]]
then
ext=dir
else
ext="${file##*.}"
fi
mv "${file}" "${file}.$ext"
done
First, if you are using bash use bash shebang (#! /bin/bash).
Then use
#! /bin/bash
for d in "$#"
do
echo "Do something with $d"
done
to iterate over the command line arguments (dirs in your case)
#!/bin/sh
for dir in "$#"; do
for file in "$dir"/*; do
echo "Doing something with '$file'"
done
done
The following shell script changes current the directory to the desktop.
v=~/Desktop/
cd $v
pwd # desktop
The following script changes the current directory to home directory instead of generating error.
cd $undefined_variable
pwd # home directory
echo $? # 0
I'm afraid that the script will remove important files if I misspelled a variable for new current directory.
Generally, how do you safely change current directory with variable in shell script?
Use:
cd ${variable:?}
if $variable is not defined or empty then bash will throw an error and exit. It's like the set -u option but not global through the file.
You can set -u to make bash exit with an error each time you expand an undefined variable.
You could use the test -d condition (checks whether the specified variable is a directory), i.e.
if [[ -d $undefined_variable ]]
then
cd $undefined_variable
echo "This will not be printed if $undefined_variable is not defined"
fi
See also here for further test options...
The Bourne Shells have a construct to substitute a value for undefined variables, ${varname-subtitution}. You can use this to have a safe fallback directory in case the variable is undefined:
cd "${undefined-/tmp/backupdir}"
If there is a variable named undefined, its value is substituted, otherwise /tmp/backupdir is substituted.
Note that I also put the variable expansion in double quotes. This is used to prevent word splitting on strings containing spaces (very common for Windows directories). This way it works even for directories with spaces.
For the gory details on all the shell substitution constructs (there are seven more for POSIX shells), read your shell manual's Parameter Substitution section.
You have to write a wrapper (this work in bash):
cd() {
if [ $# -ne 1 ] ;then
echo "cd need exactly 1 argument" >&2
return 2
fi
builtin cd "$1"
}
yes, that's shell
if you type cd without parameter it will jump to home dir.
You can can check the variable of null or empty before you cd command.
check like (cd only be called if targetDir is not empty):
test -z "$targetDir" || cd $targetDir
check like (cd only be called if targetDir really exist):
test -d "$targetDir" && cd $targetDir
Note: Thanks for -1, should read the last sentence too. So I added the real answer.
The way you would normally include a script is with "source"
eg:
main.sh:
#!/bin/bash
source incl.sh
echo "The main script"
incl.sh:
echo "The included script"
The output of executing "./main.sh" is:
The included script
The main script
... Now, if you attempt to execute that shell script from another location, it can't find the include unless it's in your path.
What's a good way to ensure that your script can find the include script, especially if for instance, the script needs to be portable?
I tend to make my scripts all be relative to one another.
That way I can use dirname:
#!/bin/sh
my_dir="$(dirname "$0")"
"$my_dir/other_script.sh"
I know I am late to the party, but this should work no matter how you start the script and uses builtins exclusively:
DIR="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi
. "$DIR/incl.sh"
. "$DIR/main.sh"
. (dot) command is an alias to source, $PWD is the Path for the Working Directory, BASH_SOURCE is an array variable whose members are the source filenames, ${string%substring} strips shortest match of $substring from back of $string
An alternative to:
scriptPath=$(dirname $0)
is:
scriptPath=${0%/*}
.. the advantage being not having the dependence on dirname, which is not a built-in command (and not always available in emulators)
If it is in the same directory you can use dirname $0:
#!/bin/bash
source $(dirname $0)/incl.sh
echo "The main script"
I think the best way to do this is to use the Chris Boran's way, BUT you should compute MY_DIR this way:
#!/bin/sh
MY_DIR=$(dirname $(readlink -f $0))
$MY_DIR/other_script.sh
To quote the man pages for readlink:
readlink - display value of a symbolic link
...
-f, --canonicalize
canonicalize by following every symlink in every component of the given
name recursively; all but the last component must exist
I've never encountered a use case where MY_DIR is not correctly computed. If you access your script through a symlink in your $PATH it works.
A combination of the answers to this question provides the most robust solution.
It worked for us in production-grade scripts with great support of dependencies and directory structure:
#!/bin/bash
# Full path of the current script
THIS=`readlink -f "${BASH_SOURCE[0]}" 2>/dev/null||echo $0`
# The directory where current script resides
DIR=`dirname "${THIS}"`
# 'Dot' means 'source', i.e. 'include':
. "$DIR/compile.sh"
The method supports all of these:
Spaces in path
Links (via readlink)
${BASH_SOURCE[0]} is more robust than $0
SRC=$(cd $(dirname "$0"); pwd)
source "${SRC}/incl.sh"
1. Neatest
I explored almost every suggestion and here is the neatest one that worked for me:
script_root=$(dirname $(readlink -f $0))
It works even when the script is symlinked to a $PATH directory.
See it in action here: https://github.com/pendashteh/hcagent/blob/master/bin/hcagent
2. The coolest
# Copyright https://stackoverflow.com/a/13222994/257479
script_root=$(ls -l /proc/$$/fd | grep "255 ->" | sed -e 's/^.\+-> //')
This is actually from another answer on this very page, but I'm adding it to my answer too!
3. The most reliable
Alternatively, in the rare case that those didn't work, here is the bullet proof approach:
# Copyright http://stackoverflow.com/a/7400673/257479
myreadlink() { [ ! -h "$1" ] && echo "$1" || (local link="$(expr "$(command ls -ld -- "$1")" : '.*-> \(.*\)$')"; cd $(dirname $1); myreadlink "$link" | sed "s|^\([^/].*\)\$|$(dirname $1)/\1|"); }
whereis() { echo $1 | sed "s|^\([^/].*/.*\)|$(pwd)/\1|;s|^\([^/]*\)$|$(which -- $1)|;s|^$|$1|"; }
whereis_realpath() { local SCRIPT_PATH=$(whereis $1); myreadlink ${SCRIPT_PATH} | sed "s|^\([^/].*\)\$|$(dirname ${SCRIPT_PATH})/\1|"; }
script_root=$(dirname $(whereis_realpath "$0"))
You can see it in action in taskrunner source: https://github.com/pendashteh/taskrunner/blob/master/bin/taskrunner
Hope this help someone out there :)
Also, please leave it as a comment if one did not work for you and mention your operating system and emulator. Thanks!
This works even if the script is sourced:
source "$( dirname "${BASH_SOURCE[0]}" )/incl.sh"
You need to specify the location of the other scripts, there is no other way around it. I'd recommend a configurable variable at the top of your script:
#!/bin/bash
installpath=/where/your/scripts/are
. $installpath/incl.sh
echo "The main script"
Alternatively, you can insist that the user maintain an environment variable indicating where your program home is at, like PROG_HOME or somesuch. This can be supplied for the user automatically by creating a script with that information in /etc/profile.d/, which will be sourced every time a user logs in.
I'd suggest that you create a setenv script whose sole purpose is to provide locations for various components across your system.
All other scripts would then source this script so that all locations are common across all scripts using the setenv script.
This is very useful when running cronjobs. You get a minimal environment when running cron, but if you make all cron scripts first include the setenv script then you are able to control and synchronise the environment that you want the cronjobs to execute in.
We used such a technique on our build monkey that was used for continuous integration across a project of about 2,000 kSLOC.
Shell Script Loader is my solution for this.
It provides a function named include() that can be called many times in many scripts to refer a single script but will only load the script once. The function can accept complete paths or partial paths (script is searched in a search path). A similar function named load() is also provided that will load the scripts unconditionally.
It works for bash, ksh, pd ksh and zsh with optimized scripts for each one of them; and other shells that are generically compatible with the original sh like ash, dash, heirloom sh, etc., through a universal script that automatically optimizes its functions depending on the features the shell can provide.
[Fowarded example]
start.sh
This is an optional starter script. Placing the startup methods here is just a convenience and can be placed in the main script instead. This script is also not needed if the scripts are to be compiled.
#!/bin/sh
# load loader.sh
. loader.sh
# include directories to search path
loader_addpath /usr/lib/sh deps source
# load main script
load main.sh
main.sh
include a.sh
include b.sh
echo '---- main.sh ----'
# remove loader from shellspace since
# we no longer need it
loader_finish
# main procedures go from here
# ...
a.sh
include main.sh
include a.sh
include b.sh
echo '---- a.sh ----'
b.sh
include main.sh
include a.sh
include b.sh
echo '---- b.sh ----'
output:
---- b.sh ----
---- a.sh ----
---- main.sh ----
What's best is scripts based on it may also be compiled to form a single script with the available compiler.
Here's a project that uses it: http://sourceforge.net/p/playshell/code/ci/master/tree/. It can run portably with or without compiling the scripts. Compiling to produce a single script can also happen, and is helpful during installation.
I also created a simpler prototype for any conservative party that may want to have a brief idea of how an implementation script works: https://sourceforge.net/p/loader/code/ci/base/tree/loader-include-prototype.bash. It's small and anyone can just include the code in their main script if they want to if their code is intended to run with Bash 4.0 or newer, and it also doesn't use eval.
Steve's reply is definitely the correct technique but it should be refactored so that your installpath variable is in a separate environment script where all such declarations are made.
Then all scripts source that script and should installpath change, you only need to change it in one location. Makes things more, er, futureproof. God I hate that word! (-:
BTW You should really refer to the variable using ${installpath} when using it in the way shown in your example:
. ${installpath}/incl.sh
If the braces are left out, some shells will try and expand the variable "installpath/incl.sh"!
I put all my startup scripts in a .bashrc.d directory.
This is a common technique in such places as /etc/profile.d, etc.
while read file; do source "${file}"; done <<HERE
$(find ${HOME}/.bashrc.d -type f)
HERE
The problem with the solution using globbing...
for file in ${HOME}/.bashrc.d/*.sh; do source ${file};done
...is you might have a file list which is "too long".
An approach like...
find ${HOME}/.bashrc.d -type f | while read file; do source ${file}; done
...runs but doesn't change the environment as desired.
This should work reliably:
source_relative() {
local dir="${BASH_SOURCE%/*}"
[[ -z "$dir" ]] && dir="$PWD"
source "$dir/$1"
}
source_relative incl.sh
Using source or $0 will not give you the real path of your script. You could use the process id of the script to retrieve its real path
ls -l /proc/$$/fd |
grep "255 ->" |
sed -e 's/^.\+-> //'
I am using this script and it has always served me well :)
Of course, to each their own, but I think the block below is pretty solid. I believe this involves the "best" way to find a directory, and the "best" way to call another bash script:
scriptdir=`dirname "$BASH_SOURCE"`
source $scriptdir/incl.sh
echo "The main script"
So this may be the "best" way to include other scripts. This is based off another "best" answer that tells a bash script where it is stored
Personally put all libraries in a lib folder and use an import function to load them.
folder structure
script.sh contents
# Imports '.sh' files from 'lib' directory
function import()
{
local file="./lib/$1.sh"
local error="\e[31mError: \e[0mCannot find \e[1m$1\e[0m library at: \e[2m$file\e[0m"
if [ -f "$file" ]; then
source "$file"
if [ -z $IMPORTED ]; then
echo -e $error
exit 1
fi
else
echo -e $error
exit 1
fi
}
Note that this import function should be at the beginning of your script and then you can easily import your libraries like this:
import "utils"
import "requirements"
Add a single line at the top of each library (i.e. utils.sh):
IMPORTED="$BASH_SOURCE"
Now you have access to functions inside utils.sh and requirements.sh from script.sh
TODO: Write a linker to build a single sh file
we just need to find out the folder where our incl.sh and main.sh is stored; just change your main.sh with this:
main.sh
#!/bin/bash
SCRIPT_NAME=$(basename $0)
SCRIPT_DIR="$(echo $0| sed "s/$SCRIPT_NAME//g")"
source $SCRIPT_DIR/incl.sh
echo "The main script"
According man hier suitable place for script includes is /usr/local/lib/
/usr/local/lib
Files associated with locally installed programs.
Personally I prefer /usr/local/lib/bash/includes for includes.
There is bash-helper lib for including libs in that way:
#!/bin/bash
. /usr/local/lib/bash/includes/bash-helpers.sh
include api-client || exit 1 # include shared functions
include mysql-status/query-builder || exit 1 # include script functions
# include script functions with status message
include mysql-status/process-checker; status 'process-checker' $? || exit 1
include mysql-status/nonexists; status 'nonexists' $? || exit 1
Most of the answers I saw here seem to overcomplicate things. This method has always worked reliably for me:
FULLPATH=$(readlink -f $0)
INCPATH=${FULLPATH%/*}
INCPATH will hold the complete path of the script excluding the script filename, regardless of how the script is called (by $PATH, relative or absolute).
After that, one only needs to do this to include files in the same directory:
. $INCPATH/file_to_include.sh
Reference: TecPorto / Location independent includes
here is a nice function you can use. it builds on what #sacii made. thank you
it will let you list any number of space separated script names to source (relative to the script calling source_files).
optionally you can pass an absolute or relative path as the first argument and it will source from there instead.
you can call it multiple times (see example below) to source scripts from different dirs
#!/usr/bin/env bash
function source_files() {
local scripts_dir
scripts_dir="$1"
if [ -d "$scripts_dir" ]; then
shift
else
scripts_dir="${BASH_SOURCE%/*}"
if [[ ! -d "$scripts_dir" ]]; then scripts_dir="$PWD"; fi
fi
for script_name in "$#"; do
# shellcheck disable=SC1091 disable=SC1090
. "$scripts_dir/$script_name.sh"
done
}
here is an example you can run to show how its used
#!/usr/bin/env bash
function source_files() {
local scripts_dir
scripts_dir="$1"
if [ -d "$scripts_dir" ]; then
shift
else
scripts_dir="${BASH_SOURCE%/*}"
if [[ ! -d "$scripts_dir" ]]; then scripts_dir="$PWD"; fi
fi
for script_name in "$#"; do
# shellcheck disable=SC1091 disable=SC1090
. "$scripts_dir/$script_name.sh"
done
}
## -- EXAMPLE -- ##
# assumes dir structure:
# /
# source_files.sh
# sibling.sh
# scripts/
# child.sh
# nested/
# scripts/
# grandchild.sh
cd /tmp || exit 1
# sibling.sh
tee sibling.sh <<- EOF > /dev/null
#!/usr/bin/env bash
export SIBLING_VAR='sibling var value'
EOF
# scripts/child.sh
mkdir -p scripts
tee scripts/child.sh <<- EOF > /dev/null
#!/usr/bin/env bash
export CHILD_VAR='child var value'
EOF
# nested/scripts/grandchild.sh
mkdir -p nested/scripts
tee nested/scripts/grandchild.sh <<- EOF > /dev/null
#!/usr/bin/env bash
export GRANDCHILD_VAR='grandchild var value'
EOF
source_files 'sibling'
source_files 'scripts' 'child'
source_files 'nested/scripts' 'grandchild'
echo "$SIBLING_VAR"
echo "$CHILD_VAR"
echo "$GRANDCHILD_VAR"
rm sibling.sh
rm -rf scripts nested
cd - || exit 1
prints:
sibling var value
child var value
grandchild var value
You can also use:
PWD=$(pwd)
source "$PWD/inc.sh"