I currently have a bash script called (log2csv). While in the current directory I can type in terminal:
log2csv *.log
This will run the script on every .log file in the current directory.
Alternatively I can run it against a single .log file with
log2csv test1.log
Instead of typing log2csv *.log, can I have the *.log included in the script? So I can just type log2csv in the directory and it runs. I know I can alias that, but I rather have the script do it.
Here is the bash script I am running:
#!/bin/bash
for path
do
base=$(basename "$path")
noext="${base/.log}"
[ -e "${noext}.csv" ] && continue
/Users/joshuacarter/bin/read_scalepack.pl "$path" > "${noext}.csv"
done
Change:
for path
to:
for path in *.log
or, perhaps better:
names=( "$#" )
if [ "${#names}" = 0 ]
then names=( *.log )
fi
for path in "${names[#]}"
and you can consider whether to set options such as shopt -s nullglob as well. This uses shell arrays to handle names with blanks etc in them. It uses command line arguments if any are given, and the list of files from *.log being expanded if there are no command line arguments.
Related
I have a bash script which works like this;
File structure;
get.sh
loop.sh
config/param1.conf
config/param2.conf
Usage of the main script, get.sh;
./get.sh <param> i.e ./get.sh param1, ./get.sh param2
So when you run the script with specific params it fetches the config files from config/<param>.conf
What I'm trying to do is to run this second script, ./loop.sh so it runs the ./get.sh <param> for you in a loop using the params inside config folder, without .conf extensions.
Here's my loop.sh;
#!/bin/bash
# run the script with the first param you found inside ~/config/
# folder without including it's .conf extension,
# wait for 5 seconds and then do the same with the 2nd param you found
for i in $(find ~/config -name '*.conf'); do
./get.sh $(basename $i) | cut -d'.' -f 1
sleep 5
done
but this one is just displaying the params inside config folder and doing nothing else.
`
Inside of the config/param1.conf;
var=Hello1
Inside of the config/param2.conf;
var=Hello2
Inside of the get.sh;
#!/bin/bash
function testFunction {
echo "$var"
}
cfg_file=$1
if [ -f "$cfg_file" ]; then
. "$cfg_file"
testFunction $1
exit 1
else
echo "$1.conf doesn't exist"
exit 1
fi
So after all, when you run the loop.sh, the expected behavior should be printing the Hello1 and Hello2 strings into shell.
How can I fix this?
In loop (using .sh extensions for bash scripts is not great practice):
#!/bin/bash
for i in ~/config/*.conf; do
i_basename=${i##*/} # change ~/config/foo.conf to just foo.conf
i_basename=${i_basename%.conf} # change foo.conf to just foo
./get "$i_basename"
sleep 5
done
The ${var##prefix} and ${var%suffix} syntax is parameter expansion; with ## it removes the longest match from the beginning (so for */, everything up to the last /); with %, it removes the shortest successful match starting from the end.
In get:
#!/bin/bash
# Using POSIX function syntax; see http://wiki.bash-hackers.org/scripting/obsolete
testFunction() {
echo "$var"
}
cfg_file="config/$1.conf"
if [ -f "$cfg_file" ]; then
. "$cfg_file"
testFunction
else
echo "$cfg_file doesn't exist"
exit 1
fi
With respect to lack of .sh extensions -- UNIX commands don't conventionally have extensions (you run ls, not ls.elf); similarly, when you install a Python module built with setuptools, it doesn't put .py extensions on shims it creates in /usr/local/bin, even if the libraries those executable shims invoke do have such extensions. Moreover, bash and POSIX sh are two different languages: Bash scripts often don't work correctly when started with sh some-bash-only-script.sh (as unlike bash, sh isn't guaranteed to support language features like arrays), but the extension implies that they will.
-name expects the parameter to be just a filename, not a whole pathname. You should use the directory as a regular argument to find, not as part of -name.
for i in $(find ~/config -name '*.conf'); do
Don't use basename, you should pass the entire pathname to script.sh.
Then in script.sh you should should use $1 as the whole path to the config file, rather than concatenating it with a directory prefix.
cfg_file=$1
I don't see any point in this:
case $1 in
$1) cfg=$1 ;;
esac
The case will always be true, how can $1 not match $1? Remove that code.
I am trying to move files from a directory if it is not empty. This script is added in cronjob but it is always executing regardless of files are present or not? What is wrong in this thing?
#!/bin/bash
logFolder="/dstDir/`date '+%Y-%m-%d/%H-%M'`"
tempLogFolder="sourceDir"
if [ -z "$(ls -A $tempLogFolder | grep *.log)" ]; then
mkdir -p $logFolder
mv $tempLogFolder/*.log $logFolder/
fi
Don't parse the output of ls. You don't need any external commands to count files in a directory. The shell can do it itself.
Try creating a helper function that checks how many arguments are passed to it. Then have the shell expand the glob "$tempLogFolder"/*.log. No need for ls or grep. The only trick is enabling the nullglob option so if no files exist the glob expands to nothing.
files_exist() { (($# > 0)); }
shopt -s nullglob
if files_exist "$tempLogFolder"/*.log; then
mkdir -p "$logFolder"
mv "$tempLogFolder"/*.log "$logFolder"/
fi
grep accepts regex, not glob. If you want to use glob like *.log you put it in your ls something like ls .... *.log (using ls output is not 100% safe).
If you want to use grep, please use regex, I guess what you meant is \.log$.
If you want to check if grep has found matches, you should check the return code ($?) of grep, instead of using -z test.
If you put it in crontab, the script will always be executed.
I think you want,
if [ -n "$(ls -A $tempLogFolder | grep *.log)" ]; then
...
because you are only interested in whether the string returned has a non-zero length.
I got a recursive script which iterates a list of names, some of which are files and some are directories.
If it's a (non-empty) directory, I should call the script again with all of the files in the directory and check if they are legal.
The part of the code making the recursive call:
if [[ -d $var ]] ; then
if [ "$(ls -A $var)" ]; then
./validate `ls $var`
fi
fi
The part of code checking if the files are legal:
if [[ -f $var ]]; then
some code
fi
But, after making the recursive calls, I can no longer check any of the files inside that directory, because they are not in the same directory as the main script, the -f $var if cannot see them.
Any suggestion how can I still see them and use them?
Why not use find? Simple and easy solution to the problem.
Always quote variables, you never known when you will find a file or directory name with spaces
shopt -s nullglob
if [[ -d "$path" ]] ; then
contents=( "$path"/* )
if (( ${#contents[#]} > 0 )); then
"$0" "${contents[#]}"
fi
fi
you're re-inventing find
of course, var is a lousy variable name
if you're recursively calling the script, you don't need to hard-code the script name.
you should consider putting the logic into a function in the script, and the function can recursively call itself, instead of having to spawn an new process to invoke the shell script each time. If you do this, use $FUNCNAME instead of "$0"
A few people have mentioned how find might solve this problem, I just wanted to show how that might be done:
find /yourdirectory -type f -exec ./validate {} +;
This will find all regular files in yourdirectory and recursively in all its sub-directories, and return their paths as arguments to ./validate. The {} is expanded to the paths of the files that find locates within yourdirectory. The + at the end means that each call to validate will be on a large number of files, instead of calling it individually on each file (wherein the + is replaced with a \), this provides a huge speedup sometimes.
One option is to change directory (carefully) into the sub-directory:
if [[ -d "$var" ]] ; then
if [ "$(ls -A $var)" ]; then
(cd "$var"; exec ./validate $(ls))
fi
fi
The outer parentheses start a new shell so the cd command does not affect the main shell. The exec replaces the original shell with (a new copy of) the validate script. Using $(...) instead of back-ticks is sensible. In general, it is sensible to enclose variable names in double quotes when they refer to file names that might contain spaces (but see below). The $(ls) will list the files in the directory.
Heaven help you with the ls commands if any file names or directory names contain spaces; you should probably be using * glob expansion instead. Note that a directory containing a single file with a name such as -n would trigger a syntax error in your script.
Corrigendum
As Jens noted in a comment, the location of the shell script (validate) has to be adjusted as you descend the directory hierarchy. The simplest mechanism is to have the script on your PATH, so you can write exec validate or even exec $0 instead of exec ./validate. Failing that, you need to adjust the value of $0 — assuming your shell leaves $0 as a relative path and doesn't mess around with converting it to an absolute path. So, a revised version of the code fragment might be:
# For validate on PATH or absolute name in $0
if [[ -d "$var" ]] ; then
if [ "$(ls -A $var)" ]; then
(cd "$var"; exec $0 $(ls))
fi
fi
or:
# For validate not on PATH and relative name in $0
if [[ -d "$var" ]] ; then
if [ "$(ls -A $var)" ]; then
(cd "$var"; exec ../$0 $(ls))
fi
fi
I am trying to move all the arguments that are given when the script is run, to a RecycleBin directory. I am setting RecycleBin to be a directory in my home folder using RecycleBin="$HOME/Recycled".
I then want to move all the arguments files / directories to the RecycleBin directory.
This is what I've got so far:
for i in $*
do
if [ $* -e ]
then
mv $i /path/to/RecycleBin/$*.`date +"%Y%m%d.%H%M%S"`
else
echo "The file does not exist"
fi
done
where .date +"%Y%m%d.%H%M%S" is appending the version (date command) of each file
and for i in $* is checking for all the arguments in the command.
Output will look something like this ./script.sh file1 file2 file3
- where 3 files are being moved into RecycleBin
This is not working as my loop function is incorrect.
Would appreciate help
Thanks
Pete
[ $* -e ]
That test should be this:
[ -e "$i" ]
Also in the mv command you should be using $i rather than $*. Personally I'd just get rid of the if statement entirely. mv will print out an error message if a file doesn't exist, no need to do it yourself.
To handle file names with spaces in them it's best practice to use "$#" in place of $*, and to surround your variable names with with quotes, like so:
for file in "$#"; do
mv "$file" "/path/to/RecycleBin/$file.$(date +%Y%m%d.%H%M%S)"
done
#!/bin/sh
files = 'ls /myDir/myDir2/myDir3/'
for file in $files do
echo $file
java myProg $file /another/directory/
done
What i'm trying to do is iterate through every file name under /myDir/myDir2/myDir3/, then use that file name as the first argument in calling a java program (second argument is "/another/directory")
When I run this script: . myScript.sh
I get this error:
-bash: files: command not found
What did I do wrong in my script? Thanks!
Per Neeaj's answer, strip off the whitespace from files =.
Better yet, use:
#!/bin/sh -f
dir=/myDir/MyDir2/MyDir3
for path in $dir/*; do
file=$(basename $path)
echo "$file"
java myProg "$file" arg2 arg3
done
Bash is perfectly capable of expanding the * wildcard itself, without spawning a copy of ls to do the job for it!
EDIT: changed to call basename rather than echo to meet OP's (previously unstated) requirement that the path echoed be relative and not absolute. If the cwd doesn't matter, then even better I'd go for:
#!/bin/sh -f
cd /myDir/MyDir2/MyDir3
for file in *; do
echo "$file"
java myProg "$file" arg2 arg3
done
and avoid the calls to basename altogether.
strip off the whitespace in and after files = as files=RHS of assignment
Remove the space surrounding the '=' : change
files = 'ls /myDir/myDir2/myDir3/'
into:
files='ls /myDir/myDir2/myDir3/'
and move the 'do' statement to its own line:
for file in $files
do
....
quote your variables and no need to use ls.
#!/bin/sh
for file in /myDir/myDir2/*
do
java myProg "$file" /another/directory/
done