I have a number of folders that are constantly and automatically generated. Some are garbage and need to be cleared out. Each folder produces a generations.txt which I want to count the important lines to determine whether or not the folder should be deleted. I'd like to have a bash script I can run every so often to clean things up.
Here's what I have. I can echo the command I want but I don't believe it outputs the integer to compare to 5. Any suggestions would really help me out. Please and thank you!
#!/bin/bash
SEARCHABLES="grep -Evc 'Value:' "
for d in */
do
PATH=$d'generations.txt'
COMMAND=$SEARCHABLES$PATH
if $COMMAND < 5
then
rm -rf $d
fi
done
You're not getting the output of the command, you need $(...) to execute a command and substitute its output.
To perform the arithmetic comparison, you have to put it inside ((...)).
#!/bin/bash
SEARCHABLES="grep -Evc 'Value:' "
for d in */
do
PATH="$d"'generations.txt'
COMMAND=$SEARCHABLES$PATH
if (( $($COMMAND) < 5 ))
then
rm -rf "$d"
fi
done
See BashFAQ/050 - I'm trying to put a command in a variable, but the complex cases always fail!
for a more detailed explanation.
In short, embedding a command in a variable is a faulty approach to the problem here because the single quotes in 'Value:' will be treated like literal data to search for. Syntax parsing happens before expansions, so you can't embed quotes in a variable like that. What you need is a function:
_count() {
grep -Evc 'Value:' "$1"
}
_count "$PATH"
Then compare the output of the function using an arithmetic expression:
occurrences=$( _count "$PATH" )
if (( occurrences < 5 )) ; then
...
fi
Related
Here is my code aiming to diff .in and .out file in batches. Both .in and .out files are in same directory and sorted by name. So, two files needed to test should be adjacent in directory. But when I want to use outFile=${arr[$(i++)]} to get the .out file, it shows i++: command not found. What's the error in my script?
#!/bin/sh
dir=$PATH
arr=($dir/*)
for((i=0;i<${#arr[#]};i++));do
inFile=${arr[$i]}
outFile=${arr[$(i++)]}
if diff $inFile $outFile > /dev/null; then
echo Same
else
echo $inFile
fi
done
Use $(( i++ )). That's two (not one) parenthesis.
$( ) runs shell commands.
$(( )) evaluates arithmetic expressions.
Also:
Your script uses bash features (arrays), it's best to use #!/bin/bash to avoid confusion.
I'm not sure what you expect dir=$PATH to do? $PATH is a special environment variable that is used for the lookup of commands. This is probably not what you want, if I understand the intent of the script correctly.
i++ will increment the value after being used; so here inFile and outFile are in fact the same! You probably want to use ++i (which will modify the variable and then use it), or just i + 1 (which will not modify the variable).
The stuff in brackets is already evaluated in arithmetic context, like within $(( ... )). So you can do:
for (( i=0; i < ${#arr[#]}; ));do
inFile=${arr[i++]}
outFile=${arr[i++]}
References:
https://www.gnu.org/software/bash/manual/bashref.html#Arrays
The subscript is treated as an arithmetic expression ...
https://www.gnu.org/software/bash/manual/bashref.html#Shell-Arithmetic
Within an expression, shell variables may also be referenced by name without using the parameter expansion syntax.
I have some file in the name of OBS_SURFACE1**, OBS_SURFACE101, OBS_SURFACE103. Yes, there indeed is a file named OBS_SURFACE1**, which I guess where the problem arise. I wrote a bash script which has:
for fil in ` ls OBS_DOMAIN1?? `
do
echo "appending" $fil
done
The first value of fil will be OBS_SURFACE1** OBS_SURFACE101 OBS_SURFACE103, the second OBS_SURFACE101. While I expect the first to be OBS_SURFACE1**. If there is no OBS_SURFACE1** file, there would be no problem. Why is that then?
Don't parse ls! It will only ever lead to problems. Use a glob instead:
for fil in OBS_DOMAIN1??
do
echo "appending $fil"
done
The problem that you are experiencing stems from the fact that the output of ls contains *, which are being expanded by bash. Note that I have also quoted the whole string to be echoed, which protects against word splitting inside the loop. See the links provided in the comments above for more details on that.
As pointed out in the comments (thanks Charles), you may also want to enable nullglob before your loop like this: shopt -s nullglob. This will mean that if there are no files that match the pattern, the loop will not run at all, rather than running once with $fil taking the literal value OBS_DOMAIN1??. Another option would be to check whether the file exists in within the loop, for example using:
if [[ -e "$fil" ]]; then
echo "appending $fil"
fi
or the more compact [[ -e "$fil" ]] && echo "appending $fil".
yet another way of doing this :
echo appending OBS_DOMAIN1??
this will list all files , no loop needed.
Here is a small[but complete] part of my bash script that finds and outputs all files in mydir if the have the prefix from a stored array. Strange thing I notice is that this script works perfectly if I take out the "-maxdepth 1 -name" from the script else it only gives me the files with the prefix of the first element in the array.
It would be of great help if someone explained this to me. Sorry in advance if there is some thing obviously silly that I'm doing. I'm relatively new to scripting.
#!/bin/sh
DIS_ARRAY=(A B C D)
echo "Array is : "
echo ${DIS_ARRAY[*]}
for dis in $DIS_ARRAY
do
IN_FILES=`find /mydir -maxdepth 1 -name "$dis*.xml"`
for file in $IN_FILES
do
echo $file
done
done
Output:
/mydir/Abc.xml
/mydir/Ab.xml
/mydir/Ac.xml
Expected Output:
/mydir/Abc.xml
/mydir/Ab.xml
/mydir/Ac.xml
/mydir/Bc.xml
/mydir/Cb.xml
/mydir/Dc.xml
The loop is broken either way. The reason why
IN_FILES=`find mydir -maxdepth 1 -name "$dis*.xml"`
works, whereas
IN_FILES=`find mydir "$dis*.xml"`
doesn't is because in the first one, you have specified -name. In the second one, find is listing all the files in mydir. If you change the second one to
IN_FILES=`find mydir -name "$dis*.xml"`
you will see that the loop isn't working.
As mentioned in the comments, the syntax that you are currently using $DIS_ARRAY will only give you the first element of the array.
Try changing your loop to this:
for dis in "${DIS_ARRAY[#]}"
The double quotes around the expansion aren't strictly necessary in your specific case, but required if the elements in your array contained spaces, as demonstrated in the following test:
#!/bin/bash
arr=("a a" "b b")
echo using '$arr'
for i in $arr; do echo $i; done
echo using '${arr[#]}'
for i in ${arr[#]}; do echo $i; done
echo using '"${arr[#]}"'
for i in "${arr[#]}"; do echo $i; done
output:
using $arr
a
a
using ${arr[#]}
a
a
b
b
using "${arr[#]}"
a a
b b
See this related question for further details.
#TomFenech's answer solves your problem, but let me suggest other improvements:
#!/usr/bin/env bash
DIS_ARRAY=(A B C D)
echo "Array is : "
echo ${DIS_ARRAY[*]}
for dis in "${DIS_ARRAY[#]}"
do
for file in "/mydir/$dis"*.xml
do
if [ -f "$file" ]; then
echo "$file"
fi
done
done
Your shebang line references sh, but your question is tagged bash - unless you need POSIX compliance, use a bash shebang line to take advantage of all that bash has to offer
To match files located directly in a given directory (i.e., if you don't need to traverse an entire subtree), use a glob (filename pattern) and rely on pathname expansion as in my code above - no need for find and command substitution.
Note that the wildcard char. * is UNquoted to ensure pathname expansion.
Caveat: if no matching files are found, the glob is left untouched (assuming the nullglob shell option is OFF, which it is by default), so the loop is entered once, with an invalid filename (the unexpanded glob) - hence the [ -f "$file" ] conditional to ensure that an actual match was found (as an aside: using bashisms, you could use [[ -f $file ]] instead).
I have spent a lot of time the past few weeks and posting on here. I finally think I am much closer with learning bash but I have one problem with my code I cannot for the life of me figure out why it will not run. I can run each line in the terminal and it returns a result but for some reason when I point it to run, it will do nothing. I get a a syntax error: word unexpected (expecting "do").
#!/bin/bash
image="/Home/Desktop/epubs/images"
for f in $(ls "$image"*.jpg); do
fsize=$(stat --printf= '%s' "$f");
if [ "$fsize" -eq "40318" ]; then
echo "$(basename $f)" >> results.txt
fi
done
What am I missing???
The problem might be in line endings. Make sure your script file has unix line endings, not the Windows ones.
Also, do not iterate over output of ls. Use globbing right in the shell:
for f in "$file"/*.jpg ; do
Your for loop appears to be missing a list of values to iterate over:
image="/Home/Desktop/epubs/images"
for f in $(ls "$image"*.jpg); do
Because $image does not end with a /, your ls command expands to
for f in $(ls /Home/Desktop/epubs/images*.jpg); do
which probably results in
for f in ; do
causing the syntax error. The simplest fix is
for f in $(ls "$image"/*.jpg); do
but you should take the advice in the other answers and skip ls:
for f in "$image"/*.jpg; do
Here's how I would do that.
#!/bin/bash -e
image="/Home/Desktop/epubs/images"
(cd "$image"
for f in *.jpg; do
let fsize=$(stat -c %s "$f")
if (( fsize == 40318 )); then
echo "$f"
fi
done) >results.txt
The -e means the script will exit if anything goes wrong (can't cd into the directory, for instance). Saves a lot of error checking when you're happy with that behavior.
The parentheses mean that the cd command is in a subshell; the surrounding script (including the redirection into results.txt) is still in whatever directory you started in.
Now that we're in the directory, we can just look for *.jpg, no directory prefix, and no need to call basename on anything.
Using let and (( == )) treats the size value as a number instead of a string, so we won't get tripped up by any wonkiness in the way stat chooses to format the value.
We just redirect the output the entire loop into the result file instead of appending every time through; it's more efficient. If you have existing contents in results.txt that you want to keep, you can just change the > back to a >>, but leaving it around the whole loop is still more efficient than opening the file and appending to it on every iteration.
Within my backup script, I'd like to only keep 7 days worth of backups (tried using logrotate for this and it worked perfectly, but I ran into issues with the timing of cron.daily and how it affected the "dateext"). I'm running into problems using parameter expansion to extract the date from the filenames.
Here are some examples of some of the files
foo.bar.tar.gz-20120904
bar.baz.tar.gz-20120904
...
Here is my bash script:
#!/bin/bash
path="/foo/"
today=$(date +%Y%m%d)
keepDays=7
keepSeconds=$(date -d "-$keepDays day" +%s)
for f in $path"*"; do
fileSeconds=$(date -d ${f##*-} +%s)
if [ $fileSeconds -lt $keepSeconds ]
then
rm $f
fi
done
Here is the error I'm getting:
date: extra operand `/foo/foo.bar.tar.gz-20120904'
Remove the quotes around the *, that prevents globbing:
for f in ${path}*; do
(the { } are not strictly required here, but make it easier to read)
Not part of the question, but the Bourne shell syntax [ $fileSeconds -lt $keepSeconds ] could be written as (( $fileSeconds < $keepSeconds )) which is possibly safer.
As cdarke says, remove the quotes around the * in the for loop:
for f in ${path}/*; do
What happens is that the shell executing date gets '/foo/*' and expands that into a list of file names (more than one) and then uses ${f##*-} on part of the list, and date is called with multiple names, and objects.
You'd see this if you ran with bash -x your-script.sh, for instance. When something mysterious goes on, the first step is to make sure you know what the shell is doing. Adding echo "$f" or echo $f in the loop would help you understand — though you'd get two different answers.