Within my backup script, I'd like to only keep 7 days worth of backups (tried using logrotate for this and it worked perfectly, but I ran into issues with the timing of cron.daily and how it affected the "dateext"). I'm running into problems using parameter expansion to extract the date from the filenames.
Here are some examples of some of the files
foo.bar.tar.gz-20120904
bar.baz.tar.gz-20120904
...
Here is my bash script:
#!/bin/bash
path="/foo/"
today=$(date +%Y%m%d)
keepDays=7
keepSeconds=$(date -d "-$keepDays day" +%s)
for f in $path"*"; do
fileSeconds=$(date -d ${f##*-} +%s)
if [ $fileSeconds -lt $keepSeconds ]
then
rm $f
fi
done
Here is the error I'm getting:
date: extra operand `/foo/foo.bar.tar.gz-20120904'
Remove the quotes around the *, that prevents globbing:
for f in ${path}*; do
(the { } are not strictly required here, but make it easier to read)
Not part of the question, but the Bourne shell syntax [ $fileSeconds -lt $keepSeconds ] could be written as (( $fileSeconds < $keepSeconds )) which is possibly safer.
As cdarke says, remove the quotes around the * in the for loop:
for f in ${path}/*; do
What happens is that the shell executing date gets '/foo/*' and expands that into a list of file names (more than one) and then uses ${f##*-} on part of the list, and date is called with multiple names, and objects.
You'd see this if you ran with bash -x your-script.sh, for instance. When something mysterious goes on, the first step is to make sure you know what the shell is doing. Adding echo "$f" or echo $f in the loop would help you understand — though you'd get two different answers.
Related
I have a number of folders that are constantly and automatically generated. Some are garbage and need to be cleared out. Each folder produces a generations.txt which I want to count the important lines to determine whether or not the folder should be deleted. I'd like to have a bash script I can run every so often to clean things up.
Here's what I have. I can echo the command I want but I don't believe it outputs the integer to compare to 5. Any suggestions would really help me out. Please and thank you!
#!/bin/bash
SEARCHABLES="grep -Evc 'Value:' "
for d in */
do
PATH=$d'generations.txt'
COMMAND=$SEARCHABLES$PATH
if $COMMAND < 5
then
rm -rf $d
fi
done
You're not getting the output of the command, you need $(...) to execute a command and substitute its output.
To perform the arithmetic comparison, you have to put it inside ((...)).
#!/bin/bash
SEARCHABLES="grep -Evc 'Value:' "
for d in */
do
PATH="$d"'generations.txt'
COMMAND=$SEARCHABLES$PATH
if (( $($COMMAND) < 5 ))
then
rm -rf "$d"
fi
done
See BashFAQ/050 - I'm trying to put a command in a variable, but the complex cases always fail!
for a more detailed explanation.
In short, embedding a command in a variable is a faulty approach to the problem here because the single quotes in 'Value:' will be treated like literal data to search for. Syntax parsing happens before expansions, so you can't embed quotes in a variable like that. What you need is a function:
_count() {
grep -Evc 'Value:' "$1"
}
_count "$PATH"
Then compare the output of the function using an arithmetic expression:
occurrences=$( _count "$PATH" )
if (( occurrences < 5 )) ; then
...
fi
I'm a beginner getting started with bash scripting.
I have 10 directories in current working directory dir1-dir10 + script.sh + a file called "tocopyfile".
Dir1-10 are empty.
tocopyfile is a test text file to be used for the purpose of my training
script.sh contains the following code:
dir=`pwd`
i="0"
for directory in `ls $dir`;do
while [ $i -le 10 ]
do
cp tocopyfile $directory/file$i &
i=$[$i+1]
done
done
The script should copy 10 copies of the file "tocopyfile" to every dir (dir1-10) in the naming convention file#. The problem is that ones the script is exists after the first directory without executing the while loop to the remaining remaining dirs.
Can someone explain what I'm doing wrong please?
Help is greatly appreciated.
The immediate issue is that you need to reset the value of i for each iteration of the outer loop.
for directory in `ls $dir`; do # No! but more on that in a moment
i=0
while [ $i -le 10 ]
There are a few other issues with your code.
dir=$(pwd) is almost always pointless; bash already provides a variable PWD containing the name of the current working directory. You don't actually need this, though; you can simply use ./*/ to expand to a list of directories in the current working directory.
Never use the output of ls in a script.
$[...] is obsolete syntax; use $((...)) instead.
Cleaning your code up a bit, we get
for directory in ./*/; do
i=0
while [ "$i" -le 10 ]; do
cp tocopyfile "$directory/file$i" &
i=$((i+1))
done
done
You need to initialize $i inside the for loop, such that $i == 0 upon each iteration of your while:
dir=`pwd`
for directory in `ls $dir`;do
i="0" # <===== notice the change here
while [ $i -le 10 ]
do
cp tocopyfile $directory/file$i &
i=$[$i+1]
done
done
Other things you might want to change:
Double-quote all your variables (in the event they have spaces in them).
Use $() instead of the long-deprecated back-tick syntax.
Use $(()) instead of the deprecated $[] syntax.
Tidy up your indentation.
I'm studying the bash shell and lately understood i'm not getting right recursive calls involving file searching- i know find is made for this but I'm recently asked to implement a certain search this way or another.
I wrote the next script:
#!/bin/bash
function rec_search {
for file in `ls $1`; do
echo ${1}/${item}
if[[ -d $item ]]; then
rec ${1}/${item}
fi
done
}
rec $1
the script gets as argument file and looking for it recursively.
i find it a poor solution of mine. and have a few improvement questions:
how to find files that contain spaces in their names
can i efficiently use pwd command for printing out absolute address (i tried so, but unsuccessfully)
every other reasonable improvement of the code
Your script currently cannot work:
The function is defined as rec_search, but then it seems you mistakenly call rec
You need to put a space after the "if" in if[[
There are some other serious issues with it too:
for file in `ls $1` goes against the recommendation to "never parse the output of ls", won't work for paths with spaces or other whitespace characters
You should indent the body of if and for to make it easier to read
The script could be fixed like this:
rec() {
for path; do
echo "$path"
if [[ -d "$path" ]]; then
rec "$path"/*
fi
done
}
But it's best to not reinvent the wheel and use the find command instead.
If you are using bash 4 or later (which is likely unless you running this under Mac OS X), you can use the ** operator.
rec () {
shopt -s globstar
for file in "$1"/**/*; do
echo "$file"
done
}
I have written a small bash script called "isinFile.sh" for checking if the first term given to the script can be found in the file "file.txt":
#!/bin/bash
FILE="file.txt"
if [ `grep -w "$1" $FILE` ]; then
echo "true"
else
echo "false"
fi
However, running the script like
> ./isinFile.sh -x
breaks the script, since -x is interpreted by grep as an option.
So I improved my script
#!/bin/bash
FILE="file.txt"
if [ `grep -w -- "$1" $FILE` ]; then
echo "true"
else
echo "false"
fi
using -- as an argument to grep. Now running
> ./isinFile.sh -x
false
works. But is using -- the correct and only way to prevent code/option injection in bash scripts? I have not seen it in the wild, only found it mentioned in ABASH: Finding Bugs in Bash Scripts.
grep -w -- ...
prevents that interpretation in what follows --
EDIT
(I did not read the last part sorry). Yes, it is the only way. The other way is to avoid it as first part of the search; e.g. ".{0}-x" works too but it is odd., so e.g.
grep -w ".{0}$1" ...
should work too.
There's actually another code injection (or whatever you want to call it) bug in this script: it simply hands the output of grep to the [ (aka test) command, and assumes that'll return true if it's not empty. But if the output is more than one "word" long, [ will treat it as an expression and try to evaluate it. For example, suppose the file contains the line 0 -eq 2 and you search for "0" -- [ will decide that 0 is not equal to 2, and the script will print false despite the fact that it found a match.
The best way to fix this is to use Ignacio Vazquez-Abrams' suggestion (as clarified by Dennis Williamson) -- this completely avoids the parsing problem, and is also faster (since -q makes grep stop searching at the first match). If that option weren't available, another method would be to protect the output with double-quotes: if [ "$(grep -w -- "$1" "$FILE")" ]; then (note that I also used $() instead of backquotes 'cause I find them much easier to read, and quotes around $FILE just in case it contains anything funny, like whitespace).
Though not applicable in this particular case, another technique can be used to prevent filenames that start with hyphens from being interpreted as options:
rm ./-x
or
rm /path/to/-x
I'm not entirely new to programming, but I'm not exactly experienced. I want to write small shell script for practice.
Here's what I have so far:
#!/bin/sh
name=$0
links=$3
owner=$4
if [ $# -ne 1 ]
then
echo "Usage: $0 <directory>"
exit 1
fi
if [ ! -e $1 ]
then
echo "$1 not found"
exit 1
elif [ -d $1 ]
then
echo "Name\t\tLinks\t\tOwner\t\tDate"
echo "$name\t$links\t$owner\t$date"
exit 0
fi
Basically what I'm trying to do is have the script go through all of the files in a specified directory and then display the name of each file with the amount of links it has, its owner, and the date it was created. What would be the syntax for displaying the date of creation or at least the date of last modification of the file?
Another thing is, what is the syntax for creating a for loop? From what I understand I would have to write something like for $1 in $1 ($1 being all of the files in the directory the user typed in correct?) and then go through checking each file and displaying the information for each one. How would I start and end the for loop (what is the syntax for this?).
As you can see I'm not very familiar bourne shell programming. If you have any helpful websites or have a better way of approaching this please show me!
Syntax for a for loop:
for var in list
do
echo $var
done
for example:
for var in *
do
echo $var
done
What you might want to consider however is something like this:
ls -l | while read perms links owner group size date1 date2 time filename
do
echo $filename
done
which splits the output of ls -l into fields on-the-fly so you don't need to do any splitting yourself.
The field-splitting is controlled by the shell-variable IFS, which by default contains a space, tab and newline. If you change this in a shell script, remember to change it back. Thus by changing the value of IFS you can, for example, parse CSV files by setting this to a comma. this example reads three fields from a CSV and spits out the 2nd and 3rd only (it's effectively the shell equivalent of cut -d, -f2,3 inputfile.csv)
oldifs=$IFS
IFS=","
while read field1 field2 field3
do
echo $field2 $field3
done < inputfile.csv
IFS=oldifs
(note: you don't need to revert IFS, but I generally do to make sure that further text processing in a script isn't affected after I'm done with it).
Plenty of documentation out the on both for and while loops; just google for it :-)
$1 is the first positional parameter, so $3 is the third and $4 is the fourth. They have nothing to do with the directory (or its files) the script was started from. If your script was started using this, for example:
./script.sh apple banana cherry date elderberry
then the variable $1 would equal "apple" and so on. The special parameter $# is the count of positional parameters, which in this case would be five.
The name of the script is contained in $0 and $* and $# are arrays that contain all the positional parameters which behave differently depending on whether they appear in quotes.
You can refer to the positional parameters using a substring-style index:
${#:2:1}
would give "banana" using the example above. And:
${#: -1}
or
${#:$#}
would give the last ("elderberry"). Note that the space before the minus sign is required in this context.
You might want to look at Advanced Bash-Scripting Guide. It has a section that explains loops.
I suggest to use find with the option -printf "%P\t%n\t%u\t%t"
for x in "$#"; do
echo "$x"
done
The "$#" protects any whitespace in supplied file names. Obviously, do your real work in place of "echo $x", which isn't doing much. But $# is all the junk supplied on the command line to your script.
But also, your script bails out if $# is not equal to 1, but you're apparently fully expecting up to 4 arguments (hence the $4 you reference in the early part of your script).
assuming you have GNU find on your system
find /path -type f -printf "filename: %f | hardlinks: %n| owner: %u | time: %TH %Tb %TY\n"