I was able to script the backup process, but I want to make an another script for my storage server for a basic file rotation.
What I want to make:
I want to store my files in my /home/user/backup folder. Only want to store the 10 most fresh backup files and name them like this:
site_foo_date_1.tar site_foo_date_2.tar ... site_foo_date_10.tar
site_foo_date_1.tar being the most recent backup file.
Past num10 the file will be deleted.
My incoming files from the other server are simply named like this: site_foo_date.tar
How can I do this?
I tried:
DATE=`date "+%Y%m%d"`
cd /home/user/backup/com
if [ -f site_com_*_10.tar ]
then
rm site_com_*_10.tar
fi
FILES=$(ls)
for file in $FILES
do
echo "$file"
if [ "$file" != "site_com_${DATE}.tar" ]
then
str_new=${file:18:1}
new_str=$((str_new + 1))
to_rename=${file::18}
mv "${file}" "$to_rename$new_str.tar"
fi
done
file=$(ls | grep site_com_${DATE}.tar)
filename=`echo "$file" | cut -d'.' -f1`
mv "${file}" "${filename}_1.tar"
The main issue with your code is that looping through all files in the directory with ls * without some sort of filter is a dangerous thing to do.
Instead, I've used for i in $(seq 9 -1 1) to loop through files from *_9 to *_1 to move them. This ensures we only move backup files, and nothing else that may have accidentally got into the backup directory.
Additionally, relying on the sequence number to be the 18th character in the filename is also destined to break. What happens if you want more than 10 backups in the future? With this design, you can change 9 to be any number you like, even if it's more than 2 digits.
Finally, I added a check before moving site_com_${DATE}.tar in case it doesn't exist.
#!/bin/bash
DATE=`date "+%Y%m%d"`
cd "/home/user/backup/com"
if [ -f "site_com_*_10.tar" ]
then
rm "site_com_*_10.tar"
fi
# Instead of wildcarding all files in the directory
# this method picks out only the expected files so non-backup
# files are not changed. The renumbering is also made easier
# this way.
# Loop through from 9 to 1 in descending order otherwise
# the same file will be moved on each iteration
for i in $(seq 9 -1 1)
do
# Find and expand the requested file
file=$(find . -maxdepth 1 -name "site_com_*_${i}.tar")
if [ -f "$file" ]
then
echo "$file"
# Create new file name
new_str=$((i + 1))
to_rename=${file%_${i}.tar}
mv "${file}" "${to_rename}_${new_str}.tar"
fi
done
# Check for latest backup file
# and only move it if it exists.
file=site_com_${DATE}.tar
if [ -f $file ]
then
filename=${file%.tar}
mv "${file}" "${filename}_1.tar"
fi
Related
I'm trying to automate the packaging of files and contents from various sources using a bash script.
I have a main directory which contains pdf files, a csv file, and various folders with additional contents. The folders are named with the location code they pertain to, e.g. 190, 191, etc.
A typical row in my csv file looks like this: form_letters_Part1.pdf,PX_A31_smith.adam.pdf,190,
Where the first column is the original pdf name, the second is what it will be renamed to, and the third column is the location code the person belongs to.
The first part of my script renames the pdf files from the cover letters format to the PX_A31... format, and then creates a directory for each file and moves them into it.
#!/usr/bin/tcsh bash
sed 's/"//g' rename_list_lab.csv | while IFS=, read orig new num; do
mv "$orig" "$new"
done
echo 'Rename Done.'
for file in *.pdf; do
mkdir "${file%.*}"
mv "$file" "${file%.*}"
done
echo 'Directory creation done.'
What needs to happen next is the folders with the location-specific contents get copied into those new directories just created, corresponding to the location code from the csv file.
So I tried this after the above echo 'Directory Creation Done.' line:
echo 'Directory Creation Done.'
sed 's/"//g' rename_list.csv | while IFS=, read orig new num; do
for folder in *; do
if [[ -d .* = "$num" ]]; then
cp -R "$folder" "${file%.*}"
fi
done
echo 'Code Folder Contents Sort Done.'
However this results in a syntax error:
syntax error in conditional expression
syntax error near `='
` if [[ -d .* = "$num" ]]; then'
EDIT: To clarify the second part if statement, the intended logic of the statement is as follows: For the items in the current directory, if it is a directory, and the name of the directory matches the location code from the csv, that directory should be copied to any directories which have that same corresponding location code in the csv.
In other words, if the newly created directory from the first part is PX_A31_smith.adam whose location code in the csv line above is 190, then the folder called 190 should be copied into the directory PX_A31_smith.adam.
If three other people also have the 190 code in the csv, the 190 directory should also be copied to those as well.
EDIT 2: I resolved the syntax error, and also realized I had an nonterminated do statement. Fixing those, still seem to be having trouble with the evaluation of the if statement. Updated script below:
#!/usr/bin/tcsh bash
sed 's/"//g' rename_list.csv | while IFS=, read orig new num; do
mv "$orig" "$new"
done
echo '1 Done.'
for file in *.pdf; do
mkdir "${file%.*}"
mv "$file" "${file%.*}"
done
echo '2 done.'
sed 's/"//g' rename_list.csv | while IFS=, read orig new num; do
for folder in * ; do
if [[ .* = "$num" ]]; then
cp -R "$folder" "${file%.*}"
else echo "No matches found."
fi
done
done
echo '3 Done.'
I'm not really sure if this answers your question, but I think it will at least set you on the right track. Structurally, I just combined all of the loops into one. This removes some of the possible logic errors that would not be considered syntax errors like the use of $file in the second part. This is a local variable to the loop in the first part and no longer exists. However, this would be interpreted as an empty string.
#!/usr/bin/bash
#^Fixed shebang line.
sed 's/"//g' rename_list.csv | while IFS=, read -r orig new num; do
if [[ -f $orig ]]; then #If the file we want to rename is indeed a file.
mkdir "${new%.*}" #make the directory from the file name you want
mv "$orig" "${new%.*}/$new" #Rename when we move the file into the new directory
if [[ -d $num ]]; then #If the number directory exists
cp -R "$num" "${new%.*}" #Fixed this based on your edit.
else
#Here you can handle what to do if the number directory does not exist.
echo "$num is not a directory."
fi
else
#Here you can handle what to do if the file does not exist.
echo "The file $orig does not exist."
fi
done
Edited based on your clarification
Note: This is pretty lacking as far as error checking goes. Remember, any of these functions could fail, which will have unwanted behavior. Either check if [[ $? != 0 ]] to check the exit status (0 being success) of the last issued command. You could also do something like mkdir somedir || exit 2 to exit on failure.
For some development work I need to remove some "noise" from a series of Log files all stored in a folder. (I have this on Linux, but can also do this in Windows.) A line that I want to remove would look like this:
Sep 5/2017 23:59:50:324 [MISC ]: ValueType:ST / SetId: / ObID:0002-d007^RhySta^MDIL / ObSubID:0 / Detail:Sinus Rhythm / Units: / AccessChecks: / ObxTimeStamp:
Anytime I see [MISC ]: I want to remove the whole line and leave nothing in its place. As soon as lines are deleted from the file I want to move to save the file with existing name, and then move to the next file in the folder.
I am not a scripter.. thus the request for assistance.
Here is one to do it using for, find, sed and mv. .
Sample dir:
[zee]$ find .
./test2/file1.tx
./test2
One-liner:
[zee]$ for file in $(find test1/ -type f) ; do echo "checking if file $file has a match" ; grep -q "MIS" $x ; if [ $? -eq 0 ]; then echo "$file has a match" ; sed -i '/MIS/d' $file && echo "deleted matches...moving $file" && mv $file test2/ ; fi ; done
Output of running the above one-liner:
checking if file test1/file1.tx has a match
test1/file1.tx has a match
deleted matches...moving test1/file1.tx
Here is what it does: find all files in directory "test1", checks if a given file has a match. If the file has a match, it removes the match(s) and move the file to directory "test2".
This is my first day scripting, I use linux but needed a script that I have been racking my brain until i finally ask for help. I need to check a directory that has directories already present to see if any new directories are added that are not expected.
Ok I think i have got this as simple as possible. The below works but displays all files in the directory as well. I will keep working at it unless someone can tell me how not to list the files too | I tried ls -d but it is doing the echo "nothing new". I feel like an idiot and should have got this sooner.
#!/bin/bash
workingdirs=`ls ~/ | grep -viE "temp1|temp2|temp3"`
if [ -d "$workingdirs" ]
then
echo "nothing new"
else
echo "The following Direcetories are now present"
echo ""
echo "$workingdirs"
fi
If you want to take some action when a new directory is created, used inotifywait. If you just want to check to see that the directories that exist are the ones you expect, you could do something like:
trap 'rm -f $TMPDIR/manifest' 0
# Create the expected values. Really, you should hand edit
# the manifest, but this is just for demonstration.
find "$Workingdir" -maxdepth 1 -type d > $TMPDIR/manifest
while true; do
sleep 60 # Check every 60 seconds. Modify period as needed, or
# (recommended) use inotifywait
if ! find "$Workingdir" -maxdepth 1 -type d | cmp - $TMPDIR/manifest; then
: Unexpected directories exist or have been removed
fi
done
Below shell script will show directory present or not.
#!/bin/bash
Workingdir=/root/working/
knowndir1=/root/working/temp1
knowndir2=/root/working/temp2
knowndir3=/root/working/temp3
my=/home/learning/perl
arr=($Workingdir $knowndir1 $knowndir2 $knowndir3 $my) #creating an array
for i in ${arr[#]} #checking for each element in array
do
if [ -d $i ]
then
echo "directory $i present"
else
echo "directory $i not present"
fi
done
output:
directory /root/working/ not present
directory /root/working/temp1 not present
directory /root/working/temp2 not present
directory /root/working/temp3 not present
**directory /home/learning/perl present**
This will save the available directories in a list to a file. When you run the script a second time, it will report directories that have been deleted or added.
#!/bin/sh
dirlist="$HOME/dirlist" # dir list file for saving state between runs
topdir='/some/path' # the directory you want to keep track of
tmpfile=$(mktemp)
find "$topdir" -type d -print | sort -o "$tmpfile"
if [ -f "$dirlist" ] && ! cmp -s "$dirlist" "$tmpfile"; then
echo 'Directories added:'
comm -1 -3 "$dirlist" "$tmpfile"
echo 'Directories removed:'
comm -2 -3 "$dirlist" "$tmpfile"
else
echo 'No changes'
fi
mv "$tmpfile" "$dirlist"
The script will have problems with directories that have very exotic names (containing newlines).
I've been using the following shell bin/bash script as an app which I can drop a folder on, and it will update the date modified of the folder to match the most recently modified file in that folder.
for f in each "$#"
do
echo "$f"
done
$HOME/setMod "$#"
This gets the folder name, and then passes it to this setMod script in my home folder.
#!/bin/bash
# Check that exactly one parameter has been specified - the directory
if [ $# -eq 1 ]; then
# Go to that directory or give up and die
cd "$1" || exit 1
# Get name of newest file
newest=$(stat -f "%m:%N" * | sort -rn | head -1 | cut -f2 -d:)
# Set modification date of folder to match
touch -r "$newest" .
fi
However, if I drop more than one folder on it at a time, it won't work, and I can't figure out how to make it work with multiple folders at once.
Also, I learned from Apple Support that the reason so many of my folders keep getting the mod date updated is due to some Time Machine-related process, despite the fact I haven't touched some of them in years. If anyone knows of a way to prevent this from happening, or to somehow automatically periodically update the date modified of folders to match the date/time of the most-recently-modified file in them, that would save me from having to run this step manually pretty regularly.
The setMod script current accepts only one parameter.
You could either make it accept many parameters and loop over them,
or you could make the calling script use a loop.
I take the second option, because the caller script has some mistakes and weak points. Here it is corrected and extended for your purpose:
for dir; do
echo "$dir"
"$HOME"/setMod "$dir"
done
Or to make setMod accept multiple parameters:
#!/bin/bash
setMod() {
cd "$1" || return 1
# Get name of newest file
newest=$(stat -f "%m:%N" * | sort -rn | head -1 | cut -f2 -d:)
# Set modification date of folder to match
touch -r "$newest" .
}
for dir; do
if [ ! -d "$dir" ]; then
echo not a directory, skipping: $dir
continue
fi
(setMod "$dir")
done
Notes:
for dir; do is equivalent to for dir in "$#"; do
The parentheses around (setMod "$dir") make it run in a sub-shell, so that the script itself doesn't change the working directory, the effect of the cd operation is limited to the sub-shell within (...)
Unfortunately my knowledge in bash not so well and I have very non-standard task.
I have a file with the files list.
Example: /tmp/my/file1.txt /tmp/my/file2.txt
How can I write a script which can check that files from folder /tmp/my exist and to have two types messages after script is done.
1 - Files exist and show files:
/tmp/my/file1.txt
/tmp/my/file2.txt
2 - The folder /tmp/my including files and folders which are not in your list. The files and folders:
/tmp/my/test
/tmp/my/1.txt
You speak of files and folders, which seems unclear.
Anyways, I wanted to try it with arrays, so here we go :
unset valid_paths; declare -a valid_paths
unset invalid_paths; declare -a invalid_paths
while read -r line
do
if [ -e "$line" ]
then
valid_paths=("${valid_paths[#]}" "$line")
else
invalid_paths=("${invalid_paths[#]}" "$line")
fi
done < files.txt
echo "VALID PATHS:"; echo "${valid_paths[#]}"
echo "INVALID PATHS:"; echo "${invalid_paths[#]}"
You can check for the files' existence (assuming a list of files, one filename per line) and print the existing ones with a prefix using this
# Part 1 - check list contents for files
while read thefile; do
if [[ -n "$thefile" ]] && [[ -f "/tmp/my/$thefile" ]]; then
echo "Y: $thefile"
else
echo "N: $thefile"
fi
done < filelist.txt | sort
# Part 2 - check existing files against list
for filepath in /tmp/my/* ; do
filename="$(basename "$filepath")"
grep "$filename" filelist.txt -q || echo "U: $filename"
done
The files that exist are prefixed here with Y:, all others are prefixed with N:
In the second section, files in the tmp directory that are not in the file list are labelled with U: (unaccounted for/unexpected)
You can swap the -f test which checks that a path exists and is a regular file for -d (exists and is a directory) or -e (exists)
See
man test
for more options.