I use a program to rip radio music. Sadly one can not set the temporary folder apart from the folder where the finished mp3's end up later. So I cannot set the output folder to auto add to iTunes.
I'm alright in coding java and what not but have no experience with shell scripts.
I need a script that iterates through all the files within a folder like every 10 minutes and moves them to a different location if they don't start with the string "Track". All temp files are called "Track..." so it should only move finished ones then. Could anyone give me a help getting started?
Thanks!
Here's an example script. You should set the DESTINATION directory properly before uncommenting the line which moves the files. Otherwise, you may end up moving them somewhere undesirable.
In the terminal, cd to the location where you save the snippet below and run the following commands to execute.
Prep work:
cd /save/location
chmod +x file_mover.sh # makes the file executable
Schedule a job:
crontab -e
*/10 * * * * /path/to/file_mover.sh
crontab -l # view list of scheduled jobs
With some minor tweaks you can make this accept CLI options.
#!/bin/bash
# files to skip
REGEX='^TRACK'
# location to move the files
DESTINATION=/tmp/mydir
# directory to read from
# PWD is the working directory
TARGET=${PWD}
# make the directory(ies) if it doesn't exists
if [ ! -f ${DESTINATION} ]; then
mkdir -p ${DESTINATION}
fi
# get the collection of files in the
for FILE in $( ls ${TARGET} )
do
# if the current file does not begin with TRACK, move it
if [[ ! ${FILE} =~ ${REGEX} ]]; then
echo ${FILE}
# SET THE DESTINATION DIRECTORY BEFORE UNCOMMENTING THE LINE BELOW
# if [ -f ${FILE} ]; then # uncomment if you want to
# ensure it's a file and not a directory
# mv ${FILE} ${DESTINATION} # move the file
# fi # uncomment to ensure it's a file (end if)
fi
done
Edit the crontab with EDITOR=nano crontab -e and add a line like this:
*/10 * * * * shopt -s extglob; mv ~/Music/Temp/!(Track)*.mp3 ~/Music/iTunes/iTunes\ Media/Automatically\ Add\ to\ iTunes.localized/
shopt -s extglob adds support for !(). See /usr/share/doc/bash/bash.html.
Related
I am trying to make a bash script to create directories with the same name as each file in a given directory, then move said files to their respective directories, and then rename the files.
Basically - a quantum chemistry program that I use requires that the input files be named "ZMAT". So, if I have multiple jobs, I currently need to manually create directories, and then move the ZMAT files into them (can only run one job per folder).
When I run my code, I get "binary operator expected". I am not sure what this means. Some help please.
Here is what I have so far:
#!/bin/bash
if [ -e *.ZMAT ];
then
echo "CFOUR Job Detected"
for INPFILE in *.ZMAT; do
BASENAME=$(basename $INPFILE )
INPFILE=$BASENAME.ZMAT
OUTFILE=$BASENAME.out
XYZFILE=$BASENAME.xyz
ERRORFILE=$BASENAME.slu
if [ ! -e $ERRORFILE ];
then
# Create folder in scratch directory with the basename
mkdir /scratch/CFOUR/$BASENAME
# Move the file to its directory
mv -f $INPFILE /scratch/CFOUR/$BASENAME
# cd to the new directory
cd /scratch/CFOUR/$BASENAME
# Change the file name to just ZMAT
mv -f $INPFILE ZMAT
echo "Submitting CFOUR Job"
# Submit to scheduler
#RUN_COMMAND="sbatch -J $BASENAME _CFOUR_MRCC_SLURM.SUB"
#eval $RUN_COMMAND
else
echo "Error File Detected - Not Submitting Job"
fi
done
fi
An alternative would be to create symlinks to the original files.
As you said before, each ZMAT symlink would need to be in its own directory.
The upside is that the original data doesn't move, so less risk of breaking it, but the tool you want to use should read the symlinks as if they are the files it is looking for.
This one-liner creates an out directory in the current folder that you could subsequently move wherever you want it. You could easily create it where you do want it by replacing "out" with whatever absolute path you wanted
for i in *.ZMAT; do mkdir -p out/$i ; ln -s $PWD/$i out/$i/ZMAT ; done
I believe I have solved my problem. Here is the new script, which appears to be working fine. Any input is welcome though!
#!/bin/bash
SUBDIR=$(pwd)
for i in *.ZMAT; do
BASENAME=$(basename $i .ZMAT)
INPFILE=$BASENAME.ZMAT
OUTFILE=$BASENAME.out
XYZFILE=$BASENAME.xyz
ERRORFILE=$BASENAME.slu
if [ ! -e $ERRORFILE ];
then
mkdir /scratch/CFOUR/$BASENAME # Create Scratch Folder
cp $INPFILE /scratch/cdc/CFOUR/$BASENAME # Move Input to Scratch
cd /scratch/CFOUR/$BASENAME #cd to Scratch Folder
mv -f $INPFILE ZMAT # Change Input Name
echo "Submitting CFOUR Job"
# Submit to scheduler
#RUN_COMMAND="sbatch -J $BASENAME _CFOUR_MRCC_SLURM.SUB"
#eval $RUN_COMMAND
cd $SUBDIR #Go back to SUBDIR
else
echo "Error File Already Exists"
fi
done
Basically I want to do something like this from bash.
if a file exists in a directory rename,move,whatever
if it doesn't exist loop every 60 seconds:
# Create ~/bin
cd ~/
if dir ~/bin does not exist
then mkdir ~/bin
#!/bin/bash
# Create ~/blahhed && ~/blahs
if dir ~/blahhed does not exist
then mkdir ~/blahhed
if dir ~/blahs does not exist
then mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
if ~/blahhed/file exists
then mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
else loop for 60s
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if ~/blahs/file does exist
then mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
else loop for 60s
Ok Ive rewritten it like this am I on the right track here?
# Create ~/bin
cd ~/
if [! -d ~/bin]; then
mkdir ~/bin
if [ -d ~/bin]; then
#!/bin/bash
# Create ~/blahhed && ~/blahs
if [! -d ~/blahhed]; then
mkdir ~/blahhed
if [! -d ~/blahs]; then
mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
while if [ -d ~/blahhed/file]; then
do
mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
continue
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if [! -d ~/blahs/file]; then
mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
sleep 60 seconds
You could use watch(1) which is able to run a program or script every N seconds.
To run some script every few minutes (not seconds) - or every few hours or days, use some crontab(5) entries. To run it at some given (relative or absolute) time, consider at(1) (which you might use with some here document in your shell terminal, etc...).
However, to execute commands when a file exists or changes, you might use make(1) (which you could run from watch); that command is configurable in a Makefile (see documentation of GNU make)
And if you really care about file appearing or changing (and doing something on such changes), consider using inotify(7) based facilities, e.g. incrond with incrontab(5)
To test existence of directories or files, use test(1) often spelt as a [ , e.g.
## test in a script if directory ~/foo/ exist
if [ -d ~/foo/ ]; then
echo the directory foo exists
fi
Spaces are important above. You could use [ -d "$HOME/foo/" ]
It may look that you want to mimick logrotate(8). See also syslog(3) library function and logger(1) command.
To debug your bash script, start it (-see execve(2) & bash(1) for details- temporarily, while debugging) with
#!/bin/bash -vx
and make your foo.sh script executable with chmod a+x foo.sh
To stop execution of some script for some seconds, use sleep(1)
The mkdir(1) command accepts -p (and then won't create a directory if it already exists). mv(1) has also many options (including for backup).
To search some files in a file tree, use find(1). To search some content inside files, use grep. I also like ack
Read also Advanced Bash Scripting Guide & (if coding in C ...) Advanced Linux Programming and also the documentation of GNU bash (e.g. for shell builtins and control statements).
Did you consider using some revision control system like git ? It is useful to manage the evolution of source files (including shell scripts)
I've seen solutions similar to what you are asking, but using crontab with find -mmin 1 which will search for any files with a modtime <= 60 seconds within specified location.
Something along these lines (untested):
$ -> vi /tmp/file_finder.sh
# Add the following lines
#!/bin/bash
find /path/to/check -mmin 1 -type -f | while read fname; do
echo "$fname"
done
# Change perms
$ -> chmod 755 /tmp/file_finder.sh
$ -> crontab -e
* * * * * /tmp/file_finder.sh
With the above, you have now setup the cron to run every minute, and kick off a script that will search given directory for files with a modtime <= 60 seconds (new or updated).
Caveat: You should look for files with a mod time up to 5 minutes, that way you don't consider a file which may still be in the process of being written too.
I think you answered yourself (kind of)
Some suggestions:
1- use a while loop and at the end add sleep 60
2- write your procedure in a file (ex.; test1)
and then
watch -n 60 ./test1
I have a cron job which runs every 30 minutes to generate log files with time-stamp like this:
test20130215100531.log,
test20130215102031.log
I would like to create one folder daily with date time-stamp and push log files in to respective date folder when generated.
I need to achieve this on AIX server with bash.
Maybe you are looking for a script like this:
#!/bin/bash
shopt -s nullglob # This line is so that it does not complain when no logfiles are found
for filename in test*.log; do # Files considered are the ones starting with test and ending in .log
foldername=$(echo "$filename" | awk '{print (substr($0, 5, 8));}'); # The foldername is characters 5 to 13 from the filename (if they exist)
mkdir -p "$foldername" # -p so that we don't get "folder exists" warning
mv "$filename" "$foldername"
echo "$filename $foldername" ;
done
I only tested with your sample, so do a proper testing before using in a directory that contains important stuff.
Edit in response to comments:
Change your original script to this:
foldername=$(date +%Y%m%d)
mkdir -p /home/app/logs/"$foldername"
sh sample.sh > /home/app/logs/"$foldername"/test$(date +%Y%m%d%H%M%S).log
Or if the directory is created somewhere else, just do this:
sh sample.sh > /home/app/logs/$(date +%Y%m%d)/test$(date +%Y%m%d%H%M%S).log
You should use logrotate! It can do this for you already, and you can just write to the same log file.
Check their man pages for info:
http://linuxcommand.org/man_pages/logrotate8.html
For class we have to work on a remote server that the school hosts. So far I have made a lot of files on the server and I would like to back them up in case I want to transfer them to my laptop or in case I accidentally delete a directory or make a silly error. I found a tutorial and a script to back up the file and I decided to modify it so that it would determine what directory it's in (which will be the main user's) and the cd to the Documents. It also creates the directory Backups if it doesn't exist. I am still pretty new to this sort of scripting and any additional advice or post links would be greatly appreciated.
Code:
#!/bin/bash
#######################################################
## Simple backup script..
## Created by Matthew Brunt: (openblue555#gmail.com)
## Licensed under GNU GPL v3 or later, at your option.
## http://www.gnu.org/licenses/gpl.html
##
## Further edited by Michael Garrison to backup the
## directory it is located in and print the contents.
#######################################################
mkdir -p Backup
#Defines our output file
OUTPUT= $( cd Backup && pwd )/backup_$(date +%Y%m%d).tar.gz
#Defines our directory to backup
BUDIR=$( cd Desktop && pwd )
#Display message about starting the backup
echo "Starting backup of directory $BUDIR to file $OUTPUT"
#Start the backup
tar -cZf $OUTPUT $BUDIR
#Checking the status of the last process:
if [ $? == 0 ]; then
#Display confirmation message
echo "The file:"
echo $OUTPUT
echo "was created as a backup for:"
echo $BUDIR
echo ""
echo "Items that were backed up include:"
for i in $BUDIR; do
echo $i
done
echo ""
else
#Display error message message
echo "There was a problem creating:"
echo $OUTPUT
echo "as a backup for:"
echo $BUDIR
fi
I know that the original script works and it worked until I changed the $OUTPUT variable. I currently get the following result:
./backup.sh
./backup.sh: line 15: /Users/mgarrison93/Backup/backup_20121004.tar.gz: No such file
or directory
Starting backup of directory /Users/mgarrison93/Desktop to file
tar: no files or directories specified
There was a problem creating:
as a backup for:
/Users/mgarrison93/Desktop
I can see that it is not accepting the file name, but I don't know how to correct this.
I just tried changing $OUTPUT to /Backups/file-name.tar.gz which I originally had and it works fine. The problem seems to be $( cd Backup && pwd )/backup_$(date +%Y%m%d).tar.gz. Just not sure what is wrong.
Consider these two entirely different pieces of bash syntax: first, you have the syntax for setting a variable to a value permanently (in the current script),
<variable>=<value>
and then there is the syntax for running a command with a variable temporarily set to a value ,
<variable>=<value> <command> <argument> ...
The difference between these two is the space. After the =, once bash runs into an unquoted space, it takes that to mean that the <value> has ended, and anything after it is interpreted as the <command>.
In this line of your script,
OUTPUT= $( cd Backup && pwd )/backup_$(date +%Y%m%d).tar.gz
you have a space after OUTPUT=. bash interprets that to mean that OUTPUT is to be (temporarily) set to the empty string, and the rest of the line, i.e. the result of $( cd Backup && pwd )/backup_$(date +%Y%m%d).tar.gz, is a command and arguments to be run while OUTPUT is equal to the empty string.
The solution is to remove the space. That way bash will know that you're trying to assign the rest of the line as a value to the variable.
I use to create a tempfile, delete it and recreate it as a directory:
temp=`tempfile`
rm -f $temp
# <breakpoint>
mkdir $temp
The problem is, when it runs to the <breakpoint>, there happens to be another program wants to do the same thing, which mkdir-ed a temp dir with the same name, will cause the failure of this program.
Use mktemp -d. It creates a temporary directory with a random name and makes sure that file doesn't already exist. You need to remember to delete the directory after using it though.
For a more robust solution i use something like the following. That way the temp dir will always be deleted after the script exits.
The cleanup function is executed on the EXIT signal. That guarantees that the cleanup function is always called, even if the script aborts somewhere.
#!/bin/bash
# the directory of the script
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# the temp directory used, within $DIR
# omit the -p parameter to create a temporal directory in the default location
WORK_DIR=`mktemp -d -p "$DIR"`
# check if tmp dir was created
if [[ ! "$WORK_DIR" || ! -d "$WORK_DIR" ]]; then
echo "Could not create temp dir"
exit 1
fi
# deletes the temp directory
function cleanup {
rm -rf "$WORK_DIR"
echo "Deleted temp working directory $WORK_DIR"
}
# register the cleanup function to be called on the EXIT signal
trap cleanup EXIT
# implementation of script starts here
...
Directory of bash script from here.
Bash traps.
My favorite one-liner for this is
cd $(mktemp -d)
The following snippet will safely create and then clean up a temporary directory.
The first trap line executes exit 1 command when any of the specified signals is received. The second trap line removes the $TEMPD on program's exit (both normal and abnormal). We initialize these traps after we check that mkdir -d succeeded to avoid accidentally executing the exit trap with $TEMPD in an unknown state.
#!/bin/bash
# set -x # un-comment to see what's going on when you run the script
# Create a temporary directory and store its name in a variable.
TEMPD=$(mktemp -d)
# Exit if the temp directory wasn't created successfully.
if [ ! -e "$TEMPD" ]; then
>&2 echo "Failed to create temp directory"
exit 1
fi
# Make sure the temp directory gets removed on script exit.
trap "exit 1" HUP INT PIPE QUIT TERM
trap 'rm -rf "$TEMPD"' EXIT
Here is a simple explanation about how to create a temp dir using templates.
Creates a temporary file or directory, safely, and prints its name.
TEMPLATE must contain at least 3 consecutive 'X's in last component.
If TEMPLATE is not specified, it will use tmp.XXXXXXXXXX
directories created are u+rwx, minus umask restrictions.
PARENT_DIR=./temp_dirs # (optional) specify a dir for your tempdirs
mkdir $PARENT_DIR
TEMPLATE_PREFIX='tmp' # prefix of your new tempdir template
TEMPLATE_RANDOM='XXXX' # Increase the Xs for more random characters
TEMPLATE=${PARENT_DIR}/${TEMPLATE_PREFIX}.${TEMPLATE_RANDOM}
# create the tempdir using your custom $TEMPLATE, which may include
# a path such as a parent dir, and assign the new path to a var
NEW_TEMP_DIR_PATH=$(mktemp -d $TEMPLATE)
echo $NEW_TEMP_DIR_PATH
# create the tempdir in parent dir, using default template
# 'tmp.XXXXXXXXXX' and assign the new path to a var
NEW_TEMP_DIR_PATH=$(mktemp -p $PARENT_DIR)
echo $NEW_TEMP_DIR_PATH
# create a tempdir in your systems default tmp path e.g. /tmp
# using the default template 'tmp.XXXXXXXXXX' and assign path to var
NEW_TEMP_DIR_PATH=$(mktemp -d)
echo $NEW_TEMP_DIR_PATH
# Do whatever you want with your generated temp dir and var holding its path
I need the following features:
Put all temp files into a single directory with a specific namespace for reusing.
Create temp files with filename prefix and suffix (extension).
With bash script on macOS:
$ namespace="com.namespace.mktemp"
# find directory for reusing
$ ls -d "${TMPDIR}${namespace}"*
# create directory if not exists
$ mktemp -d -t "$namespace"
/var/folders/s_/.../T/com.namespace.mktemp.HjqGT6w2
# create tempfile with directory name and file prefix
$ mktemp -t "com.namespace.mktemp.HjqGT6w2/file-prefix"
/var/folders/s_/.../T/com.namespace.mktemp.HjqGT6w2/file-prefix.sZDvjo14
# add suffix - `mktemp` on macOS does not support `--suffix`
mv "/var/folders/s_/.../file-prefix.sZDvjo14" "/var/folders/s_/.../file-prefix.sZDvjo14.txt"
The gmktemp (brew install coreutils) is a little different:
supports --suffix and --tmpdir
Xs are required in template and prefix
template should not contain directory, set TMPDIR instead
$ namespace="com.namespace.gmktemp"
# create directory if not exists
$ gmktemp -d -t "$namespace.XXXXXXXX"
/var/folders/s_/.../T/com.namespace.gmktemp.BjFtIAyZ
# set TMPDIR
TMPDIR="/var/folders/s_/.../T/com.namespace.gmktemp.BjFtIAyZ"
# create tempfile with directory name and file prefix
$ gmktemp --suffix=".txt" -t "prefix.XXXXXXXX"
/var/folders/s_/.../T/com.namespace.gmktemp.BjFtIAyZ/prefix.LWHj0G95.txt