I am working on bash to create a back up system. My code is
#!/bin/bash
if [ ! -d "BackUp" ]
then
mkdir BackUp
fi
echo "enter number of access days you want to take for back up."
read days
bak="$(find . -mtime +$days)"
for file in $bak
do
mv $file BackUp
done
tar -cvf BackUp.tgz BackUp >> backUp.log
So, currently I am only taking log file from tar. so it does not prints the full path it only takes current working directory for text in log file.My last line of code takes up input for log file.
But the path stored is
.BackUp/foo1
.BackUp/foo2
.BackUp/foo3
instead i want it to be
home/ubuntu/Downloads/BackUp/foo1
home/ubuntu/Downloads/BackUp/foo2
home/ubuntu/Downloads/BackUp/foo3
You could store the absolute path in a variable and use it in the tar command:
BackUpDirFullPath=$(cd BackUp && pwd)
As command substitution invokes a subshell you are not leaving the current directory by executing cd.
Update:
In order to make -v output absolute paths (on Mac OS) I had to change to the root directory in a subshell and execute it from there ... something like that:
(cd / && tar -cvf /$OLDPWD/BackUp.tgz $BackUpDirFullPath)
This does output absolute paths ... in order to preserve the leading / you might try -P which preserves path names.
Related
EDIT: DIR_trash="trash"
I wrote a function to move a file to current directory.
if [ "$1" == "-u" ]
then
if [ $# == 1 ]
then
echo "Something went wrong. Please make sure you're passing the name of the file/directory after '-u'."
else
if [ -f $DIR_trash/$2.zip ]
then
echo "$2.zip has been found in the trash."
cd
cd $DIR_trash
sed -i "/$2/d" $file7
mv -i /$DIR_trash/$2.zip .
unzip $2.zip
\rm $2.zip
cd
else
echo "$2.zip has not been found in the trash."
fi
fi
fi
As you can see, there is a line of code which says:
mv -i /DIR_trash/$2.zip .
So basically I'm trying to move a file that I passed in argument 2 to current directory, from trash. I always run this script from home directory, which does have trash directory. This is what I get when I run this:
Whenever I manually write this is in the Konsole (from home direcotry) it does work:
rm -u trash/d1 .
I'm out of ideas. Could anyone please help?
Let's say you run the script with the current directory being /some/where, and with the arguments -u and d1. I'll also assume that your home directory is /home/ninini. Let's look at where your script looks for files.
DIR_trash="trash"
if [ -f $DIR_trash/$2.zip ]
You check if /some/where/trash/d1.zip exists.
cd
cd $DIR_trash
Assuming both cd commands succeed, the current directory is now /home/ninini/trash.
mv -i /$DIR_trash/$2.zip .
You're saying to move /trash/d1.zip to the current directory, which is /home/ninini/trash.
Neither the source nor the destination make sense. The source /$DIR_trash doesn't make sense: why would you be looking for a directory called trash under the root directory? And the destination doesn't make sense since you just attempted to change to the trash directory, and now you're attempting to move a file out of the trash directory… into the trash directory.
I can't tell what the correct code is because you didn't say what the script is meant to do. You do say that you want to “to move a file to current directory”; then you must not change the current directory midway through the script! Assuming that the path $DIR_trash/$2.zip from the test command is the correct one, remove the cd commands and write
mv -i -- "$DIR_trash/$2.zip" .
Note that this moves the file from a directory called trash under the current directory. If this isn't what you wanted, you need to change the definition of DIR_trash. It should probably be an absolute path, perhaps
DIR_trash=~/trash
Note also that your script breaks on files containing whitespace and other special characters. Always put double quotes around variable substitutions: "$VAR", not $VAR. (Exception: when you know you need some effect that the double quotes prevent, and you understand why it's safe to leave them out.)
I am running numerous simulations on a remote server (via ssh). The outcomes of these simulations are stored as .tar archives in an archive directory on this remote server.
What I would like to do, is write a bash script which connects to the remote server via ssh and extracts the required output files from each .tar archive into separate folders on my local hard drive.
These folders should have the same name as the .tar file from which the files come (To give an example, say the output of simulation 1 is stored in the archive S1.tar on the remote server, I want all '.dat' and '.def' files within this .tar archive to be extracted to a directory S1 on my local drive).
For the extraction itself, I was trying:
for f in *.tar; do
(
mkdir ../${f%.tar}
tar -x -f "$f" -C ../${f%.tar} "*.dat" "*.def"
)
done
wait
Every .tar file is around 1GB and there is a lot of them. So downloading everything takes too much time, which is why I only want to extract the necessary files (see the extensions in the code above).
Now the code works perfectly when I have the .tar files on my local drive. However, what I can't figure out is how I can do it without first having to download all the .tar archives from the server.
When I first connect to the remote server via ssh username#host, then the terminal stops with the script and just connects to the server.
Btw I am doing this in VS Code and running the script through terminal on my MacBook.
I hope I have described it clear enough. Thanks for the help!
Stream the results of tar back with filenames via SSH
To get the data you wish to retrieve from .tar files, you'll need to pass the results of tar to a string of commands with the --to-command option. In the example below, we'll run three commands.
# Send the files name back to your shell
echo $TAR_FILENAME
# Send the contents of the file back
cat /dev/stdin
# Send EOF (Ctrl+d) back (note: since we're already in a $'' we don't use the $ again)
echo '\004'
Once the information is captured in your shell, we can start to process the data. This is a three-step process.
Get the file's name
note that, in this code, we aren't handling directories at all (simply stripping them away; i.e. dir/1.dat -> 1.dat)
you can write code to create directories for the file by replacing the forward slashes / with spaces and iterating over each directory name but that seems out-of-scope for this.
Check for the EOF (end-of-file)
Add content to file
# Get the files via ssh and tar
files=$(ssh -n <user#server> $'tar -xf <tar-file> --wildcards \'*\' --to-command=$\'echo $TAR_FILENAME; cat /dev/stdin; echo \'\004\'\'')
# Keeps track of what state we're in (filename or content)
state="filename"
filename=""
# Each line is one of these:
# - file's name
# - file's data
# - EOF
while read line; do
if [[ $state == "filename" ]]; then
filename=${line/*\//}
touch $filename
echo "Copying: $filename"
state="content"
elif [[ $state == "content" ]]; then
# look for EOF (ctrl+d)
if [[ $line == $'\004' ]]; then
filename=""
state="filename"
else
# append data to file
echo $line >> <output-folder>/$filename
fi
fi
# Double quotes here are very important
done < <(echo -e "$files")
Alternative: tar + scp
If the above example seems overly complex for what it's doing, it is. An alternative that touches the disk more and requires to separate ssh connections would be to extract the files you need from your .tar file to a folder and scp that folder back to your workstation.
ssh -n <username>#<server> 'mkdir output/; tar -C output/ -xf <tar-file> --wildcards *.dat *.def'
scp -r <username>#<server>:output/ ./
The breakdown
First, we'll make a place to keep our outputted files. You can skip this if you already know the folder they'll be in.
mkdir output/
Then, we'll extract the matching files to this folder we created (if you don't want them to be in a different folder remove the -C output/ option).
tar -C output/ -xf <tar-file> --wildcards *.dat *.def
Lastly, now that we're running commands on our machine again, we can run scp to reconnect to the remote machine and pull the files back.
scp -r <username>#<server>:output/ ./
Basically I want to do something like this from bash.
if a file exists in a directory rename,move,whatever
if it doesn't exist loop every 60 seconds:
# Create ~/bin
cd ~/
if dir ~/bin does not exist
then mkdir ~/bin
#!/bin/bash
# Create ~/blahhed && ~/blahs
if dir ~/blahhed does not exist
then mkdir ~/blahhed
if dir ~/blahs does not exist
then mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
if ~/blahhed/file exists
then mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
else loop for 60s
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if ~/blahs/file does exist
then mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
else loop for 60s
Ok Ive rewritten it like this am I on the right track here?
# Create ~/bin
cd ~/
if [! -d ~/bin]; then
mkdir ~/bin
if [ -d ~/bin]; then
#!/bin/bash
# Create ~/blahhed && ~/blahs
if [! -d ~/blahhed]; then
mkdir ~/blahhed
if [! -d ~/blahs]; then
mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
while if [ -d ~/blahhed/file]; then
do
mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
continue
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if [! -d ~/blahs/file]; then
mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
sleep 60 seconds
You could use watch(1) which is able to run a program or script every N seconds.
To run some script every few minutes (not seconds) - or every few hours or days, use some crontab(5) entries. To run it at some given (relative or absolute) time, consider at(1) (which you might use with some here document in your shell terminal, etc...).
However, to execute commands when a file exists or changes, you might use make(1) (which you could run from watch); that command is configurable in a Makefile (see documentation of GNU make)
And if you really care about file appearing or changing (and doing something on such changes), consider using inotify(7) based facilities, e.g. incrond with incrontab(5)
To test existence of directories or files, use test(1) often spelt as a [ , e.g.
## test in a script if directory ~/foo/ exist
if [ -d ~/foo/ ]; then
echo the directory foo exists
fi
Spaces are important above. You could use [ -d "$HOME/foo/" ]
It may look that you want to mimick logrotate(8). See also syslog(3) library function and logger(1) command.
To debug your bash script, start it (-see execve(2) & bash(1) for details- temporarily, while debugging) with
#!/bin/bash -vx
and make your foo.sh script executable with chmod a+x foo.sh
To stop execution of some script for some seconds, use sleep(1)
The mkdir(1) command accepts -p (and then won't create a directory if it already exists). mv(1) has also many options (including for backup).
To search some files in a file tree, use find(1). To search some content inside files, use grep. I also like ack
Read also Advanced Bash Scripting Guide & (if coding in C ...) Advanced Linux Programming and also the documentation of GNU bash (e.g. for shell builtins and control statements).
Did you consider using some revision control system like git ? It is useful to manage the evolution of source files (including shell scripts)
I've seen solutions similar to what you are asking, but using crontab with find -mmin 1 which will search for any files with a modtime <= 60 seconds within specified location.
Something along these lines (untested):
$ -> vi /tmp/file_finder.sh
# Add the following lines
#!/bin/bash
find /path/to/check -mmin 1 -type -f | while read fname; do
echo "$fname"
done
# Change perms
$ -> chmod 755 /tmp/file_finder.sh
$ -> crontab -e
* * * * * /tmp/file_finder.sh
With the above, you have now setup the cron to run every minute, and kick off a script that will search given directory for files with a modtime <= 60 seconds (new or updated).
Caveat: You should look for files with a mod time up to 5 minutes, that way you don't consider a file which may still be in the process of being written too.
I think you answered yourself (kind of)
Some suggestions:
1- use a while loop and at the end add sleep 60
2- write your procedure in a file (ex.; test1)
and then
watch -n 60 ./test1
I have a cron job which runs every 30 minutes to generate log files with time-stamp like this:
test20130215100531.log,
test20130215102031.log
I would like to create one folder daily with date time-stamp and push log files in to respective date folder when generated.
I need to achieve this on AIX server with bash.
Maybe you are looking for a script like this:
#!/bin/bash
shopt -s nullglob # This line is so that it does not complain when no logfiles are found
for filename in test*.log; do # Files considered are the ones starting with test and ending in .log
foldername=$(echo "$filename" | awk '{print (substr($0, 5, 8));}'); # The foldername is characters 5 to 13 from the filename (if they exist)
mkdir -p "$foldername" # -p so that we don't get "folder exists" warning
mv "$filename" "$foldername"
echo "$filename $foldername" ;
done
I only tested with your sample, so do a proper testing before using in a directory that contains important stuff.
Edit in response to comments:
Change your original script to this:
foldername=$(date +%Y%m%d)
mkdir -p /home/app/logs/"$foldername"
sh sample.sh > /home/app/logs/"$foldername"/test$(date +%Y%m%d%H%M%S).log
Or if the directory is created somewhere else, just do this:
sh sample.sh > /home/app/logs/$(date +%Y%m%d)/test$(date +%Y%m%d%H%M%S).log
You should use logrotate! It can do this for you already, and you can just write to the same log file.
Check their man pages for info:
http://linuxcommand.org/man_pages/logrotate8.html
I already searched for this, but I guess there was no great demand on working with paths.
So I'm trying two write a bash script to convert my music collection using tta and cue files.
My directory structure is as following: /Volumes/External/Music/Just/Some/Dirs/Album.tta for the tta files and /Volumes/External/Cuesheets/Just/Some/Dirs/Album.cue for cue sheets.
My current approach is setting /Volumes/External as "root_dir" and get the relative path of the album.tta file to $ROOT_DIR/Music (in this case this would be Just/Some/Dirs/Album.tta), then add this result to $ROOT_DIR/Cuesheets and change the suffix from .tta to .cue.
My current problem is, that dirname returns paths as they are, which means /Volumes/External/Music/Just/Some/Dirs does not get converted to ./Just/Some/Dirs/ when my current folder is $ROOT_DIR/Music and the absolute path was given.
Add://Here is the script if anybody has similar problems:
#!/bin/bash
ROOT_DIR=/Volumes/External
BASE="$1"
if [ ! -f "$BASE" ]
then
echo "Not a file"
exit 1
fi
if [ -n "$2" ]
then
OUTPUT_DIR="$HOME/tmp"
else
OUTPUT_DIR="$2"
fi
mkfdir -p "$OUTPUT_DIR" || exit 1
BASE=${BASE#"$ROOT_DIR/Music/"}
BASE=${BASE%.*}
TTA_FILE="$ROOT_DIR/Music/$BASE.tta"
CUE_FILE="$ROOT_DIR/Cuesheets/$BASE.cue"
shntool split -f "${CUE_FILE}" -o aiff -t "%n %t" -d "${OUTPUT_DIR}" "${TTA_FILE}"
exit 0
If your Cuesheets dir is always in the same directory as your Music, you can just remove root_dir from the path, and what is left is the relative path. If you have the path to your album.tta in album_path (album_path=/Volumes/External/Music/Just/Some/Dirs/Album.tta) and your root_dir set(root_dir=/Volumes/External), just do ${album_path#$root_dir}. This trims root_dir from the front of album_path, so you are left with album_path=Just/Some/Dirs/Album.tta.
See bash docs for more information on bash string manipulation
EDIT:// Changed ${$album_path#$root_dir} to ${album_path#$root_dir}
Okay so I've tackled this a couple of ways in the past. I don't recommend screwing with paths and pwd environment variables, I've seen some catastrophic events because of it.
Here's what I would do
CURRENTDIR=/Volumes/External/Music # make sure you check the existence in your script
...
SEDVAL=$(echo $CURRENTDIR | sed s/'\/'/'\\\/'/g)
#run your loops for iterating through files
for a in $(find ./ -name \*ogg); do
FILE=`echo $a | sed s/$SEDVAL/./g` # strip the initial directory and replace it with .
convert_file $FILE # whatever action to be performed
done
If this is something you might do frequently I would actually just write a separate script just for this.