I am trying to compress 5 days' worth log at a time and moving the compressed files to another location and deleting the logs files from original location. I need bash script to accomplish this. I got the files compressed using the below command, but not able to move them to the archive folder. I also need to compress based on date created. Now it's compressing all the files starting with a specific name.
#!/bin/bash
cd "C:\Users\ann\logs"
for filename in acap*.log*; do
# this syntax emits the value in lowercase: ${var,,*} (bash version 4)
mkdir -p archive
gzip "$filename_.zip" "$filename"
mv "$filename" archive
done
#!/bin/bash
mkdir -p archive
for file in $(find . -mtime +3 -type f -printf "%f ")
do
if [[ "$file" =~ ^acap.*\.log$ ]]
then
tar -czf archive/${file}.tar.gz $file
rm $file
fi
done
This finds all files in the current directory that match the regex and compresses them in an tar for every file. Then it deletes all the files.
Related
My download program automatically unrars rar archives, which is all well and good as Sonarr and Radarr need that original video file to import. But now my download HDD fills up with all these video files I no longer need.
I've tried playing around with modifying existing scripts I have, but every step seems to take me further from the goal.
Here's what I have so far (that isnt working and I clearly dont know what im doing). My main problem is I can't get it to find the files correctly yet. This script jumps right to "no files found". So I'm doing the search wrong at the very least. Or I'm pretty sure I might need to completely rewrite from scratch using a different method I'm not aware of..
#!/bin/bash
# Find video files and if it came from a rar, remove it.
# If no directory is given, work in local dir
if [ "$1" = "" ]; then
DIR="."
else
DIR="$1"
fi
# Find all the MKV files in this dir and its subdirs
find "$DIR" -type f -name '*.mkv' | while read filename
do
# If video file and rar file exists, delete mkv.
for f in ...
do
if [[ -f "$DIR/*.mkv" ]] && [[ -f "$DIR/*.rar" ]]
then
# rm $filename
printf "[Dry run delete]: $filename\n"
else
printf "No files found\n"
exit 1
fi
done
Example of directory structure before and after. Note the file names are often different to the extracted file. And I want to leave other folders that don't have rars in them alone.
Before:
/folder/moviename/Movie.that.came.from.rar.2021.dvdrip.mkv
/folder/moviename/movie.rar
/folder/moviename/movie.r00
/folder/moviename/movie.r01
/folder/moviename2/Movie.that.lives.alone.2021.dvdrip.mkv
/folder/moviename2/Movie.2021.dvdrip.nfo
After
# (deleted the mkv only from the first folder)
/folder/moviename/movie.rar
/folder/moviename/movie.r00
/folder/moviename/movie.r01
# (this mkv survives)
/folder/moviename2/Movie.that.lives.alone.2021.dvdrip.mkv
/folder/moviename2/Movie.2021.dvdrip.nfo
TL:DR I would like a script to look recursively in my download drive for video files and rar files, and if it sees both in the same folder, delete the video file.
With GNU find, you can condense this to one command:
find "${1:-.}" -type f -name '*.rar' -execdir sh -c 'echo rm *.mkv' \;
${1:-.} says "use $1, or . if $1 is undefined or empty".
For each .rar file found, this starts a new shell in the directory of the file found (that's what -execdir sh -c '...' does) and runs echo rm *.mkv.
If the list of files to delete looks correct, you can actually delete them by dropping the echo:
find "${1:-.}" -type f -name '*.rar' -execdir sh -c 'rm *.mkv' \;
Two remarks, though:
-execdir rm *.mkv \; would be shorter, but then the glob might be expanded prematurely in case there are .mkv files in the current directory
if a directory contains a .rar file, but no .mkv, this will try to delete a file called literally *.mkv and cause an error message
I have a fairly large number of directories (500+), each directory (and possible sub-directories) contains 4 or more zip files.
I managed to piece together a bash script that unzips the compressed files while maintaining zip filename as directory and all the directory hierarchy.
For example: If I have a zip file called 100011_test123.zip, and it contains 10 files. The script will uncompress all the files into 100011_test123/ directory.
The occurrence of numbers 100010 before the underscore in the filename/directoryname is totally random.
Here's the actual bash script:
#!/bin/bash
cd <directory-with-large-number-of-zip-files>
find . -name "*.zip" | while read filename; do unar -d -o "`dirname "$filename"`" "$filename"; done;
find . -name "*.zip" -type f -delete
Now I would like to update the script in order to remove the 100010_ from the .zip filename without tampering with the directory structure/hierarchy (I guess there's a way to rename the zip files before using unar command) and then uncompress the files into a directory without 100010_ at the beginning.
I have been stuck with this for more than 3 days. Any insights on this would be highly appreciated.
Thank you.
With all zip files at the same level, you don't need find, but a regular filename pattern globbing will do to iterate each zip archive.
And with bash's globstar option, you can also find the zip archives inside sub-directories
#!/usr/bin/env bash
shopt -s nullglob # Prevents iterating if no filename match
shopt -s globstar # ./**/ Allow searching inside sub-directories
# Set the basedir if you want all output directories at same place
#basedir="$PWD"
for zipfile in ./**/*.zip; do
# Extract the base directory containing the archive
zipdir="${zipfile%/*}"
# Extract the base name without the directory path
basename="${zipfile##*/}"
# Remove the .zip extension
# 100011_test123.zip -> 100011_test123
extensionless="${basename%.zip}"
# Remove everything before and first underscore 100011_
# 100011_test123 -> test123
outputdir="${basedir:-$zipdir}/${extensionless#*_}"
# Create output directory or continue with next archive
# mkdir -p test123
mkdir -p "$outputdir" || continue
# Unzip the zipfile into the outputdir and remove the zipfile if successful
# unrar -d -o test123 100011_test123.zip && rm -f -- 100011_test123.zip
unar -d -o "$outputdir" "$zipfile" && rm -f -- "$zipfile"
done
You need to parse directory name and filename first for each entry. Please check the ${fullpath%/*} and ${fullpath##*/} for this purpose. And awk for splitting filename with '_' and getting second part of it.
You can try following code.
#!/bin/bash
# cd directory
zip_files=($(find . -name "*.zip"))
for fullpath in "${zip_files[#]}"; do
echo "Processing: "$fullpath""
DIRNAME="${fullpath%/*}"
FILENAME="${fullpath##*/}"
NEW_FILENAME="`echo $FILENAME | awk -F'_' '{print $NF}'`"
echo " DIRNAME="$DIRNAME
echo " NEW_FILENAME="$NEW_FILENAME
mv $fullpath "$DIRNAME/$NEW_FILENAME"
# call unar command
unar -d -o $DIRNAME $NEW_FILENAME
# delete file if you want
done
I'm quite new in creating shell scripts.
I'm developing a shell script that will backup my files once a day only.
I need to check which *.war files are in three different folders (input folder, production folder, backup folder)
If the same files exists in the three directories, don't perform backup.
If it doesn't, it must move the files in folder 2 to folder 3.
This is what I've done so far.
===============================
TODAY=$(date +%d-%m-%Y)
INPUT=/home/bruno.ogasawara/entrada/
BACKUP=/home/bruno.ogasawara/backup/
PROD=/home/bruno.ogasawara/producao/
DIR1=$(ls $INPUT)
DIR2=$(ls $PROD)
DIR3=$(ls $BACKUP$TODAY)
for i in $DIR1; do
for j in $DIR2; do
for k in $DIR3; do
if [ $i == $j ] && [ $j == $k ]; then
exit 1; else
mv -f $PROD$j $BACKUP$TODAY
fi
done
done
done
mv -f $INPUT*.war $PROD
===============================
The verification is not working. Only thing working is the mv -f $INPUT*.war $PROD in the end.
Where am I missing something or doing something wrong?
Thanks in advance people.
What I understand is you want to sync those three folders.
In that case you should not modify the file names as we are using file names to compare them.Otherwise you should use md5 or sha checksums.But linux filesystem already has timestamps feature you don't have to attach date to filename.
In your code you used ls to list files ...but actually ls command lists files in column mode which is not comaptible with for loop in bash.
correct command is
find $DIR -maxdepth 1 -type f -exec basename {} \;
you want to sync the *.war files to all folders...then simply you can use this:
#!/bin/bash
DIR1=/home/bruno.ogasawara/entrada/
DIR2=/home/bruno.ogasawara/backup/
DIR3=/home/bruno.ogasawara/producao/
cp -n $DIR1/*.war $DIR2
cp -n $DIR1/*.war $DIR3
cp -n $DIR2/*.war $DIR1
cp -n $DIR2/*.war $DIR3
cp -n $DIR3/*.war $DIR1
cp -n $DIR3/*.war $DIR2
-n: will check if file already exists.it will not overwrite the existing file.
I develop following script to gzip yesterday files in a directory, any improvements suggestions
Yesterday=`TZ=GMT+24 date +%d-%b-%y`;
mkdir $Yesterday
mv /tmp/logs/servicemix.log.* /tmp/logs/$Yesterday
for File in /tmp/logs/$Yesterday/app.log.*;
do gzip $File;
done
Regards
1.Replace
mkdir $Yesterday
by
mkdir -p /tmp/logs/${Yesterday}
gzip the files you have moved
When you move servicemix* files, do not gzip app* files
Use following lines of codes
TIME=`date +"%b-%d-%y"` # This Command will add date in Backup File Name.
now=$(date +"%T")
FILENAME="filename-$TIME:$now.tar.gz" # Here i define Backup file name format.
SRCDIR="/home" # Location of Important Data Directory (Source of backup).
DESDIR="/home/user/backup" # Destination of backup file.
tar -cpzf $DESDIR/$FILENAME $SRCDIR #creating backup as tar file
echo
echo "Backup finished"
Were they changed or created yesterday? Use find with the correct modifier.
Yesterday=`TZ=GMT+24 date +%d-%b-%y`;
mkdir $Yesterday
# -1 is equal to last 24 hours
# ctime is for last inode modified time. If you need creation date you need a different modifier. See attached link.
find -ctime -1 -exec mv '{}' "$yesterday/" \;
But this is i think pretty much the best option: Use 'find' to determine files modified yesterday.
I have a folder that after an rsync will have a zip in it. I want to unzip it to its own folder(if the zip is L155.zip, to unzip its content to L155 folder). The problem is that I dont know it's name beforehand(although i know it will be "letter-number-number-number"), so I have to unzip an uknown file to its unknown folder and this to be done automatically.
The command “unzip *”(or unzip *.zip) works in terminal, but not in a script.
These are the commands that have worked through terminal one by one, but dont work in a script.
#!/bin/bash
unzip * #also tried .zip and /path/to/file/* when script is on different folder
i=$(ls | head -1)
y=${i:0:4}
mkdir $y
unzip * -d $y
First I unzip the file, then I read the name of the first extracted file through ls and save it in a variable.I take the first 4 chars and make a directory with it and then again unzip the files to that specific folder.
The whole procedure after first unzip is done, is because the files inside .zip, all start with a name that the zip already has, so if L155.ZIP is the zip, the files inside with be L155***.txt.
The zip file is at /path/to/file/NAME.zip.
When I run the script I get errors like the following:
unzip: cannot find or open /path/to/file/*.ZIP
unzip: cannot find or open /path/to/file//*.ZIP.zip
unzip: cannot find or open /path/to/file//*.ZIP.ZIP. No zipfiles found.
mkdir: cannot create directory 'data': File exists data
unzip: cannot find or open data, data.zip or data.ZIP.
Original answer
Supposing that foo.zip contains a folder foo, you could simply run
#!/bin/bash
unzip \*.zip \*
And then run it as bash auto-unzip.sh.
If you want to have these files extracted into a different folder, then I would modify the above as
#!/bin/bash
cp *.zip /home/user
cd /home/user
unzip \*.zip \*
rm *.zip
This, of course, you would run from the folder where all the zip files are stored.
Another answer
Another "simple" fix is to get dtrx (also available in the Ubuntu repos, possibly for other distros). This will extract each of your *.zip files into its own folder. So if you want the data in a different folder, I'd follow the second example and change it thusly:
#!/bin/bash
cp *.zip /home/user
cd /home/user
dtrx *.zip
rm *.zip
I would try the following.
for i in *.[Zz][Ii][Pp]; do
DIRECTORY=$(basename "$i" .zip)
DIRECTORY=$(basename "$DIRECTORY" .ZIP)
unzip "$i" -d "$DIRECTORY"
done
As noted, the basename program removes the indicated suffix .zip from the filename provided.
I have edited it to be case-insensitive. Both .zip and .ZIP will be recognized.
for zfile in $(find . -maxdepth 1 -type f -name "*.zip")
do
fn=$(echo ${zfile:2:4}) # this will give you the filename without .zip extension
mkdir -p "$fn"
unzip "$zfile" -d "$fn"
done
If the folder has only file file with the extension .zip, you can extract the name without an extension with the basename tool:
BASE=$(basename *.zip .zip)
This will produce an error message if there is more than one file matching *.zip.
Just to be clear about the issue here, the assumption is that the zip file does not contain a folder structure. If it did, there would be no problem; you could simply extract it into the subfolders with unzip. The following is only needed if your zipfile contains loose files, and you want to extract them into a subfolder.
With that caveat, the following should work:
#!/bin/bash
DIR=${1:-.}
BASE=$(basename "$DIR/"*.zip .zip 2>/dev/null) ||
{ echo More than one zipfile >> /dev/stderr; exit 1; }
if [[ $BASE = "*" ]]; then
echo No zipfile found >> /dev/stderr
exit 1
fi
mkdir -p "$DIR/$BASE" ||
{ echo Could not create $DIR/$BASE >> /dev/stderr; exit 1; }
unzip "$DIR/$BASE.zip" -d "$DIR/$BASE"
Put it in a file (anywhere), call it something like unzipper.sh, and chmod a+x it. Then you can call it like this:
/path/to/unzipper.sh /path/to/data_directory
simple one liner I use all the time
$ for file in `ls *.zip`; do unzip $file -d `echo $file | cut -d . -f 1`; done