With help, i have this script but i am not sure how i get rid of "./" from the variable so i can zip the folder. Please can you advise me.
thanks
Nick
#/bin/sh
BASEDIR=/tmp/
cd $BASEDIR
find . -type d | sort > newfiles.txt
DIFF=$(comm -13 oldfiles.txt newfiles.txt)
echo $DIFF
if [ "$DIFF" != "" ]
then
#echo "A new dir is found"
tar -czvf "$DIFF".tar.gz --verbose
fi
mv newfiles.txt oldfiles.txt
failed output:
+ tar -czvf ./file2.tar.gz --verbose
tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.
The error stems from this command:
tar -czvf "$DIFF".tar.gz --verbose
You specify the archive name, but not the files/folders to be added to said archive.
You would need it to change to:
tar -czvf "$DIFF".tar.gz "$DIFF"
Note that you don't need --verbose since you already specified -v with -czvf.
However, your code won't work like intended if DIFF contains multiple items (i.e. lines). You probably want something like this:
#/bin/sh
BASEDIR="/tmp/"
cd "$BASEDIR"
find . -type d | sort > newfiles.txt
DIFF=$(comm -13 oldfiles.txt newfiles.txt)
echo "$DIFF"
IFS=$'\n'
for item in $DIFF
do
#echo "A new dir is found"
tar -czvf "$item.tar.gz" "$item"
done
mv newfiles.txt oldfiles.txt
That being said, if you want to remove ./ from find output, you can use:
find . -type d -printf "%P\n" | sort > newfiles.txt
but you don't actually have to do this for your script to work.
Related
I have used a few entries here to guide me through my following bash script:
LOG_FILES=$(find ~/testing.domain.net -name "domain_*log" -printf ' %P ')
NEW_LOG_FILES=$(echo $LOG_FILES | sed -e 's/\r//g')
echo ${NEW_LOG_FILES}
NOW=$(date +"%m-%d-%Y")
echo ${NOW}
tar czf ${NOW}.tar.gz ${NEW_LOG_FILES}
RC=$? # Check whether an error occured
if [[ "$RC" == "0" ]]; then
mv ${NOW}.tar.gz archivedlogs/.
rm ${LOG_FILES}
fi
The objective of the script is to find any log files in the current and sub directories and tar zipped them all before moving tar file to an archivedlogs subdirectory and then deleting log files.
When I execute the script I'm getting:
domain_info_log subdir1/domain_error_log subdir2/domain_error_log domain_error_log
12-27-2020
tar: \r\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
./archiveTestLogsDaily.sh: line 12: syntax error: unexpected end of file
I can't seem to get rid of the carriage returns.
I have also tried replacing:
NEW_LOG_FILES=$(echo $LOG_FILES | sed -e 's/\r//g')
with:
NEW_LOG_FILES=${LOG_FILES//$'\r'/}
But same outcome.
Any ideas? My bash script knowledge is not the best but I'm guessing the find part comes back with carriage returns?
Also before anyone else suggests it, I don't have access to logrotate as this is a shared server.
$ uname -a
Linux 3.10.0-962.3.2.lve1.5.26.4.el7.x86_64 #1 SMP Wed Sep 11 10:29:54 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
I think it would be more safe to use a directory before making the tar archive in order to avoid a "tarbomb".
I mean something like:
now="$(date +"%m-%d-%Y")"
archiv_dir="archiv_dir-${now}"
mkdir -p "${archiv_dir}"
find ~/testing.domain.net -type f -a -iname "domain_*log" -exec mv {} "${archiv_dir}" \;
tar cvzf "${now}.tar.gz" "${archiv_dir}" && mv "${now}.tar.gz" "archivedlogs/" && rm -rf "${archiv_dir}"
P.S: make sure to double quote when you're using a variable value in Bash (or even using other POSIX shells), except inside [[.
EDIT to answer to that:
One more thing #Idriss, some of those subdirectories have the same file names, when mv tries to copy them it ignores them because they are already there... is there a way of making sure the parent directory is somehow appended to the name?
With a little bit of bash and awk, you could try something like:
#!/bin/bash
now="$(date +"%m-%d-%Y")"
archiv_dir="archiv_dir-${now}"
ROOT=~/testing.domain.net
find "${ROOT}" -type f -a -iname "domain_*log"|while read; do
subdir="$(echo $REPLY|awk -F '/' '{print $(NF-1)}')"
filename="$(basename "${REPLY}")"
[[ $subdir && $subdir != $(basename $ROOT) ]] && filename="${subdir}_${filename}"
echo mv "${REPLY}" "${archiv_dir}/${filename}"
done
Here is the output:
mv /Users/ineumann/testing.domain.net/subdir3/domain_error_log archiv_dir-12-28-2020/subdir3_domain_error_log
mv /Users/ineumann/testing.domain.net/subdir2/domain_error_log archiv_dir-12-28-2020/subdir2_domain_error_log
mv /Users/ineumann/testing.domain.net/subdir1/domain_error_log archiv_dir-12-28-2020/subdir1_domain_error_log
mv /Users/ineumann/testing.domain.net/domain_info_log archiv_dir-12-28-2020/domain_info_log
N.B: just remove the echo before the mv to perform the moves and I let you merge this example with your previous script that create directories, create the tar archive, etc.
Trying to create an if statement which will search with a username in a location to see if a tar has been done already. If not create said tar in a location. For some reason, my find is running through the then echo command regardless if there is a file in that location or not.
USER_LIST="$(awk '{print $3}' usernamefile.txt)"
for USER_NAME in $USER_LIST;do
echo $USER_NAME
if find /location/to/store/tarfile -type f -iname $USER_NAME;
then
echo "tar file has been found for" $USER_NAME "/location/to/store/tarfile" `date` >> /logfile/log.txt
else
FILE_LOC="$(awk -v $USER_NAME=$3 '{print $5;}' usernamefile.txt)"
tar -czvf ${USER_NAME}.tar.gz /location/to/put/tar/file $FILE_LOC
echo "tar exit code:" $? $USER_NAME "has been archived" `date` >> /logfile/log.txt
fi
done
I'm not sure why but if the find doesn't find anything. Surely it should move onto the else part of the script? the plan is creating tar files such as <username>.tar.gz
It appears that using test command would work best for my needs with either a -f or a -e flag.
The point is that you don't know how to capture the result of find: this gives a list of files, obeying the conditions you add in the find statement. So, best thing to do, is to count them: if there are none, then there are no such files, so you get something like:
if $(find /location/to/store/tarfile -type f -iname $USER_NAME | wc -l)
then
...
The | wc -l reads the number of found files. If a file has been found, the result is not equal to 0, so it gets interpreted as TRUE.
I'm a beginner in writing bash scripts for automating tasks, and I'm trying to untar all the tar files in one directory (there are way too many to do it by hand) for a bunch of source code files. They're all of the type *.tar.gz, *.tar.xz, or *.tar.bz2.
This is for a Linux from Scratch LFS installation I'm doing (I'm a first timer), and I'm not sure how else to automate this task other than using a bash script. The code for my little script to do this is down below.
#!/bin/bash
for afile in 'ls -1'; do
if [ 'afile | grep \"\.tar\.gz\"' ];
then
tar -xzf afile
elif [ 'afile | grep \"\.tar\.xz\"' ]
then
tar -xJf afile
elif [ 'afile | grep \"\.tar\.xz\"' ]
then
tar -xjf afile
else
echo "Something is wrong with the program"
fi
done;
I expected it to untar everything in the directory and create separate directories, but instead it exited with this error:
tar (child): afile: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Apparently it thinks afile is the actual file, but I don't know how to change afile to be each file that is going through my for construct. How would I write a script for this, especially since there are different types of files?
To get your script to work with minimal changes, use $afile whenever you want the variable's value. The dollar sign makes a variable reference; otherwise you just get the literal string 'afile'. Also get rid of the square brackets and instead echo the variable to grep.
for afile in `ls -1`; do
if echo "$afile" | grep '\.tar\.gz'
then
tar -xzf "$afile"
elif echo $afile | grep '\.tar\.xz'
then
tar -xJf "$afile"
elif echo "$afile" | grep '\.tar\.bz2'
then
tar -xjf "$afile"
else
echo "Something is wrong with the program"
fi
done
Since you're a bash beginner, let's look at various other ways you could write the script. I'd make a couple of improvements. For one, you shouldn't loop over ls. You can get the same thing by looping over *. Second, grep is a heavyweight tool. You can do some simple string comparisons with built-in shell constructs like [[ and ==.
for afile in *; do
if [[ "$afile" == *.tar.gz ]]; then
tar -xzf "$afile"
elif [[ "$afile" == *.tar.xz ]]; then
tar -xJf "$afile"
elif [[ "$afile" == *.tar.bz2 ]]; then
tar -xjf "$afile"
else
echo "Something is wrong with the program"
fi
done
Actually, this would be even nicer with a case statement. Let's try that. Also let's echo the error message to stderr with >&2. That's always a good idea.
for afile in *; do
case "$afile" in
*.tar.gz) tar -xzf "$afile";;
*.tar.xz) tar -xJf "$afile";;
*.tar.bz2) tar -xjf "$afile";;
*) echo "Something is wrong with the program" >&2
esac
done
We could even get rid of the error message if we just list the three types of files we want to loop over. Then there's no way to hit the else case.
for afile in *.tar.{gz,xz,bz2}; do
case "$afile" in
*.tar.gz) tar -xzf "$afile";;
*.tar.xz) tar -xJf "$afile";;
*.tar.bz2) tar -xjf "$afile";;
esac
done
Or a completely different way to do it: use find to find all the files and its -exec action to call a command for each file it finds. Here {} is a placeholder for the files it finds.
find . -name '*.tar.gz' -exec tar -xzf {} \;
find . -name '*.tar.xz' -exec tar -xJf {} \;
find . -name '*.tar.bz2' -exec tar -xjf {} \;
The requirement is to extract all the *.properties files from multiple tars and put them into a zip.
I tried this:
find . -iwholename "*/ext*/*.tar.gz"|xargs -n 1 tar --wildcards '*.properties' -xvzf | zip -# tar-properties.zip
This is creating a zip with the .properties files in all the tars.
But the issue is the tars are structured as in each tar contains a properties folder which contains the files. The above command is creating a zip with a single properties folder which contains all the files .
Is there a way to put these in the zip with a folder structure like {name of the tar}/properties/*.properties ?
You could use this script. My solution uses --transform as well. Please check first if your tar command supports it with tar --help 2>&1 | grep -Fe --transform.
#!/bin/bash
[ -n "$BASH_VERSION" ] || {
echo "You need bash to run this script." >&2
exit 1
}
TEMPDIR=/tmp/properties-files
OUTPUTFILE=$PWD/tar-properties.zip ## Must be an absolute path.
IFS=
if [[ ! -d $TEMPDIR ]]; then
mkdir -p "$TEMPDIR" || {
echo "Unable to create temporary directory $TEMPDIR." >&2
exit 1
}
fi
NAMES=()
while read -r FILE; do
NAMEOFTAR=${FILE##*/} ## Remove dir part.
NAMEOFTAR=${NAMEOFTAR%.tar.gz} to remove extension ## Remove .tar.gz.
echo "Extracting $FILE."
tar --wildcards '*.properties' -xvzf "$FILE" -C "$TEMPDIR" --transform "s#.*/#${NAMEOFTAR//#/\\#}/properties/#" || {
echo "An error occurred extracting to $TEMPDIR." >&2
exit 1
}
NAMES+=("$NAMEOFTAR")
done < <(exec find . -type f -iwholename '*/ext*/*.tar.gz')
(
cd "$TEMPDIR" >/dev/null || {
echo "Unable to change directory to $TEMPDIR."
exit 1
}
zip -a "$OUTPUTFILE" "${NAMES[#]}"
)
Save it to a script then run it on the directory where those files are to be searched with
bash /path/to/script.sh`
You can probably do the trick with tar option --transform, --xform. This option permits to manipulate path thanks to a sed expression.
find . -iwholename "*/ext*/*.tar.gz"|xargs -n 1 tar --wildcards '*.properties' -xvzf --xform 's#.*/#name_of_the_tar/properties/#' | zip -# tar-properties.zip
I havent been able to find an answer that best suites my needs, and I appologize if someone is able to find it easily.
I have a script that works to move files into folders based on their names. It worked perfectly until I realized that The files where missing their extension once I fixed this (another script was responsible for the file naming based on an email subject line) Once I fixed this problem It then started making a folder for each file. Is there anyway I can make this script drop everything in the folder name before the first (.)
Here is the script
#!/bin/bash
#folder script
#Benjamin D. Schran
MAIN_DIR=/PGHWH1/Photos
cd $MAIN_DIR
find . -maxdepth 1 -type f > SCRIPT_LOG1
find . -name '* *' | while read fname
do
new_fname=`echo $fname | tr " " "_"`
if [ -e $new_fname ]
then
echo "File $new_fname already exists. Not replacing $fname"
else
echo "Creating new file $new_fname to replace $fname"
mv "$fname" $new_fname
fi
done
find . -maxdepth 1 -type f | while read file;
do
f=$(basename "$file")
f1=${f%.*}
if [ -d "$f1" ];
then
mv "$f" "$f1"
else
mkdir "$f1"
chmod 777 "$f1"
mv "$f" "$f1"
fi
done
SCRIPTLOG=Script_log.$(date +%Y-%m-%d-%H-%M)
find . -type f > SCRIPT_LOG2
cd /PGHWH1/bin
sh scriptlog.sh > $SCRIPTLOG.html
mv $SCRIPTLOG.html /PGHWH1/log
rm $MAIN_DIR/SCRIPT_LOG1 $MAIN_DIR/SCRIPT_LOG2
What I need it to do is to take a files that is
Filename-date.%.jpg
and make
Foldername-date
then move the files of
Filename-date.1.jpg
Filename-date.2.jpg
Filename-date.3.jpg
to the appropriate folder
Foldername-date
but the current output is
Foldername-date.1
Foldername-date.2
Foldername-date.3
Any help at all would be appreciated
The following lines do the job in my bash:
#first create a tmp file with unique directory names
ls *.jpg | awk -F'.' '{print $1}' | uniq > dirs
#second create the directories
mkdir -p `cat dirs`
#third move the files
for i in `cat dirs`; do mv $i*.jpg $i/; done
#(optionally) remove the tmp file
rm dirs