Shellscript 2nd newest folder in tgz - bash

I've tried to develop a little backup shellscript.
When I Run it int the Backup_FILESERVER Folder it's creating the tgz.
But from /root I get an Error.
tar cvfz /NAS/for_tape/FILESERVER.tgz /NAS/Backup_FILESERVER/`ls -Art | tail -2 | head -n 1`
Error:
tar: Tuesday: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
In the folder "/NAS/Backup_FILESERVER" are 5 folders for each weekday. Monday, Tuesday, ...
Is it possible to make it runable?

Can you try
tar cvzf /NAS/for_tape/FILESERVER.tgz `find /NAS/Backup_FILESERVER/ -type d -exec sh -c "ls -1rt" \; | tail -2 | head -n 1`
find command with ls -1rt sorts the files based on the modification time and reverses it.
You can confirm if the command find /NAS/Backup_FILESERVER/ -type d -exec sh -c "ls -1rt" \; | tail -2 | head -n 1 gives the folder you need before starting the compression

Related

bash script to remove previous backup

I set up a daily cron job to backup my server.
In my folder backup, the backup command generates 2 files : the archive itself .tar.gz and a file .info.json like the ones below:
-rw-r--r-- 1 root root 1617 Feb 2 16:17 20200202-161647.info.json
-rw-r--r-- 1 root root 48699726 Feb 2 16:17 20200202-161647.tar.gz
-rw-r--r-- 1 root root 1617 Feb 3 06:25 20200203-062501.info.json
-rw-r--r-- 1 root root 48737781 Feb 3 06:25 20200203-062501.tar.gz
-rw-r--r-- 1 root root 1618 Feb 4 06:25 20200204-062501.info.json
-rw-r--r-- 1 root root 48939569 Feb 4 06:25 20200204-062501.tar.gz
How to I write a bash script that will only keep the last 2 archives and deletes all the others backup (targ.gz and info.json).
In this example, that would mean deleted 20200204-062501.info.json and 20200204-062501.tar.gz .
Edit:
I replace -name by -wholename in the script but when I run it, it doesn't have any effects apparently.The old archives are still there and they have not been deleted.
the script :
#!/bin/bash
DEBUG="";
DEBUG="echo DEBUG..."; #put last to safely debug without deleting files
keep=2;
for suffix in /home/archives .json .tar; do
list=( $( find . -wholename "*$suffix" ) ); #allow for zero names
if [ ${#list[#]} -gt $keep ]; then
# delete all but last $keep oldest files
${DEBUG}rm -f "$( ls -tr "${list[#]}" | head -n-$keep )";
fi
done
Edit 2:
if I run #sorin script, does it actually delete everything if I believe the script output?
The archive folder before running the script:
https://pastebin.com/7WtwVHCK
The script I run:
find home/archives/ \( -name '*.json' -o -name '*.tar.gz' \) -print0 |\
sort -zr |\
sed -z '3,$p' | \
xargs -0 echo rm -f
The script output:
https://pastebin.com/zd7a2zcq
Edit 3 :
The command find /home/archives/ -daystart \( -name '*.json' -o -name '*.tar.gz' \) -mtime +1 -exec echo rm -f {} + works and does the job.
Marked as solved
If the file is generated daily, a simple approach would be to take advantage of the -mtime find condition:
find /home/archives/ -daystart \( -name '*.json' -o -name '*.tar.gz' \) -mtime +1 -exec echo rm -f {} +
-daystart - use the start of the day for comparing modification times
\( -name '*.json' -o -name '*.tar.gz' \) - select files that end either in *.json or *.tar.gz
-mtime +1 - modification time is older than 24 hours (from the day start)
-exec echo rm -f {} + - remove the files (remove the echo after testing and verifying the result is what you want)
A simpler solution avoiding ls and it's pitfalls and not depending on the modification time of the files:
find /home/archives/ \( -name '*.json' -o -name '*.tar.gz' \) -print0 |\
sort -zr |\
sed -nz '3,$p' | \
xargs -0 echo rm -f
\( -name '*.json' -o -name '*.tar.gz' \) - find files that end in either *.json or tar.gz
-print0 - print them null separated
sort -zr - -z tells sort to use null as a line separator, -r sorts them in reverse
sed -nz '3,$p' - -z same as above. '3,$p' - print lines between 3rd and the end ($)
xargs -0 echo rm -f - execute rm with the piped arguments (remove the echo after you tested and you are satisfied with the command)
Note: not all sort and sed support the -z but most do. If you are stuck with such a situation, you might have to use a higher level language
Find the two most recent files in path:
most_recent_json=$(ls -t *.json | head -1)
most_recent_tar_gz=$(ls -t *.tar.gz | head -1)
Remove everything else ignoring the found recent files:
rm -i $(ls -I $most_recent_json -I $most_recent_tar_gz)
Automatic deleting can be hazardous to your mental state if it deletes unwanted files or aborts long scripts early due to unexpected errors. Say when there are fewer than 1+2 files in your example. Be sure the script does not fail if there are no files at all.
tdir=/home/archives/; #target dir
DEBUG="";
DEBUG="echo DEBUG..."; #put last to safely debug without deleting files
keep=2;
for suffix in .json .tar; do
list=( $( find "$tdir" -name "*$suffix" ) ); #allow for zero names
if [ ${#list[#]} -gt $keep ]; then
# delete all but last $keep oldest files
${DEBUG}rm -f "$( ls -tr "${list[#]}" | head -n-$keep )";
fi
done
Assuming that you have fewer than 10 files and that they are created in pairs, then you can do something straightforward like this:
files_to_delete=$(ls -tr1 | tail -n+3)
rm $files_to_delete
The -tr1 tells the ls command to list the files in reverse chronological order by modification time, each on a single line.
The tail -n+3 tells the tail command to start at the third line (skipping the first two lines).
If you have more than 10 files, a more complicated solution will be necessary, or you would need to run this multiple times.

Argument list too long for ls while moving files from one dir to other in bash shell

Below is the command I am using for moving files from dir a to dir b
ls /<someloc>/a/* | tail -2000 | xargs -I{} mv {} /<someloc>/b/
-bash: /usr/bin/ls: Argument list too long
folder a has files in millions ..
Need your help to fix this please.
If the locations of both directories are on the same disk/partition and folder b is originally empty, you can do the following
$ rmdir /path/to/b
$ mv /other/path/to/a /path/to/b
$ mkdir /other/path/to/a
If folder b is not empty, then you can do something like this:
find /path/to/a/ -type f -exec mv -t /path/to/b {} +
If you just want to move 2000 files, you can do
find /path/to/a/ -type f -print | tail -2000 | xargs mv -t /path/to/b
But this can be problematic with some filenames. A cleaner way would be is to use -print0 of find, but the problem is that head and tail can't process those, so you have to use awk for this.
# first 2000 files (mimick head)
find /path/to/a -type f -print0 \
| awk 'BEGIN{RS=ORS="\0"}(NR<=2000)' \
| xargs -0 mv -t /path/to/b
# last 2000 files (mimick tail)
find /path/to/a -type f -print0 \
| awk 'BEGIN{RS=ORS="\0"}{a[NR%2000]=$0}END{for(i=1;i<=2000;++i) print a[i]}' \
| xargs -0 mv -t /path/to/b
The ls in the code in the question does nothing useful. The glob (/<someloc>/a/*) produces a sorted list of files, and ls just copies it (after re-sorting it), if it works at all. See “Argument list too long”: How do I deal with it, without changing my command? for the reason why ls is failing.
One way to make the code work is to replace ls with printf:
printf '%s\n' /<someloc>/a/* | tail -2000 | xargs -I{} mv {} /<someloc>/b/
printf is a Bash builtin, so running it doesn't create a subprocess, and the "Argument list too long" problem doesn't occur.
This code will still fail if any of the files contains a newline character in its name. See the answer by kvantour for alternatives that are not vulnerable to this problem.

remove files from subfolders without the last three

I have a structure like that:
/usr/local/a/1.txt
/usr/local/a/2.txt
/usr/local/a/3.txt
/usr/local/b/4.txt
/usr/local/b/3.txt
/usr/local/c/1.txt
/usr/local/c/7.txt
/usr/local/c/6.txt
/usr/local/c/12.txt
...
I want to delete all the files *.txt in subfolders except the last three files with the greatest modification date, but here I am in current directory
ls -tr *.txt | head -n-3 |xargs rm -f
I need to combine that with the code:
find /usr/local/**/* -type f
Should I use the maxdepth option?
Thanks for helping,
aola
Added maxdepth options to find for one level, sorting files by last modification time, tail to ignore the oldest modified 3 files and xargs with -r to remove the files only if they are found.
for folder in $(find /usr/local/ -type d)
do
find $folder -maxdepth 1 -type f -name "*.txt" | xargs -r ls -1tr | tail -n+3 | xargs -r rm -f
done
Run the above command once without rm to ensure that the previous commands pick the proper files for deletion.
You've almost got the solution: use find to get the files,ls to sort them by modification date and tail to omit three most recently modified ones:
find /usr/lib -type f | xargs ls -t | tail -n +4 | xargs rm
If you would like to remove only the files at a specified depth add -mindepth 4 -maxdepth 4 to find parameters.
You can use find's -printf option, to print the modification time in front of the file name and then sort and strip the date off. This avoids using ls at all.
find /usr/local -type f -name '*.txt' -printf '%T#|%p\n' | sort -r | cut -d '|' -f 2 | head -n-3 | xargs rm -f
The other Answers using xargs ls -t can lead to incorrect results, when there are more results than xargs can put in a single ls -t command.
but for each subfolder, so when I have
/usr/local/a/1.txt
/usr/local/a/2.txt
/usr/local/a/3.txt
/usr/local/a/4.txt
/usr/local/b/4.txt
/usr/local/b/3.txt
/usr/local/c/1.txt
/usr/local/c/7.txt
/usr/local/c/6.txt
/usr/local/c/12.txt
I want to to use the code for each subfolder separately
head -n-3 |xargs rm -f
so I bet if I have it sorted by date then the files to delete:
/usr/local/a/4.txt
/usr/local/c/12.txt
I want to leave in any subfolder three newest files

Get file depth in directory tree

I'm using command find to recursively browse through directory tree, counting files, sizes, etc...
Now I need to get directory depth of each file.
Is there any portable way for both FreeBSD and CentOS?
I know that find is able to prinf actual directory depth but sadly this works only on CentOS, not FreeBSD.
Additionaly - I need to keep standard find output OR put directory depth on the beginning of output and cut it from there.
You can count the / in path :
$ find . -type f -exec bash -c 'echo '{}' | grep -o / | wc -l' \;
Or with file names :
$ mkdir -p one/two/three four/five && touch file one/two/file one/two/three/file
$ find . -type f -exec bash -c 'echo -n '{}' :; echo '{}' | grep -o / | wc -l' \;
./file :1
./one/two/file :3
./one/two/three/file :4
Try this:
find . -type d -exec bash -c 'echo $(tr -cd / <<< "$1"|wc -c):$1' -- {} \; | sort -n | tail -n 1 | awk -F: '{print $1, $2}'

Get folder (or sub-files/folders) last modification date and time

Is it possible to get the modification date and time of a folder?
I know you can use stat -f "%m" folder, but it doesn't reflect sub-files/folders changes.
Things that doesn't work:
ls -l folder - doesn't reflect changes inside the folder
stat -f "%m" folder - same as above
date -r folder - same again
find foo bar baz -printf - the printf option doesn't exist on my version of find
Versions of things:
OS: Mac OS X 10.7.1
Bash: GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin11)
Solution:
find . -exec stat -f "%m" \{} \; | sort -n -r | head -1
Explanation:
the find command traverses the current directory (.) and for each file encountered executes (-exec) the command stat -f "%m". stat -f "%m" prints the last modification unix timestamp of the file.
sort -n -r sorts the output of the find command numerically (-n) in reverse order (-r). This will list the latest modification timestamp first.
head -1 then extracts the first line of the output from sort. This is the latest modification unix timestamp of all the files.
You could try 'date -r folder' to give you a date last modified
You could always get it from ls :
ls -ld mydir | awk -F' ' '{ print $6 " "$7 }'
if you need to clear cache after build . then you can check the age of the last change and delete it like this
sh("find ~/Library/Developer/Xcode/DerivedData/ -type d -maxdepth 1 -mmin +360 -name 'Cache-folder-*' -print0 | xargs -0 -I {} /bin/rm -rf '{}' || echo 'There is no such folder! Start script execution' ; exit 0")
sh("find ~/Library/Developer/Xcode/DerivedData/ -type d -maxdepth 1 -mtime 0 -name 'Cache-folder-*' -ls -exec rm -r {} \\;")

Resources