Get the newest directory to a variable in Bash - bash

I would like to find the newest sub directory in a directory and save the result to variable in bash.
Something like this:
ls -t /backups | head -1 > $BACKUPDIR
Can anyone help?

BACKUPDIR=$(ls -td /backups/*/ | head -1)
$(...) evaluates the statement in a subshell and returns the output.

There is a simple solution to this using only ls:
BACKUPDIR=$(ls -td /backups/*/ | head -1)
-t orders by time (latest first)
-d only lists items from this folder
*/ only lists directories
head -1 returns the first item
I didn't know about */ until I found Listing only directories using ls in bash: An examination.

This ia a pure Bash solution:
topdir=/backups
BACKUPDIR=
# Handle subdirectories beginning with '.', and empty $topdir
shopt -s dotglob nullglob
for file in "$topdir"/* ; do
[[ -L $file || ! -d $file ]] && continue
[[ -z $BACKUPDIR || $file -nt $BACKUPDIR ]] && BACKUPDIR=$file
done
printf 'BACKUPDIR=%q\n' "$BACKUPDIR"
It skips symlinks, including symlinks to directories, which may or may not be the right thing to do. It skips other non-directories. It handles directories whose names contain any characters, including newlines and leading dots.

Well, I think this solution is the most efficient:
path="/my/dir/structure/*"
backupdir=$(find $path -type d -prune | tail -n 1)
Explanation why this is a little better:
We do not need sub-shells (aside from the one for getting the result into the bash variable).
We do not need a useless -exec ls -d at the end of the find command, it already prints the directory listing.
We can easily alter this, e.g. to exclude certain patterns. For example, if you want the second newest directory, because backup files are first written to a tmp dir in the same path:
backupdir=$(find $path -type -d -prune -not -name "*temp_dir" | tail -n 1)

The above solution doesn't take into account things like files being written and removed from the directory resulting in the upper directory being returned instead of the newest subdirectory.
The other issue is that this solution assumes that the directory only contains other directories and not files being written.
Let's say I create a file called "test.txt" and then run this command again:
echo "test" > test.txt
ls -t /backups | head -1
test.txt
The result is test.txt showing up instead of the last modified directory.
The proposed solution "works" but only in the best case scenario.
Assuming you have a maximum of 1 directory depth, a better solution is to use:
find /backups/* -type d -prune -exec ls -d {} \; |tail -1
Just swap the "/backups/" portion for your actual path.
If you want to avoid showing an absolute path in a bash script, you could always use something like this:
LOCALPATH=/backups
DIRECTORY=$(cd $LOCALPATH; find * -type d -prune -exec ls -d {} \; |tail -1)

With GNU find you can get list of directories with modification timestamps, sort that list and output the newest:
find . -mindepth 1 -maxdepth 1 -type d -printf "%T#\t%p\0" | sort -z -n | cut -z -f2- | tail -z -n1
or newline separated
find . -mindepth 1 -maxdepth 1 -type d -printf "%T#\t%p\n" | sort -n | cut -f2- | tail -n1
With POSIX find (that does not have -printf) you may, if you have it, run stat to get file modification timestamp:
find . -mindepth 1 -maxdepth 1 -type d -exec stat -c '%Y %n' {} \; | sort -n | cut -d' ' -f2- | tail -n1
Without stat a pure shell solution may be used by replacing [[ bash extension with [ as in this answer.

Your "something like this" was almost a hit:
BACKUPDIR=$(ls -t ./backups | head -1)
Combining what you wrote with what I have learned solved my problem too. Thank you for rising this question.
Note: I run the line above from GitBash within Windows environment in file called ./something.bash.

Related

"find | xargs | ls" not running ls on filenames from find

So I have a directory with files and sub-directories in it. I want to get all the files recursively and then list them in long format, sorted by the modified date. Here's what I came up with.
find . -type f | xargs -d "\n" | ls -lt
However this only lists the files in the current directory and not the sub-directories. I don't understand why, given that the following prints out all the files.
find . -type f | xargs -d "\n" | cat
Any help appreciated.
xargs can only start ls if it's passed ls as an argument. When you pipe from xargs into ls, only one copy of ls is started -- by the parent shell -- and it isn't given any of the filenames from find | xargs as arguments -- instead they're on its stdin, but ls never reads its stdin, so it doesn't even know that they're there.
Thus, you need to remove the | character:
# Does what you specified in the common case, but buggy; don't use this
# (filenames can contain newlines!)
# ...also, xargs -d is GNU-only
find . -type f | xargs -d '\n' ls -lt
...or, better:
# uses NUL separators, which cannot exist inside filenames
# also, while a non-POSIX extension, this is supported in both GNU and BSD xargs
find . -type f -print0 | xargs -0 ls -lt
...or, even better than that:
# no need for xargs at all here; find -exec can do the same thing
# -exec ... {} + is POSIX-mandated functionality since 2008
find . -type f -exec ls -lt {} +
Much of the content in this answer is also covered in the Actions, Complex Actions, and Actions in Bulk sections of Using Find, which is well worth reading.

How to remove all directories except for the last one

I can get the amount of dirs with command
ls -dtr */ | wc -l
But how do I specifically delete the N-1 directories, leaving the most recent one?
As usual for any bash operation on "a bunch of files", you have to be aware of the pain that is spaces and newlines, which may legally appear in file names. These break naïve xargs / for based approaches and require some extra hoops to jump.
Many tools support -z or -0 options, which use NUL bytes as line separator instead of newlines -- NUL may never be part of a file name.
Unfortunately, ls is not one of them, so we have to go through find to get the latest directory.
find . -maxdepth 1 -mindepth 1 -type d
This gets you all the directories in the current directory. (Note: As opposed to your ls -dtr */, this will also find "hidden" directories, i.e. ones starting with a .. Add ! -name ".*" to avoid that.)
-maxdepth 1 avoids recursion, -mindepth 1 keeps the parent directory (.) out of the list.
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0"
This lists the directories and their timestamps, using NUL instead of newline for line separation. (NUL can never be part of a file name.)
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0" | sort -z
This sorts the results, using NUL instead of newline for line separation.
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0" | sort -z | head -z -n -1
This takes everything but the last entry (the latest directory and its timestamp) from the list, using NUL instead of newline for line separation.
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0" | sort -z | head -z -n -1 | cut -z -d' ' -f 2-
Using NUL instead of newline for line separation and space as field delimiter, this filters the first field (the timestamp) from the output.
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0" | sort -z | head -z -n -1 | cut -z -d' ' -f 2- | xargs -0 rm -rf
Using NUL instead of newline for line separation, this calls rm -rf for each entry.
Use tail:
ls -dt */ | tail -n +2 | while read dirName; do rmdir "$dirName"; done
When specifying the -n option of tail with a +, it skips the first lines.
This will list the directories sorted by date, skip the most recent one and keep the rest, which will be deleted. You can replace rmdir with rm -rf following your use case.
You can try it out:
mkdir test && cd test
for i in {1..100}; do mkdir subdir-$i; done
ls -dt */ | tail -n +2 | while read dirName; do rmdir "$dirName"; done
ls
Output:
subdir-100
EDIT: if any of the directory names contain newline characters, this approach will fail because of tail. See #DevSolar's answer for a more complete one in that case.
This is a pure Bash solution:
shopt -s nullglob # Globs that match nothing expand to nothing
shopt -s dotglob # Globs include files whose names start with '.'
newest_dir=
for sdir in */ ; do
dir=${sdir%/} # Remove slash to enable a symlink check
[[ -L $dir ]] && continue # Skip symlinks to directories
if [[ -z $newest_dir ]] ; then
newest_dir=$dir
elif [[ $dir -nt $newest_dir ]] ; then
echo rm -rf -- "$newest_dir"
newest_dir=$dir
else
echo rm -rf -- "$dir"
fi
done
In its current form it just prints the commands to remove the old subdirectories. Remove the echos to make it functional.
So many ways, here's another. :)
rm -rf `ls -1dt * | tail -n +2 | tr '\n' ' '`
First, the ls -1dt * just lists all the directories sorted by modification time, one per line. Then tail -n +2 removes the first, most recently modified. Then tr '\n ' ' puts them all into a single line. Then the rm -rf the dirs.

remove files from subfolders without the last three

I have a structure like that:
/usr/local/a/1.txt
/usr/local/a/2.txt
/usr/local/a/3.txt
/usr/local/b/4.txt
/usr/local/b/3.txt
/usr/local/c/1.txt
/usr/local/c/7.txt
/usr/local/c/6.txt
/usr/local/c/12.txt
...
I want to delete all the files *.txt in subfolders except the last three files with the greatest modification date, but here I am in current directory
ls -tr *.txt | head -n-3 |xargs rm -f
I need to combine that with the code:
find /usr/local/**/* -type f
Should I use the maxdepth option?
Thanks for helping,
aola
Added maxdepth options to find for one level, sorting files by last modification time, tail to ignore the oldest modified 3 files and xargs with -r to remove the files only if they are found.
for folder in $(find /usr/local/ -type d)
do
find $folder -maxdepth 1 -type f -name "*.txt" | xargs -r ls -1tr | tail -n+3 | xargs -r rm -f
done
Run the above command once without rm to ensure that the previous commands pick the proper files for deletion.
You've almost got the solution: use find to get the files,ls to sort them by modification date and tail to omit three most recently modified ones:
find /usr/lib -type f | xargs ls -t | tail -n +4 | xargs rm
If you would like to remove only the files at a specified depth add -mindepth 4 -maxdepth 4 to find parameters.
You can use find's -printf option, to print the modification time in front of the file name and then sort and strip the date off. This avoids using ls at all.
find /usr/local -type f -name '*.txt' -printf '%T#|%p\n' | sort -r | cut -d '|' -f 2 | head -n-3 | xargs rm -f
The other Answers using xargs ls -t can lead to incorrect results, when there are more results than xargs can put in a single ls -t command.
but for each subfolder, so when I have
/usr/local/a/1.txt
/usr/local/a/2.txt
/usr/local/a/3.txt
/usr/local/a/4.txt
/usr/local/b/4.txt
/usr/local/b/3.txt
/usr/local/c/1.txt
/usr/local/c/7.txt
/usr/local/c/6.txt
/usr/local/c/12.txt
I want to to use the code for each subfolder separately
head -n-3 |xargs rm -f
so I bet if I have it sorted by date then the files to delete:
/usr/local/a/4.txt
/usr/local/c/12.txt
I want to leave in any subfolder three newest files

How to list only files and not directories of a directory Bash?

How can I list all the files of one folder but not their folders or subfiles. In other words: How can I list only the files?
Using find:
find . -maxdepth 1 -type f
Using the -maxdepth 1 option ensures that you only look in the current directory (or, if you replace the . with some path, that directory). If you want a full recursive listing of all files in that and subdirectories, just remove that option.
ls -p | grep -v /
ls -p lets you show / after the folder name, which acts as a tag for you to remove.
carlpett's find-based answer (find . -maxdepth 1 -type f) works in principle, but is not quite the same as using ls: you get a potentially unsorted list of filenames all prefixed with ./, and you lose the ability to apply ls's many options;
also find invariably finds hidden items too, whereas ls' behavior depends on the presence or absence of the -a or -A options.
An improvement, suggested by Alex Hall in a comment on the question is to combine shell globbing with find:
find * -maxdepth 0 -type f # find -L * ... includes symlinks to files
However, while this addresses the prefix problem and gives you alphabetically sorted output, you still have neither (inline) control over inclusion of hidden items nor access to ls's many other sorting / output-format options.
Hans Roggeman's ls + grep answer is pragmatic, but locks you into using long (-l) output format.
To address these limitations I wrote the fls (filtering ls) utility,
a utility that provides the output flexibility of ls while also providing type-filtering capability,
simply by placing type-filtering characters such as f for files, d for directories, and l for symlinks before a list of ls arguments (run fls --help or fls --man to learn more).
Examples:
fls f # list all files in current dir.
fls d -tA ~ # list dirs. in home dir., including hidden ones, most recent first
fls f^l /usr/local/bin/c* # List matches that are files, but not (^) symlinks (l)
Installation
Supported platforms
When installing from the npm registry: Linux and macOS
When installing manually: any Unix-like platform with Bash
From the npm registry
Note: Even if you don't use Node.js, its package manager, npm, works across platforms and is easy to install; try
curl -L https://git.io/n-install | bash
With Node.js installed, install as follows:
[sudo] npm install fls -g
Note:
Whether you need sudo depends on how you installed Node.js / io.js and whether you've changed permissions later; if you get an EACCES error, try again with sudo.
The -g ensures global installation and is needed to put fls in your system's $PATH.
Manual installation
Download this bash script as fls.
Make it executable with chmod +x fls.
Move it or symlink it to a folder in your $PATH, such as /usr/local/bin (macOS) or /usr/bin (Linux).
Listing content of some directory, without subdirectories
I like using ls options, for sample:
-l use a long listing format
-t sort by modification time, newest first
-r reverse order while sorting
-F, --classify append indicator (one of */=>#|) to entries
-h, --human-readable with -l and -s, print sizes like 1K 234M 2G etc...
Sometime --color and all others. (See ls --help)
Listing everything but folders
This will show files, symlinks, devices, pipe, sockets etc.
so
find /some/path -maxdepth 1 ! -type d
could be sorted by date easily:
find /some/path -maxdepth 1 ! -type d -exec ls -hltrF {} +
Listing files only:
or
find /some/path -maxdepth 1 -type f
sorted by size:
find /some/path -maxdepth 1 -type f -exec ls -lSF --color {} +
Prevent listing of hidden entries:
To not show hidden entries, where name begin by a dot, you could add ! -name '.*':
find /some/path -maxdepth 1 ! -type d ! -name '.*' -exec ls -hltrF {} +
Then
You could replace /some/path by . to list for current directory or .. for parent directory.
You can also use ls with grep or egrep and put it in your profile as an alias:
ls -l | egrep -v '^d'
ls -l | grep -v '^d'
find files: ls -l /home | grep "^-" | tr -s ' ' | cut -d ' ' -f 9
find directories: ls -l /home | grep "^d" | tr -s ' ' | cut -d ' ' -f 9
find links: ls -l /home | grep "^l" | tr -s ' ' | cut -d ' ' -f 9
tr -s ' ' turns the output into a space-delimited file
the cut command says the delimiter is a space, and return the 9th field (always the filename/directory name/linkname).
I use this all the time!
You are welcome!
ls -l | grep '^-'
Looking just for the name, pipe to cut or awk.
ls -l | grep '^-' | awk '{print $9}'
ls -l | grep '^-' | cut -d " " -f 13
{ find . -maxdepth 1 -type f | xargs ls -1t | less; }
added xargs to make it works, and used -1 instead of -l to show only filenames without additional ls info
You can one of these:
echo *.* | cut -d ' ' -f 1- --output-delimiter=$'\n'
echo *.* | tr ' ' '\n'
echo *.* | sed 's/\s\+/\n/g'
ls -Ap | sort | grep -v /
This method does not use external commands.
bash$ res=$( IFS=$'\n'; AA=(`compgen -d`); IFS='|'; eval compgen -f -X '#("${AA[*]}")' )
bash$ echo "$res"
. . .
Just adding on to carlpett's answer.
For a much useful view of the files, you could pipe the output to ls.
find . -maxdepth 1 -type f|ls -lt|less
Shows the most recently modified files in a list format, quite useful when you have downloaded a lot of files, and want to see a non-cluttered version of the recent ones.
"find '-maxdepth' " does not work with my old version of bash, therefore I use:
for f in $(ls) ; do if [ -f $f ] ; then echo $f ; fi ; done

Use GNU find to show only the leaf directories

I'm trying to use GNU find to find only the directories that contain no other directories, but may or may not contain regular files.
My best guess so far has been:
find dir -type d \( -not -exec ls -dA ';' \)
but this just gets me a long list of "."
Thanks!
You can use -links if your filesystem is POSIX compliant (i.e. a directory has a link for each subdirectory in it, a link from its parent and a link to itself, thus a count of 2 links if it has no subdirectories).
The following command should do what you want:
find dir -type d -links 2
However, it does not seems to work on Mac OS X (as #Piotr mentioned). Here is another version that is slower, but does work on Mac OS X. It is based on his version, with a correction to handle whitespace in directory names:
find . -type d -exec sh -c '(ls -p "{}"|grep />/dev/null)||echo "{}"' \;
I just found another solution to this that works on both Linux & macOS (without find -exec)!
It involves sort (twice) and awk:
find dir -type d | sort -r | awk 'a!~"^"$0{a=$0;print}' | sort
Explanation:
sort the find output in reverse order
now you have subdirectories appear first, then their parents
use awk to omit lines if the current line is a prefix of the previous line
(this command is from the answer here)
now you eliminated "all parent directories" (you're left with parent dirs)
sort them (so it looks like the normal find output)
Voila! Fast and portable.
#Sylvian solution didn't work for me on mac os x for some obscure reason. So I've came up with a bit more direct solution. Hope this will help someone:
find . -type d -print0 | xargs -0 -IXXX sh -c '(ls -p XXX | grep / >/dev/null) || echo XXX' ;
Explanation:
ls -p ends directories with '/'
so (ls -p XXX | grep / >/dev/null) returns 0 if there is no directories
-print0 && -0 is to make xargs handle spaces in directory names
I have some oddly named files in my directory trees that confuse awk as in
#AhmetAlpBalkan 's answer. So I took a slightly different approach
p=;
while read c;
do
l=${#c};
f=${p:0:$l};
if [ "$f" != "$c" ]; then
echo $c;
fi;
p=$c;
done < <(find . -type d | sort -r)
As in the awk solution, I reverse sort. That way if the directory path is a subpath of the previous hit, you can easily discern this.
Here p is my previous match, c is the current match, l is the length of the current match, f is the first l matching characters of the previous match. I only echo those hits that don't match the beginning of the previous match.
The problem with the awk solution offered is that the matching of the beginning of the string seems to be confused if the path name contains things such as + in the name of some of the subdirectories. This caused awk to return a number of false positives for me.
There is an alternative to find called rawhide (rh) that is much easier to use.
For filesystems other than btrfs:
rh 'd && nlink == 2'
For btrfs:
rh 'd && "[ `rh -red %S | wc -l` = 0 ]".sh'
A shorter/faster version for btrfs is:
rh 'd && "[ -z \"`rh -red %S`\" ]".sh'
The above commands search for directories and then list their sub-directories and only match when there are none (the first by counting the number of lines of output, and the second by checking if there is any output at all per directory).
For a version that works on all filesystems as efficiently as possible:
rh 'd && (nlink == 2 || nlink == 1 && "[ -z \"`rh -red %S`\" ]".sh)'
On normal (non-btrfs) filesystems, this will work without the need for any additional processes for each directory, but on btrfs, it will need them. This is probably best if you have a mix of different filesystems including btrfs.
Rawhide (rh) is available from https://raf.org/rawhide or https://github.com/raforg/rawhide. It works at least on Linux, FreeBSD, OpenBSD, NetBSD, Solaris, macOS, and Cygwin.
Disclaimer: I am the current author of rawhide.
What about this one ? It's portable and it doesn't depend on finnicky linking counts. Note however that it's important to put root/folder without the trailing /.
find root/folder -type d | awk '{ if (length($0)<length(prev) || substr($0,1,length(prev))!=prev) print prev; prev=($0 "/") } END { print prev }'
Here is solution which works on Linux and OS X:
find . -type d -execdir bash -c '[ "$(find {} -mindepth 1 -type d)" ] || echo $PWD/{}' \;
or:
find . -type d -execdir sh -c 'test -z "$(find "{}" -mindepth 1 -type d)" && echo $PWD/{}' \;
This awk/sort pipe works a bit better than the one originally proposed in this answer, but is heavily based on it :) It will work more reliably regardless of whether the path contains regex special characters or not:
find . -type d | sort -r | awk 'index(a,$0)!=1{a=$0;print}' | sort
Remember that awk strings are 1-indexed instead of 0-indexed, which might be strange if you're used to working with C-based languages.
If the index of the current line in the previous line is 1 (i.e. it starts with it) then we skip it, which works just like the match of "^"$0.
My 2 cents on this problem:
#!/bin/bash
(
while IFS= read -r -d $'\0' directory
do
files=$(ls -A "$directory" | wc -l)
if test $files -gt 0
then
echo "$directory"
fi
done < <(find . -type d -print0)
) | sort | uniq
It uses a subshell to capture output from the run, and lists directories which have files.

Resources