I was wondering if there was a way to list all files from the current directory, but not the directories themselves, recursively in BASh.
EXAMPLE:
# list -r
/root/foo.txt
/root/log.txt
/root/tremp/passwd.list
But not:
# list -r
/root/
/root/foo.txt
/root/log.txt
/root/temp/
/root/tremp/passwd.list
Use find:
find . -type f
This will recursively search all files in the current directory. For instance:
$ mkdir temp
$ touch temp/passwd.list
$ touch log.txt
$ touch foo.txt
$ find . -type f
./foo.txt
./log.txt
./temp/passwd.list
You can also grep the provided output with this:
grep -v ./$
That is, list everything but the lines which finish with "/" (directory)
Related
Given a root path, I am trying to loop through the sub-directories to loop through the files in each subdirectory and print the names of the files.
The directory structure is like this:
Root directory
dir2,
file{1..10}
dir3,
file{1..10}
dir4
file{1..10}
I want to loop through dir2 and print all the filenames in it. Then loop through dir3 and print all the file names...and so on
Here is what I have so far:
#!/bin/bash
#!/bin/sh
cd /the/root/directory
for dir in */
do
for FILE in dir
do
echo "$FILE"
done > /the/root/directory/filenames.txt
done
This is the output I get in filenames.txt:
dir
My expected output is supposed to be:
file{1..10}
file{1..10}
file{1..10}
I am a beginner to bash scripting...well scripting in general.
Any help is greatly appreciated!
You didn't mention what your end goal is, so I'll speculate here.
If your end goal is to only see the files recursively in a list, you can run just a simple find command:
find . -type f
Or if you want to see the details:
find . -type f -ls
A nice way to view them with colors and nice-looking ansi bars is to install the tree command. Example:
https://www.tecmint.com/linux-tree-command-examples/
If your needs are simple, for example, you want to do an action such as a tail -n1 on each file, you can pipe the command to xargs like this:
find . -type f | xargs tail -n1
But if your end goal is to use bash to process them in some way, then you can continue down the bash looping method as mentioned by #tjm3772.
You mentioned you were just looking for the filenames so you can just run:
find . -type f | sed 's/.*\///'
If you want to write that to a file, just redirect the output to a filename of your choice:
find . -type f | sed 's/.*\///' > filename.txt
You can use the find command, and this will loop through the directories without needing the for loop
my_bash_script.sh:
find * -type d > filenames.txt
Put this script in the same level of the directories, or you point it to the location by changing the * to the path
note: if it says permission denied in the terminal run this: chmod u+x the_script_name.sh
You forgot to expand $dir in your inner loop, so the loop is executing one time with FILE set to the literal string 'dir' instead of the directory name.
After that, you need a globbing pattern to expand to the filenames inside the directory.
Fixed example:
#!/bin/bash
cd /the/root/directory
for dir in */
do
for FILE in "$dir/"*
do
echo "$FILE"
done > /the/root/directory/filenames.txt
done
The bash's way to do it is with the globstar expansion which expands recursively in directories
#!/usr/bin/env bash
shopt -s globstar # This enables recursively expanding files in directories
# This prints all the files in all the directories starting from /the/root/directory
printf '%s\n' /the/root/directory/**
I have the following problem.
There are two nested folders A and B. They are mostly identical, but B has a few files that A does not. (These are two mounted rootfs images).
I want to create a shell script that does the following:
Find out which files are contained in B but not in A.
copy the files found in 1. from B and create a tar.gz that contains these files, keeping the folder structure.
The goal is to import the additional data from image B afterwards on an embedded system that contains the contents of image A.
For the first step I put together the following code snippet. Note to grep "Nur" : "Nur in" = "Only in" (german):
diff -rq <A> <B>/ 2>/dev/null | grep Nur | awk '{print substr($3, 1, length($3)-1) "/" substr($4, 1, length($4)-1)}'
The result is the output of the paths relative to folder B.
I have no idea how to implement the second step. Can someone give me some help?
Using diff for finding files which don't exist is severe overkill; you are doing a lot of calculations to compare the contents of the files, where clearly all you care about is whether a file name exists or not.
Maybe try this instead.
tar zcf newfiles.tar.gz $(comm -13 <(cd A && find . -type f | sort) <(cd B && find . -type f | sort) | sed 's/^\./B/')
The find commands produce a listing of the file name hierarchies; comm -13 extracts the elements which are unique to the second input file (which here isn't really a file at all; we are using the shell's process substitution facility to provide the input) and the sed command adds the path into B back to the beginning.
Passing a command substitution $(...) as the argument to tar is problematic; if there are a lot of file names, you will run into "command line too long", and if your file names contain whitespace or other irregularities in them, the shell will mess them up. The standard solution is to use xargs but using xargs tar cf will overwrite the output file if xargs ends up calling tar more than once; though perhaps your tar has an option to read the file names from standard input.
With find:
$ mkdir -p A B
$ touch A/a A/b
$ touch B/a B/b B/c B/d
$ cd B
$ find . -type f -exec sh -c '[ ! -f ../A/"$1" ]' _ {} \; -print
./c
./d
The idea is to use the exec action with a shell script that tests the existence of the current file in the other directory. There are a few subtleties:
The first argument of sh -c is the script to execute, the second (here _ but could be anything else) corresponds to the $0 positional parameter of the script and the third ({}) is the current file name as set by find and passed to the script as positional parameter $1.
The -print action at the end is needed, even if it is normally the default with find, because the use of -exec cancels this default.
Example of use to generate your tarball with GNU tar:
$ cd B
$ find . -type f -exec sh -c '[ ! -f ../A/"$1" ]' _ {} \; -print > ../list.txt
$ tar -c -v -f ../diff.tar --files-from=../list.txt
./c
./d
Note: if you have unusual file names the --verbatim-files-from GNU tar option can help. Or a combination of the -print0 action of find and the --null option of GNU tar.
Note: if the shell is POSIX (e.g., bash) you can also run find from the parent directory and get the path of the files relative from there, if you prefer:
$ mkdir -p A B
$ touch A/a A/b
$ touch B/a B/b B/c B/d
$ find B -type f -exec sh -c '[ ! -f A"${1#B}" ]' _ {} \; -print
B/c
B/d
I want to recursively rename all files in directory path by changing their prefix.
For Example
XYZMyFile.h
XYZMyFile.m
XYZMyFile1.h
XYZMyFile1.m
XYZMyFile2.h
XYZMyFile2.m
TO
ABCMyFile.h
ABCMyFile.m
ABCMyFile1.h
ABCMyFile1.m
ABCMyFile2.h
ABCMyFile2.m
These files are under a directory structure with many layers. Can someone help me with a shell script for this bulk task?
A different approach maybe:
ls *.{h,m} | while read a; do n=ABC$(echo $a | sed -e 's/^XYZ//'); mv $a $n; done
Description:
ls *.{h,m} --> Find all files with .h or .m extension
n=ABC --> Add a ABC prefix to the file name
sed -e 's/^XYZ//' --> Removes the XYZ prefix from the file name
mv $a $n --> Performs the rename
Set globstar first and then use rename like below:
# shopt -s globstar # This will cause '**' to expand to each and everything
# ls -R
.:
nXYZ1.c nXYZ2.c nXYZ3.c subdir XYZ1.m XYZ2.m XYZ3.m
nXYZ1.h nXYZ2.h nXYZ3.h XYZ1.c XYZ2.c XYZ3.c
nXYZ1.m nXYZ2.m nXYZ3.m XYZ1.h XYZ2.h XYZ3.h
./subdir:
nXYZ1.c nXYZ1.m nXYZ2.h nXYZ3.c nXYZ3.m XYZ1.h XYZ2.c XYZ2.m XYZ3.h
nXYZ1.h nXYZ2.c nXYZ2.m nXYZ3.h XYZ1.c XYZ1.m XYZ2.h XYZ3.c XYZ3.m
# rename 's/^XYZ(.*.[mh])$/ABC$1/;s/^([^\/]*\/)XYZ(.*.[mh])$/$1ABC$2/' **
# ls -R
.:
ABC1.h ABC2.m nXYZ1.c nXYZ2.c nXYZ3.c subdir XYZ3.c
ABC1.m ABC3.h nXYZ1.h nXYZ2.h nXYZ3.h XYZ1.c
ABC2.h ABC3.m nXYZ1.m nXYZ2.m nXYZ3.m XYZ2.c
./subdir:
ABC1.h ABC2.h ABC3.h nXYZ1.c nXYZ1.m nXYZ2.h nXYZ3.c nXYZ3.m XYZ2.c
ABC1.m ABC2.m ABC3.m nXYZ1.h nXYZ2.c nXYZ2.m nXYZ3.h XYZ1.c XYZ3.c
# shopt -u globstar # Unset gobstar
This may be the simplest way to achieve your objective.
Note1 : Here I am not changing nXYZ to nABC as you have noticed. If they are meant to be changed the simplified rename command would be
rename 's/XYZ(.*.[mh])$/ABC$1/' **
Note2 : The question has mentioned nothing about multiple occurrences of XYZ. So nothing done in this regard.
Easy find and rename (the binary in /usr/bin, not the Perl function mentioned)
Yes, there is a command to do this non-recursive already.
rename XYZ ABC XYZ*
rename --help
Usage:
rename [options] expression replacement file...
Options:
-v, --verbose explain what is being done
-s, --symlink act on symlink target
-h, --help display this help and exit
-V, --version output version information and exit
For more details see rename(1).
edit: missed the "many layers of directory" part of the question, b/c it's a little messy. Adding the find.
Easiest to remember:
find . -type f -name "*.pdf" -exec rename XYZ ABC {} \;
Probably faster to finish:
find . -type d -not -path "*/\.*" -not -name ".*" -exec rename XYZ ABC {}/*.pdf \;
I'm not sure how to get easier than one command line of code.
For non-recursive, you can use rename which is a perl script:
rename -v -n 's/^.+(?=MyFile)/what-you-want/' *.{h,m}
test:
dir > ls | cat -n
1 XYZMyFile1.h
2 XYZMyFile1.m
3 XYZMyFile.h
4 XYZMyFile.m
dir >
dir > rename -v -n 's/^.+(?=MyFile)/what-you-want/' *.{h,m}
rename(XYZMyFile1.h, what-you-wantMyFile1.h)
rename(XYZMyFile1.m, what-you-wantMyFile1.m)
rename(XYZMyFile.h, what-you-wantMyFile.h)
rename(XYZMyFile.m, what-you-wantMyFile.m)
dir >
and for recursive,use find + this command
If you do not have access to rename, you can use perl directly like so:
perl -le '($old=$_) && s/^xzy/abc/g && rename($old,$_) for <*.[mh]>'
and here is a screen-shot
and with renrem, a CLI I developed using C++, specifically for renaming
I need a Script that writes the directory and subdirectory in a text-file.
For example the script lies in /Mainfolder and in this folder are four other folders. Each contains several files.
Now I would like the script to write the path of each file in the textfile.
Subfolder1/File1.dat
Subfolder1/File2.dat
Subfolder2/File1.dat
Subfolder3/File1.dat
Subfolder4/File1.dat
Subfolder4/File2.dat
Important is that there is no slash in front of the listing.
Use the find command:
find Mainfolder > outputfile
and if you only want the files listed, do
find Mainfolder -type f > outputfile
You can also strip the leading ./ if you search the current directory, with the %P format option:
find . -type f -printf '%P\n' > outputfile
If your bash version is high enough, you can do it like that:
#!/bin/bash
shopt -s globstar
echo ** > yourtextfile
This solution assumes that the subdirectories contain only files -- they do not contain any directory in turn.
find . -type f -print | sed 's|^.*/S|S|'
I have created a single file in each of the four subdirectories. The original output is:
./Subfolder1/File1.dat
./Subfolder4/File4.dat
./Subfolder2/File2.dat
./Subfolder3/File3.dat
The filtered output is:
Subfolder1/File1.dat
Subfolder4/File4.dat
Subfolder2/File2.dat
Subfolder3/File3.dat
You can use this find with -exec:
find . -type f -exec bash -c 'f="{}"; echo "${f:2}"' \;
This will print all files starting from current paths by removing ./ from front.
I have a directory which contains multiple sub-directories with mov and jpg files.
/dir/
/subdir-a/ # contains a-1.jpg, a-2.jpg, a-1.mov
/subdir-b/ # contains b-1.mov
/subdir-c/ # contains c-1.jpg
/subdir-d/ # contains d-1.mov
... # more directories with the same pattern
I need to find a way using command-line tools (on Mac OSX, ideally) to move all the mov files to a new location. However, one requirement is to keep directory structure i.e.:
/dir/
/subdir-a/ # contains a-1.mov
/subdir-b/ # contains b-1.mov
# NOTE: subdir-c isn't copied because it doesn't have mov files
/subdir-d/ # contains d-1.mov
...
I am familiar with find, grep, and xargs but wasn't sure how to solve this issue. Thank you very much beforehand!
It depends slightly on your O/S and, more particularly, on the facilities in your version of tar and whether you have the command cpio. It also depends a bit on whether you have newlines (in particular) in your file names; most people don't.
Option #1
cd /old-dir
find . -name '*.mov' -print | cpio -pvdumB /new-dir
Option #2
find . -name '*.mov' -print | tar -c -f - -T - |
(cd /new-dir; tar -xf -)
The cpio command has a pass-through (copy) mode which does exactly what you want given a list of file names, one per line, on its standard input.
Some versions of the tar command have an option to read the list of file names, one per line, from standard input; on MacOS X, that option is -T - (where the lone - means 'standard input'). For the first tar command, the option -f - means (in the context of writing an archive with -c, write to standard output); in the second tar command, the -x option means that the -f - means 'read from standard input'.
There may be other options; look at the manual page or help output of tar rather carefully.
This process copies the files rather than moving them. The second half of the operation would be:
find . -name '*.mov' -exec rm -f {} +
ASSERT: No files have newline characters in them. Spaces, however, are AOK.
# TEST FIRST: CREATION OF FOLDERS
find . -type f -iname \*.mov -printf '%h\n' | sort | uniq | xargs -n 1 -d '\n' -I '{}' echo mkdir -vp "/TARGET_FOLDER_ROOT/{}"
# EXECUTE CREATION OF EMPTY TARGET FOLDERS
find . -type f -iname \*.mov -printf '%h\n' | sort | uniq | xargs -n 1 -d '\n' -I '{}' mkdir -vp "/TARGET_FOLDER_ROOT/{}"
# TEST FIRST: REVIEW FILES TO BE MOVED
find . -type f -iname \*.mov -exec echo mv {} /TARGET_FOLDER_ROOT/{} \;
# EXECUTE MOVE FILES
find . -type f -iname \*.mov -exec mv {} /TARGET_FOLDER_ROOT/{} \;
Being large files, if they are on the same file system you don't want to copy them, but just to replicate their directory structure while moving.
You can use this function:
# moves a file (or folder) preserving its folder structure (relative to source path)
# usage: move_keep_path source destination
move_keep_path () {
# create directories up to one level up
mkdir -p "`dirname "$2"`"
mv "$1" "$2"
}
Or, adding support to merging existing directories:
# moves a file (or folder) preserving its folder structure (relative to source path)
# usage: move_keep_path source destination
move_keep_path () {
# create directories up to one level up
mkdir -p "`dirname "$2"`"
if [[ -d "$1" && -d "$2" ]]; then
# merge existing folder
find "$1" -depth 1 | while read file; do
# call recursively for all files inside
mv_merge "$file" "$2/`basename "$file"`"
done
# remove after merge
rmdir "$1"
else
# either file or non-existing folder
mv "$1" "$2"
fi
}
It is easier to just copy the files like:
cp --parents some/folder/*/*.mov new_folder/
from the parent directory of "dir execute this:
find ./dir -name "*.mov" | xargs tar cif mov.tar
Then cd to the directory you want to move the files to and execute this:
tar xvf /path/to/parent/directory/of"dir"/mov.tar
This should work if you want to move all mov files to a directory called new location -
find ./dir -iname '*.mov' -exec mv '{}' ./newlocation \;
However, if you wish to move the mov files along with their sub-dirs then you can do something like this -
Step 1: Copy entire structure of /dir to a new location using cp
cp -iprv dir/ newdir
Step 2: Find jpg files from newdir and delete them.
find ./newdir -iname "*.jpg" -delete
Test:
[jaypal:~/Temp] ls -R a
a.mov aa b.mov
a/aa:
aaa c.mov d.mov
a/aa/aaa:
e.mov f.mov
[jaypal:~/Temp] mkdir d
[jaypal:~/Temp] find ./a -iname '*.mov' -exec mv '{}' ./d \;
[jaypal:~/Temp] ls -R d
a.mov b.mov c.mov d.mov e.mov f.mov
I amended the function of #djjeck, because it didn't work as I needed. The function below moves a source file to a destination directory also creating the needed levels of hierarchy in the source file path (see the example below):
# moves a file, creates needed levels of hierarchy in destination
# usage: move_with_hierarchy source_file destination top_level_directory
move_with_hierarchy () {
path_tail=$(dirname $(realpath --relative-to="$3" "$1"))
cd "$2"
mkdir -p $path_tail
cd - > /dev/null
mv "$1" "${2}/${path_tail}"
}
example:
$ ls /home/sergei/tmp/dir1/dir2/bla.txt
/home/sergei/tmp/dir1/dir2/bla.txt
$ rm -rf tmp2
$ mkdir tmp2
$ move_with_hierarchy /home/sergei/tmp/dir1/dir2/bla.txt /home/sergei/tmp2 /home/sergei/tmp
$ tree ~/tmp2
/home/sergei/tmp2
└── dir1
└── dir2
└── bla.txt
2 directories, 1 file