Find all directories that contain only hidden files and/or hidden directories - bash

Issue
I have been struggling with writing a Bash command that is able to recursively search a directory and then return the paths of every sub-directory (up to a certain max-depth) that contains exclusively hidden files and/or hidden directories.
Visual Explanation
Consider the following File System excerpt:
+--- Root_Dir
| +--- Dir_A
| | +--- abc.txt
| | +--- 123.txt
| | +--- .hiddenfile
| | +--- .hidden_dir
| | | +--- normal_sub_file_1.txt
| | | +--- .hidden_sub_file_1.txt
| |
| +--- Dir_B
| | +--- abc.txt
| | +--- .hidden_dir
| | | +--- normal_sub_file_2.txt
| | | +--- .hidden_sub_file_2.txt
| |
| +--- Dir_C
| | +--- 123.txt
| | +--- program.c
| | +--- a.out
| | +--- .hiddenfile
| |
| +--- Dir_D
| | +--- .hiddenfile
| | +--- .another_hiddenfile
| |
| +--- Dir_E
| | +--- .hiddenfile
| | +--- .hidden_dir
| | | +--- normal_sub_file_3.txt # This is OK because its within a hidden directory, aka won't be checked
| | | +--- .hidden_sub_file_3.txt
| |
| +--- Dir_F
| | +--- .hidden_dir
| | | +--- normal_sub_file_4.txt
| | | +--- .hidden_sub_file_4.txt
Desired Output
The command I am looking for would output
./Root_Dir/Dir_D
./Root_Dir/Dir_E
./Root_Dir/Dir_F
Dir_D because it only contains hidden files.
Dir_E because it only contains a hidden file and a hidden directory at the level I am searching.
Dir_F because it only contains a hidden directory at the level I am searching.
Attempts
I have attempted to use the find command to get the results I am looking for but I can't seem to figure out what other command I need to pipe the output to or what other options I should be using.
I think the command that will work would look something like this:
$ find ./Root_Dir -mindepth 2 -maxdepth 2 -type d -name "*." -type -f -name "*." | command to see if these are the only files in that directory

Parsing find's output is not a good idea; -exec exists, and sh can do the filtering without breaking anything.
find . -type d -exec sh -c '
for d; do
for f in "$d"/*; do
test -e "$f" &&
continue 2
done
for f in "$d"/.[!.]* "$d"/..?*; do
if test -e "$f"; then
printf %s\\n "$d"
break
fi
done
done' sh {} +
You can adjust the depth using whatever extension your find provides for it.

Assuming this structure, you don't need find.
Adjust your pattern as needed.
for d in $ROOT_DIR/Dir_?/; do
lst=( $d* ); [[ -e "${lst[0]}" ]] && continue # normal files, skip
lst=( $d.* ); [[ -e "${lst[2]}" ]] || continue # NO hidden, so skip
echo "$d"
done
I rebuilt your file structure in my /tmp dir and saved this as tst, so
$: ROOT_DIR=/tmp ./tst
/tmp/Dir_D/
/tmp/Dir_E/
/tmp/Dir_F/
Note that the confirmation of hidden files uses "${lst[2]}" because the first 2 will always be . and .., which don't count.
You could probably use for d in $ROOT_DIR/*/.
I suspect this'll do for you. (mindepth=2, maxdepth=2)
If you needed deeper subdirectories (mindepth=3, maxdepth=3) you could add a level -
for d in $ROOT_DIR/*/*/
and/or both (mindepth=2, maxdepth=3)
for d in $ROOT_DIR/*/ $ROOT_DIR/*/*/
or if you didn't want a mindepth/maxdepth,
shopt -s globstar
for d in $ROOT_DIR/**/

This will work if your filenames contain no newlines.
find -name '.*' | awk -F/ -v OFS=/ '{ --NF } !a[$0]++'
Learn awk: https://www.gnu.org/software/gawk/manual/gawk.html

Related

Moving all files from subfolders to main folders with duplicate file names

I've been trying to write a little script to sort image files in my Linux server.
I tried multiple solution found all over StackExchange but it never meets my requirements.
Explanation:
photo_folder are filled with images (various extensions).
Mostly, images are already in this folder.
But sometime, like the example below, images are hidden in one or multiple photo_subfolder and file names are often the same such as 1.jpg, 2.jpg... in each of them.
Basically, I would like to move all image files from photo_subfolder to their photo_folder and all duplicated filenames to be renamed before merging together.
Example:
|parent_folder
| |photo_folder
| | |photo_subfolder1
| | | 1.jpg
| | | 2.jpg
| | | 3.jpg
| | |photo_subfolder2
| | | 1.jpg
| | | 2.jpg
| | | 3.jpg
| | |photo_subfolder3
| | | 1.jpg
| | | 2.jpg
| | | 3.jpg
Expectation:
|parent_folder
| |photo_folder
| | 1_a.jpg
| | 2_a.jpg
| | 3_a.jpg
| | 1_b.jpg
| | 2_b.jpg
| | 3_b.jpg
| | 1_c.jpg
| | 2_c.jpg
| | 3_c.jpg
Note that files names are just an example. Could be anything.
Thank you!
You can replace the / of the subdirectories with another character, e.g. _ , and then cp/mv the original file to the parent directory.
I try to recreate an example of your directory tree here - very simple, but I hope it can be adapted to your case. Note that I am using bash.
#!/bin/bash
bd=parent
mkdir ${bd}
for i in $(seq 3); do
mkdir -p "${bd}/photoset_${i}/subset_${i}"
for j in $(seq 5); do
touch "${bd}/photoset_${i}/${j}.jpg"
touch "${bd}/photoset_${i}/${j}.png"
touch "${bd}/photoset_${i}/subset_${i}/${j}.jpg"
touch "${bd}/photoset_${i}/subset_${i}/${j}.gif"
done
done
Here is the script that will cp the files from the subdirectories to the parent directory. Basically
find all the files recursively in the subdirectories and loop on them
use sed to replace \ with '_' and store this in a variable new_filepath (I also remove the initial parent_, but this is optional)
copy (or move) the old filepath into parent with filename new_filepath
for xtension in jpg png gif; do
while IFS= read -r -d '' filepath; do
new_filepath=$(echo "${filepath}" | sed s#/#_#g)
cp "${filepath}" "${bd}/${new_filepath}"
done < <(find ${bd} -type f -name "*${xtension}" -print0)
done
ls ${bd}
If you want to remove also the additional parent_ from the new_filepath you can replace the new_filepath above with:
new_filepath=$(echo ${filepath} | sed s#/#_#g | sed s/${bd}_//g)
I assumed that you define all the possible extension in the script. Otherwise to find all the extensions in the directory tree you can use the following snippet from a previous answer
find . -type f -name '*.*' | sed 's|.*\.||' | sort -u

How can I get the metadata of dependencies in gradle?

I want to get all information I can get from the Gradle about each dependency I added,
I can get the dependency tree by run
grade dependencies
we get something like this
+--- io.springfox:springfox-swagger2:2.9.2
| +--- io.swagger:swagger-annotations:1.5.20
| +--- io.swagger:swagger-models:1.5.20
| | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.5 -> 2.9.0
| | +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.26
| | \--- io.swagger:swagger-annotations:1.5.20
| +--- io.springfox:springfox-spi:2.9.2 (*)
| +--- io.springfox:springfox-schema:2.9.2
| | +--- io.springfox:springfox-core:2.9.2 (*)
| | \--- io.springfox:springfox-spi:2.9.2 (*)
| +--- io.springfox:springfox-swagger-common:2.9.2
| | +--- io.swagger:swagger-annotations:1.5.20
| | +--- io.swagger:swagger-models:1.5.20 (*)
| | +--- io.springfox:springfox-spi:2.9.2 (*)
| | +--- io.springfox:springfox-schema:2.9.2 (*)
| | +--- io.springfox:springfox-spring-web:2.9.2 (*)
| | +--- com.google.guava:guava:20.0
| | +--- com.fasterxml:classmate:1.4.0
| | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.26
| | +--- org.springframework.plugin:spring-plugin-core:1.2.0.RELEASE (*)
| | \--- org.springframework.plugin:spring-plugin-metadata:1.2.0.RELEASE (*)
| +--- io.springfox:springfox-spring-web:2.9.2 (*)
| +--- com.google.guava:guava:20.0
| +--- com.fasterxml:classmate:1.4.0
| +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.26
| +--- org.springframework.plugin:spring-plugin-core:1.2.0.RELEASE (*)
| +--- org.springframework.plugin:spring-plugin-metadata:1.2.0.RELEASE (*)
| \--- org.mapstruct:mapstruct:1.2.0.Final
But can't get all other information about the dependencies
on how can I achieve this?
Update
I want to get information about the creator, date, homepage of the library, library language, license type or any information that can be provided.
Gradle does not offer APIs for doing this.
If your dependencies use Maven POM file or Ivy XML as their metadata, you can access these files through an ArtifactQuery. There is an example in the documentation to do that.
You will however have to decide which information to extract and how to present it.

bash get dirname from urls.txt

$ cat urls.txt
/var/www/example.com.com/upload/email/email-inliner.html
/var/www/example.com.com/upload/email/email.html
/var/www/example.com.com/upload/email/email2-inliner.html
/var/www/example.com.com/upload/email/email2.html
/var/www/example.com.com/upload/email/AquaTrainingBag.png
/var/www/example.com.com/upload/email/fitex/fitex-ecr7.jpg
/var/www/example.com.com/upload/email/fitex/fitex-ect7.jpg
/var/www/example.com.com/upload/email/fitex/fitex-ecu7.jpg
/var/www/example.com.com/upload/email/fitex/fitex.html
/var/www/example.com.com/upload/email/fitex/logo.png
/var/www/example.com.com/upload/email/fitex/form.html
/var/www/example.com.com/upload/email/fitex/fitex.txt
/var/www/example.com.com/upload/email/bigsale.html
/var/www/example.com.com/upload/email/logo.png
/var/www/example.com.com/upload/email/bigsale.png
/var/www/example.com.com/upload/email/bigsale-shop.html
/var/www/example.com.com/upload/email/bigsale.txt
Can anyone help me to get dirname for this?
dirname /var/www/example.com.com/upload/email/sss.png works fine, but what about a list of URLs?
Is it possible to achieve this without the use of any form of a loop (for or while). As the number of URLs can be more than several tens of millions. The best way would be with the help of redirection (tee) to a file
As always when it boils down to things like this, Awk comes to the rescue:
awk 'BEGIN{FS=OFS="/"}{NF--}1' <file>
Be aware that this is an extremely simplified version of dirname and does not have the complete identical implementation as dirname, but it will work for most cases. A correct version, which covers all cases is:
awk 'BEGIN{FS=OFS="/"}{gsub("/+","/")}
{s=$0~/^\//;NF-=$NF?1:2;$0=$0?$0:(s?"/":".")};1' <file>
The following table shows the difference:
| path | dirname | awk full | awk short |
|------------+---------+----------+-----------|
| . | . | . | |
| / | / | / | |
| foo | . | . | |
| foo/ | . | . | foo |
| foo/bar | foo | foo | foo |
| foo/bar/ | foo | foo | foo/bar |
| /foo | / | / | |
| /foo/ | / | / | /foo |
| /foo/bar | /foo | /foo | /foo |
| /foo/bar/ | /foo | /foo | /foo/bar |
| /foo///bar | /foo | /foo | /foo// |
note: various alternative solutions can be found in Extracting directory name from an absolute path using sed or awk. The solutions of Kent will all work, the solution of Solid Kim just needs a tiny tweak to fix the multiple slashes (and misses upvotes!)

How to check if a folder has any tab delimited file in it?

I am trying to search for all the tab delimited file in one folder, and if found any then I need to transfer all of them to a another folder using bash.
In my code, I am currently trying to find all files, but somehow it is not working.
Here is my code:
>nul 2>nul dir /a-d "folderName\*" && (echo Files exist) || (echo No file found)
Thanks in advance :)
For a simple move (or copy -- replace mv with cp) of files, #tripleee's answer is sufficient. To recursively search for files and run a command on each, find comes in handy.
Example:
find <src> -type f -name '*.tsv' -exec cp {} <dst> \;
Where <src> is the directory to copy from, and <dst> is the directory to copy to. Note that this searches recursively, so any files with duplicate names will cause overwrites. You can pass -i to cp to have it prompt before overwriting:
find <src> -type f -name '*.tsv' -exec cp -i {} <dst> \;
Explained:
find <src> -type f -name '*.tsv' -exec cp -i {} <dst> \;
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^^
| | | | | | | | | | | ||
| | | | | | | | | | | | --- terminator
| | | | | | | | | | | --- escape for terminator
| | | | | | | | | | --- destination directory
| | | | | | | | | --- the path of each file found
| | | | | | | | --- prompt before overwriting
| | | | | | | --- the copy command
| | | | | | --- flag for executing a command (cp in this case)
| | | | | --- pattern of files to match
| | | | --- flag for specifying file name pattern
| | | --- 'f' for a regular file (as opposed to e.g. 'd' for directory)
| | --- flag for specifying the file type
| --- location to search
--- the find command, useful for searching for files
To get a feel for what happens without actually having find run the real command, you can prefix it with echo to just print each command instead of running it:
find <src> -type f -name '*.tsv' -exec echo cp -i {} <dst> \;
Your attempt has very little valid Bash script in it.
mv foldername/*.tsv otherfolder/
There will be an error message if there are no matching files.
"it is not working". That means very little on stackoverflow.
Let's first examine what you've done:
>nul 2>nul dir /a-d "folderName\*"
So, you're doing a dir (most Linux users would use ls, but soit) on
/a-d
everything under folderName
and the output is in the file nul. For debugging purposes, it would be good to see what is in nul (do cat nul). I would bet it is something like:
dir: cannot access '/a-d': No such file or directory
dir: cannot access 'folderName\*': No such file or directory
That means that dir exits with an error. So, echo No file found will be executed.
This means that your output is probably
No file found
Which is exactly as expected.
In your code, you said you want to find all files. That means you want the output of
ls folderName
or
find folderName
if you want to do things recursively. Because find has been explained above by jsageryd, I won't elaborate on that.
If you just want to look in that specific directory, you might do:
if dir folderName/*.tsv > nul 2>nul ; then
echo "Files exist"
else
echo "No file found"
fi
and go from there.

Find references to files, recursively

In a project where XML/JS/Java files can contain references to other such files, I'd like to be able to have a quick overview of what has to be carefully checked, when one file has been updated.
So, it means I need to eventually have a look at all files referencing the modified one, and all files referencing files which refer to the modified one, etc. (recursively on matched files).
For one level, it's quite simple:
grep -E -l -o --include=*.{xml,js,java} -r "$FILE" . | xargs -n 1 basename
But how can I automate that to match (grand-(grand-))parents?
And how can that be, maybe, made more readable? For example, with a tree structure?
For example, if the file that interests me is called modified.js...
show-referring-files-to modified.js
... I could wish such an output:
some-file-with-ref-to-modified.xml
|__ a-file-referring-to-some-file-with-ref-to-modified.js
another-one-with-ref-to-modified.xml
|__ a-file-referring-to-another-one-with-ref-to-modified.js
|__ a-grand-parent-file-having-ref-to-ref-file.xml
|__ another-file-referring-to-another-one-with-ref-to-modified.js
or any other output (even flat) which allows for quickly checking which files are potentially impacted by a change.
UPDATE -- Results of current proposed answer:
ahmsff.js
|__ahmsff.xml
| |__ahmsd.js
| | |__ahmsd.xml
| | | |__ahmst.xml
| | | | |__BESH.java
| |__ahru.js
| | |__ahru.xml
| | | |__ahrut.xml
| | | | |__ashrba.js
| | | | | |__ashrba.xml
| | | | | | |__STR.java
| | |__ahrufrp.xml
| | | |__ahru.js
| | | | |__ahru.xml
| | | | | |__ahrut.xml
| | | | | | |__ashrba.js
| | | | | | | |__ashrba.xml
| | | | | | | | |__STR.java
| | | | |__ahrufrp.xml
| | | | | |__ahru.js
| | | | | | |__ahru.xml
| | | | | | | |__ahrut.xml
| | | | | | | | |__ashrba.js
| | | | | | | | | |__ashrba.xml
| | | | | | | | | | |__STR.java
| | | | | | |__ahrufrp.xml
(...)
I'd use a shell function (for the recursion) inside an shell script:
Assuming the filenames are unique have no characters that need escaping in them:
File: /usr/local/bin/show-referring-files-to
#!/bin/sh
get_references() {
grep -F -l --include=*.{xml,js,java} -r "$1" . | grep -v "$3" | while read -r subfile; do
#read each line of the grep result into the variable subfile
subfile="$(basename "$subfile")"
echo "$2""$subfile"
get_references "$subfile" ' '"$2" "$3"'\|'"$subfile"
done
}
while test $# -gt 0; do
#loop so more than one file can be given as argument to this script
echo "$1"
get_references "$1" '|__' "$1"
shift
done
There still are lots of performance enhancements possible.
Edit: Added $3 to prevent infinite-loop.

Resources