Iterate over files in a subfolder - bash

new here, learning bash for first time.
I'm trying to iterate over files named "list.txt" placed in subfolders, manipulate and create a new files, under the same subfolder. The nest could be like this:
inventory/product_names1/list.txt
inventory/product_names2/list.txt
As product_names is completly random, I would like to iterate over all list.txt files with unix cms like sed/grep/cut and create a new file, under the same random product_names folders.
for f in $( find . -name 'list.txt'); do for list in $f; do cat $f | cut -d']' -f2- > "$f/new_file.txt" ; done ; done
I can access files into the nest using find command. How can I redirect output in the right subfolder if the product_names is random?
inventory/product_names1/list.txt
inventory/product_names1/new_file.txt
inventory/product_names2/list.txt
inventory/product_names2/new_file.txt
This script is intended to work in the root folder, pointing and working with entime path "inventory". $f access to inventory/product_names1/list.txt but I need the output in inventory/product_names1. How can I redirect correctly if I don't have the right value/variable?

You can either use parameter expansion to remove the file name from the path, or you can iterate over all the directories and only work on them if they contain the list.txt file.
#!/bin/bash
for list in inventory/*/list.txt ; do
new=${list%/*}/new_list.txt
echo "$list" "$new"
done
# OR
for dir in inventory/* ; do
if [[ -f $dir/list.txt ]] ; then
echo "$dir"/list.txt "$dir"/new_list.txt
fi
done

find can not only find files but also execute commands when a file is found:
find . -type f -name 'list.txt' -execdir sh -c 'cut -d"]" -f2 list.txt > new_file.txt' \;
Explanations:
-type f condition added to skip directories named list.txt. If some of your list.txt files can be symbolic links and you want to consider them too, use -type f,l with GNU find. With other find you may need to use \(-type f -o -type l\).
-execdir runs the command in the directory where the file was found.
By default find does not print when -execdir is used. If you need it add the -print command:
find . -type f -name 'list.txt' -execdir sh -c 'cut -d"]" -f2 list.txt > new_file.txt' \; -print

Related

use command find and copy but using a list.txt with names

I have a list.txt with different filenames and I want to find all those 3600 filename in subdirectories and then copy to /destination_folder.
Can I use the command find /path/ {file.txt} then copy to /destination_folder ?
The list.txt should have the following filenames/lines:
test_20180724004008_4270.txt.bz2
test_20180724020008_4278.txt.bz2
test_20180724034009_4288.txt.bz2
test_20180724060009_4302.txt.bz2
test_20180724061009_4303.txt.bz2
test_20180724062010_4304.txt.bz2
test_20180724063010_4305.txt.bz2
test_20180724065010_4307.txt.bz2
test_20180724070010_4308.txt.bz2
test_20180724071010_4309.txt.bz2
test_20180724072010_4310.txt.bz2
test_20180724072815_4311.txt.bz2
test_20180724073507_4312.txt.bz2
test_20180724074608_4314.txt.bz2
test_20180724075041_4315.txt.bz2
test_20180724075450_4316.txt.bz2
test_20180724075843_4317.txt.bz2
test_20180724075843_4317.txt.bz2
test_20180724080207_4318.txt.bz2
test_20180724080522_4319.txt.bz2
test_20180724080826_4320.txt.bz2
test_20180724081121_4321.txt.bz2
................................
You will probably want to make a list of all of the files in a directory, then use your list to iterate through the list of files found.
First save your list of files found to a file
find . -type f > foundFiles.txt
Then you need to use your file to search the other
cat list.txt | while read line
do
if [ `grep -c "${line}" foundFiles.txt` ]
then
cp -v $(grep "${line}" foundFiles.txt) /destination_folder/
fi
done
I'll let you take the base and make it into a script for use again.
You could use echo and sed:
echo $(sed "s/.*/\"\0\"/;s/^/ -name /;s/$/ -o/;$ s/-o//" list.txt)
This outputs a list of files to be used in find command:
-name "file1.txt.bz2" -o -name "file2.txt.bz2" -o -name "file3.txt.bz2"
Then use -exec cp -t targetDir {} + in find to copy the files:
find \( $(eval echo $(sed "s/.*/\"\0\"/;s/^/ -name /;s/$/ -o/;$ s/-o//" list.txt)) \) -exec cp -t targetDir {} +
Loop through the file and append the results to your destination folder:
for i in `cat list.txt`;
do cp `find * -name $i` destination_folder/;
done
This finds all the files in list.txt and copies those files to destination_folder/.
The for i in `cat list.txt` creates a variable i that loops through the entire file.
The cp `find * -name $i` destination_folder/ finds the path to file and copies it to the destination_folder/.

Rename files in several subdirectories

I want to rename a file present in several subdirectories using bash script.
my files are in folders:
./FolderA/ABCD/ABCD_Something.ctl
./FolderA/EFGH/EFGH_Something.ctl
./FolderA/WXYZ/WXYZ_Something.ctl
I want to rename all of the .ctl file with the same name (name.ctl).
I tried several command using mv or rename but didnt work.
Working from FolderA:
find . -name '*.ctl' -exec rename *.ctl name.ctl '{}' \;
or
for f in ./*/*.ctl; do mv "$f" "${f/*.ctl/name .ctl}"; done
or
for f in $(find . -type f -name '*.ctl'); do mv $f $(echo "$f" | sed 's/*.ctl/name.ctl/'); done
Can you help me using bash?
thanks
You can do this with one line with:
find . -name *.ctl -exec sh -c 'mv "$1" `dirname "$1"`/name.ctl' x {} \;
The x just allows the filename to be positional character 1 rather than 0 which (in my opinion) wrong to use as a parameter.
Try this:
find . -name '*.ctl' | while read f; do
dn=$(dirname "${f}")
# remove the echo after you sanity check the output
echo mv "${f}" "${dn}/name.ctl"
done
find should get all the files you want, dirname will get just the directory name, and mv will perform the rename. You can remove the quotes if you're sure that you'll never have spaces in the names.

Bash: List directories with a type of file, but missing another type of file

I'm new(ish) to using Bash and I'm trying to figure out how to combine a few different things into one script.
I'm looking for file transfers that were interrupted. These folders contain image files (either jpgs or pngs), but are missing another specific file (finished.txt).
Here is what I'm using to find folders with images (from here):
for f in */incoming/ ; do
log_f="${f//\//}"
echo "searching $f"
find "$f" -iname "*jpg*" -o -iname "*png*" > "/output/${log_f}.txt"
echo "$f finished"
done
Then, I'm running this command to find folders that are missing the finished.txt file (from here):
find -mindepth 2 -maxdepth 2 -type d '!' -exec test -e "{}/finished.txt" ';' -print
Is there a way to combine them so I have a list of folders which have jpg or png files, but don't have finished.txt? Also, If I want to add -mtime, where do I put that?
Alternatively, if there's a better/faster way to do this, I'm interested in that too.
Thanks!
From the first pass when you get the files with jpg/png you can get the directory by using dirname. The list of directories can then be used for iterating over and looking for finished.txt file. On finding you can skip the directory if not print it out.
Something as below should do the needful
for i in `find "$f" -iname "*jpg*" -o -iname "*png*" -exec dirname {} \;`
do
ls $i | grep finished >/dev/null
if [ $? -eq 1 ]; then
echo $i
fi
done
Add " | sort | uniq" at the end of find command to perhaps remove the duplicates. Something like
find "$f" -iname "jpg" -o -iname "png" -exec dirname {} \; | sort | uniq

Bash: Look for files in certain folders and then create output lists

I'm new to Bash, and I'm trying to find files in a certain set of folders. I want to create a txt report for image files in each /check/ folder.
Here's what I've been working with:
# Find images
for f in */check/ ; do
find ./ -iname "*jpg*" -o -iname "*png*" > find_images.txt
echo "finished $f"
done
I can't figure out how to only look at subfolders named "check", and I also want to pass the variable so that I get separate text files named after the parent folders. Any suggestions?
You're close, but you're not using $f which contains the folder's name:
# Find images
for f in */check/ ; do
# Removing front-slashes from $f to use in log name
# http://mywiki.wooledge.org/BashGuide/Parameters#Parameter_Expansion
log_f="${f//\//_}"
# Only search inside $f, saving results to find_images_[foldername].txt
find "$f" -iname "*jpg*" -o -iname "*png*" > "find_images_${log_f}.txt"
echo "finished $f"
done
Use grep command and pipe it with find command
find . | grep check
The find command supports searching for directories (folders), e.g.
find . -name "check" -type d
You could use these results to then look for the files you want. The variable $f will be the name of the folder, so use that in the inner find command. Then if you want separate output files each time through the loop, use a variable in the filename. The $f variable will have slashes in the content, so you probably don't want to use that in the name of your output file. In my example, I use a counter to make sure each output file has a unique name.
count=1
for f in `find . -name "check" -type d` ; do
find $f -iname "*jpg*" -o -iname "*png*" > find_images_$count_.txt
count=$((count+1))
done

Doing something to all files in an entire tree

The scenario is that I want to convert all of my music files from .mp3 to .ogg. They are in a folder called "Music". In this folder there are folders and files. The files are .mp3s. The directories may contain .mp3s or directories which further contain .mp3s or directories, and so on. This is because some artists have albums which have parts and some do not, etc.
I want to write a script that converts each file using avconv.
Basically, what I am going to do is manually cd into every directory and run the following:
for file in $(ls); do avconv -i $file `echo \`basename $file .mp3\`.ogg`; done
This successfully gets me what I want. However, this is not great as I have a lot of folders, and manually going into each of them and executing this is slow.
My question, then, is how do I write a script that runs this in any directory that has .mp3s, and then goes into any subdirectory it finds and recursively calls itself? My intuition tells me to use Perl or Python because of the complex nature of this.
Thanks for any suggestions!
I'm not familiar with avconv but assuming your command is:
avconv -i inputname outputname
And you want to convert all inputname.mp3 to inputname.ogg in their original directories below Music, then the following should work in bash:
#!/bin/bash
while read -r fname; do
avconv -i "$fname" "${fname%.mp3}.ogg"
done < <(find /path/to/Music -type f -name "*.mp3")
Note: this does not remove the original .mp3, and the space between < < is required. Also note, for file in $(ls) is filled with potential for errors.
You can do it with bash in one liner:
First you find all files (of type file (-type f) ) that match next pattern "*.mp3". To read each one you use 'while' and invoke avconf.
For exchange extension I prefer 'sed' command, that keep folder so you don't need the 'cd' command.
Notice that you must put quotes on $FN variable because it can contain spaces.
find -type f -iname "*.mp3" | while read "FN" ; do avconf -i "$FN" $(echo "$FN" | sed 's/\.mp3/\.ogg/g') ; done
find <music-folder> -type f -name '*.mp3' | \
xargs -I{} bash -c 'mp3="$0"; ogg="${mp3%.mp3}.ogg"; avconv -i "$mp3" "$ogg";' {}
This should survive in cases of "weird" filenames with spaces, quotes and other strange symbols within.
You can list directories with absolute paths and recursively cd into every directory using find $PWD -type d syntax:
Just inside from Music directory run:
for d in $(find $PWD -type d)
do
cd $d
for file in $(find . -maxdepth 1 -type f)
do
echo $file
avconv -i $file `echo \`basename $file .mp3\`.ogg`
done
done

Resources