So I would like to do a simple find in a dir with:
find /HOME/ | grep .properties
Then with this list I want to weed out certain files, lets say one is server.properties and another is testing.properties.
After those have been taken out, I want to do a quick for loop that will pass each remaning file that didn't get filtered out into a function one by one. The function call is just something like
extractHash FILE OUTPUTFILE
I hope this makes sense, I'll try to be more clear if it's not.
Thanks
for file in "`find ~ -name \*.properties |grep -v -e server.properties -e testfile.properties`"; do
extractHash $file output
done
Use while, not for, for iterating over files: for will not work as you expect for iterating over the output of a backtick-ed program if there is extraneous whitespace:
find /HOME -name \*.properties \! -name server.propertiees \! -name testing.properties` |
while read -r file; do
extractHash "$file" OUTPUTFILE
done
If all your files are in the current directory, use an extended globbing pattern, and for is appropriate to iterate over filename wildcards:
shopt -s extglob
for file in !(server|testing).properties; do
extractHash "$file" out
done
In csh you would use foreach:
#!/bin/csh
set files=`find /HOME/ | grep .properties`
foreach file ($files)
set outfile = $file.out
extractHash $file $outfile
end
not sure about bash - it has a similar for loop but I never learned it :)
First, I would recommend using the -name argument for find instead of piping every filename through grep. Then you can do something like:
for file in `find /HOME -name \*.properties \! -name server.propertiees \! -name testing.properties`; do
extractHash "$file" OUTPUTFILE
done
Related
I'm writing a script that aim to automate the fulfill of some variables and I'm looking for help to achieve this:
I have a nginx sites-enabled folder which contain some reverses proxied sites.
I need to:
check if a pattern $var1 is found in any of the files in "/volume1/nginx/sites-enabled/"
return the name of the file containing $var1 as $var2
Many thanks for your attention and help!
I have found some lines but none try any files in a folder
if grep -q $var1 "/volume1/nginx/sites-enabled/testfile"; then
echo "found"
fi
find and grep can be used to produce a list of matching files:
find /volume1/nginx/sites-enabled/ -type f -exec grep -le "${var1}" {} +
The ‘trick’ is using find’s -exec and grep’s -l.
If you only want the filenames you could use:
find /volume1/nginx/sites-enabled/ -type f -exec grep -qe "${var1}" {} \; -exec basename {} \;
If you want to assign the result to a variable use command substitution ($()):
var2="$(find …)"
Don’t forget to quote your variables!
This command is the most traditional and efficient one which works on any Unix
without the requirement to have GNU versions of grep with special features.
The efficiency is, that xargs feeds the grep command as many filenames as arguments as it is possible according to the limits of the system (how long a shell command may be) and it excecutes the grep command by this only as least as possible.
With the -l option of grep it shows you only the filename once on a successful pattern search.
find /path -type f -print | xargs grep -l pattern
Assuming you have GNU Grep, this will store all files containing the contents of $var1 in an array $var2.
for file in /volume1/nginx/sites-enabled/*
do
if grep --fixed-strings --quiet "$var1" "$file"
then
var2+=("$file")
fi
done
This will loop through NUL-separated paths:
while IFS= read -d'' -r -u9 path
do
…something with "$path"
done 9< <(grep --fixed-strings --files-without-match --recursive "$var1" /volume1/nginx/sites-enabled)
The scenario is that I want to convert all of my music files from .mp3 to .ogg. They are in a folder called "Music". In this folder there are folders and files. The files are .mp3s. The directories may contain .mp3s or directories which further contain .mp3s or directories, and so on. This is because some artists have albums which have parts and some do not, etc.
I want to write a script that converts each file using avconv.
Basically, what I am going to do is manually cd into every directory and run the following:
for file in $(ls); do avconv -i $file `echo \`basename $file .mp3\`.ogg`; done
This successfully gets me what I want. However, this is not great as I have a lot of folders, and manually going into each of them and executing this is slow.
My question, then, is how do I write a script that runs this in any directory that has .mp3s, and then goes into any subdirectory it finds and recursively calls itself? My intuition tells me to use Perl or Python because of the complex nature of this.
Thanks for any suggestions!
I'm not familiar with avconv but assuming your command is:
avconv -i inputname outputname
And you want to convert all inputname.mp3 to inputname.ogg in their original directories below Music, then the following should work in bash:
#!/bin/bash
while read -r fname; do
avconv -i "$fname" "${fname%.mp3}.ogg"
done < <(find /path/to/Music -type f -name "*.mp3")
Note: this does not remove the original .mp3, and the space between < < is required. Also note, for file in $(ls) is filled with potential for errors.
You can do it with bash in one liner:
First you find all files (of type file (-type f) ) that match next pattern "*.mp3". To read each one you use 'while' and invoke avconf.
For exchange extension I prefer 'sed' command, that keep folder so you don't need the 'cd' command.
Notice that you must put quotes on $FN variable because it can contain spaces.
find -type f -iname "*.mp3" | while read "FN" ; do avconf -i "$FN" $(echo "$FN" | sed 's/\.mp3/\.ogg/g') ; done
find <music-folder> -type f -name '*.mp3' | \
xargs -I{} bash -c 'mp3="$0"; ogg="${mp3%.mp3}.ogg"; avconv -i "$mp3" "$ogg";' {}
This should survive in cases of "weird" filenames with spaces, quotes and other strange symbols within.
You can list directories with absolute paths and recursively cd into every directory using find $PWD -type d syntax:
Just inside from Music directory run:
for d in $(find $PWD -type d)
do
cd $d
for file in $(find . -maxdepth 1 -type f)
do
echo $file
avconv -i $file `echo \`basename $file .mp3\`.ogg`
done
done
I'm having a bit of trouble with globs in Bash. For example:
echo *
This prints out all of the files and folders in the current directory.
e.g. (file1 file2 folder1 folder2)
echo */
This prints out all of the folders with a / after the name.
e.g. (folder1/ folder2/)
How can I glob for just the files?
e.g. (file1 file2)
I know it could be done by parsing ls but also know that it is a bad idea. I tried using extended blobbing but couldn't get that to work either.
WIthout using any external utility you can try for loop with glob support:
for i in *; do [ -f "$i" ] && echo "$i"; done
I don't know if you can solve this with globbing, but you can certainly solve it with find:
find . -type f -maxdepth 1
You can do what you want in bash like this:
shopt -s extglob
echo !(*/)
But note that what this actually does is match "not directory-likes."
It will still match dangling symlinks, symlinks pointing to not-directories, device nodes, fifos, etc.
It won't match symlinks pointing to directories, though.
If you want to iterate over normal files and nothing more, use find -maxdepth 1 -type f.
The safe and robust way to use it goes like this:
find -maxdepth 1 -type f -print0 | while read -d $'\0' file; do
printf "%s\n" "$file"
done
My go to in this scenario is to use the find command. I just had to use it, to find/replace dozens of instances in a given directory. I'm sure there are many other ways of skinning this cat, but the pure for example above, isn't recursive.
for file in $( find path/to/dir -type f -name '*.js' );
do sed -ie 's#FIND#REPLACEMENT#g' "$file";
done
I have a number of clients running a piece of software within their public_html directory. The software includes a file named version.txt that contains the version number of their software (the number and nothing else).
I want to write a bash script that will look for a file named version.txt directly within every user's /home/xxx/public_html/ and output both the path to the file, and the contents of the file, i.e:
/home/matt/public_html/version.txt: 3.4.07
/home/john/public_html/version.txt: 3.4.01
/home/sam/public_html/version.txt: 3.4.03
So far all I have tried is:
#!/bin/bash
for file in 'locate "public_html/version.txt"'
do
echo "$file"
cat $file
done
But that does not work at all.
find /home -type f -path '*public_html/version.txt' -exec echo {} " " `cat {}` \;
Might work for you, but you can go without echo and cat ("tricking" grep):
find /home -type f -path '*public_html/version.txt' -exec grep -H "." {} \;
Or do it using find:
find /home -name "*/public_html/version.txt" -exec grep -H "" {} \;
for i in /home/*/public_html/version.txt; do
echo $i
cat $i
done
will find all the relevant files (using shell wildcarding), echo the filename out and cat out the file.
If you want a more concise output, you should investigate grep and replace the echo/cat with an appropriate regular expression e.g.
grep "[0-9]\.[0-9]" $i
I'm trying to find a bash script that will recursively look for files with a .bx extension, and remove this extension. The filenames are in no particular format (some are hidden files with "." prefix, some have spaces in the name, etc.), and not all files have this extension.
I'm not sure how to find each file with the .bx extension (in and below my cwd) and remove it. Thanks for the help!
find . -name '*.bx' -type f | while read NAME ; do mv "${NAME}" "${NAME%.bx}" ; done
find -name "*.bx" -print0 | xargs -0 rename 's/\.bx//'
Bash 4+
shopt -s globstar
shopt -s nullglob
shopt -s dotglob
for file in **/*.bx
do
mv "$file" "${file%.bx}"
done
Assuming you are in the folder from where you want to do this
find . -name "*.bx" -print0 | xargs -0 rename .bx ""
for blah in *.bx ; do mv ${blah} ${blah%%.bx}
Here is another version which does the following:
Finds out files based on $old_ext variable (right now set to .bx) in and below cwd, stores them in $files
Replaces those files' extension to nothing (or something new depending on $new_ext variable, currently set to .xyz)
The script uses dirname and basename to find out file-path and file-name respectively.
#!/bin/bash
old_ext=".bx"
new_ext=".xyz"
files=$(find ./ -name "*${old_ext}")
for file in $files
do
file_name=$(basename $file $old_ext)
file_path=$(dirname $file)
new_file=${file_path}/${file_name}${new_ext}
#echo "$file --> $new_file"
mv "$file" "$new_file"
done
Extra: How to remove any extension from filenames
find -maxdepth 1 -type f | sed 's/.\///g'| grep -E [.] | while read file; do mv $file ${file%.*}; done
will cut starting from last dot, i.e. pet.cat.dog ---> pet.cat
find -maxdepth 1 -type f | sed 's/.\///g'| grep -E [.] | while read file; do mv $file ${file%%.*}; done
will cut starting from first dot, i.e. pet.cat.dog ---> pet
"-maxdepth 1" limits operation to current directory, "-type f" is used to select files only. Sed & grep combination is used to pick only filenames with dot. Number of percent signs in "mv" command will define actual cut point.