I'm trying to find multiple patterns (I have a file of them) in multiple differents files with a lot of subdirs.
I'm trying to use exit codes for not outputting all patterns found (because I need only the ones which are NOT found), but exit codes doesn't work as I understand them.
while read pattern; do
grep -q -n -r $pattern ./dir/
if [ $? -eq 0 ]; then
: #echo $pattern ' exists'
else
echo $pattern " doesn't exist"
fi
done <strings.tmp
You can use this in bash:
while read -r pattern; do
grep -F -q -r "$pattern" ./dir/ || echo $pattern " doesn't exist"
done < strings.tmp
Use read -r to safely read regex patterns
Use quoting in "$pattern" to avoid shell escaping
No need to use -n since you're using -q (quiet) flag
#anubhava's solution should work. If it doesn't for some reason, try the following
while read -r pattern; do
lines=`grep -q -r "$pattern" ./dir/ | wc -l`
if [ $lines -eq 0 ]; then
echo $pattern " doesn't exist"
else
echo $pattern "exists"
fi
done < strings.tmp
Related
I have a list of folders and files whose names contain spaces. How can I change the names into camel case?
for oldname in *
do
newname=`echo $oldname | sed -e 's/ /_/g'`
if [ "$newname" = "$oldname" ]
then
continue
fi
if [ -e "$newname" ]
then
echo Skipping "$oldname", because "$newname" exists
else
mv "$oldname" "$newname"
fi
done
I have found this but it changes the spaces into underscores.
Try this Shellcheck-clean Bash code:
#! /bin/bash -p
lowers=abcdefghijklmnopqrstuvwxyz
uppers=ABCDEFGHIJKLMNOPQRSTUVWXYZ
for oldname in *; do
[[ $oldname == *[[:space:]]* ]] || continue
read -r -d '' -a parts <<<"$oldname"
newname=''
for p in "${parts[#]}"; do
char1=${p:0:1}
if [[ $lowers == *"$char1"* ]]; then
tmp=${lowers%"$char1"*}
uchar1=${uppers:${#tmp}:1}
newname+=${uchar1}${p:1}
else
newname+=$p
fi
done
if [[ -e $newname ]]; then
printf "Skipping '%s', because '%s' exists\\n" "$oldname" "$newname" >&2
else
echo mv -v -- "$oldname" "$newname"
fi
done
The code is intended to work with (the now ancient) Bash 3 because my understanding is that that is still the current version of the standard Bash on macOS. The code for uppercasing the first letter of filename parts is much more complicated than it would be with later versions of Bash (which have built-in mechanisms for case conversion). See How to convert a string to lower case in Bash? for information about changing case in various ways in various versions of Bash.
The code just prints the mv command that would be run. Remove the echo to make it actually do the mv.
See the accepted, and excellent, answer to Why is printf better than echo? for an explanation of why I replaced echo with printf for the "Skipping" message.
For comparison, this is Bash 4+ code:
#! /bin/bash -p
for oldname in *; do
[[ $oldname == *[[:space:]]* ]] || continue
read -r -d '' -a parts <<<"$oldname"
newname=''
for p in "${parts[#]}"; do
newname+=${p^}
done
if [[ -e $newname ]]; then
printf "Skipping '%s', because '%s' exists\\n" "$oldname" "$newname" >&2
else
echo mv -v -- "$oldname" "$newname"
fi
done
You can use the regular expression aptitude to deal with upper and lower case translations, regarding your current local collation (LC_ALL, check with the locale command).
If your filename's "words" are separated with a space and are all in lower case, you can use a simple shell script like this :
#!/bin/sh
while read -r FILENAME ; do
NEWNAME="`echo \"${FILENAME}\" | sed 's/ *\([^ ]\)/\u\1/g'`"
if [ ! "${NEWNAME}" ] ; then
NEWNAME="${FILENAME}";
fi
if [ "${FILENAME}" = "${NEWNAME}" ]; then
printf "No change : %s\\n" "${FILENAME}" >&2;
else
if [ -e "${NEWNAME}" ] ; then
printf "Already changed : %s => %s\\n" "${FILENAME}" "${NEWNAME}" >&2;
else
echo "mv \"${FILENAME}\" \"${NEWNAME}\"";
fi
fi
done
Remove the echo on echo "mv \"${FILENAME}\" \"${NEWNAME}\""; to do the mv.
Note that it should work fine with accented letters or any unicode letter having lower and upper code.
The script takes the file list to operate from stdin, so to use it "as is", you can use something like the following examples :
find . -type 'f' | theScript.sh
For a whole tree of files.
For folders, you'll have to operate them separately. List them and sort them in a descending order.
ls -1 | theScript.sh
For files in the current folder.
If your files may have all or partial upper cases at start and you look to force them entirely to camel case, you can change the line :
NEWNAME="`echo \"${FILENAME}\" | sed 's/ *\([^ ]\)/\u\1/g'`"
With:
NEWNAME="\`echo \"${FILENAME}\" | sed 's/\(.*\)/\l\1/;s/ *\([^ ]\)/\u\1/g'\`"
If you have rename installed, then all you need to do is :
rename 's/ /_/g' *
I am i need to find & remove empty files. The definition of empty files in my use case is a file which has zero lines.
I did try testing the file to see if it's empty However, this behaves strangely as in even though the file is empty it doesn't detect it so.
Hence, the best thing I could write up is the below script which i way too slow given it has to test several hundred thousand files
#!/bin/bash
LOOKUP_DIR="/path/to/source/directory"
cd ${LOOKUP_DIR} || { echo "cd failed"; exit 0; }
for fname in $(realpath */*)
do
if [[ $(wc -l "${fname}" | awk '{print $1}') -eq 0 ]]
then
echo "${fname}" is empty
rm -f "${fname}"
fi
done
Is there a better way to do what I'm after or alternatively, can the above logic be re-written in a way that brings better performance please?
Your script is slow beacuse wc reads every file to the end, which is not needed for your purpose. This might be what you're looking for:
#!/bin/bash
lookup_dir='/path/to/source/directory'
cd "$lookup_dir" || exit
for file in *; do
if [[ -f "$file" && -r "$file" && ! -L "$file" ]]; then
read < "$file" || echo rm -f -- "$file"
fi
done
Drop the echo after making sure it works as intended.
Another version, calling the rm only once, could be:
#!/bin/bash
lookup_dir='/path/to/source/directory'
cd "$lookup_dir" || exit
for file in *; do
if [[ -f "$file" && -r "$file" && ! -L "$file" ]]; then
read < "$file" || files_to_be_deleted+=("$file")
fi
done
rm -f -- "${files_to_be_deleted[#]}"
Explanation:
The core logic is in the line
read < "$file" || rm -f -- "$file"
The read < "$file" command attempts to read a line from the $file. If it succeeds, that is, a line is read, then the rm command on the right-hand side of the || won't be executed (that's how the || works). If it fails then the rm command will be executed. In any case, at most one line will be read. This has great advantage over the wc command because wc would read the whole file.
if ! read < "$file"; then rm -f -- "$file"; fi
could be used instead. The two lines are equivalent.
To check a "$fname" is a file and is empty or not, use [ -s "$fname" ]:
#!/usr/bin/env sh
LOOKUP_DIR="/path/to/source/directory"
for fname in "$LOOKUP_DIR"*/*; do
if ! [ -s "$fname" ]; then
echo "${fname}" is empty
# remove echo when output is what you want
echo rm -f "${fname}"
fi
done
See: help test:
File operators:
...
-s FILE True if file exists and is not empty.
Yet another method
wc -l ~/tmp/* 2>/dev/null | awk '$1 == 0 {print $2}' | xargs echo rm
This will break if any of your files have whitespace in the name.
To work around that, with awk still
wc -l ~/tmp/* 2>/dev/null \
| awk 'sub(/^[[:blank:]]+0[[:blank:]]+/, "")' \
| xargs echo rm
This works because the sub function returns the number of substitutions made, which can be treated as a boolean zero/not-zero condition.
Remove the echo to actually delete the files.
I have a find script that automatically opens a file if just one file is found. The way I currently handle it is doing a word count on the number of lines of the search results. Is there an easier way to do this?
if [ "$( cat "$temp" | wc -l | xargs echo )" == "1" ]; then
edit `cat "$temp"`
fi
EDITED - here is the context of the whole script.
term="$1"
temp=".aafind.txt"
find src sql common -iname "*$term*" | grep -v 'src/.*lib' >> "$temp"
if [ ! -s "$temp" ]; then
echo "ΓΈ - including lib..." 1>&2
find src sql common -iname "*$term*" >> "$temp"
fi
if [ "$( cat "$temp" | wc -l | xargs echo )" == "1" ]; then
# just open it in an editor
edit `cat "$temp"`
else
# format output
term_regex=`echo "$term" | sed "s%\*%[^/]*%g" | sed "s%\?%[^/]%g" `
cat "$temp" | sed -E 's%//+%/%' | grep --color -E -i "$term_regex|$"
fi
rm "$temp"
Unless I'm misunderstanding, the variable $temp contains one or more filenames, one per line, and if there is only one filename it should be edited?
[ $(wc -l <<< "$temp") = "1" ] && edit "$temp"
If $temp is a file containing filenames:
[ $(wc -l < "$temp") = "1" ] && edit "$(cat "$temp")"
Several of the results here will read through an entire file, whereas one can stop and have an answer after one line and one character:
if { IFS='' read -r result && ! read -n 1 _; } <file; then
echo "Exactly one line: $result"
else
echo "Either no valid content at all, or more than one line"
fi
For safely reading from find, if you have GNU find and bash as your shell, replace <file with < <(find ...) in the above. Even better, in that case, is to use NUL-delimited names, such that filenames with newlines (yes, they're legal) don't trip you up:
if { IFS='' read -r -d '' result && ! read -r -d '' -n 1 _; } \
< <(find ... -print0); then
printf 'Exactly one file: %q\n' "$result"
else
echo "Either no results, or more than one"
fi
Well, given that you are storing these results in the file $temp this is a little easier:
[ "$( wc -l < $temp )" -eq 1 ] && edit "$( cat $temp )"
Instead of 'cat $temp' you can do '< $temp', but it might take away some readability if you are not very familiar with redirection 8)
If you want to test whether the file is empty or not, test -s does that.
if [ -s "$temp" ]; then
edit `cat "$temp"`
fi
(A non-empty file by definition contains at least one line. You should find that wc -l agrees.)
If you genuinely want a line count of exactly one, then yes, it can be simplified substantially;
if [ $( wc -l <"$temp" ) = 1 ]; then
edit `cat "$temp"`
fi
You can use arrays:
x=($(find . -type f))
[ "${#x[*]}" -eq 1 ] && echo "just one || echo "many"
But you might have problems in case of filenames with whitespace, etc.
Still, something like this would be a native way
no this is the way, though you're making it over-complicated:
if [ "`wc -l $temp | cut -d' ' -f1`" = "1" ]; then
edit "$temp";
fi
what's complicating it is:
useless use of cat,
unuseful use of xargs
and I'm not sure if you really want the editcat $temp`` which is editing the file at the content of $temp
I've searched here, but still can't find the answer to my globbing problems.
We have files "file.1" through "file.5", and each one should contain the string "completed" if our overnight processing went ok.
I figure it's a good thing to first check that there are some files, then I want to grep them to see if I find 5 "completed" strings. The following innocent approach doesn't work:
FILES="/mydir/file.*"
if [ -f "$FILES" ]; then
COUNT=`grep completed $FILES`
if [ $COUNT -eq 5 ]; then
echo "found 5"
else
echo "no files?"
fi
Thanks for any advice....Lyle
Per http://mywiki.wooledge.org/BashFAQ/004, the best approach to counting files is to use an array (with the nullglob option set):
shopt -s nullglob
files=( /mydir/files.* )
count=${#files[#]}
If you want to collect the names of those files, you can do it like so (assuming GNU grep):
completed_files=()
while IFS='' read -r -d '' filename; do
completed_files+=( "$filename" )
done < <(grep -l -Z completed /dev/null files.*)
(( ${#completed_files[#]} == 5 )) && echo "Exactly 5 files completed"
This approach is somewhat verbose, but guaranteed to work even with highly unusual filenames.
try this:
[[ $(grep -l 'completed' /mydir/file.* | grep -c .) == 5 ]] || echo "Something is wrong"
will print "Something is wrong" if doesn't find 5 completed lines.
Corrected the missing "-l" - the explanation
$ grep -c completed file.*
file.1:1
file.2:1
file.3:0
$ grep -l completed file.*
file.1
file.2
$ grep -l completed file.* | grep -c .
2
$ grep -l completed file.* | wc -l
2
You can do this to prevent globbing:
echo \'$FILES\'
but it seems you have a different problem
I have used the find command for this, but it doesnt return any message when a file is not found.
And I want the search to be recursive and return a message "not found" when a file is not found.
Here's the code I have done so far. Here "input.txt" contains the list of files to be searched.
set `cat input.txt`
echo $#
for i in $#
do
find $HOME -name $i
done
Try this:
listfile=input.txt
exec 3>&1
find | \
grep -f <( sed 's|.*|/&$|' "$listfile" ) | \
tee /dev/fd/3 | \
sed 's|.*/\([^/]*\)$|\1|' | \
grep -v -f - "$listfile" | \
sed 's/$/ Not found/'
exec 3>&-
open file descriptor 3
find the files
see if they're on the list (use sed to
send a copy of the found ones to file descriptor 3
strip off the directory name
get a list of the ones that don't appear
add the "Not found" message
close file descriptor 3
Output looks like:
/path/to/file1
/path/somewhere/file2
foo Not found
bar Not found
No loops necessary.
Whats wrong with using a script. I hope this will do.
#!/bin/bash -f
for i in $#
do
var=`find $HOME -name $i`
if [ -z "$var"]
then
var="File not found"
fi
echo $var
done
You can use the shell builtin 'test' to test the existence of a file. There is also an alternative syntax using square brackets:
if [ -f $a ]; then # Don't forget the semicolon.
echo $a
else
echo 'Not Found'
fi
Here is one way - create a list of all the files to grep against. If your implementation supports
grep -q otherwise use grep [pattern] 2&>1 >/dev/null....
find $HOME -type f |
while read fname
do
echo "$(basename $fname) $fname"
done > /tmp/chk.lis
while read fname
do
grep -q "^$fname" /tmp/chk.lis
[ $? -eq 0 ] && echo "$fname found" || echo "$fname not found"
done < /tmp/chk.lis
All of this is needed because POSIX find does not return an error when a file is not found
perl -nlE'say-f$_?$_:"not found: $_"' file