My goal is to automate the following process: search within all second-level sub-directories, and for all files called "Test.pptx" in said sub-directories, rename to "Test - Appended.pptx". Based on responses I have seen to other questions on StackOverflow I have attempted the following code:
for D in *; do
if [ -d "${D}" ]; then
echo "${D}"
for E in "${D}"; do
if [ -d "${E} ]; then
echo "${E}"
for f in "Test.pptx"; do mv "$f" "Test - Appended.pptx"; done
fi
done
fi
done
I set the script executable (using chmod +x) and run it, but get the following errors:
line 7: unexpected EOF while looking for matching `"'
line 12: syntax error: unexpected end of file
I am a relative newcomer to Bash scripts, so I would appreciate any help in diagnosing the errors and achieving the initial goal. Thank you!
use find:
while read -r pptx
do
mv -n "${pptx}" "${pptx%.pptx} - Appended.pptx"
done < <( find . -mindepth 3 -maxdepth 3 -type f -name "*.pptx" )
Be aware, that I did not test it and it might need adaptions on your special case.
As long as the -n option is set in mv it will not overwrite anything.
sub-loops not really needed.
for f in */*/Test.pptx; do mv "$f" "${f%/*}/Test - Appended.pptx"; done
${f%/*} is the current file's full path with everything from the last slash (/) forward stripped off, so if the file is a/b/Test.pptx then ${f%/*} is a/b.
Related
I'm trying to create a bash script that finds all the files in my dotfiles directory, and symlinks them into ~. If the directory a file is in does not exist, it shall be created.
The script as it is now is just trying to find files and create the two paths, when that works "real" and "symlink" will be used with ln -s. However when trying to save the strings in "real" and "symlink" all i get is line 12: ./.zshrc: Permission denied
What am I doing wrong?
#!/bin/bash
dotfiles=()
readarray -d '' dotfiles < <(find . -type f ! -path '*/.git/*' ! -name "README.md" -type f -print0)
for file in ${dotfiles[#]}; do
dir=$(dirname $file | sed s#.#$HOME#1)
[ ! -d $dir ] && echo "directory $dir not exist!" && mkdir -p $dir
# Create path strings
real=$("$file" | sed s#.#$PWD#1)
symlink=$("$file" | sed s#.#$HOME#1)
echo "Real path: $cur"
echo "Symbolic link path: $new"
done
exit
P.S, I'm a bash noob and am mostly doing this script as a learning experience.
Here is a refactoring which attempts to fix the obvious problems. See comments with # ... for details. Probably also check with http://shellcheck.net/ before you ask for human assistance.
The immediate reason for the error message is that $("$file" ...) does indeed attempt to run $file as a command and capture its output. You probably meant $(echo "$file" ...) but that can often be avoided.
#!/bin/bash
dotfiles=()
readarray -d '' dotfiles < <(find . -type f ! -path '*/.git/*' ! -name "README.md" -type f -print0)
# ... Fix broken quoting throughout
for file in "${dotfiles[#]}"; do
dir=$(dirname "$file" | sed "s#^\.#$HOME#") # ... 1 is not required or useful
# ... Use standard error for diagnostics
[ ! -d "$dir" ] && echo "$0: directory $dir does not exist!" >&2 && mkdir -p "$dir"
# Create path strings
# ... Use parameter substitution
real=$PWD/${file#./}
symlink=$HOME/${$file#./}
# ... You mean $real, not $cur?
echo "Real path: $real"
# ... You mean $symlink, not $new?
echo "Symbolic link path: $symlink"
done
# ... No need to explicitly exit; trust me, you will anyway
#exit
These are just syntactic fixes; I would probably avoid storing the results from find in an array and just loop over them directly, and I haven't checked if the logic actually does what (we might guess) you are perhaps trying to do.
See also Looping over pairs of values in bash for a similar topic, and When to wrap quotes around a shell variable?.
The script still has a pesky assumption that it will be run in the dotfiles directory. Probably a better design would be to explicitly run find in that directory, and refactor accordingly.
sed 's/x/y/' will replace the first occurrence of x with y on each line by definition and by design; there is no reason or need to explicitly add 1 after the final delimiter. (Some sed variants allow a number there to select a different match than the first, but this is not portable; and of course specifying the first match when that's already the default is simply silly. There is a g flag to replace all matches instead of just the first which many beginners want to use everywhere, but of course that's not necessary or useful here either.)
Within a certain directory I have many directories containing a bunch of text files. I’m trying to write a script that concatenates only those files in each directory that have the string ‘R1’ in their filename into one file within that specific directory, and those that have ‘R2’ in another . This is what I wrote but it’s not working.
#!/bin/bash
for f in */*.fastq; do
if grep 'R1' $f ; then
cat "$f" >> R1.fastq
fi
if grep 'R2' $f ; then
cat "$f" >> R2.fastq
fi
done
I get no errors and the files are created as intended but they are empty files. Can anyone tell me what I’m doing wrong?
Thank you all for the fast and detailed responses! I think I wasn't very clear in my question, but I need the script to only concatenate the files within each specific directory so that each directory has a new file ( R1 and R2). I tried doing
cat /*R1*.fastq >*/R1.fastq
but it gave me an ambiguous redirect error. I also tried Charles Duffy's for loop but looping through the directories and doing a nested loop to run though each file within a directory like so
for f in */; do
for d in "$f"/*.fastq;do
case "$d" in
*R1*) cat "$d" >&3
*R2*) cat "$d" >&4
esac
done 3>R1.fastq 4>R2.fastq
done
but it was giving an unexpected token error regarding ')'.
Sorry in advance if I'm missing something elementary, I'm still very new to bash.
A Note To The Reader
Please review edit history on the question in considering this answer; several parts have been made less relevant by question edits.
One cat Per Output File
For the purpose at hand, you can probably just let shell globbing do all the work (if R1 or R2 will be in the filenames, as opposed to the directory names):
set -x # log what's happening!
cat */*R1*.fastq >R1.fastq
cat */*R2*.fastq >R2.fastq
One find Per Output File
If it's a really large number of files, by contrast, you might need find:
find . -mindepth 2 -maxdepth 2 -type f -name '*R1*.fastq' -exec cat '{}' + >R1.fastq
find . -mindepth 2 -maxdepth 2 -type f -name '*R2*.fastq' -exec cat '{}' + >R2.fastq
...this is because of the OS-dependent limit on command-line length; the find command given above will put as many arguments onto each cat command as possible for efficiency, but will still split them up into multiple invocations where otherwise the limit would be exceeded.
Iterate-And-Test
If you really do want to iterate over everything, and then test the names, consider a case statement for the job, which is much more efficient than using grep to check just one line:
for f in */*.fastq; do
case $f in
*R1*) cat "$f" >&3
*R2*) cat "$f" >&4
esac
done 3>R1.fastq 4>R2.fastq
Note the use of file descriptors 3 and 4 to write to R1.fastq and R2.fastq respectively -- that way we're only opening the output files once (and thus truncating them exactly once) when the for loop starts, and reusing those file descriptors rather than re-opening the output files at the beginning of each cat. (That said, running cat once per file -- which find -exec {} + avoids -- is probably more overhead on balance).
Operating Per-Directory
All of the above can be updated to work on a per-directory basis quite trivially. For example:
for d in */; do
find "$d" -name R1.fastq -prune -o -name '*R1*.fastq' -exec cat '{}' + >"$d/R1.fastq"
find "$d" -name R2.fastq -prune -o -name '*R2*.fastq' -exec cat '{}' + >"$d/R2.fastq"
done
There are only two significant changes:
We're no longer specifying -mindepth, to ensure that our input files only come from subdirectories.
We're excluding R1.fastq and R2.fastq from our input files, so we never try to use the same file as both input and output. This is a consequence of the prior change: Previously, our output files couldn't be considered as input because they didn't meet the minimum depth.
Your grep is searching the file contents instead of file name. You could rewrite it this way:
for f in */*.fastq; do
[[ -f $f ]] || continue
if [[ $f = *R1* ]]; then
cat "$f" >> R1.fastq
elif [[ $f = *R2* ]]; then
cat "$f" >> R2.fastq
fi
done
Find in a forloop might suit this:
for i in R1 R2
do
find . -type f -name "*${i}*" -exec cat '{}' + >"$i.txt"
done
I'm working on a unix script that has 2 input parameters - path and size.
The script will check all the files in the given path with the given size and deletes them. If the delete operation fails, the respective file-name is recorded into a file. For any other case, the file is rendered without any action.
I have written a short code (don't know whether it works).
find $path -type f -size +${byte_size}c -print | xargs -I {}
if $?=1;
then
rm -rf {};
else
echo {} >> Error_log_list.txt'
where
$path is the path where we search for the files.
size is the input size.
Error_log_list.txt is the file where we send the non-deletable filenames.
Can anyone please help me verify whether it is correct?
GNU find has a -delete option for this exact use case. More information (and a number of different approaches) in the find documentation.
find $path -type f -size +${byte_size}c -delete
Executing your script results in the following syntax error:
./test.sh: line 9: unexpected EOF while looking for matching `''
./test.sh: line 11: syntax error: unexpected end of file
Moreover the condition of the if statement seems not correct.
If I am not wrong it tests the return code of the "rm" command before to
execute the command.
I am not familiar with xargs and I tried to rewrite your script
using a while loop construct. Here my script
#!/bin/bash
path=$1
byte_size=$2
find $path -type f -size +${byte_size}c -print | while read file_name
do
rm -f $file_name
if [ ! $? -eq 0 ]; then
echo $file_name >> Error_log_list.txt
fi
done
I tested it trying to delete files without the right permission and it works.
I wrote a script, please check this functionality
a=`find . -type f -size +{$size}c -print`
#check if $a is empty
if [ -z "$a" ]
then
echo $a > error_log.txt
#if a is not empty then remove them
else
rm $a
fi
Let me explain what we are doing here.
First assigning the file_names in current directory (which satisfy
size requirement) to a variable 'a'
Checking if that variable is
empty (empty means there is no file with your size requirement) if a
has some values then delete them
I used this command to find all the directories containing .mp3 in the current directory, and filtered out only the directory names:
find . -iname "*.mp3" | sed -e 's!/[^/]*$!!' -e 's!^\./!!' | sort -u
I now want the opposite, but I found it a little harder. I can't just add a '!' to the find command since it'll only exclude .mp3 when printing them not find directories that do not contain .mp3.
I googled this and searched on stackoverflow and unix.stackexchange.com.
I have tried this script so far and it returns this error:
#!/bin/bash
find . -type d | while read dir
do
if [[! -f $dir/*.mp3 ]]
then
echo $dir
fi
done
/home/user/bin/try.sh: line 5: [[!: command not found
#!/bin/bash
find . -type d | while read dir
do
if [! -f $dir/*.mp3 ]
then
echo $dir
fi
done
/home/user/bin/try.sh: line 5: [!: command not found
#!/bin/bash
find . -type d | while read dir
do
if [[! -f "$dir/*.mp3" ]]
then
echo $dir
fi
done
/home/user/bin/try.sh: line 5: [!: command not found
I'm thinking it has to do with multiple arguments for the test command.
Since I'm testing all the directories the variable is going to change, and I use a wildcard for the filenames.
Any help is much appreciated. Thank You.
[ "$(echo $dir/*.mp3)" = "$dir/*.mp3" ]
should work.
Or simply add a space between '[' and '!'
A method that is probably significantly faster is
if find "$dir" -name '*.mp3' -quit ; then
: # there are mp3-files in there.
else
; # no mp3:s
fi
Okay, I solved my own answer by using a counter.
I don't know how efficient it is, but it works. I know it can be made better. Please feel free to critique.
find . -type d | while read dir
do
count=`ls -1 "$dir"/*.mp3 2>/dev/null | wc -l`
if [ $count = 0 ]
then
echo $dir
fi
done
This prints all directories not containing MP3s It also shows sub-directories thanks to the find command printing directories recursively.
I ran a script to automatically download cover art for my mp3 collection. It put a file called "cover.jpg" in the directory for each album for which it could retrieve the art. I needed to check for which albums the script had failed - i.e. which CDs (directories) did not contain a file called cover.jpg. This was my effort:
find . -maxdepth 1 -mindepth 1 -type d | while read dir; do [[ ! -f $dir/cover.jpg ]] && echo "$dir has no cover art"; done
The maxdepth 1 stops the find command from descending into a hidden directory which my WD My Cloud NAS server had created for each album and placed a default generic disc image. (This got cleared during the next scan.)
Edit: cd to the MP3 directory and run it from there, or change the . in the command above to the path to point to it.
Have documents stored in a file system which includes "daily" directories, e.g. 20050610. In a bash script I want to list the files in a months worth of these directories. So I'm running a find command find <path>/200506* -type f >> jun2005.lst. Would like to check that this set of directories is not a null set before executing the find command. However, if I use if[ -d 200506* ] I get a "too many arguements error. How can I get around this?
Your "too many arguments" error does not come from there being a huge number of files and exceeding the command line argument limit. It comes from having more than one or two directories that match the glob. Your glob "200506*" expands to something like "20050601 20050602 20050603..." and the -d test only expects one argument.
$ mkdir test
$ cd test
$ mkdir a1
$ [ -d a* ] # no error
$ mkdir a2
$ [ -d a* ]
-bash: [: a1: binary operator expected
$ mkdir a3
$ [ -d a* ]
-bash: [: too many arguments
The answer by zed_0xff is on the right track, but I'd use a different approach:
shopt -s nullglob
path='/path/to/dirs'
glob='200506*/'
outfile='jun2005.lst'
dirs=("$path"/$glob) # dirs is an array available to be iterated over if needed
if (( ${#dirs[#]} > 0 ))
then
echo "directories found"
# append may not be necessary here
find "$path"/$glob -type f >> "$outfile"
fi
The position of the quotes in "$path"/$glob versus "$path/$glob" is essential to this working.
Edit:
Corrections made to exclude files that match the glob (so only directories are included) and to handle the very unusual case of a directory named literally like the glob ("200506*").
prefix="/tmp/path"
glob="200611*"
n_dirs=$(find $prefix -maxdepth 1 -type d -wholename "$prefix/$glob" |wc -l)
if [[ $n_dirs -gt 0 ]];then
find $prefix -maxdepth 2 -type f -wholename "$prefix/$glob"
fi
S=200506*
if [ ${#S} -gt 6 ]; then
echo haz filez!
else
echo no filez
fi
not a very elegant one, but w/o any external tools/commands (if don't think of "[" as an external one)
the clue is if there is some files matched, then "S" variable will contain their names delimited with space. Otherwise it will contain a "200506*" string itself.
You could us ls like this:
if [ -n "$(ls -d | grep 200506)" ]; then
# There are directories with this pattern
fi
Because there is a limit on command line length in most shells: anything like "$(ls -d | grep 200506)" or /path/200506* will run the risk of overflowing the limit. I'm not sure if substitutions and glob expansions count towards it in BASH, but I assume so. You would have to test it and check the bash docs and source to be sure.
The answer is in simplifying your question.
find <path>/200506* -type f -exec somescript '{}' \;
Where somescript is a shell script that does the test. Something like this perhaps:
#!/bin/sh
[ -d "$#" ] && echo "$#" >> june2005.lst
Passing the june2005.lst to the script (advice: use an environment variable), and dealing with any possibility that 200506* may expand to tooo huge a file path, being left as an exercise for the OP ;)
Integrating the whole thing into a pipe line or adapting a more general scripting language would yield performance boosts, by minimizing the number of shells spawned. Now that would be fun. Here is a hint for that, use -exec and another program (awk, perl, etc) to do the directory test as part of a one line filter, and keep the >>june2005.lst on the find command.