Line in my script trying to get the output of find to be in the here-string but I keep getting "put {}" with the obvious "{}: No such file or directory"
find "$SOURCE_DIR" -type f -name "*.txt" -exec sshpass -p "$PASSWORD" sftp -oPort=$PORT $USER#$HOST:$HOST_DIR <<< $'put' {} 2>&1 \;
How to I pass the filename into the here-string so that sftp will put the file?
My previous line in the script was this which I had no problems with. However, I can no longer use curl in this script.
find "$SOURCE_DIR" -type f -name "*.txt" -exec curl -T {} sftp://$USER:$PASSWORD#$HOST:$PORT$HOST_DIR 2>&1 \;
find -exec doesn't implicitly start a shell, so it doesn't run shell operations or redirections.
You could make it start a shell (this is discussed in the Complex Actions section of Using Find), but it's just as easy to write a NUL-delimited list of filenames, and read them into your shell:
while IFS= read -r -d '' file <&3; do
sshpass -p "$PASSWORD" sftp -oPort="$PORT" "$USER#$HOST:$HOST_DIR" <<<"put $file"
done 3< <(find "$SOURCE_DIR" -type f -name "*.txt" -print0)
As an additional optimization, think about only running sftp once, not once per file (note that this is using the GNU -printf extension to find):
find "$SOURCE_DIR" -type f -name "*.txt" -printf 'put %p\n' |
sshpass -p "$PASSWORD" sftp -oPort="$PORT" "$USER#$HOST:$HOST_DIR"
Related
I am on macOS and using find . -type f -not -xattrname "com.apple.FinderInfo" -print0 to create a list of files. I want to store that list and be able to pass it to multiple commands in my script. However, I can't use tee because I need them to be sequential and wait for each to complete. The issue I am having is that since print0 uses the null character if I put it into a variable then I can't use it in commands.
To load 0-delimited data into a shell array (Much better than trying to store multiple filenames in a single string):
bash 4.4 or newer:
readarray -t -d $'\0' files < <(find . -type f -not -xattrname "com.apple.FinderInfo" -print0)
some_command "${files[#]}"
other_command "${files[#]}"
Older bash, and zsh:
while read -r -d $'\0' file; do
files+=("$file")
done < <(find . -type f -not -xattrname "com.apple.FinderInfo" -print0)
some_command "${files[#]}"
other_command "${files[#]}"
This is a bit verbose, but works with the default bash 3.2:
eval "$(find ... -print0 | xargs -0 bash -c 'files=( "$#" ); declare -p files' bash)"
Now the files array should exist in your current shell.
You will want to expand the variable with "${files[#]}" including the quotes, to pass the list of files.
I would like to recursively go through all subdirectories and remove the oldest two PDFs in each subfolder named "bak":
Works:
find . -type d -name "bak" \
-exec bash -c "cd '{}' && pwd" \;
Does not work, as the double quotes are already in use:
find . -type d -name "bak" \
-exec bash -c "cd '{}' && rm "$(ls -t *.pdf | tail -2)"" \;
Any solution to the double quote conundrum?
In a double quoted string you can use backslashes to escape other double quotes, e.g.
find ... "rm \"\$(...)\""
If that is too convoluted use variables:
cmd='$(...)'
find ... "rm $cmd"
However, I think your find -exec has more problems than that.
Using {} inside the command string "cd '{}' ..." is risky. If there is a ' inside the file name things will break and might execcute unexpected commands.
$() will be expanded by bash before find even runs. So ls -t *.pdf | tail -2 will only be executed once in the top directory . instead of once for each found directory. rm will (try to) delete the same file for each found directory.
rm "$(ls -t *.pdf | tail -2)" will not work if ls lists more than one file. Because of the quotes both files would be listed in one argument. Therefore, rm would try to delete one file with the name first.pdf\nsecond.pdf.
I'd suggest
cmd='cd "$1" && ls -t *.pdf | tail -n2 | sed "s/./\\\\&/g" | xargs rm'
find . -type d -name bak -exec bash -c "$cmd" -- {} \;
You have a more fundamental problem; because you are using the weaker double quotes around the entire script, the $(...) command substitution will be interpreted by the shell which parses the find command, not by the bash shell you are starting, which will only receive a static string containing the result from the command substitution.
If you switch to single quotes around the script, you get most of it right; but that would still fail if the file name you find contains a double quote (just like your attempt would fail for file names with single quotes). The proper fix is to pass the matching files as command-line arguments to the bash subprocess.
But a better fix still is to use -execdir so that you don't have to pass the directory name to the subshell at all:
find . -type d -name "bak" \
-execdir bash -c 'ls -t *.pdf | tail -2 | xargs -r rm' \;
This could stll fail in funny ways because you are parsing ls which is inherently buggy.
You are explicitely asking for find -exec. Usually I would just concatenate find -exec find -delete but in your case only two files should be deleted. Therefore the only method is running subshell. Socowi already gave nice solution, however if your file names do not contain tabulator or newlines, another workaround is find while read loop.
This will sort files by mtime
find . -type d -iname 'bak' | \
while read -r dir;
do
find "$dir" -maxdepth 1 -type f -iname '*.pdf' -printf "%T+\t%p\n" | \
sort | head -n2 | \
cut -f2- | \
while read -r file;
do
rm "$file";
done;
done;
The above find while read loop as "one-liner"
find . -type d -iname 'bak' | while read -r dir; do find "$dir" -maxdepth 1 -type f -iname '*.pdf' -printf "%T+\t%p\n" | sort | head -n2 | cut -f2- | while read -r file; do rm "$file"; done; done;
find while read loop can also handle NUL terminated file names. However head can not handle this, so I did improve other answers and made it work with nontrivial file names (only GNU + bash)
replace 'realpath' with rm
#!/bin/bash
rm_old () {
find "$1" -maxdepth 1 -type f -iname \*.$2 -printf "%T+\t%p\0" | sort -z | sed -zn 's,\S*\t\(.*\),\1,p' | grep -zim$3 \.$2$ | xargs -0r realpath
}
export -f rm_old
find -type d -iname bak -execdir bash -c 'rm_old "{}" pdf 2' \;
However bash -c might still exploitable, to make it more secure let stat %N do the quoting
#!/bin/bash
rm_old () {
local dir="$1"
# we don't like eval
# eval "dir=$dir"
# this works like eval
dir="${dir#?}"
dir="${dir%?}"
dir="${dir//"'$'\t''"/$'\011'}"
dir="${dir//"'$'\n''"/$'\012'}"
dir="${dir//$'\047'\\$'\047'$'\047'/$'\047'}"
find "$dir" -maxdepth 1 -type f -iname \*.$2 -printf '%T+\t%p\0' | sort -z | sed -zn 's,\S*\t\(.*\),\1,p' | grep -zim$3 \.$2$ | xargs -0r realpath
}
find -type d -iname bak -exec stat -c'%N' {} + | while read -r dir; do rm_old "$dir" pdf 2; done
find . -iname "*.txt" -exec program '{}' \; | sed 's/Value= //'
-"program" returns a different value for each file, and the output is prefixed with "Value= "
In this time the output will be "Value= 128" and the after sed just 128.
How can I take just the value "128" and have the input file be renamed to 128.txt
but also have this find run thought multiple files.
sorry for bad descriptions.
I will try to clear if needed
First write a shell script capable of renaming an argument:
mv "$1" "$(program "$1" | sed "s/Value= //").txt"
Then embed that script in your find command:
find . -iname "*.txt" \
-exec sh -c 'mv "$1" "$(program "$1" | sed "s/Value= //").txt"' _ {} \;
I'd like to run the following bash command for every file in a folder (outputting a unique JSON file for each processed .csv), via a Makefile:
csvtojson ./file/path.csv > ./file/path.json
Here's what I've managed, I'm struggling with the stdin/out syntax and arguments:
find ./ -type f -name "*.csv" -exec csvtojson {} > {}.json \;
Help much appreciated!
You're only passing a single argument to csvtojson -- the filename to run.
The > outputfile isn't an argument at all; instead, it's an instruction to the shell that parses and invokes the relevant command to connect the command's stdout to the given filename before actually starting that command.
Thus, above, that redirection is parsed before the find command is run -- because that's the only place a shell is involved at all.
If you want to involve a shell, consider doing so as follows:
find ./ -type f -name "*.csv" \
-exec sh -c 'for arg; do csvtojson "$arg" >"${arg}.json"; done' _ {} +
...or, as follows:
find ./ -type f -name '*.csv' -print0 |
while IFS= read -r -d '' filename; do
csvtojson "$filename" >"$filename.json"
done
...or, if you want to be able to set shell variables inside the loop and have them persist after its exit, you can use a process substitution to avoid the issues described in BashFAQ #24:
bad=0
good=0
while IFS= read -r -d '' filename; do
if csvtojson "$filename" >"$filename.json"; then
(( ++good ))
else
(( ++bad ))
fi
done < <(find ./ -type f -name '*.csv' -print0)
echo "Converting CSV files to JSON: ${bad} failures, ${good} successes" >&2
See UsingFind, particularly the Complex Actions section and the section on Actions In Bulk.
Is there a shell-independent equivalence of Bash substring replacement:
foo=Hello
echo ${foo/o/a} # will output "Hella"
Most of the time I can use bash so that is not a problem, however when combined with find -exec it does not work. For instance, to rename all .cpp files to .c, I'd like to use:
# does not work
find . -name '*.cpp' -exec mv {} {/.cpp$/.c}
For now, I'm using:
# does work, but longer
while read file; do
mv "$file" "${file/.cpp$/.c}";
done <<< $(find . -name '*.cpp')
Ideally a solution that could be used in scripts is better!
Using find and -exec you can do this:
find . -name '*.cpp' -exec bash -c 'f="$1"; mv "$f" "${f/.cpp/.c}"' - '{}' \;
However this will fork bash -c for each filename so using xargs or a for loop like this is better for performance reasons:
while IFS= read -d '' -r file; do
mv "$file" "${file/.cpp/.c}"
done < <(find . -name '*.cpp' -print0)
Btw, an alternative to using bash would be to use rename. If you have the cool version of the rename command, which is shipped along with perl you can do:
find -name '*.cpp' -exec rename 's/\.cpp$/.c/' {} +
The above example assumes that you have GNU findutils, having this you don't need to pass the current directory since it is the default. If you don't have GNU findutils, you need to explicitly pass it:
find . -name '*.cpp' -exec rename 's/\.cpp$/.c/' {} +