I have a mongodump which I want to import apparently I'm looking to do this using the find command. Something like this:
find *.bson -type f -exec echo mongoimport --db=abc --collection=$(echo '{}' | sed s/.bson//g) {} \;
What I'm looking isn't get evaluate what I need is
mongoimport --db=abc --collection=a a.bson
but I'm getting is
mongoimport --db=abc --collection=a.bson a.bson
My version of using sed to strip the .bson suffix from '{}' isn't working. I know its not a blocker but I felt if that is possible.
Any suggestions?
The problem twofold:
Shell expansions: Before a command is executed in a shell environment, the shell (sh/bash/ksh/zsh) will perform a sequence of expansions to build up the actual command that is being executed. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion. Hence, before the find command will be executed, it will perform all substitutions, including the command substitution located in the exec statement. Ergo, the command is equivalent to:
$ find *.bson -type f -exec echo mongoimport --db=abc --collection={} {} \;
A way forward would be to prohibit the command substitution by using single-quotes, however this leads to problem two.
find's exec statement is limited: The command that -exec can execute is limited to an external utility with optional arguments. Various shell features are therefor not recognized. To use shell built-ins, functions, conditionals, pipelines, redirections etc. directly with -exec is not possible, unless wrapped in something like a sh -c child shell.
Hence the answer would be something in the line of:
$ find *.bson -type f -exec /usr/bin/sh -c 'echo mongoimport --db=abc --collection=$(echo {} | sed s/.bson//g) {}' \;
Suggesting different strategy to this problem.
Use find with option -printf to prepare your commands.
The result will be list of commands to execute (command per line).
After inspecting and testing the commands, save find command output into a file and run the file (as a bash script).
Or just run directly into bash command.
1. find result inspection:
find . -type f -name "*.bson" -printf "mongoimport --db=abc --collection=%f %f\n" | sed s/.bson//
Notice sed replacement only on first .bson match. Do not use g option.
2. Run processed and inspected find output.
bash <<< $(find . -type f -name "*.bson" -printf "mongoimport --db=abc --collection=%f %f\n" | sed s/.bson//)
Related
I've been trying:
find dev-other -name '*.flac' -type f -exec echo $(echo {} | sed 's,^[^/]*/,,') \;
I expect to see a list of paths to .flac files within dev-other, but without a prepended dev-other/, e.g.:
4515/11057/4515-11057-0095.flac
4515/11057/4515-11057-0083.flac
4515/11057/4515-11057-0040.flac
4515/11057/4515-11057-0105.flac
4515/11057/4515-11057-0017.flac
4515/11057/4515-11057-0001.flac
Instead I see
dev-other/4515/11057/4515-11057-0095.flac
dev-other/4515/11057/4515-11057-0083.flac
dev-other/4515/11057/4515-11057-0040.flac
dev-other/4515/11057/4515-11057-0105.flac
dev-other/4515/11057/4515-11057-0017.flac
Why isn't the sed replace working here even though it works on its own
$ echo $(echo dev-other/4515/11057/4515-11057-0047.flac | sed 's,^[^/]*/,,')
4515/11057/4515-11057-0047.flac
I first tried with expansions:
find dev-other -name '*.flac' -type f -exec a={} echo ${a#*/} \;
But got the errors:
find: a=dev-other/700/122866/700-122866-0001.flac: No such file or directory
find: a=dev-other/700/122866/700-122866-0030.flac: No such file or directory
find: a=dev-other/700/122866/700-122866-0026.flac: No such file or directory
find: a=dev-other/700/122866/700-122866-0006.flac: No such file or directory
find: a=dev-other/700/122866/700-122866-0010.flac: No such file or directory
You can just use parameter expansion for your use-case when using find with the -exec option,
find dev-other -name '*.flac' -type f -exec bash -c 'x=$1; y="${x#*/}"; echo "$y"' bash {} \;
I used a separate shell (use bash or sh) using bash -c because to involve separate string operations involving parameter expansion. Think of each output of find result to be passed as argument to this sub-shell where this manipulation takes place.
When bash -c executes a command, the next argument after the command is used as $0 (the script's "name" in the process listing), and subsequent arguments become the positional parameters ($1, $2, etc.). This means that the filename passed by find (in place of the {}) becomes the first parameter of the script -- and is referenced by $1 inside the mini-script
If you don't want to use an extra bash, use _ in-place
find dev-other -name '*.flac' -type f -exec bash -c 'x=$1; y="${x#*/}"; echo "$y"' _ {} \;
where _ i is a bash predefined variable (not defined in dash for instance): "At shell startup, set to the absolute path-name used to invoke the shell or shell script being executed as passed in the environment or argument list" ( See man bash - Special Parameters section)
Worth looking at Using Find - Complex Actions
So I have the following command which looks for a series of files and appends three lines to the end of everything found. Works as expected.
find /directory/ -name "file.php" -type f -exec sh -c "echo -e 'string1\string2\nstring3\n' >> {}" \;
What I need to do is also look for any instance of string1, string2, or string3 in the find ouput of file.php prior to echoing/appending the lines so I don't append a file unnecessarily. (This is being run in a crontab)
Using | grep -v "string" after the find breaks the -exec command.
How would I go about accomplishing my goal?
Thanks in advance!
That -exec command isn't safe for strings with spaces.
You want something like this instead (assuming finding any of the strings is reason not to add any of the strings).
find /directory/ -name "file.php" -type f -exec sh -c "grep -q 'string1|string2|string3' \"\$1\" || echo -e 'string1\nstring2\nstring3\n' >> \"\$1\"" - {} \;
To explain the safety issue.
find places {} in the command it runs as a single argument but when you splat that into a double-quoted string you lose that benefit.
So instead of doing that you pass the file as an argument to the shell and then use the positional arguments in the shell command with quotes.
The command above simply chains the echo to a failure from grep to accomplish the goal.
I am trying to build and run a find command from a script. But I get a very cryptic error message from find. The following basically sums up how I build the command line and run it
$ xx="find . -name 'p*' -mmin +10 -exec echo {} \\;"
$ echo "$xx" #.....and I get the same print from echo $xx
find . -name 'p*' -mmin +10 -exec echo {} \;
$ $xx
find: missing argument to `-exec'
$ find . -name 'p*' -mmin +10 -exec echo {} \;
./p2.sh
./p1.sh
$ read xx
find . -name 'p*' -mmin +2 -exec echo {} \\;
$ $xx
find: missing argument to `-exec'
I am stuck and will appreciate your help. I am also wondering what's causing this. I am using bash 3.2.51 on SLES.
The actual command I want to execute is a little bit longer but I used echo here just to illustrate.
Thanks
Dinesh
Trying to store complicated commands in bash variables and then evaluate the variables pretty well never works.
If you need to build a command in pieces, use an array. See this useful Bash FAQ: I'm trying to put a command in a variable, but the complex cases always fail!.
Here's the basic strategy:
# Make an array
declare -a findcmd=(find .)
# Add some arguments
findcmd+=(-name 'p*')
findcmd+=(-mmin +10)
findcmd+=(-exec echo {} \;)
# Run the command
"${findcmd[#]}"
You need to understand how bash quoting works. Remember that the quoting (and de-quoting) only happens once, when you type the command (or when bash reads it from a script file). Quotes which get into the values of variables are just ordinary characters.
If you're experimenting with set -x, remember also that set -x inserts quotes in order to remove ambiguities. These quotes are not part of the variables. While that is clearly essential, it seems to be confusing to programmers who are not familiar with the bash execution model.
I have written an executable in c++, which is designed to take input from a file, and output to stdout (which I would like to redirect to a single file). The issue is, I want to run this on all of the files in a folder, and the find command that I am using is not cooperating. The command that I am using is:
find -name files/* -exec ./stagger < {} \;
From looking at examples, it is my understanding that {} replaces the file name. However, I am getting the error:
-bash: {}: No such file or directory
I am assuming that once this is ironed out, in order to get all of the results into one file, I could simply use the pattern Command >> outputfile.txt.
Thank you for any help, and let me know if the question can be clarified.
The problem that you are having is that redirection is processed before the find command. You can work around this by spawning another bash process in the -exec call:
find files/* -exec bash -c '/path/to/stagger < "$1"' -- {} \;
The < operator is interpreted as a redirect by the shell prior to running the command. The shell tries redirecting input from a file named {} to find's stdin, and an error occurs if the file doesn't exist.
The argument to -name is unquoted and contains a glob character. The shell applies pathname expansion and gives nonsensical arguments to find.
Filenames can't contain slashes. The argument to -name can't work even if it were quoted. If GNU find is available, -path can be used to specify a glob pattern files/*, but this doesn't mean "files in directories named files", for that you need -regex. Portable solutions are harder.
You need to specify one or more paths for find to start from.
Assuming what you really wanted was to have a shell perform the redirect, Here's a way with GNU find.
find . -type f -regex '.*foo/[^/]*$' -exec sh -c 'for x; do ./stagger <"$x"; done' -- {} +
This is probably the best portable way using find (-depth and -prune won't work for this):
find . -type d -name files -exec sh -c 'for x; do for y in "$x"/*; do [ -f "$y" ] && ./stagger <"$y"; done; done' -- {} +
If you're using Bash, this problem is a very good candidate for just using a globstar pattern instead of find.
#!/usr/bin/env bash
shopt -s extglob globstar nullglob
for x in **/files/*; do
[[ -f "$x" ]] && ./stagger <"$x"
done
Simply escape the less-than symbol, so that redirection is carried out by the find command rather than the shell it is running in:
find files/* -exec ./stagger \< {} \;
Can someone show me to use xargs properly? Or if not xargs, what unix command should I use?
I basically want to input more than (1) file name for input <localfile>, third input parameter.
For example:
1. use `find` to get list of files
2. use each filename as input to shell script
Usage of shell script:
test.sh <localdir> <localfile> <projectname>
My attempt, but not working:
find /share1/test -name '*.dat' | xargs ./test.sh /staging/data/project/ '{}' projectZ \;
Edit:
After some input from everybody and trying -exec, I am finding that my <localfile> filename input with find is also giving me the full path. /path/filename.dat instead of filename.dat. Is there a way to get the basename from find? I think this will have to be a separate question.
I'd just use find -exec here:
% find /share1/test -name '*.dat' -exec ./test.sh /staging/data/project/ {} projectZ \;
This will invoke ./test.sh with your three arguments once for each .dat file under /share1/test.
xargs would pack up all of these filenames and pass them into one invocation of ./test.sh, which doesn't look like your desired behaviour.
If you want to execute the shell script for each file (as opposed to execute in only once on the whole list of files), you may want to use find -exec:
find /share1/test -name '*.dat' -exec ./test.sh /staging/data/project/ '{}' projectZ \;
Remember:
find -exec is for when you want to run a command on one file, for each file.
xargs instead runs a command only once, using all the files as arguments.
xargs stuffs as many files as it can onto the end of the command line.
Do you want to execute the script on one file at a time or all files? For one at a time, use file's exec, which it looks like you're already using the syntax for, and which xargs doesn't use:
find /share1/test -name '*.dat' -exec ./test.sh /staging/data/project/ '{}' projectZ \;
xargs does not have to combine arguments, it's just the default behavior. this properly uses xargs, to execute the commands, as intended.
find /share1/test -name '*.dat' -print0 | xargs -0 -I'{}' ./test.sh /staging/data/project/ '{}' projectZ
When piping find to xargs, NULL termination is usually preferred, I recommend appending the -print0 option to find. After which you must add -0 to xargs, so it will expect NULL terminated arguments. This ensures proper handling of filenames. It's not POSIX proper, but considered well supported. You can always drop the NULL terminating options, if your commands lack support.
Remeber while find's purpose is finding files, xargs is much more generic. I often use xargs to process non-filename arguments.