In bash I want to copy all .yml.sample files in a Git repository (recursively) and rename them to just have a .yml extension.
Eg. test.yml.sample would be copied to test.yml
Here’s as close as I’ve got, but I'm not clear on how to strip .sample off the end of the file name when I copy.
find . -depth -name "*.yml.sample" -exec sh -c 'cp "$1" "${1%/.sample/}"' _ {} \;
This should work:
find . -depth -name "*.yml.sample" -exec sh -c 'cp -p "$1" "${1%.yml.sample}.yml"' _ {} \;
The first *.yml.sample finds the files via find. Then after the -exec part, the magic happens via cp taking the results of that find via $1 and then the file extension for the copied file is set via ${1%.yml.sample}.yml where .yml.sample is the source extension, and .yml is the new destination extension.
Note I also added the -p attribute to preserve the attributes from the source file to the copied file. You might not need that, but I think it can be helpful when doing copies like this.
And—since this shell logic can be confusing—in terms of the _ {} \;, it breaks down as this:
_ {}: As explained in this answer on the Unix/Linux Stack Exchange site, “The way this works is bash takes the parameters after -c as arguments, _ {} is needed so that the contents of {} is assigned to $1 not l.”
\;: When you run find with a -exec parameter, everything that happens after that is parsed through a new shell. Meaning the main find command runs in one parent shell and stuff after -exec runs in another child shell command. If you run it as _ {} ;, the child shell command would terminate. So instead, you escape it as \; so you get _ {} \; which means only the parent sell find would interpret that ; as a “terminate” and thus the paren find command can successfully run iterative commands via -exec without stopping that child shell command. Read up on -exec command ; here.
I think you can use a tool like mmv, to mass rename all the files you need.
mmv \*.yml.sample \#1.yml
The above line should work... just make sure to test it first. Hope this helps!
Edit: If you want to copy and rename, all in one step, you can use the -c flag. That will preserve the original file, and will make a copy using the rename mask.
mmv -c \*.yml.sample \#1.yml
Related
I would like to find txt files with find command and move the directory of the found file, and then apply a command to the file using a bash shell one-liner
For example, this command works, but the acmd is executed in the current directory.
$ find . -name "*.txt" | xargs acmd
I would like to run acmd in the txt file's direcotry.
Does anyone have good idea?
From the find man page:--
-execdir command ;
-execdir command {} +
Like -exec, but the specified command is run from the subdirec‐
tory containing the matched file, which is not normally the
directory in which you started find. This a much more secure
method for invoking commands, as it avoids race conditions dur‐
ing resolution of the paths to the matched files. As with the
-exec action, the `+' form of -execdir will build a command line
to process more than one matched file, but any given invocation
of command will only list files that exist in the same subdirec‐
tory. If you use this option, you must ensure that your $PATH
environment variable does not reference `.'; otherwise, an
attacker can run any commands they like by leaving an appropri‐
ately-named file in a directory in which you will run -execdir.
The same applies to having entries in $PATH which are empty or
which are not absolute directory names. If find encounters an
error, this can sometimes cause an immediate exit, so some pend‐
ing commands may not be run at all. The result of the action
depends on whether the + or the ; variant is being used;
-execdir command {} + always returns true, while -execdir com‐
mand {} ; returns true only if command returns 0.
Just for completeness, the other option would be to do:
$ find . -name \*.txt | xargs -i sh -c 'echo "for file $(basename {}), the directory is $(dirname '{}')"'
for file schedutil.txt, the directory is ./Documentation/scheduler
for file devices.txt, the directory is ./Documentation/admin-guide
for file kernel-parameters.txt, the directory is ./Documentation/admin-guide
for file gdbmacros.txt, the directory is ./Documentation/admin-guide/kdump
...
i.e. have xargs "defer to a shell". In usecases where -execdir suffices, go for it.
So, I have this simple script which converts videos in a folder into a format which the R4DS can play.
#!/bin/bash
scr='/home/user/dpgv4/dpgv4.py';mkdir -p 'DPG_DS'
find '../Exports' -name "*1080pnornmain.mp4" -exec python3 "$scr" {} \;
The problem is, some of the videos are invalid and won't play, and I've moved those videos to a different directory inside the Exports folder. What I want to do is check to make sure the files are in a folder called new before running the python script on them, preferably within the find command. The path should look something like this:
../Exports/(anything here)/new/*1080pnornmain.mp4
Please note that (anything here) text does not indicate a single directory, it could be something like foo/bar, foo/b/ar, f/o/o/b/a/r, etc.
You cannot use -name because the search is on the path now. My first solution was:
find ./Exports -path '**/new/*1080pnornmain.mp4' -exec python3 "$scr" {} \;
But, as #dan pointed out in the comments, it is wrong because it uses the globstar wildcard (**) unnecessarily:
This checks if /new/ is somewhere in the preceding path, it doesn't have to be a direct parent.
So, the star is not enough here. Another possibility, using find only, could be this one:
find ./Exports -regex '.*/new/[^\/]*1080pnornmain.mp4' -exec python3 "$scr" {} \;
This regex matches:
any number of nested folders before new with .*/new
any character (except / to leave out further subpaths) + your filename with [^\/]*1080pnornmain.mp4
Performances could degrade given that it uses regular expressions.
Generally, instead of using the -exec option of the find command, you should opt to passing each line of find output to xargs because of the more efficient thread spawning, like:
find ./Exports -regex '.*/new/[^\/]*1080pnornmain.mp4' | xargs -0 -I '{}' python3 "$scr" '{}'
I am a bash newbie, and I'm trying to do something that seems fairly straightforward but am having issues.
I am trying to search for a file with a pretty generic but nonunique name (e.g. analysis.uniqueExt, but also maybe sorted_result.uniqueExt) that can be within one specific subdirectory of a directory that was found from a different 'find' query. Then I would like to copy that file to my personal directory whilst also renaming the file to something more descriptive that hints to its origin location.
Here is an example of what I have tried:
case=/home/data/ABC_123 # In reality this is coming from a different query successfully
specific_id=ABC_123 # This was extracted from the previous variable
OUTDIR=/my/personal/directory
mkdir -p $OUT_DIR/$this_folder
find $case/subfolder/ -type f -name "*.uniqueExt" -exec sh -c 'cp "$1" ${OUT_DIR}/${specific_id}/$(basename "$1")' sh {} \;
This doesn't work because OUT_DIR and specific_id are not scoped in the inner shell created by the -exec command.
So I tried to do this another way:
find $case/subfolder/ -type f -name "*.uniqueExt" -exec cp {} ${OUT_DIR}/${specific_id}/$(basename {}) \;
However now I cannot extract the basename of the file found in the 'find' query as I have not invoked a shell to do so.
Is there a way I can either properly scope my variables in example #1 or execute the basename function in example #2 to accomplish this? Or maybe there is a totally different solution (possibly involving multiple -exec calls? Or maybe just piping the find results to xargs?).
Thanks for your help!
You need to export the variables since you're using them in a different shell process than the one you assigned them in.
Exporting variables makes them available in descendant processes.
export specific_id=ABC_123 # This was extracted from the previous variable
export OUTDIR=/my/personal/directory
However, you don't really need to use the shell for this. You can use
find $case/subfolder/ -type f -name "*.uniqueExt" -exec cp -t "$OUTDIR/$specific_id/" {} +
You don't have to call basename yourself, because copying a file to a target directory automatically uses the basename as the destination filename.
In my version, I use the -t option so I can put the destination directory first. This allows it to use the + variant to put all the found filenames in a single command, rather than running cp separately for each file.
First I made a question here: Unzip a file and then display it in the console in one step
It works and helped me a lot. (please read)
Now I have a second issue. I do not have a single zipped log file but I have a lot of them in defferent folders, which I need to find first. The files have the same names. For example:
/somedir/server1/log.gz
/somedir/server2/log.gz
/somedir/server3/log.gz
and so on...
What I need is a way to:
find all the files like: find /somedir/server* -type f -name log.gz
unzip the files like: gunzip -c log.gz
use grep on the content of the files
Important! The whole should be done in one step.
I cannot first store the extracted files in the filesystem because it is a readonly filesystem. I need somehow to connect, with pipes, the output from one command to the input of the next.
Before, the log files were in text format (.txt), therefore I had not to unzip them first. In this case it was easy:
ex.
find /somedir/server* -type f -name log.txt | xargs grep "term"
Now I have to deal with zipped files. That means, after I find the files, I need first somehow do unzip them and then send the contents to grep.
With one file I do:
gunzip -p /somedir/server1/log.gz | grep term
But for multiple files I don't know how to do it. For example how to pass the output of find to gunzip and the to grep?!
Also if there is another way / "best practise" how to do that, it is welcome :)
find lets you invoke a command on the files it finds:
find /somedir/server* -type f -name log.gz -exec gunzip -c '{}' + | grep ...
From the man page:
-exec command {} +
This variant of the -exec action runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of
invocations of the command will be much less than the number
of matched files. The command line is built in much the same
way that xargs builds its command lines. Only one instance of
{} is allowed within the command, and (when find is being
invoked from a shell) it should be quoted (for example, '{}')
to protect it from interpretation by shells. The command is
executed in the starting directory. If any invocation with
the + form returns a non-zero value as exit status, then
find returns a non-zero exit status. If find encounters an
error, this can sometimes cause an immediate exit, so some
pending commands may not be run at all. This variant of -exec
always returns true.
I'd like to copy a file_list to another location. This is being called in a python script. I have
find <sourceaddress> -exec cp '{}' <destaddress> | .* rm
but it tells me an exact parameter is missing. It runs though it gives a prompt from the command line and from the script just does nothing.
I think you are missing "\;" at the end. I am not sure what the .* rm does. Assuming you want to remove the files you can use the 'mv' command instead of 'cp'.
For copying files only from one directory to another ,
find <srcdirectory> -exec cp '{}' <destdirectory> \;
If you want to move the files, use 'mv' instead use below.
find <srcdirectory> -exec mv '{}' <destdirectory> \;