Apreciate any help and excuse me if my terminology is incorrect.
What I am trying to do is write a scrpit/.bat file that will do the following:
copy 1 directory(and subdirectories) from pointA, to point B.
Then in pointB(and subdirectories) unzip the files which will give *.csv files
Then in pointB(and subdirectories) I want to delete some rows from all these csv files
This unix command, run on cygwin, will copy all the files from /cygdrive/v/pointA/* to the current directory . (i.e. the dot is the current working directory)
cp /cygdrive/v/pointA/* .
This unix command, run on cygwin, will go through all the files in the directory and subdirectories that end with .zip
and unzip them
find -iname *.zip -execdir unzip {} \;
This unix command, run on cygwin, will go through all the files in the directory and subdirectories that end with .csv
For each file it deletes the 1st 6 rows and the last row and that's the returned file.
find ./ -iname '*.csv' -exec sed -i '1,6d;$ d' '{}' ';'
I was looking to do this in one script/bat file but I am having trouble with the first find command
I am having trouble with the find and unzip commands on the one line and am wondering how and if this can be done
chdir C:\pointA
C:\cygwin\bin\cp.exe /cygdrive/v/pointB/* .
::find -iname *.zip -execdir unzip {} \;
::find ./ -iname '*.csv' -exec sed -i '1,6d;$ d' '{}' ';'
I did try something like this:
C:\cygwin\bin\find.exe -iname *.zip -execdir C:\cygwin\bin\unzip.exe {} \;
but I get the following:
/usr/bin/find: missing argument to `-execdir'
Can anyone advise if/how this can be done?
The Cygwin tools use their own kind of paths, e.g. /cygdrive/c/cygwin/bin/unzip.exe though sometimes the Windows paths with backslashes work, the backslashes do tend to confuse the Cygwin tools.
I highly recommend you write your tool in Bash shell script instead of a cmd.exe Windows batch file. In my experience (1) it's much easier to do flow control in bash scripts than in batch files, and (2) the Cygwin environment works better from Bash. You can open a bash shell and run bash yourscript.sh.
Your Bash script might look something like this: (untested)
#!/bin/bash
# This script would be run from a Cygwin Bash shell.
# You can use the Mintty program or run C:\cygwin\bin\bash --login
# to start a bash shell from Windows Command Prompt.
# Configure bash so the script will exit if a command fails.
set -e
cd /cygdrive/c/pointA
cp /cygdrive/v/pointB/* .
# I did try something like this:
# 1. Make sure you quote wildcards so the shell doesn't expand them
# before passing them to the 'find' program.
#
# 2. If you start bash with the --login option, the PATH will be
# configured so that C:\cygwin\bin is in your PATH, and you can
# just call 'find', 'cp' etc. without specifying full path to it.
# This will unzip all .zip files in all subdirectories under this one.
find -iname '*.zip' -execdir unzip {} \;
Related
I would like to find txt files with find command and move the directory of the found file, and then apply a command to the file using a bash shell one-liner
For example, this command works, but the acmd is executed in the current directory.
$ find . -name "*.txt" | xargs acmd
I would like to run acmd in the txt file's direcotry.
Does anyone have good idea?
From the find man page:--
-execdir command ;
-execdir command {} +
Like -exec, but the specified command is run from the subdirec‐
tory containing the matched file, which is not normally the
directory in which you started find. This a much more secure
method for invoking commands, as it avoids race conditions dur‐
ing resolution of the paths to the matched files. As with the
-exec action, the `+' form of -execdir will build a command line
to process more than one matched file, but any given invocation
of command will only list files that exist in the same subdirec‐
tory. If you use this option, you must ensure that your $PATH
environment variable does not reference `.'; otherwise, an
attacker can run any commands they like by leaving an appropri‐
ately-named file in a directory in which you will run -execdir.
The same applies to having entries in $PATH which are empty or
which are not absolute directory names. If find encounters an
error, this can sometimes cause an immediate exit, so some pend‐
ing commands may not be run at all. The result of the action
depends on whether the + or the ; variant is being used;
-execdir command {} + always returns true, while -execdir com‐
mand {} ; returns true only if command returns 0.
Just for completeness, the other option would be to do:
$ find . -name \*.txt | xargs -i sh -c 'echo "for file $(basename {}), the directory is $(dirname '{}')"'
for file schedutil.txt, the directory is ./Documentation/scheduler
for file devices.txt, the directory is ./Documentation/admin-guide
for file kernel-parameters.txt, the directory is ./Documentation/admin-guide
for file gdbmacros.txt, the directory is ./Documentation/admin-guide/kdump
...
i.e. have xargs "defer to a shell". In usecases where -execdir suffices, go for it.
Pretty much I want to cd to the output of the find command:
find ~ -name work_project_linux
cd the_output
In general the best way to execute an arbitrary command on the results of find is with find -exec. Curly braces {} are placeholders for the file names it finds, and the entire command ends with + or \;. For example, this will run ls -l on all of the files found:
find ~ -name work_project_linux -exec ls -l {} +
It doesn't work with some special commands like cd, though. -exec runs binaries, such as those found in /usr/bin, and cd isn't a binary. It's a shell builtin, a special type of command that the shell executes directly instead of calling out to some executable on disk. For shell builtins you can use command substitution:
cd "$(find ~ -name work_project_linux)"
This wouldn't work if find finds multiple files. It's only good for a single file name. Command substitution also won't handle some unusual file names correctly, such as those with embedded newlines—unusual, but legal.
I have a folder designed in the following way:
-parentDirectory
---folder1.zip
----item1
-----item1.zip
-----item2.zip
-----item3.zip
---folder2.zip
----item1
-----item1.zip
-----item2.zip
-----item3.zip
---folder3.zip
----item1
-----item1.zip
-----item2.zip
-----item3.zip
I would like to write a bash script that will loop through and unzip the folders and then go into each subdirectory of those folders and unzip the files and name those files a certain way.
I have tried the following
cd parentDirectory
find ./ -name \*.zip -exec unzip {} \;
count=1
for fname in *
do
unzip
mv $fname $attempt{count}.cpp
count=$(($count + 1))
done
I thought the first two lines would go into the parentDirectory folder and unzip all zips in that folder and then the for loop would handle the unzipping and renaming. But instead, it unzipped everything it could and placed it in the parentDirectory. I would like to maintain the same directory structure I have.
Any help would be appreciated
excerpt from man unzip
[-d exdir]
An optional directory to which to extract files. By default, all files and subdirectories are recreated in the current directory; the -d option allows extraction in an arbitrary directory (always assuming one has permission to write to the directory).
It's doing exactly what you told it, and what would happen if you had done the same on the command line. Just tell it where to extract, since you want it to extract there.
see Ubuntu bash script: how to split path by last slash? for an example of splitting the path out of fname.
putting it all together, your command executed in the parentDirectory is
find ./ -name \*.zip -exec unzip {} \;
But you want unzip to extract to the directory where it found the file. I was going to just use backticks on dirname {} but I can't get it to work right, as it either executes on the "{}" literal before find, or never executes.
The easiest workaround was to write my own script for unzip which does it in place.
> cat unzip_in_place
unzip $1 -d `dirname $1`
> find . -name "*.zip" -exec ./unzip_in_place {} \;
You could probably alias unzip to do that automatically, but that is unwise in case you ever use other tools that expect unzip to work as documented.
In bash I want to copy all .yml.sample files in a Git repository (recursively) and rename them to just have a .yml extension.
Eg. test.yml.sample would be copied to test.yml
Here’s as close as I’ve got, but I'm not clear on how to strip .sample off the end of the file name when I copy.
find . -depth -name "*.yml.sample" -exec sh -c 'cp "$1" "${1%/.sample/}"' _ {} \;
This should work:
find . -depth -name "*.yml.sample" -exec sh -c 'cp -p "$1" "${1%.yml.sample}.yml"' _ {} \;
The first *.yml.sample finds the files via find. Then after the -exec part, the magic happens via cp taking the results of that find via $1 and then the file extension for the copied file is set via ${1%.yml.sample}.yml where .yml.sample is the source extension, and .yml is the new destination extension.
Note I also added the -p attribute to preserve the attributes from the source file to the copied file. You might not need that, but I think it can be helpful when doing copies like this.
And—since this shell logic can be confusing—in terms of the _ {} \;, it breaks down as this:
_ {}: As explained in this answer on the Unix/Linux Stack Exchange site, “The way this works is bash takes the parameters after -c as arguments, _ {} is needed so that the contents of {} is assigned to $1 not l.”
\;: When you run find with a -exec parameter, everything that happens after that is parsed through a new shell. Meaning the main find command runs in one parent shell and stuff after -exec runs in another child shell command. If you run it as _ {} ;, the child shell command would terminate. So instead, you escape it as \; so you get _ {} \; which means only the parent sell find would interpret that ; as a “terminate” and thus the paren find command can successfully run iterative commands via -exec without stopping that child shell command. Read up on -exec command ; here.
I think you can use a tool like mmv, to mass rename all the files you need.
mmv \*.yml.sample \#1.yml
The above line should work... just make sure to test it first. Hope this helps!
Edit: If you want to copy and rename, all in one step, you can use the -c flag. That will preserve the original file, and will make a copy using the rename mask.
mmv -c \*.yml.sample \#1.yml
Apreciate any help and excuse me if my terminology is incorrect.
this is a script(*.sh file) that:
1-goes to a specific dir A
2-copies files from another dir B to dir A
3-#comented out# it also unzips the files in dir A and its subdirectories
4-#comented out# it also removes rows 1-6 and the last row of all *.csv files in dir A
#!/bin/bash
# Configure bash so the script will exit if a command fails.
set -e
#cd to the dir you want to copy to:
cd /cygdrive/c/path/I/want/to/copy/to
#echo Hello
#cp the files i want
#include the subdirectories
cp -r /cygdrive/c/path/I/want/to/copy/from/* .
# This will unzip all .zip files in all subdirectories under this one.
# -o is required to overwrite everything that is in there
#find -iname '*.zip' -execdir unzip -o {} \;
#find ./ -iname '*.csv' -exec sed -i '1,6d;$ d' '{}' ';'
Now I can get this script to work in cygwin by going to the dir where the file is stored and giving the following commands:
./filename.sh
or
/cygdrive/c/path/where/the/file/is/filename.sh
or
bash filename.sh
I can also do this in CMD/Windows DOS by doing the following:
C:\cygwin\bin\bash.exe -l
to get into a bash terminal and then give the following command:
/cygdrive/c/path/where/the/file/is/filename.sh
In task scheduler(in Windows) I have tried to schedule the following:
C:\cygwin\bin\bash.exe -l /cygdrive/c/path/where/the/file/is/filename.sh
but this does not work, even though the seperate commands work in CMD/Windows DOS as I have said above
Now what I want to do is be able to schedule this script(filename.sh) like I would a .vbs or .bat file in windows using task scheduler? Can anyone advise on this?
Note I have tried to write a Windows batch file(.bat) to do this(see below), but I could not get my unzip and sed commands to work,see here. So I have tried to write the Bash shell script above.
chdir C:\pointA
C:\cygwin\bin\cp.exe /cygdrive/v/pointB/* .
::find -iname *.zip -execdir unzip {} \;
::find ./ -iname '*.csv' -exec sed -i '1,6d;$ d' '{}' ';'
A solution is to associate .sh files with a batch file that runs bash. That way whenever you tell windows to execute an sh file it'll use the correct launcher - whether that's via a double click or a scheduled task. Here's mine:
#echo off
d:
chdir d:\cygwin\bin
bash --login %*
Associating a file type means that when you try to execute a file of that type, windows will use the path to that file as an argument passed to the program you've specified. For example, I have LibreOffice4 associated with .ods files. So if I doubleclick a .ods file, or just enter the path to a .ods file at the command prompt, windows will run open office calc, with the first parameter being the ods file. So if I have Untitled.ods on my desktop. I doubleclick it. That's effectively the same as opening up command prompt, typing
D:\Program Files (x86)\LibreOffice 4\program\scalc.exe" "C:\Users\Adam\Desktop\Untitled.ods".
and hitting enter. Indeed, if I do it, the expected happens: open office calc starts up and loads the file.
You can see how this works if you change the association to echo.exe (which I found in D:\cygwin\bin).
If I change the association to echo, open up the command prompt and type
"C:\Users\Adam\Desktop\Untitled.ods"
I'll just see echo.exe echo the filename back to me.
So what I'm suggesting you do is this:
create a batch file to run bash scripts using cygwin's bash (or use mine).
change the association for .sh files to that batch file
execute those .sh files directly, as though they were .exe or .bat files.
why not creating a batchfile (.bat) that loads your cygwin bashscript and schedule this batchfile? like this you dont have to deal with the way M$ handles paramters