I am trying to do the following without any success so far:
remove:
for file in $(shell find src/ -name migrations -type d); do rm $(wildcard "$(file)/0*.py"); done
If I understood well, you want to remove all files named:
src/**/migrations/0*.py
where ** can be anything, including nothing at all. find can do this alone:
find src -path '*/migrations/0*.py' -delete
And as make recipes are just bare shell, you do not need $(shell... or anything like this:
remove:
find src -path '*/migrations/0*.py' -delete
But before running this potentially dangerous rule, you should maybe use this one until you are satisfied:
remove:
find src -path '*/migrations/0*.py' -ok rm {} \;
The only difference is that it will ask you confirmation before deleting files. We can also make all this a bit easier:
remove-safe: DELCMD := -ok rm {} \\;
remove-unsafe: DELCMD := -delete
remove-dry-run: DELCMD := -print
remove-safe remove-unsafe remove-dry-run:
find src -path '*/migrations/0*.py' $(DELCMD)
Use one or the other goal depending of what you want:
remove-dry-run to print the list of files that would be deleted without deleting them,
remove-safe to delete the files with confirmation,
remove-unsafe to delete the files without confirmation.
EDIT 1: if this an exercise about loops and make, just remember that make recipes are shell scripts (one per line) that are first expanded by make and next passed to the shell. So:
If you want to use shell variables, use them in one single line (but you can break lines with a trailing \). Else they will be from different shell invocations and it will not work as expected.
If you use the $ sign in your recipe for shell expansion, double them to escape the first expansion by make.
In the following I assume that you do not have directory or file names with spaces or special characters. If you do, it is time to ask a question about your shell.
So, with the bash shell, for instance, we can first build an array of target directories and then loop over them to delete the files:
SHELL := bash
remove:
dirs=( $$(find src -type d -name migrations) ); \
for d in "$${dirs[#]}"; do \
rm -f "$$d"/0*.py; \
done
EDIT 2: you could also use the builtin make loop system with one (phony) clean target per directory.
DIRS := $(shell find src -type d -name migrations)
clean-targets := $(addprefix clean-,$(DIRS))
.PHONY: remove $(clean-targets)
remove: $(clean-targets)
$(clean-targets): clean-%:
rm -f "$*"/0*.py
The tricky part is the:
$(clean-targets): clean-%:
rm -f "$*"/0*.py
rule. It is equivalent to one rule per migrations directory that would be:
clean-DIR: clean-%:
rm -f "$*"/0*.py
where DIR is the directory path. And these rules are static pattern rules. In the recipe the $* make automatic variable is expanded by make as the stem of the clean-% pattern. So, if DIR is src/foo/bar/migrations, the rule for it is equivalent to:
clean-src/foo/bar/migrations:
rm -f src/foo/bar/migrations/0*.py
The main advantage of this style over the shell loop is that make can run all recipes in parallel. If you have hundreds of migrations directories and if you have 8 or 16 cores on your computer, it can really make a difference. Just try:
make -j1 remove
make -j16 remove
and you will see the difference.
The short answer is you can't do that.
It's critical when writing makefile recipes to be clear in your own understanding of how they work: first, make will expand all make variables and functions. Then, make will pass the resulting string to the shell to be run. Finally, make waits until the shell completes and examines the exit code to find out whether it succeeded or not.
Here you're trying to use a shell for loop where the body of the loop invokes a make wildcard function. This cannot work.
Consider what you are wanting make (and the shell) to do: the shell would run its for loop, then every time it executed the body of the loop it would communicate back to make the value of some shell variable and make would expand the make wildcard function using that variable, then return the results to the shell for more processing. This is simply not possible.
Best is to write the recipe using shell constructs. A good rule of thumb is that it's never a good idea to use make's shell function inside a recipe: the recipe is already being run in a shell, so you don't need the shell function at all. It's simply confusing. Similarly, you rarely need make's wildcard function in a recipe because the shell does file globbing for you automatically.
I would write this as:
remove:
find src/ -name migrations -type d | while read -r dir; do rm "$$file"/0*.py; done
Note we have escaped $$file with two dollar signs to keep make from expanding it as a make variable.
Related
I have a lot of kernel .deb files from my custom kerenls. I would like to write a bash that would delete all the old files except the ones associated with the currently installed kernel version. My script:
#!/bin/bash
version='uname -r'
$version
dir=~/Installed-kernels
ls | grep -v '$version*' | xargs rm
Unfortunately, this deletes all files in the dir.
How can I get the currently installed kernel version and set said version as a perimeter with? Each .deb I want to keep contains the kernel version (5.18.8) but have other strings in their name (linux-headers-5.18.8_5.18.8_amd64.deb).
Edit: I am only deleting .deb files inside the noted directory. The current list of file names in the tree are
linux-headers-5.18.8-lz-xan1_5.18.8-lz-1_amd64.deb
linux-libc-dev_5.18.8-lz-1_amd64.deb
linux-image-5.18.8-lz-xan1_5.18.8-lz-1_amd64.deb
This can be done as a one-liner, though I've preserved your variables:
#!/bin/bash
version="$(uname -r)"
dir="$HOME/Installed-kernels"
find "$dir" -maxdepth 1 -type f -not -name "*$version*" -print0 |xargs -0 rm
To set a variable to the output of a command, you need either $(…) or `…`, ideally wrapped in double-quotes to preserve spacing. A tilde isn't always interpreted correctly when passed through variables, so I expanded that out to $HOME.
The find command is much safer to parse than the output of ls, plus it lets you better filter things. In this case, -maxdepth 1 will look at just that directory (no recursion), -type f seeks only files, and -not -name "*$version*" removes paths or filenames that match the kernel version (which is a glob, not a regex—you'd otherwise have to escape the dots). Also note those quotes; we want find to see the asterisks, and without the quotes, the shell will expand the glob prematurely. The -print0 and corresponding -0 ensure that you preserve spacing by delimiting entries with null characters.
You can remove the prompts regarding read-only files with rm -f.
If you also want to delete directories, remove the -type f part and add -r to the end of that final line.
I have a file I would like to copy into about 300,000 different directories, these are themselves split between two directories, e.g.
DirA/Dir001/
...
DirB/Dir149000/
However when I try:
cp file.txt */*
It returns:
bash: /bin/cp: Argument list too long
What is the best way of copying a file into multiple directories, when you have too many to use cp?
The answer to the question as asked is find.
find . -mindepth 2 -maxdepth 2 -type d -exec cp script.py {} \;
But of course #triplee is right... why make so many copies of a file?
You could, of course, instead create links to the file...
find . -mindepth 2 -maxdepth 2 -type d -exec ln script.py {} \;
The options -mindepth 2 -maxdepth 2 limit the recursive search of find to elements exactly two levels deep from the current directory (.). The -type d matches all directories. -exec then executes the command (up to the closing \;), for each element found, replacing the {} with the name of the element (the two-levels-deep subdirectory).
The links created are hard links. That means, you edit the script in one place, the script will look different in all places. The script is, for all intents and purposes, in all the places, with none of them being any less "real" than the others. (This concept can be surprising to those not used to it.) Use ln -s if you instead want to create "soft" links, which are mere references to "the one, true" script.py in the original location.
The beauty of find ... -exec ... {}, as opposed to many other ways to do it, is that it will work correctly even for filenames with "funny" characters in them, including but not limited to spaces or newlines.
But still, you should really only need one script. You should fix the part of your project where you need that script in every directory; that is the broken part...
Extrapolating from the answer to your other question you seem to have code which looks something like
for TGZ in $(find . -name "file.tar.gz")
do
mkdir -p work
cd work
tar xzf $TGZ
python script.py
cd ..
rm -rf work
done
Of course, the trivial fix is to replace
python script.py
with
python ../script.py
and voilá, you no longer need a copy of the script in each directory at all.
I woud further advice to refactor out the cd and changing script.py so you can pass it the directory to operate on as a command-line argument. (Briefly, import sys and examine the value of sys.argv[1] though you'll often want to have option parsing and support for multiple arguments; argparse from the Python standard library is slightly intimidating, but there are friendly third-party wrappers like click.)
As an aside, many beginners seem to think the location of your executable is going to be the working directory when it executes. This is obviously not the case; or /bin/ls woul only list files in /bin.
To get rid of the cd problem mentioned in a comment, a minimal fix is
for tgz in $(find . -name "file.tar.gz")
do
mkdir -p work
tar -C work -x -z -f "$tgz"
(cd work; python ../script.py)
rm -rf work
done
Again, if you can change the Python script so it doesn't need its input files in the current directory, this can be simplified further. Notice also the preference for lower case for your variables, and the use of quoting around variables which contain file names. The use of find in a command substitution is still slightly broken (it can't work for file names which contain whitespace or shell metacharacters) but maybe that's a topic for a separate question.
I have never in my life made a shell script (although I have had to find and delete multiple troll ones my friends put on my school account when i am not looking).
I have some .o files under the working directory. I want a shell script that, by being given a simple (no path) .o file name, finds the matching file under the current directory and then runs the shell command
arm-none-eabi-objdump -D <found file>
So if I give it example.o, it will find dir1/dir2/example.o and then run
arm-none-eabi-objdump -D dir1/dir2/example.o
A shell script isn't especially needed for this, but I will attempt to cover all approaches. This assumes that the shell of choice is bash, but this may work for other shells as well. First, you need to consider whether you may have multiple object files with the same name, and what you may want to do if you do. If you want to dump only the first match, then this should work for you:
find ./ -name example.o -exec arm-none-eabi-objdump -D '{}' \; -quit
If, however, you want to dump all found matches, you can either remove the -quit (which will concatenate the output) or put the command in a loop:
find ./ -name example.o |
while read file; do
arm-none-eabi-objdump -D "$file" | less
done
If you wish to save yourself the typing (or reverse search) and put this in a shell script, all you need to do is put the same text in a file, add at the beginning of the file #!/bin/bash on its own line, and then make the file executable via chmod a+rx my-script.sh. Then you can run the script by typing ./my-script.sh example.o (assuming you are in the same directory as the script). Note that unless you put the script somewhere in your PATH environment variable, then you do need the ./ before the file name.
I'm working on a C kernel and I want to make it easier to compile all of the sources by using a bash script file. I need to know how to do a foreach loop and only get the files with a .c extension, and then get the filename of each file I find so I can make gcc compile each one.
Use find to walk through your tree
and then read the list it generates using while read:
find . -name \*.c | while read file
do
echo process $file
done
If the action that you want to do with file is not so complex
and can be expressed using ore or two commands, you can avoid while
and make all things with the find itself. For that you will use -exec:
find . -name \*.c -exec command {} \;
Here you write your command instead of command.
You can also use -execdir:
find . -name \*.c -execdir command {} \;
In this case command will be executed in the directory of found file (for each file that was found).
If you're using GNU make, you can do this using only make's built-in functions, which has the advantage of making it independent of the shell (but the disadvantage of being slower than find for large trees):
# Usage: $(call find-recursive,DIRECTORY,PATTERN)
find-recursive = \
$(foreach f,$(wildcard $(1)/*),\
$(if $(wildcard $(f)/.),\
$(call find-recursive,$(f),$(2)),\
$(filter $(2),$(f))))
all:
#echo $(call find-recursive,.,%.c)
There are often times that I want to execute a command on all files (including hidden files) in a directory. When I try using
chmod g+w * .*
it changes the permissions on all the files I want (in the directory) and all the files in the parent directory (that I want left alone).
Is there a wildcard that does the right thing or do I need to start using find?
You will need two glob patterns to cover all the potential “dot files”: .[^.]* and ..?*.
The first matches all directory entries with two or more characters where the first character is a dot and the second character is not a dot. The second picks up entries with three or more characters that start with .. (this excludes .. because it only has two characters and starts with a ., but includes (unlikely) entries like ..foo).
chmod g+w .[^.]* ..?*
This should work well in most all shells and is suitable for scripts.
For regular interactive use, the patterns may be too difficult to remember. For those cases, your shell might have a more convenient way to skip . and ...
zsh always excludes . and .. from patterns like .*.
With bash, you have to use the GLOBIGNORE shell variable.
# bash
GLOBIGNORE=.:..
echo .*
You might consider setting GLOBIGNORE in one of your bash customization files (e.g. .bash_profile/.bash_login or .bashrc).
Beware, however, becoming accustomed to this customization if you often use other environments.
If you run a command like chmod g+w .* in an environment that is missing your customization, then you will unexpectedly end up including . and .. in your command.
Additionally, you can configure the shells to include “dot files” in patterns that do not start with an explicit dot (e.g. *).
# zsh
setopt glob_dots
# bash
shopt -s dotglob
# show all files, even “dot files”
echo *
Usually I would just use . .[a-zA-Z0-9]* since my file names tend to follow certain rules, but that won't catch all possible cases.
You can use:
chmod g+w $(ls -1a | grep -v '^..$')
which will basically list all the files and directories, strip out the parent directory then process the rest. Beware of spaces in file names though, it'll treat them as separate files.
Of course, if you just want to do files, you can use:
find . -maxdepth 0 -type f -exec chmod g+w {} ';'
or, yet another solution, which should do all files and directories except the .. one:
for i in * .* ; do if [[ ${i} != ".." ]] ; then chmod g+w "$i"; fi done
but now you're getting into territory where scripts or aliases may be necessary.
What i did was
tar --directory my_directory --file my_directory.tar --create `ls -A mydirectory/`
Works just fine the ls -A my_directory expands to everything in the directory except . and ... No wierd globs, and on a single line.
ps: Perhaps someone will tell me why this is not a good idea. :p
How about:
shopt -s dotglob
chmod g+w ./*
Since you may not want to set dotglob for the rest of your bash session you can set it for a single set of commands by running in a subprocess like so:
$ (shopt -s dotglob; chmod g+w ./*)
If you are sure that two character hidden file names will never be used, then the simplest option is just be to do:
chmod g+w * ...*