My make version on macOS High Sierra Version 10.13.6 looks like this:
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
My Bash version looks like this:
$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
Copyright (C) 2007 Free Software Foundation, Inc.
My current directory looks like this:
$ ls -1
Makefile
a.txt
b.txt
c.txt
My Makefile looks like this:
hello:
echo hello
%.txt: FORCE
cat "$#"
FORCE:
Case 1: Without bash-completion
When I do not have bash-completion installed, I can conveniently autocomplete filenames in the current directory as the target argument for make. I mean, if I type:
make a<TAB>
it would automatically complete to:
make a.txt
Case 2: With bash-completion
When I do have bash-completion (such as with brew install bash-completion), I can no longer use this behaviour.
If I type:
make a<TAB>
it does not autocomplete at all. It does autocomplete other targets now such as make he<TAB> and make FO<TAB> which was not possible earlier but I lose the autocomplete any arbitrary filenames.
Question
Why does installing bash-completion disable the default behaviour of autocompleting current filenames? I think that's a very convenient behaviour to have.
How can I get that behaviour back without removing bash-completion. If I can do this, I can have the best of both worlds. Use auto-completions provided by bash-completion as well as auto-complete filenames in the current directory.
To answer the first part of your question, I think most people would not want Bash to offer files in the working directory as suggested targets for make.
But you can customise the behaviour without too much trouble.
Some useful docs on Bash Tab Completion can be found here.
It looks like the source code for the Bash Completion used on Mac OS X is here. The line you care about is this one:
COMPREPLY=( $( compgen -W "$( make -qp $makef $makef_dir 2>/dev/null | \
awk -F':' '/^[a-zA-Z0-9][^$#\/\t=]*:([^=]|$)/ \
{split($1,A,/ /);for(i in A)print A[i]}' )" \
-- "$cur" ) )
You can see that the compgen builtin there generates the list of possible completions, based on the output of make -qp Makefile ..
You can hack this to work the way you want it to this way. First, copy the make-specific autocomplete file to your user bash completion file:
cp /usr/local/Cellar/bash-completion/1.3_3/etc/bash_completion.d/make ~/.bash_completion
Then edit that file to specify a slightly different behaviour. Here, I'll show the diff:
--- /usr/local/Cellar/bash-completion/1.3_3/etc/bash_completion.d/make 2019-02-06 18:35:00.000000000 +1100
+++ /Users/alexharvey/.bash_completion 2019-02-06 18:34:36.000000000 +1100
## -62,7 +62,7 ##
COMPREPLY=( $( compgen -W "$( make -qp $makef $makef_dir 2>/dev/null | \
awk -F':' '/^[a-zA-Z0-9][^$#\/\t=]*:([^=]|$)/ \
- {split($1,A,/ /);for(i in A)print A[i]}' )" \
+ {split($1,A,/ /);for(i in A)print A[i]}' ; ls * )" \
-- "$cur" ) )
fi
Notice all I did there was to append the output of ls * to the response returned to compgen so that it considers files in the working directory to be also possible tab completions.
And that's it. Start a new shell.
Related
I'm using the following command on Ubuntu to list all files containing a given pattern:
for f in *; do if grep -zoPq "foo\nbar" $f; then echo $f; fi; done
But on macos, I'm geting the following error:
grep: invalid option -- z
There's no -z option to treat files as a big string with macos grep, unlike gnu grep.
Is there another option on macos grep equivalent to `-z ? If not, what alternative can I use to get the same result ?
-P (PERL regex) is only supported in gnu grep but not on BSD grep found on Mac OS.
You can either use home brew to install gnu grep or else use this equivalent awk command:
awk 'p ~ /foo$/ && /^bar/ {print FILENAME; nextfile}; {p=$0}' *
Please note that this eliminates the need to use shell for loop.
You can install pcregrep via home brew, and then use it with the -M option:
By default, each line that matches a pattern is copied to the
standard output, and if there is more than one file, the file name is
output at the start of each line, followed by a colon. However, there
are options that can change how pcregrep behaves. In particular, the
-M option makes it possible to search for patterns that span line
boundaries. What defines a line boundary is controlled by the -N
(--newline) option.
With ripgrep
rg -lU 'foo\nbar'
This will list all filenames containing foo\nbar in the current directory. -U option allows to match multiple lines. Unlike grep -z, whole file isn't read in one-shot, so this is safe to use even for larger input files.
ripgrep recursively searches by default. Use rg -lU --max-depth 1 'foo\nbar' if you don't want to search sub-directories.
However, note that by default, rigprep ignores
files and directories that match rules specified by ignore files like .gitignore
hidden files and directories
binary files
You can change that by using:
-u or --no-ignore
-uu or --no-ignore --hidden
-uuu or --no-ignore --hidden --binary
It seems you are searching for files which have the sequence foo\nbar. With GNU awk (brew install gawk), you can set the record separatorRS to this sequence and check if the record matches:
gawk 'BEGIN{RS="foo\nbar"}{exit (RT!=RS)}' file
This will try to split your files in records which are separated by the record separator RS, if so, it will terminate with exit code 0, otherwise with exit code 1. The behaviour is the same as the proposed grep
If you just want the files listed, you can do:
gawk 'BEGIN{RS="foo\nbar"}(RT==RS){print FILENAME}{nextfile}' *
TL;DR:
This does not work in a Makefile in MacOS (catalina):
grab_setup_rule != cat $(all_makefiles_except_current) | grep -h -E '.+-setup:.*##' | awk '{ print $1 }'
but on executed on the command line, with $(all_makefiles_except_current) replaced with the correct paths, it works.
In depth explanation
Hey,
I have the following code in a Makefile:
BASEPATH = ..
repos = project1 project2
possible_includes = $(foreach repo,$(repos), $(BASEPATH)/$(repo)/Makefile)
-include $(possible_includes)
all_makefiles_except_current := $(wordlist 2,$(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
grab_setup_rule != cat $(all_makefiles_except_current) | grep -h -E '.+-setup:.*##' | awk '{ print $1 }'
What this does is include all the Makefiles specified in the repos variable, get a list of all included files, set in all_makefiles_except_current, and then try to grab all the targets from those Makefiles that end with -setup.
This is an example of what is in the included Makefiles:
projectName-targetX: ### desc
code
projectName-setup: ## desc
code
<.. more targets ..>
This is a self-documenting makefile.
If I run the shell commands:
cat ../project1/Makefile | grep -h -E '.+-setup:.*##' | awk '{ print $1 }'
NOTE: on windows and linux the grep regex has to be without single quotes, ex.: grep -h -E .+-setup:.*##
on the command line, IT WORKS, I get the targets that I want and this will work on Windows (msys2), on Linux (ubuntu) and on MacOS (catalina).
But on the Makefile the grab_setup_rule will always be empty in MacOS. Seems it doesn't get processed correctly.
What am I doing wrong in the grab_setup_rule on MacOS?
The != operator was added in GNU make 4.0. It is not available in older releases.
Apple is scared of the GPLv3 license and refuses to ship any GNU software using that license, so they will never provide a version of GNU make newer than 3.81 (released in 2006). They have similarly old and broken versions of other GNU software including their standard interactive shell, bash.
You can either download the GNU make source code and build it yourself, or use brew or similar to get a newer version, or else use the := $(shell ...) method of invoking a shell command and assigning it to a variable which works in GNU make 3.81 as well.
I'd just like to change this
cc211_AMBER_13062012i.II cc211_GROMOS_13062012i.II
cc211_CHARM_13062012i.II cc211_OPLS_13062012i.II
to
cc211_AMBER_15062012i.II cc211_GROMOS_15062012i.II
cc211_CHARM_15062012i.II cc211_OPLS_15062012i.II
I tried,
find -name "*.13 *" | xargs rename ".13" ".15"
There is normally no space between the 3 and the second asterix, thats just makes it italics on from what I can see. Basically there's a lot of answers for what to do when it's at the end of the filename, where asterix seem to work, but here I can't make it work.
Anything you've got would make my life a lot easier!
Edit 1: Trial
-bash-4.1$ ls
cc211_AMBER_13062012.II cc211_GROMOS_13062012.II
cc211_CHARM_13062012.II cc211_OPLS_13062012.II
-bash-4.1$ rename 's/_13/_15/' cc*
-bash-4.1$ ls
cc211_AMBER_13062012.II cc211_GROMOS_13062012.II
cc211_CHARM_13062012.II cc211_OPLS_13062012.II
How about this:
for i in *.II; do mv $i $(echo $i | sed 's/_13/_15/g'); done
This will replace _13 with _15 in all files with extension .II
More information on sed here.
A pure bash solution:
for i in cc*; do
mv "$i" "${i/_13/_15}"
done
rename 's/_13/_15/' cc*
Should do what you want. The regular expression s/_13/_15/ replaces _13 by _15 in all files starting 'cc'.
$ ls
cc211_AMBER_13062012.II cc211_GROMOS_13062012.II
cc211_CHARM_13062012.II cc211_OPLS_13062012.II
$ rename 's/_13/_15/' cc*
$ ls
cc211_AMBER_15062012.II cc211_GROMOS_15062012.II
cc211_CHARM_15062012.II cc211_OPLS_15062012.II
This will only work with the newer perl version of rename. To check which version you have do man rename. If the top of the page says
Perl Programmers Reference Guide
you have the perl version. If it says:
Linux Programmer's Manual
you have the standard (older) version.
For the older version, the command should be:
rename _13 _15 cc*
I'm using a pure Linux solution:
### find all files that contains _DES in name and duplicate them adding _AUXLOCAL
for f in **/*_DES*; do
cp "$f" "${f%.DES}_AUXLOCAL"
done
###Rename all _AUXLOCAL files, removing _DES to _LOCAL
for f in **/*_AUXLOCAL*; do
mv "$f" "${f/_DES/_LOCAL}"
done
###Rename all _AUXLOCAL files, removing _AUXLOCAL
for f in **/*_AUXLOCAL*; do
mv "$f" "${f/_AUXLOCAL/}"
done
I hope it helps
In this case, you can use this command:
rename -v "_130" "_150" *.II
I'm trying to use SED to extract text from a log file. I can do a search-and-replace without too much trouble:
sed 's/foo/bar/' mylog.txt
However, I want to make the search case-insensitive. From what I've googled, it looks like appending i to the end of the command should work:
sed 's/foo/bar/i' mylog.txt
However, this gives me an error message:
sed: 1: "s/foo/bar/i": bad flag in substitute command: 'i'
What's going wrong here, and how do I fix it?
Update: Starting with macOS Big Sur (11.0), sed now does support the I flag for case-insensitive matching, so the command in the question should now work (BSD sed doesn't reporting its version, but you can go by the date at the bottom of the man page, which should be March 27, 2017 or more recent); a simple example:
# BSD sed on macOS Big Sur and above (and GNU sed, the default on Linux)
$ sed 's/ö/#/I' <<<'FÖO'
F#O # `I` matched the uppercase Ö correctly against its lowercase counterpart
Note: I (uppercase) is the documented form of the flag, but i works as well.
Similarly, starting with macOS Big Sur (11.0) awk now is locale-aware (awk --version should report 20200816 or more recent):
# BSD awk on macOS Big Sur and above (and GNU awk, the default on Linux)
$ awk 'tolower($0)' <<<'FÖO'
föo # non-ASCII character Ö was properly lowercased
The following applies to macOS up to Catalina (10.15):
To be clear: On macOS, sed - which is the BSD implementation - does NOT support case-insensitive matching - hard to believe, but true. The formerly accepted answer, which itself shows a GNU sed command, gained that status because of the perl-based solution mentioned in the comments.
To make that Perl solution work with foreign characters as well, via UTF-8, use something like:
perl -C -Mutf8 -pe 's/öœ/oo/i' <<< "FÖŒ" # -> "Foo"
-C turns on UTF-8 support for streams and files, assuming the current locale is UTF-8-based.
-Mutf8 tells Perl to interpret the source code as UTF-8 (in this case, the string passed to -pe) - this is the shorter equivalent of the more verbose -e 'use utf8;'.Thanks, Mark Reed
(Note that using awk is not an option either, as awk on macOS (i.e., BWK awk and BSD awk) appears to be completely unaware of locales altogether - its tolower() and toupper() functions ignore foreign characters (and sub() / gsub() don't have case-insensitivity flags to begin with).)
A note on the relationship of sed and awk to the POSIX standard:
BSD sed and awk limit their functionality mostly to what the POSIX sed and
POSIX awk specs mandate, whereas their GNU counterparts implement many more extensions.
Editor's note: This solution doesn't work on macOS (out of the box), because it only applies to GNU sed, whereas macOS comes with BSD sed.
Capitalize the 'I'.
sed 's/foo/bar/I' file
Another work-around for sed on Mac OS X is to install gsedfrom MacPorts or HomeBrew and then create the alias sed='gsed'.
If you are doing pattern matching first, e.g.,
/pattern/s/xx/yy/g
then you want to put the I after the pattern:
/pattern/Is/xx/yy/g
Example:
echo Fred | sed '/fred/Is//willma/g'
returns willma; without the I, it returns the string untouched (Fred).
The sed FAQ addresses the closely related case-insensitive search. It points out that a) many versions of sed support a flag for it and b) it's awkward to do in sed, you should rather use awk or Perl.
But to do it in POSIX sed, they suggest three options (adapted for substitution here):
Convert to uppercase and store original line in hold space; this won't work for substitutions, though, as the original content will be restored before printing, so it's only good for insert or adding lines based on a case-insensitive match.
Maybe the possibilities are limited to FOO, Foo and foo. These can be covered by
s/FOO/bar/;s/[Ff]oo/bar/
To search for all possible matches, one can use bracket expressions for each character:
s/[Ff][Oo][Oo]/bar/
The Mac version of sed seems a bit limited. One way to work around this is to use a linux container (via Docker) which has a useable version of sed:
cat your_file.txt | docker run -i busybox /bin/sed -r 's/[0-9]{4}/****/Ig'
Use following to replace all occurrences:
sed 's/foo/bar/gI' mylog.txt
I had a similar need, and came up with this:
this command to simply find all the files:
grep -i -l -r foo ./*
this one to exclude this_shell.sh (in case you put the command in a script called this_shell.sh), tee the output to the console to see what happened, and then use sed on each file name found to replace the text foo with bar:
grep -i -l -r --exclude "this_shell.sh" foo ./* | tee /dev/fd/2 | while read -r x; do sed -b -i 's/foo/bar/gi' "$x"; done
I chose this method, as I didn't like having all the timestamps changed for files not modified. feeding the grep result allows only the files with target text to be looked at (thus likely may improve performance / speed as well)
be sure to backup your files & test before using. May not work in some environments for files with embedded spaces. (?)
Following should be fine:
sed -i 's/foo/bar/gi' mylog.txt
I've got a makefile (developed for gmake on Linux) that I'm attempting to port to MacOS, but it seems like sed doesn't want to cooperate. What I do is use GCC to autogenerate dependency files, and then tweak them a bit using sed. The relevant portion of the makefile:
$(OBJ_DIR)/%.d: $(SRC_DIR)/%.cpp
$(CPPC) -MM -MD $< -o $#
sed -i 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1_utest.o:|' $#
While this runs with no trouble under GNU/Linux, I get errors like the following when attempting to build on MacOS:
sed: 1: "test/obj/equipmentConta ...": undefined label 'est/obj/equipmentContainer_utest.d'
sed: 1: "test/obj/dice_utest.d": undefined label 'est/obj/dice_utest.d'
sed: 1: "test/obj/color-string_u ...": undefined label 'est/obj/color-string_utest.d'
It would seem like sed is chopping off a character, but I can't see the solution.
OS X sed handles the -i argument differently to the Linux version.
You can generate a command that might "work" for both by adding -e in this way:
# vv
sed -i -e 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1_utest.o:|' $#
OS X sed -i interprets the next thing after the -i as a file extension for a backup copy of the in-place edit. (The Linux version only does this if there is no space between the -i and the extension.) Obviously a side affect of using this is that you will get a backup file with -e as an extension, which you may not want. Please refer to other answers to this question for more details, and cleaner approaches that can be used instead.
The behaviour you see is because OS X sed consumes the s||| as the extension (!) then interprets the next argument as a command - in this case it begins with t, which sed recognizes as a branch-to-label command expecting the target label as an argument - hence the error you see.
If you create a file test you can reproduce the error:
$ sed -i 's|x|y|' test
sed: 1: "test": undefined label 'est'
Actually, doing
sed -i -e "s/blah/blah/" files
doesn't do what you expect in MacOS either. Instead it creates backup files with -e extension.
The proper command for MacOS is
sed -i "" -e "s/blah/blah/" files
On Linux, remove the space between -i and "" (see related answer)
sed -i"" -e "s/blah/blah/" files
The currently accepted answer is flawed in two very important ways.
With BSD sed (the OSX version), the -e option is interpreted as
a file extension and therefore creates a backup file with a -e
extension.
Testing for the darwin kernel as suggested is not a reliable
approach to a cross platform solution since GNU or BSD sed could
be present on any number of systems.
A much more reliable test would be to simply test for the --version option which is only found in the GNU version of sed.
sed --version >/dev/null 2>&1
Once the correct version of sed is determined, we can then execute the command in its proper syntax.
GNU sed syntax for -i option:
sed -i -- "$#"
BSD sed syntax for -i option:
sed -i "" "$#"
Finally put it all together in a cross platform function to execute an in place edit sed commend:
sedi () {
sed --version >/dev/null 2>&1 && sed -i -- "$#" || sed -i "" "$#"
}
Example usage:
sedi 's/old/new/g' 'some_file.txt'
This solution has been tested on OSX, Ubuntu, Freebsd, Cygwin, CentOS, Red Hat Enterprise, & Msys.
martin clayton's helpful answer provides a good explanation of the problem[1], but his solution - as he states - has a potentially unwanted side effect.
Here are side-effect-free solutions:
Caveat: Solving the -i syntax problem alone, as below, may not be enough, because there are many other differences between GNU sed and BSD/macOS sed (for a comprehensive discussion, see this answer of mine).
Workaround with -i: Create a backup file temporarily, then clean it up:
With a non-empty suffix (backup-file filename extension) option-argument (a value that is not the empty string), you can use -i in a way that works with both BSD/macOS sed and GNU sed, by directly appending the suffix to the -i option.
This can be utilized to create a backup file temporarily that you can clean up right away:
sed -i.bak 's/foo/bar/' file && rm file.bak
Obviously, if you do want to keep the backup, simply omit the && rm file.bak part.
Workaround that is POSIX-compliant, using a temporary file and mv:
If only a single file is to be edited in-place, the -i option can be bypassed to avoid the incompatibility.
If you restrict your sed script and other options to POSIX-compliant features, the following is a fully portable solution (note that -i is not POSIX-compliant).
sed 's/foo/bar' file > /tmp/file.$$ && mv /tmp/file.$$ file
This command simply writes the modifications to a temporary file and, if the sed command succeeds (&&), replaces the original file with the temporary one.
If you do want to keep the original file as a backup, add another mv command that renames the original first.
Caveat: Fundamentally, this is what -i does too, except that it tries to preserve permissions and extended attributes (macOS) of the original file; however, if the original file is a symlink, both this solution and -i will replace the symlink with a regular file.
See the bottom half of this answer of mine for details on how -i works.
[1] For a more in-depth explanation, see this answer of mine.
This isn't quite an answer to the question, but one can get linux-equivalent behavior through
brew install gnu-sed
# Add to .bashrc / .zshrc
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
(previously there was a --with-default-names option to brew install gnu-sed but that has recently been removed)
I came across this issue as well and thought of the following solution:
darwin=false;
case "`uname`" in
Darwin*) darwin=true ;;
esac
if $darwin; then
sedi="/usr/bin/sed -i ''"
else
sedi="sed -i"
fi
$sedi 's/foo/bar/' /home/foobar/bar
Works for me ;-), YMMV
I work in a multi-OS team where ppl build on Windows, Linux and OS X. Some OS X users complained because they got another error - they had the GNU port of sed installed so I had to specify the full path.
I've corrected the solution posted by #thecarpy:
Here's a proper cross-platform solution for sed -i:
sedi() {
case $(uname) in
Darwin*) sedi=('-i' '') ;;
*) sedi='-i' ;;
esac
LC_ALL=C sed "${sedi[#]}" "$#"
}
I avoid using sed -i when writing scripts and i came up with simple solution:
printf '%s' "$(sed 's/foo/bar' file)" > file
much compatible and is POSIX-compliant. It is doing pretty much the same as sed -i, but this one does not create temp files, it directly redirect the changes to file.
As a noob idk what's the cons of doing this, the only matters is "It works"