If I run mkdir -pv, the -p works, but I do not get verbose output; however, if I run just mkdir -v, the verbose output appears as expected. Also potentially of note, the longform of -v does not seem to work at all.
From my testing:
mkdir -p a/b/c: creates a/, a/b/, and a/b/c/, gives no output to terminal (as expected)
mkdir -v d: creates d/ and outputs mkdir: created directory 'd' (as expected)
mkdir -pv e/f/g: creates e/, e/f/, and e/f/g/, gives no output to terminal (why?)
mkdir --verbose h: gives a illegal option -- - error (why?)
Update: I filed a bug report with Apple for this issue, and received the following reply:
Their answer that "-v is not applicable" does not make much sense to me, since mkdir -v works just fine, but since there are various workarounds and I am no longer even using OSX, I don't think it's worth pursuing any further to me.
Macs use BSD-based code that is (mostly) POSIX compliant but (mostly) without GNU extensions (such as double-dash long options). The man page does document -v and -p, and -p works but does seem to suppress the -v option (which is probably when it is most useful).
One of your options is to file a bug with the Darwin or BSD teams, or with Apple. That's the way it is; it is arguably not the way it should be. (GNU mkdir supports -v and prints the directories it creates when used with -p, which makes more sense, and supports the 'it is a bug' contention.)
With thanks to SnoringFrog:
Another option is to install and use the GNU mkdir command on OSX. It is part of GNU coreutils and you could install it as explained in How to replace Mac OS X utilities with GNU core utilities at Ask Different. Then, you could alias mkdir to point to gmkdir to get the expected behavior (assuming you don't use --default-names when installing the GNU tools).
You could also do this if logging is critical (since the -v flag works in the simple case):
mkdir -v path &&
mkdir -v path/to &&
mkdir -v path/to/{destination1,destination2,etc} ;
Think Different or Think Again?!
Related
I'm trying to reproduce a situation which happens only in some machines. To reproduce it, I create a directory with 2000 files:
mkdir /tmp/test
cd /tmp/test
for f in $(seq 1 2000); do touch $f.txt; done
Then I use the following Makefile (simplified from the real use case):
FILES:=$(shell find . -name '*.txt')
%.done: %.txt
#echo "done $#"
toolong:
#$(foreach file,$(sort $(FILES)), \
if $(MAKE) $(file); \
then echo "did $(file)" >> $#; \
else echo "failed $(file)" >> $#; fi; )
Running make produces, unsurprisingly, an error:
make: execvp: /bin/sh: Argument list too long
This question presents a solution which does work. However, I need to understand exactly why this error does not happen on my colleagues' computers. I tried the following things:
Increasing stack limit (ulimit -s gives the same result on both machines, 8192, and increasing it does not change anything);
Checking getconf ARG_MAX (2097152 in both machines);
Checking MAX_ARG_STRLEN (131072 in both machines);
Using a different shell (zsh is being used in both machines; I also tried bash, dash and sh, via export SHELL=<shell> make, and also by replacing the symlink /bin/sh -> /bin/bash with a link to dash).
Finally, I tried recompiling Make from source, and realized that, even when I compile the same version of Make (4.1) in my Ubuntu test machine, I get the same behavior as I had in my Fedora, that is, the error "argument list too long".
make --version only shows a single difference between them:
Version from the apt package:
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Version compiled from source (./configure && make):
GNU Make 4.1
Built for x86_64-unknown-linux-gnu
I even tried compiling make-dfsg, which results in an identical make --version, but the result is the same as with my other manually-compiled make.
By increasing the number of files on Ubuntu, I managed to identify that the actual limits in the size of the generated command line are:
Fedora or Arch Linux (both with Make 4.2.1), or Ubuntu with manually-compiled Make 4.1: 128 KB (~1200 files);
Debian Sid or Ubuntu, both with Make 4.1 installed from apt package: 2 MB (~19300 files).
I'd really like to understand (1) why this difference exists, and (2) how could I compile Make to obtain the higher limit, so that I can have the exact same behavior on both machines.
Your recipe expands as a one-line shell compound command (because of the line ending backslashes). And this line is probably way too long. Did you check what it looks like (the shell compound command) on all machines? I suggest that you wrap it in $(info...):
toolong:
#$(info $(foreach file,$(sort $(FILES)), \
if $(MAKE) $(file); \
then echo "did $(file)" >> $#; \
else echo "failed $(file)" >> $#; fi; ))
and see if the output is the same on your machine and on the others. If, for any reason, yours is longer, it could be the explanation. Else it must be a OS difference...
Note: you could have one rule per target, instead of one single rule with a huge recipe for all.
Note: your recipe (independently from its fantastic length) does nothing useful. As the files already exist and do not have pre-requisites, all your sub-make calls will just tell you that the files are up-to-date.
Edit: Issue has been resolved after update to Git for Windows >= 2.9.0.windows1
Disclaimer
Some comments are referring to full "story" behind this issue but I decided to shorten it because it was getting too long and hard to follow. I present you as succinct failing example as possible. For those who are interested in knowing full context of the problem: it is available in previous revision of the question.
This: basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')") is the first (excluding hashbang) line in scripts generated by npm after installing any package which comes with CLI.
For some reason basedir is incorrectly resolved and that's why node can't find module and crashes. I managed to narrow down problem to the pipe in subshell on latest Git for Windows' git-bash. Executing:
echo -n "1:"
echo "a" | cat
echo -n "2:"
echo "$(echo "a" | cat)"
echo -n "3:"
echo "$(echo "a")"
prints:
1:a
2:
3:a
I can't find other people with this issue so I think that it's something wrong with my env (Windows 10 Pro, Git for Windows 2.8.4) and personally I'm out of ideas where it might come from.
My findings:
downgrading Git for Windows to 2.6.4 fixes the problem. Still I don't like being stuck on old version ;/
it works fine on clean Windows 10 VM
pipe output seems to be completely empty because running following snippet doesn't return any result.
On clean installation of mingw + msys problem doesn't occur
snippet:
echo $(echo foobar | cat > bazzzzzzzzzz ; ) ; cat bazzzzzzzzzz
find /c -name bazzzzzzz* 2> /dev/null # /c, /d and /x are my Windows partitions
find /d -name bazzzzzzz* 2> /dev/null # I did test if it actually works for existing file and it does
find /x -name bazzzzzzz* 2> /dev/null
Thanks to agc for invaluable help on figuring this out to this point.
My PATH variable looks like this:
PATH=/c/Users/ja/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/ja/bin:/c/Windows:/c/Windows/System32:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Program Files/nodejs:/c/ProgramData/Oracle/Java/javapath:/c/program files/graphicsmagick-1.3.23-q16:/c/ProgramData/chocolatey/lib/getopt/binaries:/c/Program Files (x86)/Windows Kits/8.1/Windows Performance Toolkit:/c/Program Files/nodejs:/c/Program Files (x86)/Microsoft VS Code/bin:/c/Users/ja/AppData/Roaming/npm:/c/Program Files (x86)/MacType:/usr/bin/vendor_perl:/usr/bin/core_perl
also
$ which sed
/usr/bin/sed
$ which echo
/usr/bin/echo
$ which cat
/usr/bin/cat
$ echo $SHELL
/usr/bin/bash
Summarizing the comments, the short (tl;dr) version: either downgrade, upgrade, and/or re-install MSYS and MinGW that come with Git for Windows.
MSYS supplements MinGW, and the version provided by Git for Windows may be modified from the original maintainers of MSYS. There was a bug reported against MSYS for what appears to be this same issue (using "mingw version: 64 bit bundled with git version 2.8.3.windows.1"), but was marked as "works for me" (i.e., "can't reproduce"). But there was a comment that the problem could be in the repackaging:
"Please be advised that MSYS, as bundled with git for windows, may be modified from our official distribution, (and 64-bit MinGW certainly isn't ours); thus we don't formally support either of these." https://sourceforge.net/p/mingw/bugs/2303/
Long story short, looks like a bug.
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")
This seems like an example of the XY problem. Let us break down this line:
echo "$0"
This is usually the path to the script, for example ./alfa.sh
sed -e 's,\\,/,g'
This replaces backslashes with forward slashes. This is where this line starts
to fall apart:
You dont need the -e, you can just do sed 's,\\,/,g'
You probably dont need the g, usually just going to be one slash as shown
above
Changing the slashes doesnt really make sense. Bash, even on Windows, is
going to be using forward slashes already
If for some reason the slashes do need to be changed, Sed is not the right
tool for this anyway, cygpath is:
$ cygpath -m 'C:\Program Files\Mozilla Firefox\firefox.exe'
C:/Program Files/Mozilla Firefox/firefox.exe
dirname
Now you are calling dirname after sed/cygpath. It should be called before, that
way sed/cygpath dont have to replace as much:
basedir=$(cygpath -m "$(dirname "$0")")
Finally, the sed command is bad for another reason; if you are going to be
spitting out a path, it should be an absolute one, because why not?
basedir=$(cygpath -am "$(dirname "$0")")
Notice now that no pipe is even involved. I will also add that this problem was
introduced recently to the NPM repo. You might comment to the devs there.
I have a Makefile that has this kind of command:
browserify:
rm -rf ./dist
mkdir dist
# Browserify
echo -n "/* ${NPM_PACKAGE} ${NPM_VERSION} ${GITHUB_PROJ} */" \
> dist/pica.js
So when I do make browserify it should output the comment to the top of the file without \n. But for some reasons... the output looks like this
-n /* package 0.0.0 */
...more things...
I'm using zsh on osx.
It doesn't matter what shell you are using. Make will always use /bin/sh as the shell it invokes (unless you specifically set the SHELL make variable to something else). Think what a disaster it would be if make used whatever shell the user was using to invoke recipes!
On many GNU/Linux systems, /bin/sh is actually a link to bash. On other GNU/Linux systems, /bin/sh is a link to dash which is a small, POSIX-standard shell without all the extensions bash uses (dash is good for running portable shell scripts fast, but not good for a user's interactive shell as it's missing too many expected features). On non-GNU-based systems (like OSX) /bin/sh might be ksh or something else even.
There is no portable, standard way to invoke echo in such a way that it doesn't print the trailing newline. There is an echo program, and different ones work differently. Many shells, including bash and zsh also have an echo built-in to the shell:
$ type -a echo
echo is a shell builtin
echo is /bin/echo
and these versions of echo also work differently than the program echo. Some versions have no way to suppress newlines. Some use the -n flag. Some use \c at the end to suppress printing the newline. Some support a combination of them.
The short answer is that if you want to print a line in the shell without a newline in a portable and reliable way, you should use the printf program to do it, not echo:
browserify:
rm -rf ./dist
mkdir dist
# Browserify
printf %s "/* ${NPM_PACKAGE} ${NPM_VERSION} ${GITHUB_PROJ} */" \
> dist/pica.js
I am trying to build some code and the location where the compiler is present has a space in it 'blahblah/Source Code/blahblah' .I am not sure how to add this to the environment variable.I use bash and tried to use the normal
export PATH="$PATH:/blahblah/Source Code/blahblah"
but it doesnt seem to work(I also tried using \before spaces).Throws me errors like No such file or directory.Am I missing out on something?
I just did a little experimentation on my own:
$ mkdir 'Source Code'
$ cd Source\ Code/
$ vim testme.pl
$ chmod 755 testme.pl
$ cat testme.pl
#! /usr/bin/perl
print "I worked\n";
$ ./testme.pl
I worked
$ cd ..
$ export PATH="$PATH:/home/bchittenden/Source Code"
$ testme.pl
I worked
This indicates that the problem is not the whitespace in $PATH... bash seems to handle that correctly... you'll have to give us more information.
This topic is about the util 'ls'
The BSD version uses the parameter '-G' to color up the output,
while the Linux version uses parameter '--color'
Also the environment variable to set the colors is different:
BSD: $LSCOLORS
Linux: $LS_COLORS
But now the problem is: I want to determine which version is installed (using a small Shell script), so I can set alias ls and the environment appropriate in my .bachrc file.
As I mentioned above this seems to me to be the handiest method
if ls --color -d . >/dev/null 2>&1; then
GNU_LS=1
elif ls -G -d . >/dev/null 2>&1; then
BSD_LS=1
else
SOLARIS_LS=1
fi
I've essentially this in my l script, which I use on various platforms to tweak ls output as I like
Just run 'ls' and see whether it throws an error, e.g. on my mac:
$ ls --color 1>/dev/null 2>&1
$ echo $?
1
Whereas
$ ls -G 1>/dev/null 2>&1
$ echo $?
0
Indicating -G is supported, but --color is not.
Ironically, the --version switch Kimmo mentions is not supported on most BSD systems :-)
Writing a portable configuration file for your particular setup can be a Herculean task. In your case, if you're sure your .bashrc is going to be used only on GNU/Linux and on a BSD system, you can check for switches that exist in one of the ls' but not in the other: for example, -D doesn't seem to be an accepted switch by ls on my BSD machines (FreeBSD and Mac OS X), whereas it is for GNU ls. Conversely, -P is accepted on BSD, but not on GNU/Linux. Knowing this, you can distinguish between the two ls' and set up environment variables accordingly.
$ ls --version
ls (GNU coreutils) 6.10
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Richard Stallman and David MacKenzie.
combining the methods described, here's an easy way to use a bash function instead of an alias in order to make colors work regardless of if you are using BSD or GNU ls.
ll () {
if ls --version &>/dev/null; then
ls --color=auto -lahtr
else
ls -Glahtr
fi
}
inspired by a particular conda env recipe pulling in GNU ls on my macOS system where my ls aliases were all hard-coded for stock BSD ls only.