Pipes inside backticks fail under MinGW - bash

When I try to run some Bash scripts on a Windows XP machine using MinGW, I get the following error:
./autogen.sh: pipe error: No such file or directory
I have localised the problem to Bash lines such as the following, which have a pipe inside backticks:
__copyright="`grep Copyright ./autogen.sh | head -1`"
However, pipes not inside backticks work just fine:
grep Copyright ./autogen.sh | head -1
All the programs you expect (sh, head, grep) are available and run happily on the command line and in Bash.
What should I do to resolve this error? I cannot run most Bash build tools without it.

$ msysinfo | head -3
msysinfo-1.3: Send this to the MSYS support list:
MSYS 1.0.11(0.46/3/2) 2004-04-30 18:55 i686 unknown; targ=MINGW32
$ echo '__copyright="`grep Copyright ./autogen.sh | head -1`" && echo $__copyright' >test.sh
$ cat test.sh
__copyright="`grep Copyright ./autogen.sh | head -1`" && echo $__copyright
$ sh test.sh
# Copyright 2005, 2006, 2007, 2008 by
For this test, I copied autogen.sh from some place like... randomly... http://svn.ghostscript.com/ghostscript/tags/freetype-2.3.7/autogen.sh
Probably this means your question needs more information.
But... sometimes when I run into snarly scenarios, it can help to enclose the breaking code in ( ). Technically, also, you do not need " " around backticks, and that complicates some use, but your example does not seem problematic.
I really hate to propose the following as it seems completely unnecessary:
$ for item in `grep Copyright ./autogen.sh`;
do
__copyright="$__copyright $item";
done;
echo $__copyright
What is more weird is that your error message seems to imply autogen.sh itself generated the error as though autogen.sh was grepping itself so I did:
$ sh test.sh
__copyright="`grep Copyright ./test.sh | head -1`" && echo $__copyright
after modifying test.sh to grep itself, and even that worked.
It therefore sounds like a sequencing issue and not a backtick issue, you know, which came first, the chicken or the egg?
Is autogen.sh trying to read its own Copyright comment? One way to let comments serve as data is to wrap them in here documents:
_Copyright ()
{
cat <<-END_OF_TEXT
# Copyright 2012, me
END_OF_TEXT
}
_Copyright

Related

Why can't I pass these parameters to my command?

B=master U='my.email#email.com' curl -u "${U}" "https://bitbucket.org/myteam/pod-dev/raw/${B}/install.bash"
I get asked:
Enter host password for user '':
See, the "U" is missing.
And also curl is performed:
GET /myteam/pod-dev/raw//install.bash HTTP/1.1
Also here, the B is missing (branch)
macOS Sierra 10.12.2
$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)
Copyright (C) 2007 Free Software Foundation, Inc.
Variables assignments like you are using them are part of the command. In your case the shell will try to expand "${A}", "${B}" before they have been actually set - during parsing the command.
You can separate the variable assignments and the actual command by ;:
B=master; U='my.email#email.com'; curl -u "${U}" "https://bitbucket.org/myteam/pod-dev/raw/${B}/install.bash"
That way they are 3 separate commands.
bash will try to expand the variables first, but you haven't set them at this point. See for example:
$ x=foo echo "$x"
>
$ x=foo; echo "$x"
> foo
Try like this instead of your command:
export B=master; export U='my.email#email.com' ; curl -u "$U" "https://bitbucket.org/myteam/pod-dev/raw/$B/install.bash"

pipe is returning empty string in bash in git for windows

Edit: Issue has been resolved after update to Git for Windows >= 2.9.0.windows1
Disclaimer
Some comments are referring to full "story" behind this issue but I decided to shorten it because it was getting too long and hard to follow. I present you as succinct failing example as possible. For those who are interested in knowing full context of the problem: it is available in previous revision of the question.
This: basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')") is the first (excluding hashbang) line in scripts generated by npm after installing any package which comes with CLI.
For some reason basedir is incorrectly resolved and that's why node can't find module and crashes. I managed to narrow down problem to the pipe in subshell on latest Git for Windows' git-bash. Executing:
echo -n "1:"
echo "a" | cat
echo -n "2:"
echo "$(echo "a" | cat)"
echo -n "3:"
echo "$(echo "a")"
prints:
1:a
2:
3:a
I can't find other people with this issue so I think that it's something wrong with my env (Windows 10 Pro, Git for Windows 2.8.4) and personally I'm out of ideas where it might come from.
My findings:
downgrading Git for Windows to 2.6.4 fixes the problem. Still I don't like being stuck on old version ;/
it works fine on clean Windows 10 VM
pipe output seems to be completely empty because running following snippet doesn't return any result.
On clean installation of mingw + msys problem doesn't occur
snippet:
echo $(echo foobar | cat > bazzzzzzzzzz ; ) ; cat bazzzzzzzzzz
find /c -name bazzzzzzz* 2> /dev/null # /c, /d and /x are my Windows partitions
find /d -name bazzzzzzz* 2> /dev/null # I did test if it actually works for existing file and it does
find /x -name bazzzzzzz* 2> /dev/null
Thanks to agc for invaluable help on figuring this out to this point.
My PATH variable looks like this:
PATH=/c/Users/ja/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/ja/bin:/c/Windows:/c/Windows/System32:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Program Files/nodejs:/c/ProgramData/Oracle/Java/javapath:/c/program files/graphicsmagick-1.3.23-q16:/c/ProgramData/chocolatey/lib/getopt/binaries:/c/Program Files (x86)/Windows Kits/8.1/Windows Performance Toolkit:/c/Program Files/nodejs:/c/Program Files (x86)/Microsoft VS Code/bin:/c/Users/ja/AppData/Roaming/npm:/c/Program Files (x86)/MacType:/usr/bin/vendor_perl:/usr/bin/core_perl
also
$ which sed
/usr/bin/sed
$ which echo
/usr/bin/echo
$ which cat
/usr/bin/cat
$ echo $SHELL
/usr/bin/bash
Summarizing the comments, the short (tl;dr) version: either downgrade, upgrade, and/or re-install MSYS and MinGW that come with Git for Windows.
MSYS supplements MinGW, and the version provided by Git for Windows may be modified from the original maintainers of MSYS. There was a bug reported against MSYS for what appears to be this same issue (using "mingw version: 64 bit bundled with git version 2.8.3.windows.1"), but was marked as "works for me" (i.e., "can't reproduce"). But there was a comment that the problem could be in the repackaging:
"Please be advised that MSYS, as bundled with git for windows, may be modified from our official distribution, (and 64-bit MinGW certainly isn't ours); thus we don't formally support either of these." https://sourceforge.net/p/mingw/bugs/2303/
Long story short, looks like a bug.
basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")
This seems like an example of the XY problem. Let us break down this line:
echo "$0"
This is usually the path to the script, for example ./alfa.sh
sed -e 's,\\,/,g'
This replaces backslashes with forward slashes. This is where this line starts
to fall apart:
You dont need the -e, you can just do sed 's,\\,/,g'
You probably dont need the g, usually just going to be one slash as shown
above
Changing the slashes doesnt really make sense. Bash, even on Windows, is
going to be using forward slashes already
If for some reason the slashes do need to be changed, Sed is not the right
tool for this anyway, cygpath is:
$ cygpath -m 'C:\Program Files\Mozilla Firefox\firefox.exe'
C:/Program Files/Mozilla Firefox/firefox.exe
dirname
Now you are calling dirname after sed/cygpath. It should be called before, that
way sed/cygpath dont have to replace as much:
basedir=$(cygpath -m "$(dirname "$0")")
Finally, the sed command is bad for another reason; if you are going to be
spitting out a path, it should be an absolute one, because why not?
basedir=$(cygpath -am "$(dirname "$0")")
Notice now that no pipe is even involved. I will also add that this problem was
introduced recently to the NPM repo. You might comment to the devs there.

Stale file descriptor with /dev/stdin

I'm attempting to write a script to loop over entries in .ssh/authorized_keys and do things with them, namely print their fingerprint and append them to a new place. This is what I have so far:
echo "$SSH_KEYS" | while read key ; do
ssh-keygen -lf /dev/stdin <<< "$key"
echo "$key" >> newplace
done
This unfortunately gives me the following error:
/dev/stdin: Stale file handle
I'm running Bash 4.3.11 on Ubuntu 14.04 kernel 3.13.0-24-generic.
On the same kernel running Bash 4.3.8, it works fine. Changing my version of Bash doesn't look to be an option at this point, this is an automated script for something in production.
I found this solution in another question here on StackOverflow but it seems to not work with this later version of Bash.
I think you're on the right track, but you want something like:
while read key; do
ssh-keygen -lf /dev/stdin <<< "$key"
echo "$key" >> newplace
done < .ssh/authorized_keys
As opposed to:
echo "$SSH_KEYS" | while read key ; do
ssh-keygen -lf /dev/stdin <<< "$key"
echo "$key" >> newplace
done
Note that instead of piping the output of echo, simply feed the file directly into the stdin of the while loop.
This worked for me on:
$ bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
I'm also on Ubuntu 14.04, but it seems that someone has also maybe seen this problem: https://github.com/docker/docker/issues/2067
A work-around is to write each key to a tempfile, process it, then remove the file.

Different exit codes

I'm trying to grep for a version number from my subversion command, so that I can check we have the write subversion module loaded in a bash script.
Interactively, this is an example use:
> svn --version | head -n1 | grep -q '1.7'; echo $?
0
However, when I put this same line (and nothing else) in a script and run the script:
> ./setup_svn.sh
1
Also, the script runs noticeably faster than the interactive shell command. Does anyone have ideas of what I might be missing that explains this result?
Edit
It turns out that my interactive bash script was using the wrong svn command. Not sure exactly why, but I think that might be a question for the Unix StackExchange.
It's almost certainly to do with the module system on our workstations, running interactively I get:
> module list
Currently Loaded Modulefiles:
...
12) subversion/1.7.7
> which svn
/usr/bin/svn
> svn --version
svn, version 1.7.7 (r1393599)
...
Running in the script, I get:
> ./setup_svn.sh
Currently Loaded Modulefiles:
...
12) subversion/1.7.7
/usr/bin/svn
svn, version 1.6.17 (r1128011)
...
Further edit
It seems that if I start a new shell, I also get the same issues:
> bash
> module list
Currently Loaded Modulefiles:
...
12) subversion/1.7.7
> svn --version
svn, version 1.6.17 (r1128011)
I think I'll find out what our module system does to the environment and use that to work out what's going wrong.
I know it will not correct the issue, but I think the head command is useless. You can use the --quiet option.
$ svn --version --quiet
1.7.5
EDIT:
If it's a semicolon issue as mentioned by konsolebox, you can also use this syntax :
echo $(svn --version --quiet | grep -q '1.7')$?
It could be how the shell handles the semicolon. Try to place in two lines.
#!/bin/sh
svn --version | head -n1 | grep -q '1.7'
echo "$?"
If you're in bash also, make sure that pipefail is not enabled:
#!/bin/bash
shopt -u -o pipefail
svn --version | head -n1 | grep -q '1.7'
echo "$?"

How to check the ls version

This topic is about the util 'ls'
The BSD version uses the parameter '-G' to color up the output,
while the Linux version uses parameter '--color'
Also the environment variable to set the colors is different:
BSD: $LSCOLORS
Linux: $LS_COLORS
But now the problem is: I want to determine which version is installed (using a small Shell script), so I can set alias ls and the environment appropriate in my .bachrc file.
As I mentioned above this seems to me to be the handiest method
if ls --color -d . >/dev/null 2>&1; then
GNU_LS=1
elif ls -G -d . >/dev/null 2>&1; then
BSD_LS=1
else
SOLARIS_LS=1
fi
I've essentially this in my l script, which I use on various platforms to tweak ls output as I like
Just run 'ls' and see whether it throws an error, e.g. on my mac:
$ ls --color 1>/dev/null 2>&1
$ echo $?
1
Whereas
$ ls -G 1>/dev/null 2>&1
$ echo $?
0
Indicating -G is supported, but --color is not.
Ironically, the --version switch Kimmo mentions is not supported on most BSD systems :-)
Writing a portable configuration file for your particular setup can be a Herculean task. In your case, if you're sure your .bashrc is going to be used only on GNU/Linux and on a BSD system, you can check for switches that exist in one of the ls' but not in the other: for example, -D doesn't seem to be an accepted switch by ls on my BSD machines (FreeBSD and Mac OS X), whereas it is for GNU ls. Conversely, -P is accepted on BSD, but not on GNU/Linux. Knowing this, you can distinguish between the two ls' and set up environment variables accordingly.
$ ls --version
ls (GNU coreutils) 6.10
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Richard Stallman and David MacKenzie.
combining the methods described, here's an easy way to use a bash function instead of an alias in order to make colors work regardless of if you are using BSD or GNU ls.
ll () {
if ls --version &>/dev/null; then
ls --color=auto -lahtr
else
ls -Glahtr
fi
}
inspired by a particular conda env recipe pulling in GNU ls on my macOS system where my ls aliases were all hard-coded for stock BSD ls only.

Resources