I have been trying to use GNU parallel for some time, but I have never been able to get it to function at all!
For example, running (in a non-empty directory!):
ls | parallel echo # Outputs single new line
ls | parallel echo echo echo # Outputs three new lines.
ls | parallel echo {} # /bin/bash: {}: command not found
ls | parallel echo '{}' # /bin/bash: {}: command not found
ls | parallel 'echo {}' # Outputs: {}
ls | parallel -IMM 'echo MM' # Outputs: MM
It seems that it is simply executing each argument as a command, which makes no sense.
I have tried bash, zsh, tcsh, csh, and sh, to no avail.
As I was about to complete writing this question, I ran parallel --version to report the version, only to find:
WARNING: YOU ARE USING --tollef. IF THINGS ARE ACTING WEIRD USE --gnu.
It is not clear to me why that flag is set by default. Needless to say, using --gnu worked!
Thought I would post this to save someone hours of frustration and confusion.
EDIT:
To fix this permanently (in Ubuntu at least), delete the --tollef flag in /etc/parallel/config
Depending on your operating system, you should check whether you're actually running the GNU version.
$ parallel --version
parallel: invalid option -- '-'
parallel [OPTIONS] command -- arguments
for each argument, run command with argument, in parallel
parallel [OPTIONS] -- commands
run specified commands in parallel
If this is the case, you're not running the GNU version. Ubuntu 12.04 is like this, and you'll need to manually install GNU parallel to get the functionality you expect.
Had issues running parallel as an external command from FREEMAT (MATLAB lookalike); the argumentFile was not fed to the command properly solved it by:
Adding --gnu to options
Not using cmdString syntax involving ["]
Code:
cmdString = 'parallel --gnu command ::: ';
while j<=jLength
cmdString = [cmdString argumentFilePath(j,:) ' '];
j=j+1;
end
system(cmdString)
Thank you for that :)
Im on Ubuntu 12.04 as well.
For me it was same issue but different problem. Just running parallel command was exiting silently. Also parallel --version was saying invalid option error. In my Path there was just one parallel executable binary but still it was not detecting.
I was able to fix it as below:
Run whereis parallel. This gives all the paths where executables named parallel is present. For my case there was just one path /usr/local/bin/parallel. Running using this path works just fine.
You can add an alias for this in ~/.bashrc or ~/.zshrc file like alias parallel='/usr/local/bin/parallel'
And now parallel works like charm.
dev-dsk % parallel --version
GNU parallel 20190322
Copyright (C) 2007-2019 Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.
Related
my gnu bash scripts
GNU bash, version 5.1.0(1)-release (x86_64-apple-darwin19.6.0)
on macos hang when they contain a heredoc with greater than 512 characters,
e.g. the USAGE heredoc below works unless I add 1 more character to it
cat <<'USAGE'
--all List all tasks, TASK_IDs will be ignored
--name NAME Only list tasks with specified NAME
--logs list log messages
--pending Only list tasks that have not been scheduled
--active same as --pending
--scheduled Only List tasks that have been scheduled, whether running or finished
--running Only List tasks that are currently executing / running
--finished Only List tasks that have been run, i.e., have finished
12345678901234567890
USAGE
note: there are no variable expansions, quotes, etc. just literal text.
If i break all of the text into multiple heredocs they all work ... but if i combine them in anyway to create a heredoc with >512 characters bash hangs
what am i doing wrong?
Well - the problem has disappeared. Maybe related to recent bash upgrade to
GNU bash, version 5.1.4(1)-release (x86_64-apple-darwin19.6.0)
which did have changes to heredoc processing related to size of heredoc wrt buffer sizes.
The same thing recently started happening to me. I recognize that this isn't a very satisfying answer, but I just switched back to bash 3.2.57(1)-release, that comes pre-installed with mac. (I spent several hours trying to figure out how to use homebrew to rollback to an earlier version of bash, but, as of 2020, this no longer seems to be a supported feature.)
I ran chsh -s /bin/bash, to change my default shell to the Mac-default bash.
I then then re-arranged my path so that it would find /bin/bash before /usr/local/bin/bash. (If a script has a #!/usr/bin/env bash shebang, it finds the older version.)
I have encounter the same issue. With help of a colleague we trace a quite odd behaviour in one of our tools to this exact issue, when the string to pipe into <<< exceeds 512 characters the command hangs.
We both have the same environment, it works for him it does not for me.
macOS Catalina 10.15.7
Bash installed with brew
Bash version GNU bash, version 5.1.8(1)-release (x86_64-apple-darwin19.6.0)
Kernel Darwin Kernel Version 19.6.0
When using Mac included Bash it works (GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19) but I cannot use that version.
This is driving us nuts.
Experienced this with bash version 5.1.16(1)-release (x86_64-apple-darwin21.1.0) and fixed via reboot.
This is not really an answer to the question but a suggestion to simplify the code:
echo "
--all List all tasks, TASK_IDs will be ignored
--name NAME Only list tasks with specified NAME
--logs list log messages
--pending Only list tasks that have not been scheduled
--active same as --pending
--scheduled Only List tasks that have been scheduled, whether running or finished
--running Only List tasks that are currently executing / running
--finished Only List tasks that have been run, i.e., have finished
12345678901234567890 "
It will do the same but in a easier way.
In which file are stored the info about the command
bash --version
?
I've tried in some directories like /bin or /var, /etc, but without results...
I need both for Linux and MAC OS x. Thanks!
Do you have any reason to believe that it is not hard-coded in the bash-executable itself?
To find out which bash is used, run
which bash
For me, it gives /bin/bash. Now, to see whether the executable has any string-like byte sequences in it that contain the substring "version", run
strings /bin/bash | grep version
On this machine, it gives:
shell_version_string
build_version
sccs_version
rl_library_version
rl_do_lowercase_version
show_shell_version
rl_readline_version
dist_version
GNU bash, version %s-(%s)
GNU bash, version %s (%s)
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
#(#)Bash version 4.3.48(1) release GNU
display-shell-version
-l do not print tilde-prefixed versions of directories relative
HOSTTYPE The type of CPU this version of Bash is running under.
OSTYPE The version of Unix this version of Bash is running on.
version, type `enable -n test'.
do-lowercase-version
.gnu.version
.gnu.version_r
Thus, "Bash version 4.3.48" is hard-coded in the executable as an ordinary string.
strings $(which bash) | grep "Bash version"
Output (e.g.):
#(#)Bash version 4.3.48(1) release GNU
I built a toolchain script to prepare a Linux build environment. The script can be found here: https://github.com/mynameismevin/prometheus/blob/toolchain/ptool-make.sh
The script runs perfectly until after the Perl section around line 416. After Perl is done, when it goes to unzip sed, it complains with this error:
tar (child): sed-4.2.2.tar.bz2: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
However, if I run the sections by hand, it completes without errors. If I split the script into ptool-make1.sh (which ends at Perl) and ptool-make2.sh (which starts at sed) and run them sequentially then they both complete without any issues at all. At this point, I assert the issue isn't with the script, and I would like to debug the shell to see if it's an issue with the shell.
Here are some useful configurations:
user#ubuntu:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
user#ubuntu:~$ bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
user#ubuntu:~$ ls -lh $(which bash)
-rwxr-xr-x 1 root root 998K Oct 7 2014 /bin/bash
I don't think Ubuntu bash comes with debugging symbols, so I would assume I would have to recompile to include those?
How would I use gdb to debug the shell when I run a script? Or how would I have the shell log to a file while the script runs so I can open it with gdb after it's done? I know how to debug a shell script, I don't want to debug the script, I want to debug the shell.
Edit: It doesn't look like Ubuntu bash comes with debugging symbols out of the box. Reading symbols from bash...(no debugging symbols found)...done.
Edit: $PROMETHEUS is set in my root shell. At the end of Perl cd .. results in the same results a cd $PROMETHEUS/sources/.
Shells come with their own debugging tools. The most simple is running them with -x (bash -x ...script...) which will print each command after variable expansion but before it is executed. That's usually enough to determine the problem.
For more ideas, see How to debug a bash script?
You should also consider to write helper functions to reduce the size of the script to just a few lines. You could move special options to configure or post-build steps into extra files and run them from the helper function if they exist.
Looking at the code, it seems that this line is the culrit:
cd $PROMETHEUS/sources/
everywhere else, you just use cd ... If PROMETHEUS isn't defined (the script doesn't define it, that becomes cd /sources/ which should also fail but doesn't abort your script. Try
cd $PROMETHEUS/sources/ || exit 1
instead. Also #!/bin/bash -u might be useful (abort on undefined variables).
Lastly, you can use pushd and popd to navigate the folders.
While Googling for another bash tip, I found this page on .inputrc http://ss64.com/bash/syntax-inputrc.html, which exactly fixed what I was trying to do (put a trailing / on symlinks to directories), but as I read further, I saw that there was a way to skip completed text as well. "Awesome", I said to myself, "this bugs me about twice a week". So, I put that into my .inputrc as well, but it doesn't work, or even seem to recognize what's going on. After logging out, and logging back in, this is what I'm seeing:
$ tail .inputrc
set mark-symlinked-directories on
set skip-completed-text on
$ bash --version
GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin10.0)
Copyright (C) 2007 Free Software Foundation, Inc.
$ bind -V | grep completed
$ bind -V | grep skip
$
That is, bash doesn't seem to recognize it's there at all... Is there something wrong with the version of bash supplied in Mountain Lion? Anything blindingly obvious I've forgotten to do?
skip-completed-text is new to bash version 4.
The: skip-completed-text is only in bash4.
On my Mac:
$ bash --version
GNU bash, version 4.2.37(2)-release (i386-apple-darwin12.1.0)
If you want install some opensource utilities i recommending you using macports (www.macports.org). Installing macports will give you one command called port and with it you can install things dead simply, like:
$ port install bash
will install bash 4 for you.
Bash v4 won't make it to OSX. Might I suggest zsh, which is not GPL3.
I've got this Makefile that is presenting some odd behaviour:
$ javac -d Classes -sourcepath .. -classpath `for x in \`ls Classes/jars/*\`; do echo -n $x:; done` PCA/PCAClassifier.java
compiles the java just fine. But for somereason when I call
make PCA
I get:
Compiling PCAClassifier
javac -d Classes -sourcepath .. -classpath `for x in \`ls Classes/jars/*\`; do echo -n $x:; done` PCA/PCAClassifier.java
javac: invalid flag: Classes/jars/Jama.jar:
Usage: javac <options> <source files>
use -help for a list of possible options
make: * [Classes/RobotSuite/PCA/PCAClassifier.class] Error 2
I am so confused. Anyone have a solution?
Make version info:
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin10.0
Background: Working on a group project. I'm running Ubuntu, my partner is running Mac OS X. For whatever reason, This makefile works just fine on my computer, but not on his. Even though the command works in his BASh terminal, somehow Make isn't sending it correctly.
You have given us almost no information to go on. Nonetheless, you are in luck!
On Mac OS X, bash has been built with --enable-strict-posix-default and hence the xpg_echo shell option defaults to being on in POSIX mode. POSIX mode is on when the shell has been invoked as /bin/sh, as it has been when it is invoked by Make, unless you instruct it otherwise by setting the $(SHELL) make variable (which you probably shouldn't).
This is the difference between Linux and Mac OS that is killing your makefile. When xpg_echo is on, the shell's built-in echo treats -n as just another argument to be printed (and hence also prints a newline). So the single classpath argument that you're trying to construct ends up as a bunch of separate arguments (half of which are "-n") and javac gets confused.
(This doesn't happen on the command line, even on Mac OS X, because then the shell has been invoked as /bin/bash, so it's not in POSIX mode and xpg_echo is off.)
So you have a number of options for fixing this:
Use /bin/echo -n in your shell snippet; unlike the built-in one, the real echo command these days mostly always understands -n;
Construct the classpath argument in a less roundabout way than your shell for loop; for example
... -classpath `ls Classes/jars/* | tr '\n' :` ...
or, if you are already assuming GNU Make, using make wildcards and functions instead of a shell snippet;
Have your colleague add shopt -u xpg_echo to an appropriate bash startup file on their machine (however this will just lead to future confusion when your next Mac OS-using colleague comes along and you've all long since forgotten how you fixed this this time).
Finally, a general note about debugging makefiles: when a javac ... command in a makefile recipe is giving incomprehensible error messages, change it to echo javac ... instead. Then you'll be able to see exactly how it's being invoked -- which, as seen here, may not be what you intended.