I want to test whether pbsdsh works or not in my system. I use the following programme
I am using a linux server with two 64-bit quad-core Intel Xeon (24 compute nodes
try.sh
#!/bin/bash
# Set up the path
PATH=/usr/local/bin:$PATH
export PATH
# Make all hosts print out "Hello World"
pbsdsh echo Hello World
then I issue the command
qsub -l nodes=2 try.sh
I got the following error message in try.sh.e2013
/var/spool/torque/mom_priv/jobs/2013.hpc.SC: line 8: pbsdsh: command not found
This error message is misleading as pbsdsh does exist in the system.
Could anyone help me out with this?
Thanks
"This error message is misleading as
pbsdsh does exist in the system."
Does it exist in PATH? And, in particular, does it exist in PATH on the compute nodes on which it's being run? Keep in mind that the $PATH variable in the environment of this script may be different than in an interactive shell on the head node; you can find this out by echoing $PATH in your script. Try submitting an interactive job
$ qsub -I -l nodes=2
and then, when you have an interactive shell on one of the nodes, find pbsdsh and make sure it's where you think it is.
Related
In my .profile (SUSE Linux with Korn shell) I have had the following code active for many years:-
case $0 in
-ksh|ksh)
set -o ignoreeof
set -a
set -o vi
PS1=$(print '\033[34m$(tput bold)ksh:$(hostname)->$(tput sgr0)\033[00m ')
PS2="continue-> "
PS3=": "
PS4="$0.$LINENO+ "
FCEDIT=/usr/bin/vi
HISTFILE=~/.histories/${TTY}_$(hostname)_ksh_his
HISTSIZE=500
EDITOR=/bin/vi
VISUAL=/bin/vi
TERM=gnome
set +a ;;
*) SHELL=${SHELL} ;;
But lately that PS1 entry has been causing some unexpected problems. When I source a file of ~3K environment variables and then try to get onto to a UGE compute host, the following error causes everything to break down:-
$ qrsh -V -j y -pe mt 1 -l "os_version=SUSE12.0,model=EMT3500,cpu_code=E5-2667v4" -P iheavy -now no
ksh: /usr/bin/tput: Argument list too long
ksh: /bin/hostname: Argument list too long
ksh: /usr/bin/tput: Argument list too long
ksh:->
If I request a CentOS system (os_version=CS7.0) I do not see the error--it is specific to SUSE. Also, if I eliminate the PS1 entry altogether, I can get onto a SUSE system without any errors. This is the simplest way to capture the bigger problem: when I issue a qsub for batch computing tasks, my .profile ends up partially initialized and jobs fail to launch.
I often change environments and shells, so color-coding my terminal prompt in a shell-specific and host/queue-specific way has always been helpful. I would rather not retire that PS1 entry just for one project that has such a long list of environment variable names.
I have done various searches and learned that increasing ulimit settings can sometimes help in situations like this; however, I tried that (increased stack size and number-of-open-files to their maximum) and the outcome did not change.
Is there a practical way to avoid this problem without removing the PS1 entry?
I am trying to run a sh file on git bash on Windows 10. I am even seeing the statements which I don't want to be shown like variable assignment in the console. But the same stuff I can't see on either Windows Subsystem for Linux(WSL) or other linux distributions. It becomes a bit annoying to see all the code lines in the sh file when trying to run that, when I am only interested in seeing the statements which I want to echo. Can someone please help in this ?
Sample sh file :
#!/usr/bin/env sh
VARIABLE="OK"
echo $VARIABLE
Output on git bash:
Output on WSL(or any other Linux Systems):
That only happens when both of these occur:
set -x is used
PS4 is set
If you fix either one, the messages will go away.
If you want to fix number one, you can do set +x, or you can find the file that
is calling set -x and remove that line. Probably ~/.bash_profile or
~/.bashrc.
If you want to fix number two, you can do PS4=, and you can add it to one of
those files above to persist if you want.
This question already has answers here:
How to obtain the first letter in a Bash variable?
(7 answers)
Closed 3 years ago.
I am trying to my a custom terminal command. I just learned I am supposed to do it using the Unix script? I don't really know much of what that is and am still trying to figure it out. What I do know is that $1 is an arg is it possible to make it a variable and then get the first letter like you could in python?
EX:
str = 'happy'
str[0] = 'h'
You're asking a few different things here.
I am trying to my a custom terminal command.
That could mean a few different things, but the most obvious meaning is that you want to add an executable to your path so that when you type it at the terminal, it runs just like any other executable on your system. This requires just a few things:
the executable permission must be set.
the file must specify how it can be executed. For interpreted programs such as bash scripts or python scripts, you can do so by beginning the file with a "shebang line" that specifies the interpreter for the file.
the file must be in one of the locations specified by your $PATH.
I just learned I am supposed to do it using the Unix script?
there's no such thing as a "unix script", but what you seem to be referring to is a "shell script". Though these are commonly associated with unix, they're no more inherently a unix script than any other language. A shell, such as bash, sh, or any other, is just an interpreted language that is designed so that it is convenient to be used interactively by a human as well as being programmatically executed as part of a saved file.
I don't really know much of what that is and am still trying to figure it out.
Let's get into some specifics.
First I edit a file called 'hello-world' to contain:
#!/bin/bash
echo "Hello, world!"
Note that this filename has no "extension". Though heuristics based on file extension are sometimes used (espeically in windows) to determine a file type, unix typically sees a file "extension" as part of the arbitrary file name. The thing that makes this a potentially executable bash script is the specification of that interpreter on the shebang line.
We can run our script right now from bash, just as we could if we wrote a python script.
$ bash hello-world
hello, world!
To make the bash implicit, we mark the file as executable. This enables the linux operating system to consult the beginning "magic bytes" of the file to determine how to run it. Thes beginning bytes might signify an ELF file (a compiled executable, written in eg C, C++, or go). Or, it might be #! which just so happens means , "read the rest of this first line to determine the command to run, and pass the rest of this file into that command to be interpreted.
$ chmod +x hello-world
ls -l will show us the "permissions" on the file (more accurately called the "file mode", hence chmod rather than chperm) . The x stands for executable, so we have enabled the use of the leading bytes to determine method of execution. Remember, the first two bytes of this file, and the rest of that first line, then specify that this file should be "run through bash" so to speak.
$ ls -l hello-world
-rwxr-xr-x 1 danfarrell staff 33 Dec 27 20:02 hello-world
Now we can run the file from the current directory:
$ ./hello-world
hello, world!
At this point, the only difference between this command and any other on the system, is that you have to specify its location. That's because my current directory is not in the system path. In short, the path (accessible in a unix shell via the $PATH variable) specifies an ordered list of locations that should be searched for a specified command whose location is not otherwise specified.
For example, there's a very common program called whoami. I can run it directly from my terminal without specifying a location of the executable:
$ whoami
danfarrell
This is because there's a location in my $PATH in which the shell was able to find that command. Let's take a closer look. First, here's my path:
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin
And there's also a convenient program called whereis which can help show which path elements supply a named executable:
$ whereis whoami
/usr/bin/whoami
Sure enough, whoami is in one of the elements of the $PATH. (Actually I shared a simplified $PATH. Yours might be somewhat longer).
Finally, then, we can get to the last thing. If I put hello-world in one of the $PATH elements, I will be able to invoke it without a path. There are two ways to do this: we can move the executable to a location specified in the path, or we can add a new location to the path. For simplicity's sake I'll choose the first of these.
$ sudo cp hello-world /usr/local/bin/
Password:
I needed to use sudo to write to /usr/local/bin because it's not accessible as my user directly - that's quite standard.
Finally, I've achieved the goal of being able to run my very important program from any location, without specifying the executable's location.
$ hello-world
hello, world!
$ which hello-world
/usr/local/bin/hello-world
It works! I've created what might be described as a "custom terminal command".
What I do know is that $1 is an arg is it possible to make it a variable and then get the first letter like you could in python?
Well, one option would be to simply write the custom terminal command in python. If python is available,
$ which python
/usr/bin/python
You can specify it in a shebang just like a shell can be:
#!/usr/bin/env python
print("hello, world!"[0])
$ hello-world
h
it works!
Okay, confession time. I actually used #!/usr/bin/env python, not /usr/bin/python. env helps find the correct python to use in the user's environment, rather than hard coding one particular python. If you've been using python during the very long running python 2 to python 3 migration, you can no doubt understand why I"m reticent to hard code a python executable in my program.
It's certainly possible to get the first letter of a string in a bash script. But it's also very possible to write a custom command in a program other than shell. Python is an excellent choice for string manipulation, if you know it. I often use python for shell one-liners that need to interact with json, a format that doesn't lend itself well to standard unix tool stream editing.
Anyway, at the expense of incurring SO community's ire by reanswering an "already answered" question, I'll include a version in shell (Credit goes to David C Rankin)
#!/bin/bash
echo "${1:0:1}"
$ hello-world hiworld
h
A formely working bash script no longer works after switching computers. I get the following error:
No such file or directory.
Before going on, please excuse any mistakes you may find since english is not my native language.
The script was used in cygwin under Windows XP. I now had to switch to cygwin64 under Windwos 7 (64bit).
The script is used as a checkhandler for the program SMSTools3 to split a file with a specific format into multiple smaller ones, which the program then uses to send SMS to multiple recipients. The script was copied directly from the page of SMSTools3 and uses the package formail.
After looking up the error the most likely problem was that the environmantle path was not set up to look in the right path (/usr/bin). I therefore added it to the path but to no avail.
I then deleted other entries in the enviromental path of windows which contained spaces because that could have been another explanation, but again to no avail.
Following is a minimal example of the code which produces the error.
#!/bin/bash
# Sample script to allow multiple recipients in one message file.
# Define this script as a checkhandler.
echo $PATH
which formail
outgoing="/var/spool/sms/outgoing"
recipients=`formail -zx "To:" < "$1"`
I added the lines the lines echo $Path and which formail to show if the script can find the correct file. Both results look fine, the second command gives me the right output '/usr/bin/formail'
But the line recipients=... throws me the error:
No such file or directory.
I do not have much experience with bash scripting, or cygwin in general. So if someone on this wonderful board could help me solve this problem, I would be really grateful. Thank you all for your help.
EDIT:
First of all thank you all for your comments.
Secondly, I would like to apologize for the late reply. The computer in question is also used for other purposes and my problem is part of a background routine, so I have to wait for "free time" on the pc to test things.
For the things #shellter pruposed: The ls command returned an error: '': No such file or directory.
The which -a formail as well as the echo $(which -a formail) commands that #DougHenderson pruposed returned the 'right' path of /usr/bin/formail. echo \$1 = $1 before the recipent line returned the path to the checkhandler file (/usr/local/bin/smsd_checkhandler.sh), the same command after the recipent line seems to show a empty string ($1 = ). Also, the pruposed change to the recipent line did not change the error.
For the dos2unix conversion that #DennisWilliamson pruposed, I opened the file in notepad++ to use their build in converion, but it showed me that the file is in unix format with Unix style line endings.
This keeps happening to me all the time:
1) I write a script(ruby, shell, etc).
2) run it, it works.
3) put it in crontab so it runs in a few minutes so I know it runs from there.
4) It doesnt, no error trace, back to step 2 or 3 a 1000 times.
When I ruby script fails in crontab, I can't really know why it fails cause when I pipe output like this:
ruby script.rb >& /path/to/output
I sorta get the output of the script, but I don't get any of the errors from it and I don't get the errors coming from bash (like if ruby is not found or file isn't there)
I have no idea what environmental variables are set and whether or not it's a problem. Turns out that to run a ruby script from crontab you have to export a ton of environment variables.
Is there a way for me to just have crontab run a script as if I ran it myself from my terminal?
When debugging, I have to reset the timer and go back to waiting. Very time consuming.
How to test things in crontab better or avoid these problems?
"Is there a way for me to just have crontab run a script as if I ran it myself from my terminal?"
Yes:
bash -li -c /path/to/script
From the man page:
[vindaloo:pgl]:~/p/test $ man bash | grep -A2 -m1 -- -i
-i If the -i option is present, the shell is interactive.
-l Make bash act as if it had been invoked as a login shell (see
INVOCATION below).
G'day,
One of the basic problems with cron is that you get a minimal environment being set by cron. In fact, you only get four env. var's set and they are:
SHELL - set to /bin/sh
LOGNAME - set to your userid as found in /etc/passwd
HOME - set to your home dir. as found in /etc/passwd
PATH - set to "/usr/bin:/bin"
That's it.
However, what you can do is take a snapshot of the environment you want and save that to a file.
Now make your cronjob source a trivial shell script that sources this env. file and then executes your Ruby script.
BTW Having a wrapper source a common env. file is an excellent way to enforce a consistent environment for multiple cronjobs. This also enforces the DRY principle because it gives you just one point to update things as required, instead of having to search through a bunch of scripts and search for a specific string if, say, a logging location is changed or a different utility is now being used, e.g. gnutar instead of vanilla tar.
Actually, this technique is used very successfully with The Build Monkey which is used to implement Continuous Integration for a major software project that is common to several major world airlines. 3,500kSLOC being checked out and built several times a day and over 8,000 regression tests run once a day.
HTH
'Avahappy,
Run a 'set' command from inside of the ruby script, fire it from crontab, and you'll see exactly what's set and what's not.
To find out the environment in which cron runs jobs, add this cron job:
{ echo "\nenv\n" && env|sort ; echo "\nset\n" && set; } | /usr/bin/mailx -s 'my env' you#example.com
Or send the output to a file instead of email.
You could write a wrapper script, called for example rbcron, which looks something like:
#!/bin/bash
RUBY=ruby
export VAR1=foo
export VAR2=bar
export VAR3=baz
$RUBY "$*" 2>&1
This will redirect standard error from ruby to the standard output. Then you run rbcron in your cron job, and the standard output contains out+err of ruby, but also the "bash" errors existing from rbcron itself. In your cron entry, redirect 2>&1 > /path/to/output to get output+error messages to go to /path/to/output.
If you really want to run it as yourself, you may want to invoke ruby from a shell script that sources your .profile/.bashrc etc. That way it'll pull in your environment.
However, the downside is that it's not isolated from your environment, and if you change that, you may find your cron jobs suddenly stop working.