I am learning DEBUG signal of bash.
The followings are my test code to regenerate the phenomenon of my question. So it does not have much meaning. Please don't care details.
It prepares two traps, one is called by EXIT signal to clean up temporal script originally but it is a dummy function here. And the second one is called by DEBUG signal to calculate the line where the debugger is scanning.
My question is DEBUG signal may be dispached at clean_up_debugger with LINENO = 0. Why is the LINENO 0 at that time? I add detail of my question after following output by bash -x. Please tell me why it happens.
Thank you very much.
#!/bin/bash
# file name is debug.working
source "bash_debugger_functions.sh"
trap clean_up_debugger_func EXIT
no_of_line_until_here=${LINENO} # *** no_of_line_until_here is 12 ***
trap "show_line_scanned \$(( \${LINENO} - ${no_of_line_until_here} - 1 ))" DEBUG
#!/bin/bash
echo "echo_sring = $1"
The following is a library file
#!/bin/bash
# file name is 'bash_debugger_functions.sh'
clean_up_debugger_func() {
echo "dummy"
}
show_line_scanned() {
echo "At line $1"
}
The following is a part of output by bash -x debug.working
+ (debug.working:17): echo 'echo_sring = test_message'
++ (debug.working:1): show_line_scanned -12
Just after "echo 'echo_sring = test_message'" is called, show_line_scanned is called with negative value, -12. no_of_line_until_here is +12. So it seems LINENO is 0 at that time. I don't know why the show_line_scanned is called here because I supposed that DEBUG signal is dispatched at each line but there is no new line after "echo "echo_sring = $1"". And I would like to know why LINENO is 0 here.
Please teach me the mechanism here.
I supposed that DEBUG signal is dispatched at each line but there is no new line after "echo "echo_sring = $1"".
Still there is a line executed after the last line of your debug.working script, because you set up the trap clean_up_debugger_func EXIT command, and for that clean_up_debugger_func command you get the DEBUG command dispatch which puzzles you.
And I would like to know why LINENO is 0 here.
The execution of the script has just ended at this time, and man bash states about LINENO:
When not in a
script or function, the value substituted is not guaranteed to
be meaningful.
Related
I'm trying to find a way to emulate the behavior of set -e in a function, but only within the scope of that function.
Basically, I want a function where if any simple command would trigger set -e it returns 1 up one level. The goal is to isolate sets of risky jobs into functions so that I can gracefully handle them.
If you want any failing command to return 1, you can achieve that by following each command with || return 1.
For instance:
false || return 1 # This will always return 1
I am a big fan of never letting any command fail without explicit handling. For my scripts, I am using an exception handling technique where I return errors in a way that is not return codes, and trap all errors (with bash traps). Any command with a non-zero return code automatically means an improperly handled situation or bug, and I prefer my scripts to fail as soon as such a situation occurs.
Caution: I highly advise against using this technique. If you run the function in a subshell environment, you almost get the behavior you desire. Consider:
#!/bin/bash
foo() ( # Use parens to get a sub-shell
set -e # Does not impact the main script
echo This is executed
false
echo This should *not* be executed
)
foo # Function call fails, returns 1
echo return: $?
# BUT: this is a good reason to avoid this technique
if foo; then # Set -e is invalid in the function
echo Foo returned 0!!
else
echo fail
fi
false # Demonstrates that set -e is not set for the script
echo ok
Seems like you are looking for "nested exceptions" somewhat like what Java gives. For your requirement of scoping it, how about doing a set -e at the beginning of the function and making sure to run set +e before returning from it?
Another idea, which is not efficient or convenient, is to call your function in a subshell:
# some code
(set -e; my_function)
if [[ $? -ne 0 ]]; then
# the function didn't succeed...
fi
# more code
In any case, please be aware that set -e is not the greatest way to handle errors in a shell script. There are way too many issues making it quite unreliable. See these related posts:
What does set -e mean in a bash script?
Error handling in Bash
The approach I take for large scripts that need to exist for a long time in a production environment is:
create a library of functions to do all the standard stuff
the library will have a wrapper around each standard action (say, mv, cp, mkdir, ln, rm, etc.) that would validate the arguments carefully and also handle exceptions
upon exception, the wrapper exits with a clear error message
the exit itself could be a library function, somewhat like this:
--
# library of common functions
trap '_error_handler' ERR
trap '_exit_handler' EXIT
trap '_int_handler' SIGINT
_error_handler() {
# appropriate code
}
# other handlers go here...
#
exit_if_error() {
error_code=${1:-0}
error_message=${2:-"Uknown error"}
[[ $error_code == 0 ]] && return 0 # it is all good
# this can be enhanced to print out the "stack trace"
>&2 printf "%s\n" $error_message
# out of here
my_exit $error_code
}
my_exit() {
exit_code=${1:-0}
_global_graceful_exit=1 # this can be checked by the "EXIT" trap handler
exit $exit_code
}
# simple wrapper for cp
my_cp() {
# add code to check arguments more effectively
cp $1 $2
exit_if_error $? "cp of '$1' to '$2' failed"
}
# main code
source /path/to/library.sh
...
my_cp file1 file2
# clutter-free code
This, along with effective use of trap to take action on ERR and EXIT events, would be a good way to write reliable shell scripts.
Doing more research, I found a solution I rather like in Google's Shell Style Guide. There are some seriously interesting suggestions here, but I think I'm going to go with this for readability:
if ! mv "${file_list}" "${dest_dir}/" ; then
echo "Unable to move ${file_list} to ${dest_dir}" >&2
exit "${E_BAD_MOVE}"
fi
I am trying to write a bash script that echoes values of 2 variables in parallel as part of an experiment before writing a shell script that generates files with numbers appended to them in parallel for a project of mine.
Here is the shell script:
#!/bin/bash
value1=0
value2=1
for i in $(seq 1 2); do
echo "Value 1 : " $((++value1)) &
echo "Value 2 : " $((++value2))
wait
echo "Wait"
done
And here is the output I get:
Value 2 : 2
Value 1 : 1
Wait
Value 2 : 3
Value 1 : 1
Wait
I know about GNU parallel and xargs but I don't want to use them.
I would like to know why 'value2' gets printed first and why 'value1' never gets incremented.
value1 is incremented, but in a separate process. value1 in the original process is not modified. There is no guarantee which value you will see printed first; the two echo statements occur in two different processes which are scheduled at the whim of the operating system.
This is a follow-up to my previous question on SO. I am still trying to command a script deepScript from within another script shallowScript and process its output before display on terminal. Here is a code sample:
deepScript.sh
#!/bin/zsh
print "Hello - this is deepScript"
read "ans?Reading : "
print $ans
shallowScript.sh
#!/bin/zsh
function __process {
while read input; do
echo $input | sed "s/e/E/g"
done }
print "Hello - this is shallowScript"
. ./deepScript.sh |& __process
(edited : outcome of this syntax and of 2 alternatives pasted below)
[UPDATE]
I have tried alternative syntaxes for last redirection . ./deepScript.sh |& __process and each syntax has a different outcome, but of course none of them is the one I want. I'll just paste each syntax and the resulting output of ./shallowScript.sh (where I typed "input" when read was waiting for an input), together with my findings so far.
Option 1 : . ./deepScript.sh |& __process
From this link, it seems that . ./deepScript.sh is run from a subshell, but not __process. Output:
zsh : ./shallowScript.sh
Hello - this is shallowScript
HEllo - this is dEEpScript
input
REading : input
Basically, the first two lines are printed as expected, then instead of printing the prompt REading :, the script directly waits for the stdin input, and then prints the prompt and executes print $ans.
Option 2: __process < <(. ./deepScript.sh)
Zsh's manpage indicates that (. ./deepScript.sh) will run as a subprocess. To me, that looks similar to Option 1. Output:
Hello - this is shallowScript
Reading : HEllo - this is dEEpScript
input
input
So, within . ./deepScript.sh, it prints read's prompt (script line 3) before the print (script line 2). Strange.
Option 3: __process < =(. ./deepScript.sh)
According to the same manpage, (. ./deepScript.sh) here sends its output to a temp file, which is then injected in __process (I don't know if there is a subprocess or not). Output:
Hello - this is shallowScript
Reading : input
HEllo - this is dEEpScript
input
Again, deepScript's line 3 prints to the terminal before line 2, but now it waits for the read to be complete.
Two questions:
Should this be expected?
Is there a fix or a workaround?
The observed delay stems from two factors:
deepScript.sh and process run asynchronously
read reads a complete line before returning
deepScript.sh writes the prompt to standard error, but without a newline. It then waits for your input while process continues to wait for a full line to be written so that its call to read can finish.
I would like to write a script to execute the steps outlined below. If someone can provide simple examples on how to modify files and search through folders using a script (not necessarily solving my problem below), I will greatly appreciate it.
submit job MyJob in currentDirectory using myJobShellFile.sh to a queue
upon completion of MyJob, goto to currentDirectory/myJobDataFolder.
In myJobDataFolder, there are folders
myJobData.0000 myJobData.0001 myJobData.0002 myJobData.0003
I want to find the maximum number maxIteration of all the listed folders. Here it would be maxIteration=0003.\
In file myJobShellFile.sh, at the last line says
mpiexec ./main input myJobDataFolder
I want to append this line to
'mpiexec ./main input myJobDataFolder 0003'
I want to submit MyJob to the que while maxIteration < 10
Upon completion of MyJob, find the new maxIteration and change this number in myJobShellFile.sh and goto step 4.
I think people write python scripts typically to do this stuff, but am having a hard time finding out how. I probably don't know the correct terminology for this procedure. I am also aware that the script will vary slightly depending on the queing system, but any help will be greatly appreciated.
Quite a few aspects of your question are unclear, such as the meaning of “submit job MyJob in currentDirectory using myJobShellFile.sh to a que”, “append this line to
'mpiexec ./main input myJobDataFolder 0003'”, how you detect when a job is done, relevant parts of myJobShellFile.sh, and some other details. If you can list the specific shell commands you use in each iteration of job submission, then you can post a better question, with a bash tag instead of python.
In the following script, I put a ### at the end of any line where I am guessing what you are talking about. Lines ending with ### may be irrelevant to whatever you actually do, or may be pseudocode. Anyway, the general idea is that the script is supposed to do the things you listed in your items 1 to 5. This script assumes that you have modified myJobShellFile.sh to say
mpiexec ./main input $1 $2
instead of
mpiexec ./main input
because it is simpler to use parameters to modify what you tell mpiexec than it is to keep modifying a shell script. Also, it seems to me you would want to increment maxIter before submitting next job, instead of after. If so, remove the # from the t=$((1$maxIter+1)); maxIter=${t#1} line. Note, see the “Parameter Expansion” section of man bash re expansion of the ${var#txt} form, and the “Arithmetic Expansion” section re $((expression)) form. The 1$maxIter and similar forms are used to change text like 0018 (which is not a valid bash number because 8 is not an octal digit) to 10018.
#!/bin/sh
./myJobShellFile.sh MyJob ###
maxIter=0
while true; do
waitforjobcompletion ###
cd ./myJobDataFolder
maxFile= $(ls myJobData* | tail -1)
maxIter= ${maxFile#myJobData.} #Get max extension
# If you want to increment maxIter, uncomment next line
# t=$((1$maxIter+1)); maxIter=${t#1}
cd ..
if [[ 1$maxIter -lt 11000 ]] ; then
./myJobShellFile.sh MyJobDataFolder $maxIter
else
break
fi
done
Notes: (1) To test with smaller runs than 1000 submissions, replace 11000 by 10000+n; for example, to do 123 runs, replace it with 10123. (2) In writing the above script, I assumed that not-previously-known numbers of output files appear in the output directory from time to time. If instead exactly one output file appears per run, and you just want to do one run per value for the values 0000, 0001, 0002, 0999, 1000, then use a script like the following. (For testing with a smaller number than 1000, replace 1000 with (eg) 0020. The leading zeroes in these numbers tell bash to fill the generated numbers with leading zeroes.)
#!/bin/sh
for iter in {0000..1000}; do
./myJobShellFile.sh MyJobDataFolder $iter
waitforjobcompletion ###
done
(3) If the system has a command that sleeps while it waits for a job to complete on the supercomputing resource, it is reasonable to use that command in place of waitforjobcompletion in the above scripts. Otherwise, if the system has a command jobisrunning that returns true if a job is still running, replace waitforjobcompletion with something like the following:
while jobisrunning ; do sleep 15; done
This will run the jobisrunning command; if it returns true, the shell will sleep for 15 seconds and then retest. Here is an example that illustrates waiting for a file to appear and then for it to go away:
while [ ! -f abc ]; do sleep 3; echo no abc; done
while ls abc >/dev/null 2>&1; do sleep 3; echo an abc; done
The second line's test could be [ -f abc ] instead; I showed a longer example to illustrate how to suppress output and error messages by routing them to /dev/null. (4) To reverse the sense of a while statement's test, replace the word while with until. For example, while [ ! -f abc ]; ... is equivalent to until [ -f abc ]; ....
Here's a simple version of my script which displays the failure:
#!/bin/bash
${something:="false"}
${something_else:="blahblah"}
${name:="file.ext"}
echo ${something}
echo ${something_else}
echo ${name}
When I echo the variables, I get the values I put in, but it also emits an error. What am I doing wrong?
Output:
./test.sh: line 3: blahblah: command not found
./test.sh: line 4: file.ext: command not found
false
blahblah
file.ext
The first two lines are being emitted to stderr, while the next three are being output to stdout.
My platform is fedora 15, bash version 4.2.10.
You can add colon:
: ${something:="false"}
: ${something_else:="blahblah"}
: ${name:="file.ext"}
The trick with a ":" (no-operation command) is that, nothing gets executated, but parameters gets expanded. Personally I don't like this syntax, because for people not knowing this trick the code is difficult to understand.
You can use this as an alternative:
something=${something:-"default value"}
or longer, more portable (but IMHO more readable):
[ "$something" ] || something="default value"
Putting a variable on a line by itself will execute the command stored in the variable. That an assignment is being performed at the same time is incidental.
In short, don't do that.
echo ${something:="false"}
echo ${something_else:="blahblah"}
echo ${name:="file.ext"}
It's simply
variable_name=value
If you use $(variable_name:=value} bash substitutes the variable_name if it is set otherwise it uses the default you specified.