Output showing too early - bash

I made a small Bash-script to make my life easier. But I encountered a problem which I can't fix.
What I want
I've made a small script which will check for php-errors in a file each time that file gets saved/changed. This is done without me needing to run a command each time. So I run the Bash-script once on my second screen, and than each time when I save my PHP-file on screen one; I get the eventual errors shown on screen two automatically.
Basic algorithm
Get the hash of the file
Compare it to it's previous hash
If it differs, the file is changed/saved: Check if there are errors using the php -l command
Print out the result from php -l
Problem:
The result from php -l gets printed out before my code asked for it.
Code
#!/bin/bash
#Declaring basic variables here
fileToCheck="$1"
oldHash=("")
checksum=("")
#Function to get a striped line as long as the terminal width
function lineAmount {
cwidth=`tput cols`
lines=""
for i in $(seq $(expr $cwidth - 33)); do lines="$lines-";done
echo $lines
}
#Function to show the actual error
function showError {
msg=$1
time=`date +"%c"`
l=$(lineAmount)
if [ "$msg" == "No" ]
then
msg="No errors detected."
fi
printf "\n\n$time $l \n$msg\n"
}
#Start-screen------------------------------------------------
printf "Starting session for $1 at $time \n$(lineAmount)\n"
if [ ! -f $1 ]
then
echo "[Error] File $1 not found."
exit
fi
printf "\n\n\n"
#------------------------------------------
#Printing out error when file changed
while true
do
sleep 0.6
checksum=($(sha256sum $fileToCheck))
checksum=${checksum[0]}
if [ "$checksum" != "$oldHash" ]
then
error=$(php -l $fileToCheck)
oldHash=$checksum
showError $error
fi
done
Test file (test.php):
<?php
function foo() {
}
?>
Output of script:
Starting session for /home/name/Desktop/test.php at
-----------------------------------------------
Thu 11 Aug 2016 08:16:15 PM CEST -----------------------------------------------
No errors detected.
Now, in test.php I delete line 4:
<?php
function foo() {
?>
This will of course give an error, and my script shows that error:
Starting session for /home/name/Desktop/test.php at
-----------------------------------------------
Thu 11 Aug 2016 08:16:15 PM CEST -----------------------------------------------
No errors detected.
PHP Parse error: syntax error, unexpected end of file in /home/name/Desktop/test.php on line 6
Thu 11 Aug 2016 08:19:37 PM CEST ----------------------------------------------------------------------------------
Parse
But like you can see, this is not a nice output.
PHP Parse error: syntax error, unexpected end of file in /home/name/Desktop/test.php on line 6 should be printed below the second dotted line. Not below "No errors found." (The first output).
Expected output:
Starting session for /home/name/Desktop/test.php at
-----------------------------------------------
Thu 11 Aug 2016 08:16:15 PM CEST -----------------------------------------------
No errors detected.
Thu 11 Aug 2016 08:19:37 PM CEST ----------------------------------------------------------------------------------
PHP Parse error: syntax error, unexpected end of file in /home/name/Desktop/test.php on line 6
I tried a lot, I tried to change my algorithm a bit, searched up a lot; but it ain't working.
I guess the problem is somewhere on line 51, or 29. But I really can't see what's wrong.
Thanks!

Here's a stripped down and simplified version of your problem:
Why does this print an error message immediately instead of assigning it to the variable?
$ error=$(php -l test.php)
PHP Parse error: syntax error, unexpected end of file in test.php on line 5
php -l prints error messages to stderr like a good Unix citizen should. $(..) only captures stdout.
If you want to capture stdout and stderr together, you can use:
error=$(php -l $fileToCheck 2>&1)
You should also quote your variables so that the message is passed as a single parameter, since you're currently throwing away most of it (shellcheck is helpful):
showError "$error"
Being a good citizen, php also returns a useful exit code, so instead of trying to match a "No" to see if it's successful, you can just check the status directly:
if error=$(php -l $fileToCheck 2>&1)
then
echo "No problems"
else
echo "It failed with these messages: $error"
fi

Related

Bash script not recognizing existing file

I have to write a simple bash script, which does the compilation of multiple sql scripts which can contain recursive references to the other sql scripts with Oracle SQLPlus convention, so in the end the result will be only one SQL file, containing all statements from all (possibly also recursively) referenced subscripts.
I have created a simple script called oracle-script-compressor.sh with following bash code:
#!/bin/bash
#
#
function err () {
echo $* > "/dev/stderr"
}
function replace_subscripts_placeholders() {
local PROCESSING_FILENAME=$1
echo "-- processing file $PROCESSING_FILENAME"
while read script_line; do
local script_line_NO_LEAD_SPACE="$(echo -e "${script_line}" | sed -e 's/^[[:space:]]*//')"
if [[ $script_line_NO_LEAD_SPACE == \#\#* ]] ; then
local file_name="./${script_line_NO_LEAD_SPACE#\#\#}"
echo "-- found reference to file $file_name"
echo "-- starting with processing, pwd=$PWD"
# for debug purposes:
ls -la
# but this returns always false:
if [ -f "$file_name" ]; then
# so this part is never started:
. replace_subscripts_placeholders $file_name
else
err_msg="WARNING: Could not find the referenced file $file_name"
echo "-- $err_msg"
err $err_msg
fi
else
echo "$script_line"
fi
done < $PROCESSING_FILENAME
}
if test -z "$1" ; then
err "Usage: oracle-script-compressor.sh {file_name_to_process} [> output_file]"
err " Be aware, if the referenced files within {file_name_to_process} are not specified by absolute paths, you have to start the script from corresponding directory."
err " If the part [> output_file] is omitted, then this script writes to standard output"
err " If the part [>> output_file] is used, then the result will be appended to output_file it it exists before the processing"
else
if [ -f "$1" ]; then
replace_subscripts_placeholders $1
else
echo "file $1 does not exist"
fi
fi
and I am stuck on interesting problem - it seems, inside of the function replace_subscripts_placeholders() the file check
if [ -f "$file_name" ]; then
. replace_subscripts_placeholders $file_name
else
err_msg="WARNING: Could not find the referenced file $file_name"
echo "-- $err_msg"
err $err_msg
fi
never goes to the recursion call and even, if I remove the if statement and really call the function in recursion with the correct referenced file name, which exists, it is still not recognized to be found and not passed into the loop over all lines of the referenced file in the recursive call (then the error file not found comes and the loop cannot be executed).
I have added the debug messages into script like mentioned above in the script, but still I am unable to find, why the hell should bash not find the file, if it is in the same directory. The scripts are placed in
user#mycomputer:/tmp/test$ ls -la
celkem 52
drwxr-xr-x 2 user user 4096 zář 14 21:47 .
drwxrwxrwt 38 root root 36864 zář 14 21:48 ..
-rw-r--r-- 1 user user 51 zář 14 21:45 a.sql
-rw-r--r-- 1 user user 51 zář 14 21:45 b.sql
-rw-r--r-- 1 user user 590 zář 14 21:46 start.sql
user#mycomputer:/tmp/test$
and the content of the file start.sql looks like this:
spool output__UpdSch.log
whenever sqlerror exit sql.sqlcode
--
--
PROMPT a.sql - starting
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as START_TIMESTAMP from dual;
##a.sql
PROMPT a.sql - finished
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as FINISH_TIMESTAMP from dual;
--
--
PROMPT b.sql - starting
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as START_TIMESTAMP from dual;
##b.sql
PROMPT b.sql - finished
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as FINISH_TIMESTAMP from dual;
--
--
spool off
and if I execute the script, it seems it decoded the filenames correctly, but there is still the problem in bash - the testing of the file existence returns always false:
user#mycomputer:/tmp/test$ ~/tmp/oracle-script-compressor.sh start.sql
-- processing file start.sql
spool output__UpdSch.log
whenever sqlerror exit sql.sqlcode
--
--
PROMPT a.sql - starting
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as START_TIMESTAMP from dual;
-- found reference to file ./a.sql
-- starting with processing, pwd=/tmp/test
celkem 52
drwxr-xr-x 2 user user 4096 zář 14 21:47 .
drwxrwxrwt 38 root root 36864 zář 14 21:48 ..
-rw-r--r-- 1 user user 51 zář 14 21:45 a.sql
-rw-r--r-- 1 user user 51 zář 14 21:45 b.sql
-rw-r--r-- 1 user user 590 zář 14 21:46 start.sql
-- WARNING: Could not find the referenced file ./a.sql
WARNING: Could not find the referenced file ./a.sql
PROMPT a.sql - finished
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as FINISH_TIMESTAMP from dual;
--
--
PROMPT b.sql - starting
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as START_TIMESTAMP from dual;
-- found reference to file ./b.sql
-- starting with processing, pwd=/tmp/test
celkem 52
drwxr-xr-x 2 user user 4096 zář 14 21:47 .
drwxrwxrwt 38 root root 36864 zář 14 21:48 ..
-rw-r--r-- 1 user user 51 zář 14 21:45 a.sql
-rw-r--r-- 1 user user 51 zář 14 21:45 b.sql
-rw-r--r-- 1 user user 590 zář 14 21:46 start.sql
-- WARNING: Could not find the referenced file ./b.sql
WARNING: Could not find the referenced file ./b.sql
PROMPT b.sql - finished
select to_char(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF3') as FINISH_TIMESTAMP from dual;
--
--
spool off
user#mycomputer:/tmp/test$
The script has read access to files, everything is in the same folder, where I start the script, It also does not matter, if I reference the file with the current directory prefix "./" or not, it is just never found. Interesting is, the check during the start of the script passes correctly, the only problem is the searching within the function... I have also tried to call the function with preceded "." and without it, it makes no difference... There are also no trailing spaces in the referenced file names... It also makes no difference if I declare the variables inside of the function as local or not.
So I have no idea, what could be the problem, maybe I just looked too long on it and cannot see something - seems it has to be something trivial, like the function is started in some different current directory - (but pwd and ls shows the directory is always correct...???) - Any help or pointers will be appreciated.
Thank you for the comments, it brought me to the solution. I found out, this problem was related to the fact, the test sql files were created on Windows, so they had in the end of line always the <CR><LF> characters. This leads to the fact in the original bash script I posted, the script line
while read script_line
puts into the variable not only the given file name from the line, preceded by the characters ##
##a.sql
but also the <CR> character - in the variable is then the value
##a.sql<CR>
what was the cause the file a.sql etc. could never be found. Of course the character is invisible, so therefore it has not been shown on any debug echoing I made here - I had to put the content of $file_name between some another characters and then i could see it in the echoed test... I made also some other corrections and the final working script looks in the following way, if somebody needs to join referenced SQL scripts into one, this does it:
#!/bin/bash
#
#
function err () {
echo $* > "/dev/stderr"
}
function replace_subscripts_placeholders() {
local PROCESSING_FILENAME=$1
echo "-- processing file $PROCESSING_FILENAME"
while read script_line || [ -n "$script_line" ]; do
local script_line_NO_LEAD_SPACE="$(echo -e "${script_line}" | sed -e 's/^[[:space:]]*//' | sed -e 's/^M$//')"
if [[ $script_line_NO_LEAD_SPACE == \#\#* ]] ; then
local file_name="${script_line_NO_LEAD_SPACE#\#\#}"
echo "-- found reference to file $file_name"
if [ -f "$file_name" ]; then
replace_subscripts_placeholders $file_name
else
err_msg="WARNING: Could not find the referenced file $file_name"
echo "-- $err_msg"
err $err_msg
fi
else
echo "$script_line"
fi
done < $PROCESSING_FILENAME
}
if test -z "$1" ; then
err "Usage: oracle-script-compressor.sh {file_name_to_process} [> output_file]"
err " Be aware, if the referenced files within {file_name_to_process} are not specified by absolute paths, you have to start the script from corresponding directory."
err " If the part [> output_file] is omitted, then this script writes to standard output"
err " If the part [>> output_file] is used, then the result will be appended to output_file it it exists before the processing"
else
if [ -f "$1" ]; then
replace_subscripts_placeholders $1
else
echo "file $1 does not exist"
fi
fi

Conditionals not being tested properly

I've got this script that runs every time my Ubuntu machine wakes up and goes to sleep.
#!/bin/sh
if [ "${1}" == "pre" ]; then
# Do the thing you want before suspend here, e.g.:
echo "we are suspending at $(date)..." > /home/albin/stuff/suspend_test
elif [ "${1}" == "post" ]; then
# Do the thing you want after resume here, e.g.:
echo "...and we are back from $(date)" >> /home/albin/stuff/suspend_test
else
echo ".2..neither pre or post . $1 . $(date)" >> /home/albin/stuff/suspend_test
fi
This is found in the output file:
.2..neither pre or post . pre . tis 11 sep 2018 20:44:42 CEST
.2..neither pre or post . post . tis 11 sep 2018 20:45:06 CEST
.2..neither pre or post . pre . tis 11 sep 2018 22:12:52 CEST
.2..neither pre or post . post . ons 12 sep 2018 06:55:21 CEST
.2..neither pre or post . pre . ons 12 sep 2018 06:55:22 CEST
.2..neither pre or post . post . ons 12 sep 2018 06:55:43 CEST
.2..neither pre or post . pre . ons 12 sep 2018 07:13:28 CEST
.2..neither pre or post . post . ons 12 sep 2018 07:14:00 CEST
I've checked multiple guides on how to write bash conditionals without finding any issues. The $1 variable is available but ignored in the if statements
You are using == instead of = inside [ ], but that's a bashism. Recent versions of Ubuntu use dash as /bin/sh, and it does not support ==; it gets an error, which is interpreted as the test failing:
$ if [ foo == foo ]; then echo "The strings match"; else echo "The strings do *not* match"; fi
dash: 1: [: foo: unexpected operator
The strings do *not* match
Solution: switch to =
$ if [ foo = foo ]; then echo "The strings match"; else echo "The strings do *not* match"; fi
The strings match
If you're going to use /bin/sh instead of /bin/bash, you really need to watch out for bashisms. There's a good page on this in the Ubuntu wiki. Either that, or switch to /bin/bash.
P.s. If you're going to use bash extensions, I'd recommend using [[ ]] instead of [ ] for conditionals -- it fixes most of the syntactic oddities of [ ], and adds some useful new capabilities like pattern and regular expression matching.
Mostly likely, I think you got your redirection output wrong on line 4.
Must be :
echo "we are suspending at $(date)..." >> /home/albin/stuff/suspend_test
instead of
echo "we are suspending at $(date)..." > /home/albin/stuff/suspend_test
Also, I think $1 is neither equal to 'pre' or 'post'. From looking at the output you posted, I can't understand from why these strings "tis" and "ons" have come from.
Probably caller of this script passing you other arguments than post or pre.
Change log writing to use >> instead of > (to prevent overwriting log all time) and try add echo “${1}” >> log at first line, then you can see what arguments you got in your script

bash to verify file type integrity and create log

I am trying to use bash to verify the integrity of specific downloads .bam files. There are two parts (bash 1) runs the command to verify the .bam files which creates .txt files, and creates a process.log. That part works perfect, what I am getting an error in is checking each of the .txt files for a string (SUCCESS) and if it is found then in the process.log that file is verified if it is not found then that file is corrupt. Currently the terminal displays the status and then gives an error. Thank you :).
bash part 1
logfile=/home/cmccabe/Desktop/NGS/API/5-4-2016/process.log
for f in /home/cmccabe/Desktop/NGS/API/5-4-2016/*.bam ; do
echo "Start bam validation creation: $(date) - File: $f"
bname=`basename $f`
pref=${bname%%.bam}
bam validate --in $f --verbose 2> /home/cmccabe/Desktop/NGS/API/5-4-2016/bam_validation/${pref}_validation.txt
echo "End bam validation creation: $(date) - File: $f"
done >> "$logfile"
echo "Start verifying $(date) - File: $file"
value=$( grep -ic "(SUCCESS)" /home/cmccabe/Desktop/NGS/API/5-4-2016/bam_validation/*.txt )
bash part 2
if [ $value -eq 1 ]
then
echo "bam file is verified and complete"
else
echo "bam is corrupt, check log for reason"
echo "End bam verify: $(date) - File: $f"
fi
done >> "$logfile"
erorr
Start verifying Thu May 5 12:49:10 CDT 2016 - File:
/home/cmccabe/Desktop/loop.sh: line 11: [: too many arguments
bam is corrupt, check log for reason
End bam verify: Thu May 5 12:49:10 CDT 2016 - File: /home/cmccabe/Desktop/NGS/API/5-4-2016/NA19240.bam
/home/cmccabe/Desktop/loop.sh: line 18: syntax error near unexpected token `done'
/home/cmccabe/Desktop/loop.sh: line 18: `done >> "$logfile"'
file that is created to check for SUCCESS
Number of records read = 24723078
Number of valid records = 24723078
TotalReads(e6) 24.72
MappedReads(e6) 24.57
PairedReads(e6) 0.00
ProperPair(e6) 0.00
DuplicateReads(e6) 7.33
QCFailureReads(e6) 0.00
MappingRate(%) 99.38
PairedReads(%) 0.00
ProperPair(%) 0.00
DupRate(%) 29.66
QCFailRate(%) 0.00
TotalBases(e6) 4332.46
BasesInMappedReads(e6) 4325.68
Returning: 0 (SUCCESS)
The error is due to simplest of the reasons, in bash if you use a single parentheses [] to evaluate a condition, it is nothing but an implicit way of using the bash test expression which expands the $value as a string containing spaces, special characters as separate parameters. Here you have left the variable ambiguous which could be expanded to multiple parameters for some error cases.
All you need to do is enclose that variable within double quotes, so that it is treated as one single string.
if [ "$value" == 1 ]; then

Single echo statement displaying values in 2 lines

The echo statement below is displaying values in 2 line instead of single line (Redirect File).
echo ">>"`date`": Value1: " $VAL1 "Value2:" $VAL2>>$RES_FILE
Actual Result:
>>Tue Dec 1 10:20:24 IST 2015: Value1:1
Value2:2
Expected Result:
>>Tue Dec 1 10:20:24 IST 2015: Value1:1 Value2:2
Tell me whats wrong with the statement.
There's no problem with your usage of echo. From your output, it looks like there's a newline in VAL1.
You try removing it:
VAL1="${VAL1//$'\n'}"

How to get the real line number of a failing Bash command?

In the process of coming up with a way to catch errors in my Bash scripts, I've been experimenting with "set -e", "set -E", and the "trap" command. In the process, I've discovered some strange behavior in how $LINENO is evaluated in the context of functions. First, here's a stripped down version of how I'm trying to log errors:
#!/bin/bash
set -E
trap 'echo Failed on line: $LINENO at command: $BASH_COMMAND && exit $?' ERR
Now, the behavior is different based on where the failure occurs. For example, if I follow the above with:
echo "Should fail at: $((LINENO + 1))"
false
I get the following output:
Should fail at: 6
Failed on line: 6 at command: false
Everything is as expected. Line 6 is the line containing the single command "false". But if I wrap up my failing command in a function and call it like this:
function failure {
echo "Should fail at $((LINENO + 1))"
false
}
failure
Then I get the following output:
Should fail at 7
Failed on line: 5 at command: false
As you can see, $BASH_COMMAND contains the correct failing command: "false", but $LINENO is reporting the first line of the "failure" function definition as the current command. That makes no sense to me. Is there a way to get the line number of the line referenced in $BASH_COMMAND?
It's possible this behavior is specific to older versions of Bash. I'm stuck on 3.2.51 for the time being. If the behavior has changed in later releases, it would still be nice to know if there's a workaround to get the value I want on 3.2.51.
EDIT: I'm afraid some people are confused because I broke up my example into chunks. Let me try to clarify what I have, what I'm getting, and what I want.
This is my script:
#!/bin/bash
set -E
function handle_error {
local retval=$?
local line=$1
echo "Failed at $line: $BASH_COMMAND"
exit $retval
}
trap 'handle_error $LINENO' ERR
function fail {
echo "I expect the next line to be the failing line: $((LINENO + 1))"
command_that_fails
}
fail
Now, what I expect is the following output:
I expect the next line to be the failing line: 14
Failed at 14: command_that_fails
Now, what I get is the following output:
I expect the next line to be the failing line: 14
Failed at 12: command_that_fails
BUT line 12 is not command_that_fails. Line 12 is function fail {, which is somewhat less helpful. I have also examined the ${BASH_LINENO[#]} array, and it does not have an entry for line 14.
For bash releases prior to 4.1, a special level of awful, hacky, performance-killing hell is needed to work around an issue wherein, on errors, the system jumps back to the function definition point before invoking an error handler.
#!/bin/bash
set -E
set -o functrace
function handle_error {
local retval=$?
local line=${last_lineno:-$1}
echo "Failed at $line: $BASH_COMMAND"
echo "Trace: " "$#"
exit $retval
}
if (( ${BASH_VERSION%%.*} <= 3 )) || [[ ${BASH_VERSION%.*} = 4.0 ]]; then
trap '[[ $FUNCNAME = handle_error ]] || { last_lineno=$real_lineno; real_lineno=$LINENO; }' DEBUG
fi
trap 'handle_error $LINENO ${BASH_LINENO[#]}' ERR
fail() {
echo "I expect the next line to be the failing line: $((LINENO + 1))"
command_that_fails
}
fail
BASH_LINENO is an array. You can refer to different values in it: ${BASH_LINENO[1]}, ${BASH_LINENO[2]}, etc. to back up the stack. (Positions in this array line up with those in the BASH_SOURCE array, if you want to get fancy and actually print a stack trace).
Even better, though, you can just inject the correct line number in your trap:
failure() {
local lineno=$1
echo "Failed at $lineno"
}
trap 'failure ${LINENO}' ERR
You might also find my prior answer at https://stackoverflow.com/a/185900/14122 (with a more complete error-handling example) interesting.
That behaviour is very reasonable.
The whole picture of the call stack provides comprehensive information whenever an error occurs. Your example had demonstrated a good error message; you could see where the an error actually occurred and which line triggered the function, etc.
If the interpreter/compiler can't precisely indicate where the error actually occurs, you could be more easily confused.

Resources