Running a python script within a bash file within a Yii project - bash

I have a Yii project that allows importing files.
Within this project I call the following command to try and convert xls files to csv:
$file = fopen($model->importfile->tempname,'r');
$filetype = substr($model->importfile, strrpos($model->importfile, '.')+1);
if ($filetype === 'xls')
{
$tempxls = $model->importfile->tempname;
$outputArr = array();
exec(Yii::app()->basePath."/commands/xlstocsv.sh " . $tempxls, $outputArr);
PropertiesController::xlsToConsoleV7Format($tempxls, $log);
}
xlstocsv.sh:
#!/bin/bash
# Try to autodetect OOFFICE and OOOPYTHON.
OOFFICE=`ls /usr/bin/libreoffice /usr/lib/libreoffice/program/soffice /usr/bin/X11/libreoffice | head -n 1`
OOOPYTHON=`ls /usr/bin/python3 | head -n 1`
XLS='.xls'
CSV='.csv'
INPUT=$1$XLS
OUTPUT=$1$CSV
cp $1 $INPUT
if [ ! -x "$OOFFICE" ]
then
echo "Could not auto-detect OpenOffice.org binary"
exit
fi
if [ ! -x "$OOOPYTHON" ]
then
echo "Could not auto-detect OpenOffice.org Python"
exit
fi
echo "Detected OpenOffice.org binary: $OOFFICE"
echo "Detected OpenOffice.org python: $OOOPYTHON"
# Start OpenOffice.org in listening mode on TCP port 2002.
$OOFFICE "-accept=socket,host=localhost,port=2002;urp;StarOffice.ServiceManager" -norestore -nofirststartwizard -nologo -headless &
# Wait a few seconds to be sure it has started.
sleep 5s
# Convert as many documents as you want serially (but not concurrently).
# Substitute whichever documents you wish.
$OOOPYTHON /fullpath/DocumentConverter.py $INPUT $OUTPUT
# Close OpenOffice.org.
cp $OUTPUT $1
DocumentConverter.py:
This can be found here: https://github.com/mirkonasato/pyodconverter. It has been slightly modified to have correct syntax for python3.
Ok, the issue is, when running the php code from the terminal, it correctly creates the csv file from the excel file.
However, when running it from within the browser, it still runs the script and creates the output file, but it has not correctly converted it into csv.
It all works perfectly for every file I have thrown at it so far when running from console, but for some reason when running it from within a browser, it fails to convert the file properly.
Any ideas for what could be going wrong?

Thanks alejandro, permission errors seemed to be the issue. Also I needed to move the .config/librroffice folder into apaches home directory.

Related

Get file size not working in scheduled job

I have a bash script running on Ubuntu 18.04. I scheduled it using SYSTEMD timer.
#!/bin/bash
backupdb(){
/usr/bin/mysqldump -u backupuser -pbackuppassword --add-locks --extended-insert --hex-blob $1 > /opt/mysqlbackup/$1.sql
/bin/gzip -c /opt/mysqlbackup/$1.sql > /opt/mysqlbackup/$1-$(date +%A).sql.gz
rm -rf /opt/mysqlbackup/$1.sql
echo `date "+%h %d %H:%M:%S"`": " $1 "- Size:" `/usr/bin/stat -c%s "${1}-$(date +%A).sql.gz"` >> /opt/mysqlbackup/backupsql.log
}
# List of databases to backup
backupdb cardb
backupdb bikedb
When I run this script interactively, the backup log get 2 entries:
Jun 16 20:15:03: cardb - Size: 200345
Jun 16 20:15:12: bikedb - Size: 150123
However, when this is run as a SYSTEMD timer service, the log still gets 2 entries but no file size is given in the log file. Not 0, it's simply blank. The backup file, cardb.sql.gz is created and is non-zero. I can unzip it and it does contain a valid SQL file.
I can't figure out why this is happening.
You need to specify the absolute path of your file
Without specifying the absolute path you are making the assumption that the systemd timer is running your script from the same directory you tested it from. To remedy this, you can either use the absolute path or change directories before accessing your file.
echo `date "+%h %d %H:%M:%S"`": " $1 "- Size:" `/usr/bin/stat -c%s "/opt/mysqlbackup/${1}-$(date +%A).sql.gz"` >> /opt/mysqlbackup/backupsql.log

on-download-complete can't work with aria2

i am not root user,so i install aria2-1.34.0 with ./configure --prefix=/home/xxx/.local, everything works fine but on-download-complete
i set on-download-complete=/home/xxx/aria2/a.sh
#!/bin/bash
echo 123 > 1.txt
when a task download completed, the log shows everything is ok
2019-10-28 19:04:11.295587 [NOTICE] [RequestGroup.cc:1216] Download complete: /home/xxx/aria2/data/0.png
2019-10-28 19:04:11.295598 [INFO] [DefaultBtProgressInfoFile.cc:415] The segment file /home/xxx/aria2/data/0.png.aria2 does not exist.
2019-10-28 19:04:11.295612 [INFO] [util.cc:2239] Executing user command: /home/xxx/aria2/a.sh b031d9399fb9d93f 1 /home/xxx/aria2/data/0.png
but actually nothing happened, it didn't work!!!
aria2c path:
/home/xxx/.local/aria2c
aria2.conf
enable-rpc=true
rpc-allow-origin-all=true
rpc-listen-all=false
max-concurrent-downloads=1
continue=true
max-connection-per-server=5
min-split-size=5M
split=5
max-overall-download-limit=5M
max-download-limit=2M
max-overall-upload-limit=0
max-upload-limit=0
dir=/home/xxx/aria2/data
file-allocation=prealloc
on-download-complete=/home/xxx/aria2/a.sh
log=/home/xxx/aria2/aria2.log
log-level=info
i don't know how to deal with it, could u give me any suggestions, thanks.
got a solution for me
put --on-download-complete in cmd instead of aria2.conf
just run aria2 like:
aria2c --conf-path=/xxx/xxx/arai2.conf --on-download-complete=/xxx/xxx/xxx.sh -D
and it works
but i have no idea why it didn't work if set --on-download-complete in conf file
Accroding to the official document
-D, --daemon [true|false]
Run as daemon. The current working directory will be changed to / and standard input, standard output and standard error will be redirected to /dev/null. Default: false
You should modify /home/xxx/aria2/a.sh to
#!/bin/bash
echo 123 > $(dirname $0)/1.txt
to write the output to the right place.

How in envoy show debugging info?

Working with envoy in laravel 5.7 I see in examples that echo command is used for debugging purpose.
But whe I write echo in my envoy file, like:
#setup
$server_login_user= 'lardeployer';
$timezone= 'Europe/Kiev';
$path= '/var/www/html/AppDir';
$current = $path . '/current';
$repo= 'git#bitbucket.org:myaccount/votes.git'';
$branch= 'master';
echo "Step # 01";
$writableDirs= [
'/storage/logs',
'/bootstrap/cache'
];
echo "Step # 02";
...
echo "Step # 03";
#endsetup
#servers(['production' => $server_login_user.'#NNN.NN.NN.N])
#task( 'clone', ['on'=>$on] )
...
running envoy script I do not see any echo messages in my console.
I see echo command mentioned in 5.0 version documentation : https://laravel.com/docs/5.0/envoy But that does not
work in my 5.7/5.8 apps. Is echo still supported in laravel ? Or is that some config option ?
Thanks!
if you are going to use echo you should use it inside tasks as mention on laravel too
You may access the options in your tasks via Blade's "echo" syntax.
You may also use if statements and loops within your tasks.
for more detail click here

How to break shell script if a script it calls produces an error

I'm currently debugging a shell script, which acts as a master-script in a data pipeline. In order to run the pipeline, you feed a bunch of arguments into the shell script. From there, the shell script sequentially calls 6 different scripts [4 in R, 2 in Python], writes out stuff to log files, and so on. Basically, my idea is to use this script to automate a data pipeline that takes a long time to run.
Right now, if any of the individual R or Python scripts break within the shell script, it just jumps to the next script that it's supposed to call. However, running script 03.py requires the data input to scripts 01.R and 02.R to be fully run and processed, otherwise 03 will produce erroneous output data which will then be written out and further processed in later scripts.
What I want to do is,
1. Break the overall shell script if there's an error in any of the R scripts
2. Output a message telling me where this error happened [line of individual R / python script]
Here's a sample of the master.sh shell script which calls the individual scripts.
#############
# STEP 2 : RUNNING SCRIPTS
#############
# A - 01.R
#################################################################
# log_file - this needs to be reassigned for every individual script
log_file=01.log
current_time=$(date)
echo "Current time: $current_time"
echo "Now running script 01. Log file output being written to $log_file_dir$log_file."
Rscript 01.R -f $input_file -s $sql_db > $log_file_dir$log_file
# current time/date
current_time=$(date)
echo "Current time: $current_time"
# B - 02.R
#################################################################
log_file=02.log
current_time=$(date)
echo "Current time: $current_time"
echo "Now running script 02. Log file output being written to $log_file_dir$log_file"
Rscript 02.R -f $input_file -s $sql_db > $log_file_dir$log_file
# PRINT OUT TIMINGS
current_time=$(date)
echo "Current time: $current_time"
This sequence is repeated throughout the master.sh script until script 06.R, after which it collates some data retrieved from output files and log files, and prints them to stout.
Here's some sample output that gets printed by my current master.sh, which shows how the script just keeps moving even though 01.R has produced an error.
file: test-data/minisample.txt
There are a total of 101 elements in file.
Using the main database.
Writing log-files to this directory: log_files/minisample/.
Writing output-csv with classifications to output/minisample.csv.
Current time: Wed Nov 14 18:19:53 UTC 2018
Now running script 01. Log file output being written to log_files/minisample/01.log.
Loading required package: stringi
Loading required package: dplyr
Attaching package: ‘dplyr’
The following objects are masked from ‘package:stats’:
filter, lag
The following objects are masked from ‘package:base’:
intersect, setdiff, setequal, union
Loading required package: RMySQL
Loading required package: DBI
Loading required package: methods
Loading required package: hms
Error: The following 2 arguments need to be provided:
-f <input file>.csv
-s <MySQL db name>
Execution halted
Current time: Wed Nov 14 18:19:54 UTC 2018
./master.sh: line 95: -1: substring expression < 0
./master.sh: line 100: -1: substring expression < 0
./master.sh: line 104: -1: substring expression < 0
Total time taken to run script 01.R:
Average time taken per user to run script 01.R:
Total time taken to run pipeline so far [01/06]:
Average time taken per user to run pipeline so far [01/06]:
Current time: Wed Nov 14 18:19:54 UTC 2018
Now running script 02. Log file output being written to log_files/minisample/02.log
Seeing as the R script 01.R produces an error, I want the script master.sh to stop. But how?
Any help would be greatly appreciated, thanks in advance!
As another user mentioned, simply running set -e will make your script terminate on first error. However, if you want more control, you can also check the exit status with ${?} or simply $? assuming your program gives an exit code of 0 on success, and non-zero otherwise.
#!/bin/bash
url=https://nosuchaddress1234.com/nosuchpage.html
error_file=errorFile.txt
wget ${url} 2> ${error_file}
exit_status=${?}
if [ ${exit_status} -ne 0 ]; then
echo -n "wget ${url} "
if [ ${exit_status} -eq 4 ]; then
echo "- Network failure."
elif [ ${exit_status} -eq 8 ]; then
echo "- Server issued an error response."
else
echo "- Other error"
fi
echo "See ${error_file} for more details"
exit ${exit_status};
fi
I like to put some boilerplate at the top of most scripts like this -
trap 'echo >&2 "ERROR in $0 at line $LINENO, Aborting"; exit $LINENO;' ERR
set -u
While coding at debugging, I usually add
set -x
And a lot of trace "comments" with colons -
: this will parse its args but only show under set -x
Then the trick is to make sure any errors you know are ok are handled.
Conditionals consume the errors, so those are safe.
if grep foo nonexistantfile
then : do the success stuff
else : if you *want* a failout here, just call false
false here will abort # args don't matter :)
fi
By the same token, if you just want to catch and ignore a known possible error -
ls $mightNotExist ||: # || says "do on fail"; : is an alias for "true"
Just always check your likely errors. Then the only thing that will crash your script is a fail.

if else statement incorrect output

I'm working on a custom Nagios script that will monitor cPanel to make sure it is running and give back a status depending on what it gets from an output of service cpanel status. This is what I have:
##############################################################################
# Constants
cpanelstate="running..."
ALERT_OK="OK - cPanel is running"
ALERT_CRITICAL="CRITICAL - cPanel is NOT running"
###############################################################################
cpanel=$(service cpanel status | head -1)
if [ "$cpanel" = "$cpanelstate" ]; then
echo $ALERT_OK
exit 0
else
echo $ALERT_CRITICAL
exit 2
fi
exit $exitstatus
When I run the script, this is the output I get:
root#shared01 [/home/mvelez]# /usr/local/nagios/libexec/check_cpanel
CRITICAL - cPanel is NOT running
When I run the script, cPanel IS RUNNING but this is the output I get. As a matter of fact, no matter what the status reports for cPanel this is the output that comes out. When I comment out the ELSE, ECHO and EXIT 2 statement:
#else
# echo $ALERT_CRITICAL
# exit 2
It gives back a blank output:
root#shared01 [/home/mvelez]# /usr/local/nagios/libexec/check_cpanel
root#shared01 [/home/mvelez]#
I'm not sure what I'm not doing correctly as I am very new to bash scripting and trying to learn as I go along. Thank you in advanced for any and all help very very much!
The code below should work, but you might need to run it with sudo, because 'service' might not be available for ordinary users.
#!/bin/bash
##############################################################################
# Constants
cpanelstate="running"
ALERT_OK="OK - cPanel is running"
ALERT_CRITICAL="CRITICAL - cPanel is NOT running"
###############################################################################
cpanel=$(service apache2 status | head -1)
echo CPANEL $cpanel
if [[ $cpanel == *$cpanelstate* ]]; then
echo $ALERT_OK
exit 0
else
echo $ALERT_CRITICAL
exit 2
fi
#Oleg Gryb's answer solves your problem, but as for why your original script didn't work:
[ "$cpanel" = "$cpanelstate" ] compared the full command output - e.g., cpsrvd (pid 10066) is running..., against a substring of the expected output, running... for equality, which will obviously fail.
The solution is to use bash's pattern matching, provided via the right-hand side of its [[ ... ]] conditional (bash's superior alternative to the [ ... ] conditional):
[[ "$cpanel" == *"$cpanelstate" ]]
* represents any sequence of characters, so that this conditional returns true, if $cpanel ends with $cpanelstate (note how * must be unquoted to be recognized as a special pattern char.)

Resources