I have a loop set up to search through a datafile and perform COMMANDS using each line of the datafile as parameters. The PUBLISH parameter can have spaces so that's why I have it escaped differently. I want to be able to output stderr and stdout for each iteration of the loop to a different logfile:
while read VARIABLE1 PUBLISH LOGFILENAME
do
/home/script_name -switch $VARIABLE1 -switch2 \"$PUBLISH\" >> /log/$LOGFILENAME 2>&1
done < $INPUTFILE
The logfile LOOKS to get created - that is - I can see it when I do an ll and it appears to have size - but when I attempt to cat the file - I'm getting:
# ll logfile
-rw-r--r-- 1 root root 3205 2014-10-15 12:52 logfile
# pg logfile
pg: logfile: No such file or directory
# cat logfile
cat: logfile: No such file or directory
# read -r logfile
^C (ctrl-c'd after no display)
# file logfile
logfile: ERROR: cannot open `logfile' (No such file or directory)
The COMMANDS are running fine - but I cannot get the logfile created. If I remove the redirect string (>> /log/$LOGFILENAME 2>&1) - the COMMAND output goes to the screen just fine. I'm executing the script and reading the logfile as root - so permissions should not be an issue.
Thanks to n.m. who got me on the right track. I changed the output filename to a hardcoded name and it worked - I was able to see the logfile and page it. I went back and looked at the original $INPUTFILE and discovered that there was a SPACE after the varialble name in the LOGFILENAME variable.
Related
I have a bash script running on Ubuntu 18.04. I scheduled it using SYSTEMD timer.
#!/bin/bash
backupdb(){
/usr/bin/mysqldump -u backupuser -pbackuppassword --add-locks --extended-insert --hex-blob $1 > /opt/mysqlbackup/$1.sql
/bin/gzip -c /opt/mysqlbackup/$1.sql > /opt/mysqlbackup/$1-$(date +%A).sql.gz
rm -rf /opt/mysqlbackup/$1.sql
echo `date "+%h %d %H:%M:%S"`": " $1 "- Size:" `/usr/bin/stat -c%s "${1}-$(date +%A).sql.gz"` >> /opt/mysqlbackup/backupsql.log
}
# List of databases to backup
backupdb cardb
backupdb bikedb
When I run this script interactively, the backup log get 2 entries:
Jun 16 20:15:03: cardb - Size: 200345
Jun 16 20:15:12: bikedb - Size: 150123
However, when this is run as a SYSTEMD timer service, the log still gets 2 entries but no file size is given in the log file. Not 0, it's simply blank. The backup file, cardb.sql.gz is created and is non-zero. I can unzip it and it does contain a valid SQL file.
I can't figure out why this is happening.
You need to specify the absolute path of your file
Without specifying the absolute path you are making the assumption that the systemd timer is running your script from the same directory you tested it from. To remedy this, you can either use the absolute path or change directories before accessing your file.
echo `date "+%h %d %H:%M:%S"`": " $1 "- Size:" `/usr/bin/stat -c%s "/opt/mysqlbackup/${1}-$(date +%A).sql.gz"` >> /opt/mysqlbackup/backupsql.log
i am not root user,so i install aria2-1.34.0 with ./configure --prefix=/home/xxx/.local, everything works fine but on-download-complete
i set on-download-complete=/home/xxx/aria2/a.sh
#!/bin/bash
echo 123 > 1.txt
when a task download completed, the log shows everything is ok
2019-10-28 19:04:11.295587 [NOTICE] [RequestGroup.cc:1216] Download complete: /home/xxx/aria2/data/0.png
2019-10-28 19:04:11.295598 [INFO] [DefaultBtProgressInfoFile.cc:415] The segment file /home/xxx/aria2/data/0.png.aria2 does not exist.
2019-10-28 19:04:11.295612 [INFO] [util.cc:2239] Executing user command: /home/xxx/aria2/a.sh b031d9399fb9d93f 1 /home/xxx/aria2/data/0.png
but actually nothing happened, it didn't work!!!
aria2c path:
/home/xxx/.local/aria2c
aria2.conf
enable-rpc=true
rpc-allow-origin-all=true
rpc-listen-all=false
max-concurrent-downloads=1
continue=true
max-connection-per-server=5
min-split-size=5M
split=5
max-overall-download-limit=5M
max-download-limit=2M
max-overall-upload-limit=0
max-upload-limit=0
dir=/home/xxx/aria2/data
file-allocation=prealloc
on-download-complete=/home/xxx/aria2/a.sh
log=/home/xxx/aria2/aria2.log
log-level=info
i don't know how to deal with it, could u give me any suggestions, thanks.
got a solution for me
put --on-download-complete in cmd instead of aria2.conf
just run aria2 like:
aria2c --conf-path=/xxx/xxx/arai2.conf --on-download-complete=/xxx/xxx/xxx.sh -D
and it works
but i have no idea why it didn't work if set --on-download-complete in conf file
Accroding to the official document
-D, --daemon [true|false]
Run as daemon. The current working directory will be changed to / and standard input, standard output and standard error will be redirected to /dev/null. Default: false
You should modify /home/xxx/aria2/a.sh to
#!/bin/bash
echo 123 > $(dirname $0)/1.txt
to write the output to the right place.
I'm currently debugging a shell script, which acts as a master-script in a data pipeline. In order to run the pipeline, you feed a bunch of arguments into the shell script. From there, the shell script sequentially calls 6 different scripts [4 in R, 2 in Python], writes out stuff to log files, and so on. Basically, my idea is to use this script to automate a data pipeline that takes a long time to run.
Right now, if any of the individual R or Python scripts break within the shell script, it just jumps to the next script that it's supposed to call. However, running script 03.py requires the data input to scripts 01.R and 02.R to be fully run and processed, otherwise 03 will produce erroneous output data which will then be written out and further processed in later scripts.
What I want to do is,
1. Break the overall shell script if there's an error in any of the R scripts
2. Output a message telling me where this error happened [line of individual R / python script]
Here's a sample of the master.sh shell script which calls the individual scripts.
#############
# STEP 2 : RUNNING SCRIPTS
#############
# A - 01.R
#################################################################
# log_file - this needs to be reassigned for every individual script
log_file=01.log
current_time=$(date)
echo "Current time: $current_time"
echo "Now running script 01. Log file output being written to $log_file_dir$log_file."
Rscript 01.R -f $input_file -s $sql_db > $log_file_dir$log_file
# current time/date
current_time=$(date)
echo "Current time: $current_time"
# B - 02.R
#################################################################
log_file=02.log
current_time=$(date)
echo "Current time: $current_time"
echo "Now running script 02. Log file output being written to $log_file_dir$log_file"
Rscript 02.R -f $input_file -s $sql_db > $log_file_dir$log_file
# PRINT OUT TIMINGS
current_time=$(date)
echo "Current time: $current_time"
This sequence is repeated throughout the master.sh script until script 06.R, after which it collates some data retrieved from output files and log files, and prints them to stout.
Here's some sample output that gets printed by my current master.sh, which shows how the script just keeps moving even though 01.R has produced an error.
file: test-data/minisample.txt
There are a total of 101 elements in file.
Using the main database.
Writing log-files to this directory: log_files/minisample/.
Writing output-csv with classifications to output/minisample.csv.
Current time: Wed Nov 14 18:19:53 UTC 2018
Now running script 01. Log file output being written to log_files/minisample/01.log.
Loading required package: stringi
Loading required package: dplyr
Attaching package: ‘dplyr’
The following objects are masked from ‘package:stats’:
filter, lag
The following objects are masked from ‘package:base’:
intersect, setdiff, setequal, union
Loading required package: RMySQL
Loading required package: DBI
Loading required package: methods
Loading required package: hms
Error: The following 2 arguments need to be provided:
-f <input file>.csv
-s <MySQL db name>
Execution halted
Current time: Wed Nov 14 18:19:54 UTC 2018
./master.sh: line 95: -1: substring expression < 0
./master.sh: line 100: -1: substring expression < 0
./master.sh: line 104: -1: substring expression < 0
Total time taken to run script 01.R:
Average time taken per user to run script 01.R:
Total time taken to run pipeline so far [01/06]:
Average time taken per user to run pipeline so far [01/06]:
Current time: Wed Nov 14 18:19:54 UTC 2018
Now running script 02. Log file output being written to log_files/minisample/02.log
Seeing as the R script 01.R produces an error, I want the script master.sh to stop. But how?
Any help would be greatly appreciated, thanks in advance!
As another user mentioned, simply running set -e will make your script terminate on first error. However, if you want more control, you can also check the exit status with ${?} or simply $? assuming your program gives an exit code of 0 on success, and non-zero otherwise.
#!/bin/bash
url=https://nosuchaddress1234.com/nosuchpage.html
error_file=errorFile.txt
wget ${url} 2> ${error_file}
exit_status=${?}
if [ ${exit_status} -ne 0 ]; then
echo -n "wget ${url} "
if [ ${exit_status} -eq 4 ]; then
echo "- Network failure."
elif [ ${exit_status} -eq 8 ]; then
echo "- Server issued an error response."
else
echo "- Other error"
fi
echo "See ${error_file} for more details"
exit ${exit_status};
fi
I like to put some boilerplate at the top of most scripts like this -
trap 'echo >&2 "ERROR in $0 at line $LINENO, Aborting"; exit $LINENO;' ERR
set -u
While coding at debugging, I usually add
set -x
And a lot of trace "comments" with colons -
: this will parse its args but only show under set -x
Then the trick is to make sure any errors you know are ok are handled.
Conditionals consume the errors, so those are safe.
if grep foo nonexistantfile
then : do the success stuff
else : if you *want* a failout here, just call false
false here will abort # args don't matter :)
fi
By the same token, if you just want to catch and ignore a known possible error -
ls $mightNotExist ||: # || says "do on fail"; : is an alias for "true"
Just always check your likely errors. Then the only thing that will crash your script is a fail.
I am trying to upload a file in redis using command:
redis-cli -p <Port> -h <Host> -n <DB> -x set <key> < /tmp/file.json
The problem is : in redis value -
It is storing a \n at the end of line and I don't want this.
"{\"items\":{\"38749\":{\"buyone\":0,\"buytwo\":10},\"38712\":{\"buyone\":0,\"buytwo\":10},\"112775\":{\"buyone\":0,\"buytwo\":10},\"38721\":{\"buyone\":0,\"buytwo\":10},\"38720\":{\"buyone\":0,\"buytwo\":10},\"38714\":{\"buyone\":0,\"buytwo\":10},\"38726\":{\"buyone\":0,\"buytwo\":10},\"38733\":{\"buyone\":0,\"buytwo\":10},\"38729\":{\"buyone\":0,\"buytwo\":10},\"113708\":{\"buyone\":0,\"buytwo\":10},\"38731\":{\"buyone\":0,\"buytwo\":10},\"38745\":{\"buyone\":0,\"buytwo\":10},\"38732\":{\"buyone\":0,\"buytwo\":10},\"100074\":{\"buyone\":0,\"buytwo\":10},\"38730\":{\"buyone\":0,\"buytwo\":10},\"38719\":{\"buyone\":0,\"buytwo\":10},\"38723\":{\"buyone\":0,\"buytwo\":10},\"38755\":{\"buyone\":0,\"buytwo\":10},\"38760\":{\"buyone\":0,\"buytwo\":10}}}\n"
Since your file contains newlines, that's what gets stored in Redis.
You'll need to strip the newlines from your file before setting its contents to Redis. Depending on your OS, the method may vary. Here's a question about this: How do I remove newlines from a text file?
Finally got this working -
cat /tmp/up.json | redis-cli -n 20 --pipe
and up.json contents
set 'PACKAGES_CONFIG' '{"checkOfferFieldPrices":true,"showPrescInfoScreen":true,"showAddOnsScreen":true,"offerText":"1 + 1 with Lenskart Gold","bannerConfig":{"isVisible":true,"primaryText":"Hi %s, You are a GOLD Member!","secondaryText":"You are eligible for Buy 1 Get 1 offer on this order!"},"isExpandedByDefault":true,"isPreSelected":true,"displayBogoTabs": true, "defaultSelectedTabId": "buy2","tabConfig":[{"id":"buy1","title":"Buy 1","subtitle":"No Offer","enabled":true},{"id":"buy2","title":"Buy 2","subtitle":"Buy 1 Get 1 Free","enabled":true}]}'
In some cases what I see in a console output is different from what I get recorded after redirection. I see this on Linux/bash but this example is ksh/OpenBSD. Is there a way around this?
For example:
# pfctl -ttable -Ttest 123.123.123.123 > result.txt
0/1 addresses match.
# more result.txt
result.txt (END)
In other words the "0/1 addresses match." is printed on the console, but I cannot for the life of me get it into a file, variable or anything. I've used $() and > which work for most commands, but every now and then there is a command that spits out stuff on the screen but I get nothing via the redirect/pipe. I hope someone can shed light on this peculiarity.
So again contrast this:
# OUTP=$(pfctl -tscanners -Ttest 123.123.123.123)
0/1 addresses match.
# echo $OUTP
#
(nothing echoing, the variable does not hold the console output) with this:
# OUTP=$(date)
# echo $OUTP
Sun Aug 21 08:33:37 PDT 2016
#
(the variable contains the entire console output)
Thanks again for any help.
Your command has 2 different output streams.
You need to rederict the second (stderr) to the first.
pfctl -ttable -Ttest 123.123.123.123 > result.txt 2>&1