I am running a Matlab script on Linux (RedHat Enterprise Linux RHEL 7.6, 64-bit) as a cron job. I am not admin on that machine, therefore, I use crontab -e to schedule the job. The installed version of Matlab is 2018b. The email which I recieve upon execution includes a couple of >> at the beginning and end which I find a bit irritating.
Here, an example of the email:
MATLAB is selecting SOFTWARE OPENGL rendering.
< M A T L A B (R) >
Copyright 1984-2018 The MathWorks, Inc.
R2018b (9.5.0.944444) 64-bit (glnxa64)
August 28, 2018
To get started, type doc.
For product information, visit www.mathworks.com.
>> >> >> >>
Matlab started: 2020-07-31 21:50:26.
>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>
Going to update from 2015-01-01 00:00:00 UTC to 2015-12-31 23:00:00 UTC.
[...]
>> Matlab closes: 2020-07-31 23:26:41.
>>
The corresponding lines at the beginning of the Matlab script look exactly like this:
close all
clearvars
% profile on % to check performance
fprintf('\nMatlab started: %s.\n', char(datetime()))
%% Database user parameters
% connects always to the soecified database on "localhost"
DB_conn_name = 'abc';
DB_username = 'def';
DB_password = 'ghi';
% Add path and subfolders
if isunix
addpath(genpath('/project/abc'));
elseif ispc
addpath(genpath('C:\Branches\abc'));
end
% Change working folder
if isunix
cd /project/abc
elseif ispc
cd C:\Branches\abc
end
% Add database driver to path
javaaddpath JDBC_driver/mysql-connector-java.jar % Forward slashes within Matlab work even on Windows
% Set default datetime format
datetime.setDefaultFormats('default','yyyy-MM-dd HH:mm:ss')
%% Begin and end of update period
% now_UTC = datetime('now','TimeZone','UTC');
% time_2 = datetime(now_UTC.Year, now_UTC.Month, now_UTC.Day-1, 22, 0, 0); % Set the end time not too late, otherwise, some data might not yet be available for some areas leading to ugly "dips" in Power BI.
% During each update, we update e.g. the past 30 days
% datetime_month_delay = time_1 - days(30);
% Override automatic dates obtained below, for testing purposes
% time_1 = datetime(2020,1,1,0,0,0);
% time_2 = datetime(2020,2,1,23,0,0);
% Updating several years, one at a time
for iYear = 2015:2019
time_1 = datetime(iYear,1,1,0,0,0);
time_2 = datetime(iYear,12,31,23,0,0);
fprintf(['\nGoing to update from ',char(time_1),' UTC to ',char(time_2),' UTC. \n'])
[...]
Looks as though each row that is outside the for loop produces an empty line and therefore such a >> prompt in the output. Also visible at the end (not included here).
The crontab -e looks like the following:
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=<my email address>
HOME=/project/abc
HTTP_PROXY=<proxy address>:8086
HTTPS_PROXY=<proxy address>:8086
# Run script regularly: minute hour day month dayofweek command
# No linebreaks allowed
15 2 * * * ~/script.sh
The shell script script.sh looks like this:
#!/bin/bash
/prog/matlab2018b/bin/matlab -nodesktop < ~/git-repos/abc/matlabscript.m
Does anyone have an idea what I need to change to get rid of these >>? That would be great! Thanks in advance!
The -nodesktop flag is still giving you an interactive shell, which is why crontab is capturing the prompts at all. You need to tell the matlab command what statement to execute.
I know you are using R2018b; but, I am going to give you BOTH answers for before and after R2019a, in case you ever upgrade.
For both answers: Because you called this in your crontab, make sure to use full path for your MATLAB executable for security reasons; and, it would be good to make sure you use the -sd flag as well so that your statement to execute is first in the path. The statement to execute is to be typed the same way you would type it on the MATLABcommand line.
Before R2019a: Per the doc page for the R2018b matlab (Linux) command, you need to run your command with the -r and -sd flags together. The -sd flag specifies your startup directory. Also, your code needs to have an exit statement at the end so that the matlab executable knows its done.
/path/before_R2019a/matlab -sd /path/startup_directory -b statement
Starting in R2019a, the -batch flag in your invocation of MATLAB is the recommended way to run automated jobs like this, per the matlab (Linux) command doc page
Note that starting in R2019a, the -r flag is NOT recommended; and, it should NOT be used with the -batch flag.
The -batch flag is simpler to use, and was added to make automation tasks easier. For starters, you no longer need to have an exit statement in your code with this approach.
Also remember that if you need quotes, starting in R2016b, MATLAB handles both double and single quoted strings. Choose appropriately in your script or cron call to handle your linux shell replacements - or avoid them.
/path/R2019a+/matlab -sd /path/startup_directory -b statement
As an added bonus, if you use the -batch flag, you can tell from inside your script whether it is running from a -batch call or interactively using the MATLAB variable batchStartupOptionUsed.
Related
I am trying to create my first bash script. The goal of this script is to check at what rate my public IP changes. It is a fairly straight forward script. First it checks if the new address is different from the old one. If so then it should update the old one to the new one and print out the date along with the new IP address.
At this point I have created a simple script in order to accomplish this. But I have two main problems.
First the script keeps on printing out the IP even tough it hasn't changed and I have updated the PREV_IP with the CUR_IP.
My second problem is that I want the output to direct to a file instead of outputting it into the terminal.
The interval is currently set to 1 second for test purposes. This will change to a higher interval in the final product.
#!/bin/bash
while true
PREV_IP=00
do
CUR_IP=$(curl https://ipinfo.io/ip)
if [ $PREV_IP != "$CUR_IP" ]; then
PREV_IP=$CUR_IP
"$(date)"
echo "$CUR_IP"
sleep 1
fi
done
I also get a really weird output. I have edited my public IP to xx.xxx.xxx.xxx:
Sat 20 Mar 09:45:29 CET 2021
xx.xxx.xxx.xxx
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
while true
PREV_IP=00
do
is the reason you are seeing ip each loop. It's the same as while true; PREV_IP=00; do. The exit status of true; PREV_IP=00 is the exit status of last command - the exit status of assignment is 0 (success) - so the loop will always execute. But PREV_IP will be reset to 00 each loop... This is a typo and you meant to set prev_ip once, before the loop starts.
"$(date)"
will try execute the output of date command, as a next command. So it will print:
$ "$(date)"
bash: sob, 20 mar 2021, 10:57:02 CET: command not found
And finally, to silence curl, read man curl first and then find out about -s. I use -sS so errors are also visible.
Do not use uppercase variables in your scripts. Prefer lower case variables. Check you scripts with http://shellcheck.net . Quote variable expansions.
I would sleep each loop. Your script could look like this:
#!/bin/bash
prev=""
while true; do
cur=$(curl -sS https://ipinfo.io/ip)
if [ "$prev" != "$cur" ]; then
prev="$cur"
echo "$(date) $cur"
fi
sleep 1
done
that I want the output to direct to a file instead of outputting it into the terminal.
Then research how redirection works in shell and how to use it. The simplest would be to redirect echo output.
echo "$(date) $cur" >> "a_file.txt"
The interval is currently set to 1 second for test purposes. This will change to a higher interval in the final product.
You are still limited with the time it takes to connect to https://ipinfo.io/ip. And from ipinfo.io documentation:
Free usage of our API is limited to 50,000 API requests per month.
And finally, I wrote a script where I tried to use many public services as I found ,get_ip_external for getting external ip address. You may take multiple public services for getting ipv4 address and choose a random/round-robin one so that rate-limiting don't kick that fast.
I'm currently debugging a shell script, which acts as a master-script in a data pipeline. In order to run the pipeline, you feed a bunch of arguments into the shell script. From there, the shell script sequentially calls 6 different scripts [4 in R, 2 in Python], writes out stuff to log files, and so on. Basically, my idea is to use this script to automate a data pipeline that takes a long time to run.
Right now, if any of the individual R or Python scripts break within the shell script, it just jumps to the next script that it's supposed to call. However, running script 03.py requires the data input to scripts 01.R and 02.R to be fully run and processed, otherwise 03 will produce erroneous output data which will then be written out and further processed in later scripts.
What I want to do is,
1. Break the overall shell script if there's an error in any of the R scripts
2. Output a message telling me where this error happened [line of individual R / python script]
Here's a sample of the master.sh shell script which calls the individual scripts.
#############
# STEP 2 : RUNNING SCRIPTS
#############
# A - 01.R
#################################################################
# log_file - this needs to be reassigned for every individual script
log_file=01.log
current_time=$(date)
echo "Current time: $current_time"
echo "Now running script 01. Log file output being written to $log_file_dir$log_file."
Rscript 01.R -f $input_file -s $sql_db > $log_file_dir$log_file
# current time/date
current_time=$(date)
echo "Current time: $current_time"
# B - 02.R
#################################################################
log_file=02.log
current_time=$(date)
echo "Current time: $current_time"
echo "Now running script 02. Log file output being written to $log_file_dir$log_file"
Rscript 02.R -f $input_file -s $sql_db > $log_file_dir$log_file
# PRINT OUT TIMINGS
current_time=$(date)
echo "Current time: $current_time"
This sequence is repeated throughout the master.sh script until script 06.R, after which it collates some data retrieved from output files and log files, and prints them to stout.
Here's some sample output that gets printed by my current master.sh, which shows how the script just keeps moving even though 01.R has produced an error.
file: test-data/minisample.txt
There are a total of 101 elements in file.
Using the main database.
Writing log-files to this directory: log_files/minisample/.
Writing output-csv with classifications to output/minisample.csv.
Current time: Wed Nov 14 18:19:53 UTC 2018
Now running script 01. Log file output being written to log_files/minisample/01.log.
Loading required package: stringi
Loading required package: dplyr
Attaching package: ‘dplyr’
The following objects are masked from ‘package:stats’:
filter, lag
The following objects are masked from ‘package:base’:
intersect, setdiff, setequal, union
Loading required package: RMySQL
Loading required package: DBI
Loading required package: methods
Loading required package: hms
Error: The following 2 arguments need to be provided:
-f <input file>.csv
-s <MySQL db name>
Execution halted
Current time: Wed Nov 14 18:19:54 UTC 2018
./master.sh: line 95: -1: substring expression < 0
./master.sh: line 100: -1: substring expression < 0
./master.sh: line 104: -1: substring expression < 0
Total time taken to run script 01.R:
Average time taken per user to run script 01.R:
Total time taken to run pipeline so far [01/06]:
Average time taken per user to run pipeline so far [01/06]:
Current time: Wed Nov 14 18:19:54 UTC 2018
Now running script 02. Log file output being written to log_files/minisample/02.log
Seeing as the R script 01.R produces an error, I want the script master.sh to stop. But how?
Any help would be greatly appreciated, thanks in advance!
As another user mentioned, simply running set -e will make your script terminate on first error. However, if you want more control, you can also check the exit status with ${?} or simply $? assuming your program gives an exit code of 0 on success, and non-zero otherwise.
#!/bin/bash
url=https://nosuchaddress1234.com/nosuchpage.html
error_file=errorFile.txt
wget ${url} 2> ${error_file}
exit_status=${?}
if [ ${exit_status} -ne 0 ]; then
echo -n "wget ${url} "
if [ ${exit_status} -eq 4 ]; then
echo "- Network failure."
elif [ ${exit_status} -eq 8 ]; then
echo "- Server issued an error response."
else
echo "- Other error"
fi
echo "See ${error_file} for more details"
exit ${exit_status};
fi
I like to put some boilerplate at the top of most scripts like this -
trap 'echo >&2 "ERROR in $0 at line $LINENO, Aborting"; exit $LINENO;' ERR
set -u
While coding at debugging, I usually add
set -x
And a lot of trace "comments" with colons -
: this will parse its args but only show under set -x
Then the trick is to make sure any errors you know are ok are handled.
Conditionals consume the errors, so those are safe.
if grep foo nonexistantfile
then : do the success stuff
else : if you *want* a failout here, just call false
false here will abort # args don't matter :)
fi
By the same token, if you just want to catch and ignore a known possible error -
ls $mightNotExist ||: # || says "do on fail"; : is an alias for "true"
Just always check your likely errors. Then the only thing that will crash your script is a fail.
I am working on a robot and my goal is to plot the state of the robot.
For now, my workflow is this:
Launch the program
Redirect the output in a file (robot/bash): rosrun explo explo_node > states.txt
Send the file to my local machine (robot/bash): scp states.txt my_desktop:/home/user
Plot the states with octave (desktop/octave): plot_data('states.txt')
Is there a simple solution to have the data in "real time"? For the octave side. I think that I can with not so much difficulty read from a file as an input and plot the data when data is added.
The problem is how do I send the data to a file?
I am opened to other solutions than octave. The thing is that I need to have 2d plot with arrows for the orientation of the robot.
Here's an example of how you could send the data over the network (as Andy suggested) and plot as it is generated (i.e. realtime). I also think this approach is the most flexible / appropriate.
To demonstrate, I will use a bash script that generates an
pair every 10th of a second, for the
function, in the range
:
#!/bin/bash
# script: sin.sh
for i in `seq 0 0.01 31.4`;
do
printf "$i, `echo "s($i)" | bc -l`\n"
sleep 0.1
done
(Don't forget to make this script executable!)
Prepare the following octave script (requires the sockets package!):
% in visualiseRobotData.m
pkg load sockets
s = socket();
bind(s, 9000);
listen(s, 1);
c = accept(s);
figure; hold on;
while ! isempty (a = str2num (char (recv (c, inf))))
plot (a(:,1), a(:,2), '*'); drawnow;
end
hold off;
Now execute things in the following order:
Run the visualiseRobotData script from the octave terminal.
(Note: this will block until a connection is established)
From your bash terminal run: ./sin.sh | nc localhost 9000
And watch the datapoints get plotted as they come in from your sin.sh script.
It's a bit crude, but you can just reload the file in a loop. This one runs for 5 minutes:
for i = 1:300
load Test/sine.txt
plot (sine(:,1), sine(:,2))
sleep (1)
endfor
You can mount remote directory via sshfs:
sshfs user#remote:/path/to/remote_dir local_dir
so you wouldn't have to load remote file. If sshfs is not installed, install it. To unmount remote directory later, execute
fusermount -u local_dir
To get a robot's data from Octave, execute (Octave code)
system("ssh user#host 'cd remote_dir; rosrun explo explo_node > states.txt'")
%% then plot picture from the data in local_dir
%% that is defacto the directory on the remote server
I'm using MediaInfo CLI version in Win 7 x64 to automatically make a CSV via template when a video file has finished encoding in StaxRip.
However, its CLI version is critical about how to apply the output template (long story short, its variables are in sections (general, video, audio, text) and you can only use one section in one block, you can't go back to a previous section further down the template), so one variable that I want elsewhere has to end up in the wrong spot for the automation to even work.
Like this:
UTC 2015-05-21 18:04:06,Episode01.mp4,211 MiB,22mn 7s,29.970 fps,1 210 Kbps,High 10#L3,120 Kbps,AAC,Japanese
UTC 2015-05-21 19:16:18,Episode02.mp4,211 MiB,22mn 6s,29.970 fps,1 212 Kbps,High 10#L3,118 Kbps,AAC,Japanese
UTC 2015-05-21 20:24:57,Episode03.mp4,211 MiB,22mn 6s,29.970 fps,1 212 Kbps,High 10#L3,119 Kbps,AAC,Japanese
What I'm looking for is the timestamp portion (first column) to become the LAST column instead:
Episode01.mp4,211 MiB,22mn 7s,29.970 fps,1 210 Kbps,High 10#L3,120 Kbps,AAC,Japanese,UTC 2015-05-21 18:04:06
I would very much love to find a solution to this in a .bat or Powershell script if possible since these are already used in the aforementioned process, but am open to small single-purpose applications. The crucial part is being able to be run from CMD or from a master .bat file.
Thank you for your time.
I tried this one out and is working.
[string] $SourceFileFullPath = "C:\Projects\INT\CSV_ColumnSwap.csv"
[Array] $SourceFileContent = Get-Content $SourceFileFullPath
[int] $ArrayLength = $SourceFileContent.length
for ($i=0; $i -lt $ArrayLength; $i++) {
$splitter1 = ","
$LineData = $SourceFileContent[$i] -split $splitter1
$DateTimeV, $Linedata = $LineData
$LineData += $DateTimeV
$LineData -join "," >> Result.csv
}
I am not particularly sure about the performance aspects. YMMV.
Cheers
I have a bash script that is sending me a text daily, for 100 days.
#! /bin/bash
EMAIL="my-phone-gateway#address.net"
MESSAGE="message_content.txt"
mail $EMAIL < $MESSAGE
Using crontab, I can have the static $MESSAGE sent to me every day.
Other than hard-coding 100 days of texts ;)
How could I implement a variable counter such that I can have my texts say:
"Today is Day #1" on the first day, "Today is Day #2" on the second day, etc. ?
Note: The location of the requested text within the $MESSAGE file doesn't matter. Last line, first line, middle, etc.
The only requirement for an answer here is that I know what day it is relative to the first, where the first day is the day the script was started.
Of course, bonus awesome points for the cleanest, simplest, shortest solution :)
For our nightly build systems, I wrote a C program that does the calculation (using local proprietary libraries that store dates as a number of days since a reference date). Basically, given a (non-changing) reference date, it reports the number of days since the reference date. So, the cron script would have a hard-wired first day in it, and the program would report the number of days since then.
The big advantage of this system is that the reference date doesn't change (very often), so the script doesn't change (very often), and there are no external files to store information in.
There probably are ways to achieve the same effect with standard Unix tools, but I've not sat down and worked out the portable solution. I'd probably think it terms of using Perl. (The C program only works up to 2999 CE; I left a note in the code for people to contact me about 50 years before it becomes a problem for the Y3K fix. It is probably trivial.)
You could perhaps work in terms of Unix timestamps...
Create a script 'days_since 1234567890' which treats the number as the reference date, gets the current time stamp (from date with appropriate format specification; on Linux, date '+%s' would do that job, and it works on Mac OS X too), takes the difference and divides by 86,400 (the number of seconds in a day).
refdate=1234567890
bc <<EOF
scale=0
($(date '+%s') - $refdate) / 86400
EOF
An example:
$ timestamp 1234567890
1234567890 = Fri Feb 13 15:31:30 2009
$ timestamp
1330027280 = Thu Feb 23 12:01:20 2012
$ refdate=1234567890
$ bc <<EOF
> scale=0
> ($(date '+%s') - $refdate) / 86400
> EOF
1104
$
So, if the reference date was 13th Feb 2009, today is day 1104. (The program bc is the calculator; its name has nothing to do with Anno Domini or Before Christ. The program timestamp is another homebrew of mine that prints timestamps according to a format that can be specified; it is a specialized variant of date originally written in the days before date had the functionality, by which I mean in the early 1980s.)
In a Perl one-liner (assuming you specify the reference date in your script):
perl -e 'printf "%d\n", int((time - 1234567890)/ 86400)'
or:
days=$(perl -e 'printf "%d\n", int((time - 1234567890)/ 86400)')
The only way to accomplish this would be to store the date in a file, and read from that file each day. I would suggest storing the epoch time.
today=$(date +%s)
time_file="~/.first_time"
if [[ -f $time_file ]]; then
f_time=$(< "$time_file")
else
f_time=$today
echo "$f_time" > "$time_file"
fi
printf 'This is day: %s\n' "$((($today - $f_time) / 60 / 60 / 24))"
Considering that your script is running only once a day, something like this should work:
#!/bin/bash
EMAIL="my-phone-gateway#address.net"
MESSAGE="message_content.txt"
STFILE=/tmp/start.txt
start=0
[ -f $STFILE ] && start=$(<$STFILE)
start=$((start+1))
MESSAGE=${MESSAGE}$'\n'"Today is Day #${start}"
echo "$start" > $STFILE
mail $EMAIL < $MESSAGE
A simple answer would be to export the current value to an external file, and read that back in again later.
So, for example, make a file called "CurrentDay.dat" that has the number 1 in it.
Then, in your bash script, read in the number and increment it.
e.g. your bash script could be:
#!/bin/bash
#Your stuff here.
DayCounter=$(<CurrentDay.dat)
#Use the value of DayCounter (i.e. $DayCounter) in your message.
DayCounter=$((DayCounter + 1))
echo $DayCounter > CurrentDay.dat
Of course, you may need to implement some additional checks to avoid something going wrong, but that should work as is.