How to populate RRD database with CPU and MEM usage data? - shell

I have a Lighttpd server (on Centos) and would like to display 4 graphs: lighttpd traffic, lighttpd requests per second, CPU usage and MEM usage. I've set place for rrd database for lighttpd config like this:
rrdtool.binary = "/usr/bin/rrdtool"
rrdtool.db-name = "/var/www/lighttpd.rrd"
And put into my WWW cgi-bin sh file that gets data from lighttpd RRD file and creates graphs of traffic and requests per second like this:
#!/bin/sh
RRDTOOL=/usr/bin/rrdtool
OUTDIR=//var/www/graphs
INFILE=/var/www/lighttpd.rrd
OUTPRE=lighttpd-traffic
WIDTH=400
HEIGHT=100
DISP="-v bytes --title TrafficWebserver \
DEF:binraw=$INFILE:InOctets:AVERAGE \
DEF:binmaxraw=$INFILE:InOctets:MAX \
DEF:binminraw=$INFILE:InOctets:MIN \
DEF:bout=$INFILE:OutOctets:AVERAGE \
DEF:boutmax=$INFILE:OutOctets:MAX \
DEF:boutmin=$INFILE:OutOctets:MIN \
CDEF:bin=binraw,-1,* \
CDEF:binmax=binmaxraw,-1,* \
CDEF:binmin=binminraw,-1,* \
CDEF:binminmax=binmaxraw,binminraw,- \
CDEF:boutminmax=boutmax,boutmin,- \
AREA:binmin#ffffff: \
STACK:binmax#f00000: \
LINE1:binmin#a0a0a0: \
LINE1:binmax#a0a0a0: \
LINE2:bin#efb71d:incoming \
GPRINT:bin:MIN:%.2lf \
GPRINT:bin:AVERAGE:%.2lf \
GPRINT:bin:MAX:%.2lf \
AREA:boutmin#ffffff: \
STACK:boutminmax#00f000: \
LINE1:boutmin#a0a0a0: \
LINE1:boutmax#a0a0a0: \
LINE2:bout#a0a735:outgoing \
GPRINT:bout:MIN:%.2lf \
GPRINT:bout:AVERAGE:%.2lf \
GPRINT:bout:MAX:%.2lf \
"
$RRDTOOL graph $OUTDIR/$OUTPRE-hour.png -a PNG --start -14400 $DISP -w $WIDTH -h $HEIGHT
$RRDTOOL graph $OUTDIR/$OUTPRE-day.png -a PNG --start -86400 $DISP -w $WIDTH -h $HEIGHT
$RRDTOOL graph $OUTDIR/$OUTPRE-month.png -a PNG --start -2592000 $DISP -w $WIDTH -h $HEIGHT
OUTPRE=lighttpd-requests
DISP="-v req --title RequestsperSecond -u 1 \
DEF:req=$INFILE:Requests:AVERAGE \
DEF:reqmax=$INFILE:Requests:MAX \
DEF:reqmin=$INFILE:Requests:MIN \
CDEF:reqminmax=reqmax,reqmin,- \
AREA:reqmin#ffffff: \
STACK:reqminmax#00f000: \
LINE1:reqmin#a0a0a0: \
LINE1:reqmax#a0a0a0: \
LINE2:req#00a735:requests"
$RRDTOOL graph $OUTDIR/$OUTPRE-hour.png -a PNG --start -14400 $DISP -w $WIDTH -h $HEIGHT
$RRDTOOL graph $OUTDIR/$OUTPRE-day.png -a PNG --start -86400 $DISP -w $WIDTH -h $HEIGHT
$RRDTOOL graph $OUTDIR/$OUTPRE-month.png -a PNG --start -2592000 $DISP -w $WIDTH -h $HEIGHT
Basically it's not my script, i get it from somewhere from the internet.
Now i would like to do the same for CPU usage and MEM usage.
I don't like to use any additional packages!
As you can see lighttpd populates lighttpd.rrd file with traffic data and requests per second. Now i would like to the system to populate second rrd file with CPU and MEM usage, so i can add to sh file code to generate graphs for this data.
How can I populate RRD file with CPU and MEM usage data?
Please, NO THIRD-PARTY tools !

the trick is to read data from files in the proc filesystem and use sed/awk to extract the actual value. Create an rrdfile with DS type GAUGE to store the data ... look at the tutorials on www.rrdtool.org to get going.

Related

Output training losses over iterations/epochs to file from trainer.py in HuggingFace Transfrormers

In the Transformer's library framework, by HuggingFace only the evaluation step metrics are outputted to a file named eval_resuls_{dataset}.txt in the "output_dir" when running run_glue.py. In the eval_resuls file, there are the metrics associated with the dataset. e.g., accuracy for MNLI and the evaluation loss.
Can a parameter be passed to run_glue.py to generate a training_results_{dataset}.txt file that tracks the training loss? Or would I have to build the functionality myself?
My file named run_python_script_glue.bash:
GLUE_DIR=../../huggingface/GLUE_SMALL/
TASK_NAME=MNLI
ID=OT
python3 run_glue.py \
--local_rank -1 \
--seed 42 \
--model_type albert \
--model_name_or_path albert-base-v2 \
--task_name $TASK_NAME \
--do_train \
--do_eval \
--data_dir $GLUE_DIR/$TASK_NAME \
--max_seq_length 128 \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 8 \
--gradient_accumulation_steps 2\
--learning_rate 3e-5 \
--max_steps -1 \
--warmup_steps 1000\
--doc_stride 128 \
--num_train_epochs 3.0 \
--save_steps 9999\
--output_dir ./results/GLUE_SMALL/$TASK_NAME/ALBERT/$ID/ \
--do_lower_case \
--overwrite_output_dir \
--label_noise 0.2\
--att_kl 0.01\
--att_se_hid_size 16\
--att_se_nonlinear relu\
--att_type soft_attention \
--adver_type ot \
--rho 0.5 \
--model_type whai \
--prior_gamma 2.70 \
--three_initial 0.0
In the trainer.py file in the transformer library, the training loss variable during the training step is called tr_loss.
tr_loss = self._training_step(model, inputs, optimizer, global_step)
loss_scalar = (tr_loss - logging_loss) / self.args.logging_steps
logs["loss"] = loss_scalar
logging_loss = tr_loss
In the code, the training loss is first scaled by the logging steps and later passed to a logs dictionary. The logs['loss'] is later printed to the terminal but not to a file. Is there a way to upgrade this to include an update to a txt file?

Check If Vim Syntax Region Exists and Remove It

Background:
Syntax highlighting for perl files is extremely slow at times for large files (1k+ lines).
I profiled using:
:syntime on
"*** Do some slow actions ***
:syntime report
There were many slowly performaning regions, like: perlStatementProc
I significantly improved performance by removing some of the slowly performing syntax regions (there are more):
:syntax clear perlStatementProc
Now I want to use this vimrc with these improvements on a different machine which may not have a specific region defined.
I am seeing this ERROR when opening Vim:
E28: No such highlight group name: perlStatementProc
How can I check if the syntax region name perlStatementProc exists?
I found out about hlexists and implemented this solution in my vimrc:
" Remove some syntax highlighting from large perl files.
function! RemovePerlSyntax()
if line('$') > 1000
let perl_syntaxes = [
\ "perlStatementProc",
\ "perlMatch",
\ "perlStatementPword",
\ "perlQR",
\ "perlQW",
\ "perlQQ",
\ "perlQ",
\ "perlStatementIndirObjWrap",
\ "perlVarPlain",
\ "perlVarPlain",
\ "perlOperator",
\ "perlStatementFiledesc",
\ "perlStatementScalar",
\ "perlStatementInclude",
\ "perlStatementNumeric",
\ "perlStatementSocket",
\ "perlFloat",
\ "perlFormat",
\ "perlStatementMisc",
\ "perlStatementFiles",
\ "perlStatementList",
\ "perlStatementIPC",
\ "perlStatementNetwork",
\ "perlStatementTime",
\ "perlStatementIOfunc",
\ "perlStatementFlow",
\ "perlStatementControl",
\ "perlHereDoc",
\ "perlHereDocStart",
\ "perlVarPlain2",
\ "perlVarBlock",
\ "perlVarBlock2",
\ "perlDATA",
\ "perlControl",
\ "perlStatementHash",
\ "perlStatementVector",
\ "perlIndentedHereDoc",
\ "perlLabel",
\ "perlConditional",
\ "perlRepeat",
\ "perlNumber",
\ "perlStatementRegexp",
\ ]
for perl_syntax in perl_syntaxes
" NEW - Was missing this check before.
if hlexists( perl_syntax )
exec "syntax clear " . perl_syntax
endif
endfor
let b:remove_perl_syntax = 1
else
let b:remove_perl_syntax = 0
endif
endfunction
augroup remove_perl_syntax
autocmd!
autocmd BufNewFile,BufRead,BufReadPost,FileType perl call RemovePerlSyntax()
augroup END

How can I avoid getting blank spaces in my RRDTool graphs?

I've created quite a few RRDTool graphs monitoring various aspects of a Raspberry Pi server.
I'm displaying 36 hours, 10 days, 45 days and 18 months for things like transferred data, CPU temperature, load averages etc.
However, the only "continuous" looking graphs are the 10-day graphs, all the others have gaps in them. I'm recording each data point at a minutely interval.
There are 28 (29) images, so I'm not going to put them all here, so I've put them on imgur for your perusal
But here's an example of what I'm talking about:
10-days works fine!
45-days, not so much.
Here's my .rrd creation script:
rrdtool create data.rrd \
--start N --step '60' \
'DS:rx:GAUGE:60:0:U' \
'DS:tx:GAUGE:60:0:U' \
'DS:rxc:COUNTER:60:0:U' \
'DS:txc:COUNTER:60:0:U' \
'DS:wrx:GAUGE:60:0:U' \
'DS:wtx:GAUGE:60:0:U' \
'DS:wrxc:COUNTER:60:0:U' \
'DS:wtxc:COUNTER:60:0:U' \
'RRA:AVERAGE:0.5:1:129600' \
'RRA:AVERAGE:0.5:2:64800' \
'RRA:AVERAGE:0.5:60:14400' \
'RRA:AVERAGE:0.5:300:12960' \
'RRA:AVERAGE:0.5:3600:13140'
rrdtool create load.rrd \
--start N \
--step '60' \
'DS:load:GAUGE:60:0:4' \
'RRA:AVERAGE:0.5:1:129600' \
'RRA:AVERAGE:0.5:2:64800' \
'RRA:AVERAGE:0.5:60:14400' \
'RRA:AVERAGE:0.5:300:12960' \
'RRA:AVERAGE:0.5:3600:13140'
rrdtool create mem.rrd \
--start N \
--step '60' \
'DS:mem:GAUGE:60:0:100' \
'RRA:AVERAGE:0.5:1:129600' \
'RRA:AVERAGE:0.5:2:64800' \
'RRA:AVERAGE:0.5:60:14400' \
'RRA:AVERAGE:0.5:300:12960' \
'RRA:AVERAGE:0.5:3600:13140'
rrdtool create pitemp.rrd \
--start N \
--step '60' \
'DS:pitemp:GAUGE:60:U:U' \
'RRA:AVERAGE:0.5:1:129600' \
'RRA:AVERAGE:0.5:2:64800' \
'RRA:AVERAGE:0.5:60:14400' \
'RRA:AVERAGE:0.5:300:12960' \
'RRA:AVERAGE:0.5:3600:13140'
My entire draw script is like over 900 lines long, so I'll just include the actual draw code here for one set of graphs ($RRDTOOL is a variable containing the path /usr/bin/rrdtool):
$RRDTOOL graph /var/www/html/images/graphs/data36h.png \
--title 'Odin Absolute Traffic (eth0)' \
--watermark "Graph Drawn `date`" \
--vertical-label 'Bytes' \
--lower-limit '0' \
--rigid \
--alt-autoscale \
--units=si \
--width '640' \
--height '300' \
--full-size-mode \
--start end-36h \
'DEF:rx=/usr/local/bin/system/data.rrd:rx:AVERAGE' \
'CDEF:cleanrx=rx,UN,PREV,rx,IF' \
'DEF:tx=/usr/local/bin/system/data.rrd:tx:AVERAGE' \
'AREA:rx#00CC00FF:Download\:' \
'GPRINT:rx:LAST:\:%8.2lf %s]' \
'STACK:tx#0000FFFF:Upload\:' \
'GPRINT:tx:LAST:\:%8.2lf %s]\n'
$RRDTOOL graph /var/www/html/images/graphs/data10d.png \
--title 'Odin Absolute Traffic (eth0) 10 days' \
--watermark "Graph Drawn `date`" \
--vertical-label 'Bytes' \
--lower-limit '0' \
--rigid \
--alt-autoscale \
--units=si \
--width '640' \
--height '300' \
--full-size-mode \
--start end-10d \
'DEF:rx=/usr/local/bin/system/data.rrd:rx:AVERAGE' \
'DEF:tx=/usr/local/bin/system/data.rrd:tx:AVERAGE' \
'AREA:rx#00CC00FF:Download\:' \
'GPRINT:rx:LAST:\:%8.2lf %s]' \
'STACK:tx#0000FFFF:Upload\:' \
'GPRINT:tx:LAST:\:%8.2lf %s]\n'
$RRDTOOL graph /var/www/html/images/graphs/data45d.png \
--title 'Odin Absolute Traffic (eth0) 45 days' \
--watermark "Graph Drawn `date`" \
--vertical-label 'Bytes' \
--lower-limit '0' \
--rigid \
--alt-autoscale \
--units=si \
--width '640' \
--height '300' \
--full-size-mode \
--start end-45d \
'DEF:rx=/usr/local/bin/system/data.rrd:rx:AVERAGE' \
'DEF:tx=/usr/local/bin/system/data.rrd:tx:AVERAGE' \
'AREA:rx#00CC00FF:Download\:' \
'GPRINT:rx:LAST:\:%8.2lf %s]' \
'STACK:tx#0000FFFF:Upload\:' \
$RRDTOOL graph /var/www/html/images/graphs/data18m.png \
--title 'Odin Absolute Traffic (eth0) 18 month' \
--watermark "Graph Drawn `date`" \
--vertical-label 'Bytes' \
--lower-limit '0' \
--rigid \
--alt-autoscale \
--units=si \
--width '640' \
--height '300' \
--full-size-mode \
--start end-1y6m \
'DEF:rx=/usr/local/bin/system/data.rrd:rx:AVERAGE' \
'DEF:tx=/usr/local/bin/system/data.rrd:tx:AVERAGE' \
'AREA:rx#00CC00FF:Download\:' \
'GPRINT:rx:LAST:\:%8.2lf %s]' \
'STACK:tx#0000FFFF:Upload\:'
And yes, I know that the title on one of the graphs is wrong, I've fixed that, but only after saving all the images to imgur.
If you choose a --step of 60 seconds, I would choose a mrhb of 120s and not also of 60s because rrdtool will disregard any updates that are more than 60s apart.
rrdtool create data.rrd \
--start N --step '60' \
'DS:rx:GAUGE:120:0:U' \
'DS:tx:GAUGE:120:0:U' \
'DS:rxc:COUNTER:120:0:U' \
'DS:txc:COUNTER:120:0:U' \
'DS:wrx:GAUGE:120:0:U' \
'DS:wtx:GAUGE:120:0:U' \
'DS:wrxc:COUNTER:120:0:U' \
'DS:wtxc:COUNTER:120:0:U' \
'RRA:AVERAGE:0.5:1:129600' \
'RRA:AVERAGE:0.5:2:64800' \
'RRA:AVERAGE:0.5:60:14400' \
'RRA:AVERAGE:0.5:300:12960' \
'RRA:AVERAGE:0.5:3600:13140'

Custom garmin map have no name

I created a Garmin map from my own OSM files (using JOSM and my own GPS records, no input from Openstreetmap).
The whole process run well, but I have just a little problem : when I load the final map to Basecamp, the name of this map is empty (blank).
Any idea ?
Here is the code. Before, some variables :
PREFIX=640000
ORIGINALNAME=$(echo ${PREFIX}00)
NAME=$(echo ${PREFIX}01)
ID_PUBLIC=64
DIR="/home/Carto"
GMAPIBUILDER="/Applications/Carto/gmapi-builder.py"
MKGMAP="/Applications/Carto/mkgmap/mkgmap.jar"
First, create img files from different layers
for f in $DIR/src/public/*.osm ; do
g=$(basename $f .osm) ;
d=$(dirname $f)
java -Xmx2G -jar $MKGMAP \
--transparent --add-pois-to-areas \
--keep-going --draw-priority=$drawpriority \
--description="[iero] "$g \
--family-name="iero Congo" \
--series-name="iero Congo" \
--mapname=$NAME --family-id=$ID --product-id=$ID \
--country-name=Congo --country-abbr=CG \
--style-file=$DIR/styles --style=iero \
--copyright-message="[iero.org] Congo $DATE" \
--product-version=$VERSION \
--latin1 --output-dir=$DIR/output/imgs/public $f 1> /dev/null;
cp $DIR/output/imgs/public/${NAME}.img $DIR/output/imgs/public/${NAME}.img
let NAME++ ;
let nbfiles++ ;
let drawpriority++ ;
done
Next, concatenate those files in unique img file
java -jar $MKGMAP --tdbfile --gmapsupp $DIR/output/imgs/public/*.img \
--keep-going \
--style-file=$DIR/styles --style=iero \
--family-name="iero Congo" \
--series-name="iero Congo" \
--description="[iero] Congo map" \
--mapname=$ORIGINALNAME --family-id=${ID_PUBLIC} --product-id=${ID_PUBLIC} \
--copyright-message="[iero.org] Congo $DATE" \
--product-version=$VERSION \
--output-dir=$DIR/output/gps/public 1> /dev/null;
Then, create gmapi files, ready for Basecamp :
python $GMAPIBUILDER -t $DIR/output/gps/public/osmmap.tdb -b $DIR/output/gps/public/osmmap.img -o $DIR/output/basecamp/mac/public $DIR/output/imgs/public/*.img
If you want to see the problem, final files can be downloaded in my website : http://www.iero.org/blog/2014/06/carte-du-congo/
Thanks !
Greg
I have done testing and only get the blank names with versions of mkgmap after they introduced the overview map feature. I built a map with r2585 and the name showed correctly.

RRDTool graph command insert into variable

I am using sh(1) shell script to generate RRDTool for day, week, month and year. My question is, how to make this code more effective without repeating the same code? I tried it like this but always get errors, use trailing escape character etc:
NEWVAR="-a PNG -v "Interrupts/s" -h 130 -w 576 \
--watermark "`hostname`:`date "+%H:%M:%S - %d/%m/%Y"`" --font TITLE:15: \
--font LEGEND:7: --font UNIT:7: --font AXIS:5: -c CANVAS#000000 -c BACK#D8D8D8 \
DEF:DI="$RRDFDI":DI:AVERAGE \
CDEF:DIx=DI,8000,LT,DI,UNKN,IF \
AREA:DIx#FFDD44:"" \
LINE1.2:DIx#31B404:"Device Interrupts" \
GPRINT:DIx:MIN:"Min\:%6.0lf" \
GPRINT:DIx:AVERAGE:"Avg\:%6.0lf" \
GPRINT:DIx:MAX:"Max\:%6.0lf" \
GPRINT:DIx:LAST:"Cur\:%6.0lf\l""
$RRDBIN graph $NEWVAR -t "Device Interrupts Days Statistic (Interrupts)" --start -1d
$RRDBIN graph $NEWVAR -t "Device Interrupts Weeks Statistic (Interrupts)" --start -1w
$RRDBIN graph $NEWVAR -t "Device Interrupts Months Statistic (Interrupts)" --start -1m
$RRDBIN graph $NEWVAR -t "Device Interrupts Years Statistic (Interrupts)" --start -1y
Thank in advance!
Use a for loop, something along the lines of following.
NEWVAR="-a PNG -v "Interrupts/s" -h 130 -w 576 \
--watermark "`hostname`:`date "+%H:%M:%S - %d/%m/%Y"`" --font TITLE:15: \
--font LEGEND:7: --font UNIT:7: --font AXIS:5: -c CANVAS#000000 -c BACK#D8D8D8 \
DEF:DI="$RRDFDI":DI:AVERAGE \
CDEF:DIx=DI,8000,LT,DI,UNKN,IF \
AREA:DIx#FFDD44:"" \
LINE1.2:DIx#31B404:"Device Interrupts" \
GPRINT:DIx:MIN:"Min\:%6.0lf" \
GPRINT:DIx:AVERAGE:"Avg\:%6.0lf" \
GPRINT:DIx:MAX:"Max\:%6.0lf" \
GPRINT:DIx:LAST:"Cur\:%6.0lf\l""
for c in d w m y ; do
$RRDBIN graph $NEWVAR -t "Device Interrupts Days Statistic (Interrupts)" --start -1$c output.file.$c
done
The most obvious reason for your errors when using this code is that you have failed to escape the embedded double quotes in your declaration of NEWVAR.
You cannot generate all three graphs simulataneously; the way you are coding it is optimal, though you need to fix your use of embedded quotes in the NEWVAR declaration. You will also need to double-escape the colons and \l endof line sequence...
NEWVAR="-a PNG -v \"Interrupts/s\" -h 130 -w 576 \
--watermark \""`hostname`":"`date '+%H:%M:%S - %d/%m/%Y'`"\" --font TITLE:15: \
--font LEGEND:7: --font UNIT:7: --font AXIS:5: -c CANVAS#000000 -c BACK#D8D8D8 \
DEF:DI=$RRDFDI:DI:AVERAGE \
CDEF:DIx=DI,8000,LT,DI,UNKN,IF \
AREA:DIx#FFDD44:\"\" \
LINE1.2:DIx#31B404:'Device Interrupts' \
GPRINT:DIx:MIN:'Min\:%6.0lf' \
GPRINT:DIx:AVERAGE:'Avg\:%6.0lf' \
GPRINT:DIx:MAX:'Max\:%6.0lf' \
GPRINT:DIx:LAST:'Cur\:%6.0lf\l'"

Resources