I usually see and use a first ogr2ogr comand to convert from .shp to .geoJSON. In a second command, I use topojson.js to convert from .geoJSON to .topoJSON format, with simplification of the precision, coordinates, arcs, and filtering to keep relevant metadata only. Example :
# DOWNLOAD: Data from http://gadm.org/
CRI_adm.zip:
curl -o CRI_adm.zip http://gadm.org/data/shp/CRI_adm.zip
CRI_adm0.shp: CRI_adm.zip
unzip CRI_adm.zip
touch CRI_adm0.shp
# PROCESS DATA: SIMLIFY, FILTER
costarica.json: CRI_adm0.shp
ogr2ogr -f GeoJSON costarica.json CRI_adm0.shp
# Require topojson: https://github.com/mbostock/topojson
# (this minifies/simplifies the data)
costarica_min_topo.json: costarica.json
topojson \
-p name=NAME \
-p name \
-q 1e4 \
-o costarica_min_topo.json \
costarica.json
But, since topojson.js can convert from .shp directly into .topoJSON, with simplification of the precision, coordinates, arcs, can we jump ogr2ogr and directly convert and filter with a single topojson.js command ? Such:
# PROCESS DATA: SIMLIFY, FILTER
topojson \
-p name=NAME \
-p name \
-q 1e4 \
-o costarica_min_topo.json \
CRI_adm0.shp
ogr2ogr has a -where parameter. You can use -where "name=NAME" to filter the features of the shp file
Related
I'm trying to use a Makefile to iterate over several date values and execute a python script for each one, here's the Makefile I'm using (Makefile.study.s1):
include Makefile
# Dates to test
SNAP_TST := 2019-10-12 2020-02-08 2020-10-10 2021-01-02 2021-07-24 2021-12-31 2022-05-27
buildDataset:
for date in $(SNAP_TST) ; do \
python src/_buildDataset.py --table $(TABLE) \
--nan-values $(NAN_CONFIG) \
--patterns-rmv $(PATTERNS_RMV) \
--target-bin $(TARGET_CLASS) \
--target-surv $(TARGET_SURV) \
--target-init $(TARGET_INIT) \
--train-file $(DATA_TRN) \
--test-file $(DIR_DATA)/main/churnvol_test_$$date.csv \
--train-date $(SNAP_TRN) \
--test-date $$date \
--config-input $(CONFIG_INPUT) \
--feats $(FEATS) ; \
done
.PHONY: buildDataset
When I run make -f Makefile.study.s1 buildDataset it replaces the value date with the string "$date" instead of one of the dates in SNAP_TST. Can you guys help me figure out what I did wrong here, and how I can fix this makefile so that $$date is replaced with one of the dates in SNAP_TST? Thank you in advance.
I want to export one certain page from a pdf document to an image and automatically fill the page number in the file name. When I run the following code:
gs \
-sDEVICE=jpeg \
-o outfile-%03.jpeg \
-dFirstPage=12 \
-dLastPage=12 \
wo.pdf
I get: outfile-001.jpeg instead of outfile-012.jpeg.
I've wrote a bash script for the job:
function extract_nth_page(){
printf -v j "outfile-%05g.png" $1
echo $j
gs -q -dNOPAUSE -sDEVICE=png16m -r600 -dFirstPage=$1 -dLastPage=$1 -sOutputFile=$j $2 -c quit
return 0
}
# Extracts page number 42 from myFile.pdf to outfile-00042.png
extract_nth_page 42 myFile.pdf
Trying to import data to RRDtool DB for a couple of temperature sensor collected from a RFXtrx433e USB-controller. Output to .txt files
My database created like this:
[code]
# Script to create rrd-file
# 24h with 2,5 min resolution
# 7d with 5 min resolution
# 1y with 10 min resolution
# 20y with 1h resolution
directory="/home/pi/temp/rrddata/"
filename="domoticz_temp.rrd"
# Check i file already exists
if [ ! -f "$directory$filename" ]
then
# File doesn't exist, create new rrd-file
echo "Creating RRDtool DB for outside temp sensor"
rrdtool create $directory$filename \
--step 120 \
DS:probe:GAUGE:120:-50:60 \
DS:xxxx1:GAUGE:120:-50:60 \
DS:vardagsrum:GAUGE:120:-50:60 \
RRA:AVERAGE:0.5:1:576 \
RRA:AVERAGE:0.5:2:2016 \
RRA:AVERAGE:0.5:4:52560 \
RRA:AVERAGE:0.5:24:175200 \
RRA:MAX:0.5:1:5760 \
RRA:MAX:0.5:2:2016 \
RRA:MAX:0.5:4:52560 \
RRA:MAX:0.5:24:175200 \
RRA:MIN:0.5:1:5760 \
RRA:MIN:0.5:2:2016 \
RRA:MIN:0.5:4:52560 \
RRA:MIN:0.5:24:175200
echo "Done!"
else
echo $directory$filename" already exists, delete it first."
fi
Import of sensor data
rrdtool update /home/pi/temp/rrddata/domoticz_temp.rrd --template probe N:`head -n 1 </home/pi/temp/output/temp_probe.txt`
The textfile imported just contain one row with a number (temperature collected from the sensor through a LUA-script)
The code for create the graph
rrdtool graph /home/pi/temp/output/img/test/hour.png \
-w 697 -h 287 -a PNG \
--slope-mode \
--start -6h --end now \
--vertical-label "Last 6 hour temperature" \
DEF:probe=/home/pi/temp/rrddata/domoticz_temp.rrd:probe:AVERAGE \
DEF:xxxx1=/home/pi/temp/rrddata/domoticz_temp.rrd:xxxx1:AVERAGE \
DEF:vardagsrum=/home/pi/temp/rrddata/domoticz_temp.rrd:vardagsrum:AVERAGE \
COMMENT:" Location Min Max Senaste\l" \
LINE1:probe#ff0000:"Utetemp" \
LINE1:0#ff0000: \
GPRINT:probe:MIN:" %5.1lf" \
GPRINT:probe:MAX:" %5.1lf" \
GPRINT:probe:LAST:" %5.1lf\n" \
LINE1:xxxx1#00ff00:"Xxxx1" \
LINE1:0#00ff00: \
GPRINT:probe:MIN:" %5.1lf" \
GPRINT:probe:MAX:" %5.1lf" \
GPRINT:probe:LAST:" %5.1lf\n" \
LINE1:vardagsrum#0000ff:"vardagsrum" \
LINE1:0#0000ff: \
GPRINT:probe:MIN:" %5.1lf" \
GPRINT:probe:MAX:" %5.1lf" \
GPRINT:probe:LAST:" %5.1lf\n" \
Gives me this graph http://i.imgur.com/lnFxTik.png
Now to my questions:
Have I created the database and the rest of script in a correct way? I think should get NAN on the values not in the DB?
How do I import the rest of the sensors? They are in several simular TXT files.
Should/can I collect data from the sensor in another better way to get them in to the RRDtool DB?
Hope anyone can help me.
New info!
My LUA-script for collection sensor data
commandArray = {}
if (devicechanged['Probe']) then
local file = io.open("/home/pi/temp/output/temp_probe.txt", "w")
file:write(tonumber(otherdevices_temperature['Probe']))
file:close()
end
if (devicechanged['Xxxx1']) then
local file = io.open("/home/pi/temp/output/temp_xxxx1.txt", "w")
file:write(tonumber(otherdevices_temperature['Xxxx1']))
file:close()
end
if (devicechanged['Vardagsrum']) then
local file = io.open("/home/pi/temp/output/temp_vardagsrum.txt", "w")
file:write(tonumber(otherdevices_temperature['Vardagsrum']))
file:close()
end
return commandArray`
Yes if a value is missing you get NaN. Your create statement looks ok ... although 20y with 1h resolution ... wow!
importing from several text files would work like this
.
A=`perl -ne 'chomp;print;exit' xx1.txt`
B=`perl -ne 'chomp;print;exit' xx2.txt`
rrdtool update domoticz_temp.rrd --template xx1:xx2 N:$A:$B
.
yes instead of writing them to a file first, I would recommend to update the rrd file directly.
# 24h with 2,5 min resolution
# 7d with 5 min resolution
# 1y with 10 min resolution
# 20y with 1h resolution
...
rrdtool create $directory$filename \
--step 120 \
DS:probe:GAUGE:120:-50:60 \
DS:xxxx1:GAUGE:120:-50:60 \
DS:vardagsrum:GAUGE:120:-50:60 \
RRA:AVERAGE:0.5:1:576 \
RRA:AVERAGE:0.5:2:2016 \
RRA:AVERAGE:0.5:4:52560 \
RRA:AVERAGE:0.5:24:175200 \
OK, you seem to have a 2min step size, and your RRAs are consolodating 1, 2, 4 and 24 steps. This corresponds to 2min, 4min, 8min and 48min, not to 2.5, 5, 10 and 1h. Maybe your step should be 150? Also, the heartbeat on your DSs is the same as your step, which might cause you to lose data. Generally speaking, the heartbeat should be about 1.5 to 2 times the step size to allow for irregular data arrival.
However none of this relates to your 'unknown' question, much of which Tobi has already answered.
You will get 'unknown' on timeslots you have not loaded, yes.
2 and 3. Since you have a single RRD you need to have all the samples updated at the same timestamp, in the same operation. In this case, you're probably better off collecting them all at once and storing them into the same file, so that you can load them together and store into the RRD together. If this is an issue, and the sensors are probed independently, then I'd advise having a separate RRD for each sensor, so that you can update them independently. You can still generate a graph over all 3 together as you can define your graph DEFs to point to different RRD files no problem. This might be a better way to do it.
And Tobi's right about a 20y RRA possibly being somewhat excessive ;)
I created a Garmin map from my own OSM files (using JOSM and my own GPS records, no input from Openstreetmap).
The whole process run well, but I have just a little problem : when I load the final map to Basecamp, the name of this map is empty (blank).
Any idea ?
Here is the code. Before, some variables :
PREFIX=640000
ORIGINALNAME=$(echo ${PREFIX}00)
NAME=$(echo ${PREFIX}01)
ID_PUBLIC=64
DIR="/home/Carto"
GMAPIBUILDER="/Applications/Carto/gmapi-builder.py"
MKGMAP="/Applications/Carto/mkgmap/mkgmap.jar"
First, create img files from different layers
for f in $DIR/src/public/*.osm ; do
g=$(basename $f .osm) ;
d=$(dirname $f)
java -Xmx2G -jar $MKGMAP \
--transparent --add-pois-to-areas \
--keep-going --draw-priority=$drawpriority \
--description="[iero] "$g \
--family-name="iero Congo" \
--series-name="iero Congo" \
--mapname=$NAME --family-id=$ID --product-id=$ID \
--country-name=Congo --country-abbr=CG \
--style-file=$DIR/styles --style=iero \
--copyright-message="[iero.org] Congo $DATE" \
--product-version=$VERSION \
--latin1 --output-dir=$DIR/output/imgs/public $f 1> /dev/null;
cp $DIR/output/imgs/public/${NAME}.img $DIR/output/imgs/public/${NAME}.img
let NAME++ ;
let nbfiles++ ;
let drawpriority++ ;
done
Next, concatenate those files in unique img file
java -jar $MKGMAP --tdbfile --gmapsupp $DIR/output/imgs/public/*.img \
--keep-going \
--style-file=$DIR/styles --style=iero \
--family-name="iero Congo" \
--series-name="iero Congo" \
--description="[iero] Congo map" \
--mapname=$ORIGINALNAME --family-id=${ID_PUBLIC} --product-id=${ID_PUBLIC} \
--copyright-message="[iero.org] Congo $DATE" \
--product-version=$VERSION \
--output-dir=$DIR/output/gps/public 1> /dev/null;
Then, create gmapi files, ready for Basecamp :
python $GMAPIBUILDER -t $DIR/output/gps/public/osmmap.tdb -b $DIR/output/gps/public/osmmap.img -o $DIR/output/basecamp/mac/public $DIR/output/imgs/public/*.img
If you want to see the problem, final files can be downloaded in my website : http://www.iero.org/blog/2014/06/carte-du-congo/
Thanks !
Greg
I have done testing and only get the blank names with versions of mkgmap after they introduced the overview map feature. I built a map with r2585 and the name showed correctly.
I have a Lighttpd server (on Centos) and would like to display 4 graphs: lighttpd traffic, lighttpd requests per second, CPU usage and MEM usage. I've set place for rrd database for lighttpd config like this:
rrdtool.binary = "/usr/bin/rrdtool"
rrdtool.db-name = "/var/www/lighttpd.rrd"
And put into my WWW cgi-bin sh file that gets data from lighttpd RRD file and creates graphs of traffic and requests per second like this:
#!/bin/sh
RRDTOOL=/usr/bin/rrdtool
OUTDIR=//var/www/graphs
INFILE=/var/www/lighttpd.rrd
OUTPRE=lighttpd-traffic
WIDTH=400
HEIGHT=100
DISP="-v bytes --title TrafficWebserver \
DEF:binraw=$INFILE:InOctets:AVERAGE \
DEF:binmaxraw=$INFILE:InOctets:MAX \
DEF:binminraw=$INFILE:InOctets:MIN \
DEF:bout=$INFILE:OutOctets:AVERAGE \
DEF:boutmax=$INFILE:OutOctets:MAX \
DEF:boutmin=$INFILE:OutOctets:MIN \
CDEF:bin=binraw,-1,* \
CDEF:binmax=binmaxraw,-1,* \
CDEF:binmin=binminraw,-1,* \
CDEF:binminmax=binmaxraw,binminraw,- \
CDEF:boutminmax=boutmax,boutmin,- \
AREA:binmin#ffffff: \
STACK:binmax#f00000: \
LINE1:binmin#a0a0a0: \
LINE1:binmax#a0a0a0: \
LINE2:bin#efb71d:incoming \
GPRINT:bin:MIN:%.2lf \
GPRINT:bin:AVERAGE:%.2lf \
GPRINT:bin:MAX:%.2lf \
AREA:boutmin#ffffff: \
STACK:boutminmax#00f000: \
LINE1:boutmin#a0a0a0: \
LINE1:boutmax#a0a0a0: \
LINE2:bout#a0a735:outgoing \
GPRINT:bout:MIN:%.2lf \
GPRINT:bout:AVERAGE:%.2lf \
GPRINT:bout:MAX:%.2lf \
"
$RRDTOOL graph $OUTDIR/$OUTPRE-hour.png -a PNG --start -14400 $DISP -w $WIDTH -h $HEIGHT
$RRDTOOL graph $OUTDIR/$OUTPRE-day.png -a PNG --start -86400 $DISP -w $WIDTH -h $HEIGHT
$RRDTOOL graph $OUTDIR/$OUTPRE-month.png -a PNG --start -2592000 $DISP -w $WIDTH -h $HEIGHT
OUTPRE=lighttpd-requests
DISP="-v req --title RequestsperSecond -u 1 \
DEF:req=$INFILE:Requests:AVERAGE \
DEF:reqmax=$INFILE:Requests:MAX \
DEF:reqmin=$INFILE:Requests:MIN \
CDEF:reqminmax=reqmax,reqmin,- \
AREA:reqmin#ffffff: \
STACK:reqminmax#00f000: \
LINE1:reqmin#a0a0a0: \
LINE1:reqmax#a0a0a0: \
LINE2:req#00a735:requests"
$RRDTOOL graph $OUTDIR/$OUTPRE-hour.png -a PNG --start -14400 $DISP -w $WIDTH -h $HEIGHT
$RRDTOOL graph $OUTDIR/$OUTPRE-day.png -a PNG --start -86400 $DISP -w $WIDTH -h $HEIGHT
$RRDTOOL graph $OUTDIR/$OUTPRE-month.png -a PNG --start -2592000 $DISP -w $WIDTH -h $HEIGHT
Basically it's not my script, i get it from somewhere from the internet.
Now i would like to do the same for CPU usage and MEM usage.
I don't like to use any additional packages!
As you can see lighttpd populates lighttpd.rrd file with traffic data and requests per second. Now i would like to the system to populate second rrd file with CPU and MEM usage, so i can add to sh file code to generate graphs for this data.
How can I populate RRD file with CPU and MEM usage data?
Please, NO THIRD-PARTY tools !
the trick is to read data from files in the proc filesystem and use sed/awk to extract the actual value. Create an rrdfile with DS type GAUGE to store the data ... look at the tutorials on www.rrdtool.org to get going.