I need to extract a portion of a large GeoJson layer. I've tried multiple times with QGis on a 16Gb Ram intel i7 11th gen processor with no result. After about one hour if the program does not crash, I am able to load the file in QGis but no way of cropping it using the Clip vector function.
The layer in question is publicly available at https://github.com/microsoft/KenyaNigeriaBuildingFootprints (the file is kenya.geojson.zip)
... and the extent I need is 36.753446978,36.821255765,-1.330521423,-1.295234032 [EPSG:4326]
Here is how you can approach that with R.
Get the data
options(timeout=max(6000, getOption("timeout")))
url <- "https://minedbuildings.blob.core.windows.net/africa/kenya.geojsonl.zip"
download.file(url, basename(url), mode="wb")
f <- unzip("kenya.geojsonl.zip")
f
#"./kenya.geojsonl"
Read a subset by specifying an extent. This takes a while because geojson is a rather inefficient format.
library(terra)
e <- ext(36.753446978, 36.821255765, -1.330521423, -1.295234032)
## with timer
system.time(v <- vect("kenya.geojsonl", extent=e))
# user system elapsed
#1390.03 8.28 1409.11
v
# class : SpatVector
# geometry : polygons
# dimensions : 18159, 0 (geometries, attributes)
# extent : 36.75303, 36.8228, -1.331356, -1.29447 (xmin, xmax, ymin, ymax)
# source : kenya.geojsonl
# coord. ref. : lon/lat WGS 84 (EPSG:4326)
And then perhaps:
writeVector(v, "buildings.shp")
Related
I am trying to use gnuplot 5.0 to plot a 2D array of data with no margins or borders or axes... just a 2D image (.png or .jpg) representing some data. I would like to have each array element to correspond to exactly one pixel in the image with no scaling / interpolation etc and no extra white pixels at the edges.
So far, when I try to set the margins to 0 and even using the pixels flag, I am still left with a row of white pixels on the right and top borders of the image.
How can I get just an image file with pixel-by-pixel representation of a data array and nothing extra?
gnuplot script:
#!/usr/bin/gnuplot --persist
set terminal png size 400, 200
set size ratio -1
set lmargin at screen 0
set rmargin at screen 1
set tmargin at screen 0
set bmargin at screen 1
unset colorbox
unset tics
unset xtics
unset ytics
unset border
unset key
set output "pic.png"
plot "T.dat" binary array=400x200 format="%f" with image pixels notitle
Example data from Fortran 90:
program main
implicit none
integer, parameter :: nx = 400
integer, parameter :: ny = 200
real, dimension (:,:), allocatable :: T
allocate (T(nx,ny))
T(:,:)=0.500
T(2,2)=5.
T(nx-1,ny-1)=5.
T(2,ny-1)=5.
T(nx-1,2)=5.
open(3, file="T.dat", access="stream")
write(3) T(:,:)
close(3)
end program main
Some gnuplot terminals implement "with image" by creating a separate png file containing the image and then linking to it inside the resulting plot. Using that separate png image file directly will avoid any issues of page layout, margins, etc. Here I use the canvas terminal. The plot itself is thrown away; all we keep is the png file created with the desired content.
gnuplot> set term canvas name 'myplot'
Terminal type is now 'canvas'
Options are ' rounded size 600,400 enhanced fsize 10 lw 1 fontscale 1 standalone'
gnuplot> set output '/dev/null'
gnuplot> plot "T.dat" binary array=400x200 format="%f" with image
linking image 1 to external file myplot_image_01.png
gnuplot> quit
$identify myplot_image_01.png
myplot_image_01.png PNG 400x200 400x200+0+0 8-bit sRGB 348B 0.000u 0:00.000
Don't use gnuplot.
Instead, write a script that reads your data and converts it into one of the Portable Anymap formats. Here's an example in Python:
#!/usr/bin/env python3
import math
import struct
width = 400
height = 200
levels = 255
raw_datum_fmt = '=d' # native, binary double-precision float
raw_datum_size = struct.calcsize(raw_datum_fmt)
with open('T.dat', 'rb') as f:
print("P2")
print("{} {}".format(width, height))
print("{}".format(levels))
raw_data = f.read(width * height * raw_datum_size)
for y in range(height):
for x in range(width):
raw_datum, = struct.unpack_from(raw_datum_fmt, raw_data, (y * width + x) * raw_datum_size)
datum = math.floor(raw_datum * levels) # assume a number in the range [0, 1]
print("{:>3} ".format(datum), end='')
print()
If you can modify the program which generates the data file, you can even skip the above step and instead generate the data directly in a PNM format.
Either way, you can then use ImageMagick to convert the image to a format of your choice:
./convert.py | convert - pic.png
This should be an easy task, however, apparently it's not.
The following might be a (cumbersome) solution because all other attempts failed. My suspicion is that some graphics library has an issue which you probably cannot solve as a gnuplot user.
You mentioned that ASCII matrix data is also ok. The "trick" here is to plot data with lines where the data is "interrupted" by empty lines, basically drawing single points. Check this in case you need to get your datafile 1:1 into a datablock.
However, if it is not already strange enough, it seems to work for png and gif terminal but not for pngcairo or wxt.
I guess the workaround is probably slow and inefficient but at least it creates the desired output. I'm not sure if there is a limit on size. Tested with 100x100 pixels with Win7, gnuplot 5.2.6. Comments and improvements are welcome.
Code:
### pixel image from matrix data without strange white border
reset session
SizeX = 100
SizeY = 100
set terminal png size SizeX,SizeY
set output "tbPixelImage.png"
# generate some random matrix data
set print $Data2
do for [y=1:SizeY] {
Line = ''
do for [x=1:SizeX] {
Line = Line.sprintf(" %9d",int(rand(0)*0x01000000)) # random color
}
print Line
}
set print
# print $Data2
# convert matrix data into x y z data with empty lines inbetween
set print $Data3
do for [y=1:SizeY] {
do for [x=1:SizeX] {
print sprintf("%g %g %s", x, y, word($Data2[y],x))
print ""
}
}
set print
# print $Data3
set margins 0,0,0,0
unset colorbox
unset border
unset key
unset tics
set xrange[1:SizeX]
set yrange[1:SizeY]
plot $Data3 u 1:2:3 w l lw 1 lc rgb var notitle
set output
### end of code
Result: (100x100 pixels)
(enlarged with black background):
Image with 400x200 pixels (takes about 22 sec on my 8 year old laptop).
What I ended up actually using to get what I needed even though the question / bounty asks for a gnuplot solution:
matplotlib has a function matplotlib.pyplot.imsave which does what I was looking for... i.e. plotting 'just data pixels' and no extras like borders, margins, axes, etc. Originally I only knew about matplotlib.pyplot.imshow and had to pull a lot of tricks to eliminate all the extras from the image file and prevent any interpolation/smoothing etc (and therefore turned to gnuplot at a certain point). With imsave it's fairly easy, so I'm back to using matplotlib for an easy yet still flexible (in terms of colormap, scaling, etc) solution for 'pixel exact' plots. Here's an example:
#!/usr/bin/env python3
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
nx = 400
ny = 200
data = np.fromfile('T.dat', dtype=np.float32, count=nx*ny)
data = data.reshape((nx,ny), order='F')
matplotlib.image.imsave('T.png', np.transpose(data), origin='lower', format='png')
OK, here is another possible solution (I separated it from my first cumbersome approach). It creates the plot immediately, less than a second. No renaming necessary or creation of a useless file.
I guess key is to use term png and ps 0.1.
I don't have a proof but I think ps 1 would be ca. 6 pixels large and would create some overlap and/or white pixels at the corner. Again, for whatever reason it seems to work with term png but not with term pngcairo.
What I tested (Win7, gnuplot 5.2.6) is a binary file having the pattern 00 00 FF repeated all over (I can't display null bytes here). Since gnuplot apparently reads 4 bytes per array item (format="%d"), this leads to an alternating RGB pattern if I am plotting with lc rgb var.
In the same way (hopefully) we can figure out how to read format="%f" and use it together with a color palette. I guess that's what you are looking for, right?
Further test results, comments, improvements and explanations are welcome.
Code:
### pixel image from matrix data without strange white border
reset session
SizeX = 400
SizeY = 200
set terminal png size SizeX,SizeY
set output "tbPixelImage.png"
set margins 0,0,0,0
unset colorbox
unset border
unset key
unset tics
set xrange[0:SizeX-1]
set yrange[0:SizeY-1]
plot "tbBinary.dat" binary array=(SizeX,SizeY) format="%d" w p pt 5 ps 0.1 lc rgb var
### end of code
Result:
I want to shade the area under the density curve for the standard normal distribution by the following ranges:
1) mean-2std , mean-std ---> in red
2) mean +std , mean+2std ---> in red
3) mean -std , mean+st --> in blue
This is a variant on the questions "Shade (fill or color) area under density curve by quantile".
the data used to draw the denisty curve is taken from a column of a dataframe.
eg: This is only part of the data. The column has 256 values.
Gap
1 -3.260010
2 -7.790009
3 -1.179993
4 2.270019
5 9.000000
6 -4.930023
7 -7.920014
To draw the plot I did the following code:
sns.kdeplot(TeslaStock18_19['Gap'], label = 'Gap Density', color = 'darkblue')
Considering all the data, I found out that the distribution is normal. This allows me to use the Empricial rule (68-95) to make some statitical consideraton.
What I would like to obtain is the following plot:
https://www.nku.edu/~statistics/images/Using_1.gif
N.B. I am starting to use Python, It is for a Univeristy project.
This is what I tried to do but it does not fill me completely the area
ptx = np.linspace(meanGap-std, meanGap+std) pty = scipy.stats.norm.pdf(ptx,meanGap,stdGap) plt.fill_between(ptx, pty, color='#0b559f', alpha='0.35')
I want to read in the right ascension (in hour angles), declination (in degrees) and size (in arcmin) of a catalogue of galaxies and draw all of them in a large image of specified pixel size.
I tried converting the ra, dec and size into pixels to create a Bounds object for each galaxy, but get an error that "BoundsI must be initialized with integer values." I understand that pixels have to be integers...
But is there a way to center the large image at a specified ra and dec, then input the ra and dec of each galaxy as parameters to draw it in?
Thank you in advance!
GalSim uses the CelestialCoord class to handle coordinates in the sky and any of a number of WCS classes to handle the conversion from pixels to celestial coordinates.
The two demos in the tutorial series that use a CelestialWCS (the base class for WCS classes that use celestial coordinates for their world coordinate system) are demo11 and demo13. So you might want to take a look at them. However, neither one does something very close to what you're doing.
So here's a script that more or less does what you described.
import galsim
import numpy
# Make some random input data so we can run this.
# You would use values from your input catalog.
ngal = 20
numpy.random.seed(123)
ra = 15 + 0.02*numpy.random.random( (ngal) ) # hours
dec = -34 + 0.3*numpy.random.random( (ngal) ) # degrees
size = 0.1 * numpy.random.random( (ngal) ) # arcmin
e1 = 0.5 * numpy.random.random( (ngal) ) - 0.25
e2 = 0.5 * numpy.random.random( (ngal) ) - 0.25
# arcsec is usually the more natural units for sizes, so let's
# convert to that here to make things simpler later.
# There are options throughout GalSim to do things in different
# units, such as arcmin, but arcsec is the default, so it will
# be simpler if we don't have to worry about that.
size *= 60 # size now in arcsec
# Some plausible location at which to center the image.
# Note that we are now attaching the right units to these
# so GalSim knows what angle they correspond to.
cen_ra = numpy.mean(ra) * galsim.hours
cen_dec = numpy.mean(dec) * galsim.degrees
# GalSim uses CelestialCoord to handle celestial coordinates.
# It knows how to do all the correct spherical geometry calculations.
cen_coord = galsim.CelestialCoord(cen_ra, cen_dec)
print 'cen_coord = ',cen_coord.ra.hms(), cen_coord.dec.dms()
# Define some reasonable pixel size.
pixel_scale = 0.4 # arcsec / pixel
# Make the full image of some size.
# Powers of two are typical, but not required.
image_size = 2048
image = galsim.Image(image_size, image_size)
# Define the WCS we'll use to connect pixels to celestial coords.
# For real data, this would usually be read from the FITS header.
# Here, we'll need to make our own. The simplest one that properly
# handles celestial coordinates is TanWCS. It first goes from
# pixels to a local tangent plane using a linear affine transformation.
# Then it projects that tangent plane into the spherical sky coordinates.
# In our case, we can just let the affine transformation be a uniform
# square pixel grid with its origin at the center of the image.
affine_wcs = galsim.PixelScale(pixel_scale).affine().withOrigin(image.center())
wcs = galsim.TanWCS(affine_wcs, world_origin=cen_coord)
image.wcs = wcs # Tell the image to use this WCS
for i in range(ngal):
# Get the celestial coord of the galaxy
coord = galsim.CelestialCoord(ra[i]*galsim.hours, dec[i]*galsim.degrees)
print 'gal coord = ',coord.ra.hms(), coord.dec.dms()
# Where is it in the image?
image_pos = wcs.toImage(coord)
print 'position in image = ',image_pos
# Make some model of the galaxy.
flux = size[i]**2 * 1000 # Make bigger things brighter...
gal = galsim.Exponential(half_light_radius=size[i], flux=flux)
gal = gal.shear(e1=e1[i],e2=e2[i])
# Pull out a cutout around where we want the galaxy to be.
# The bounds needs to be in integers.
# The fractional part of the position will go into offset when we draw.
ix = int(image_pos.x)
iy = int(image_pos.y)
bounds = galsim.BoundsI(ix-64, ix+64, iy-64, iy+64)
# This might be (partially) off the full image, so get the overlap region.
bounds = bounds & image.bounds
if not bounds.isDefined():
print ' This galaxy is completely off the image.'
continue
# This is the portion of the full image where we will draw. If you try to
# draw onto the full image, it will use a lot of memory, but if you go too
# small, you might see artifacts at the edges. You might need to
# experiment a bit with what is a good size cutout.
sub_image = image[bounds]
# Draw the galaxy.
# GalSim by default will center the object at the "true center" of the
# image. We actually want it centered at image_pos, so provide the
# difference as the offset parameter.
# Also, the default is to overwrite the image. But we want to add to
# the existing image in case galaxies overlap. Hence add_to_image=True
gal.drawImage(image=sub_image, offset=image_pos - sub_image.trueCenter(),
add_to_image=True)
# Probably want to add a little noise...
image.addNoise(galsim.GaussianNoise(sigma=0.5))
# Write to a file.
image.write('output.fits')
GalSim deals with image bounds and locations using image coordinates. The way to connect true positions on the sky (RA, dec) into image coordinates is using the World Coordinate System (WCS) functionality in GalSim. I gather from your description that there is a simple mapping from RA/dec into pixel coordinates (i.e., there are no distortions).
So basically, you would set up a simple WCS defining the (RA, dec) center of the big image and its pixel scale. Then for a given galaxy (RA, dec), you can use the "toImage" method of the WCS to figure out where on the big image the galaxy should live. Any subimage bounds can be constructed using that information.
For a simple example with a trivial world coordinate system, you can check out demo10 in the GalSim repository.
My project is to detect human activity through stored video clips.
I am successfully able to do the following:
Get the Motion History Image (MHI) from a video using OpenCV
Train and classify the set of images using Matlab
However, I want to use Matlab in order to get the Motion History Image (MHI). Is it possible, and if yes can someone guide me? Thank you.
I have attached a sample Motion History Image (MHI)
I have used the following code for MHI:
http://www.ece.iastate.edu/~alexs/classes/2007_Fall_401/code/09_MotionHistory/motempl.c
MHI is just a ways of implementing motion detection (and uses silhouettes as the basis of it).
Let suppose that the silhouette of the most recent object has been created. It also uses a timestamp to identify if the current silhouette is recent or not. The older silhouettes have to be compared with the current silhouette in order to achieve movement detection. Hence, earlier silhouettes are also saved in the image, with an earlier timestamp.
MHI describes the changes of some moving objects over the image sequence. Basically, you should only maintain an image where every pixel encodes a time information - whether the silhouette is recent or not or where the movement occurs at a given time.
Therefore the implementation of MHI is very simple e.g.:
function MHI = MHI(fg)
% Initialize the output, MHI a.k.a. H(x,y,t,T)
MHI = fg;
% Define MHI parameter T
T = 15; % # of frames being considered; maximal value of MHI.
% Load the first frame
frame1 = fg{1};
% Get dimensions of the frames
[y_max x_max] = size(frame1);
% Compute H(x,y,1,T) (the first MHI)
MHI{1} = fg{1} .* T;
% Start global loop for each frame
for frameIndex = 2:length(fg)
%Load current frame from image cell
frame = fg{frameIndex};
% Begin looping through each point
for y = 1:y_max
for x = 1:x_max
if (frame(y,x) == 255)
MHI{frameIndex}(y,x) = T;
else
if (MHI{frameIndex-1}(y,x) > 1)
MHI{frameIndex}(y,x) = MHI{frameIndex-1}(y,x) - 1;
else
MHI{frameIndex}(y,x) = 0;
end
end
end
end
end
Code from: https://searchcode.com/codesearch/view/8509149/
Update #1:
Try to draw it as follows:
% showMHI.m
% Input frame number and motion history vector to display normalized MHI
% at the specified frame.
function showMHI(n, motion_history)
frameDisp = motion_history{n};
frameDisp = double(frameDisp);
frameDisp = frameDisp ./ 15;
figure, imshow(frameDisp)
title('MHI Image');
Below is the current working code in python using PIL for highlighting the difference between the two images. But rest of the images is blacken.
Currently i want to show the background as well along with the highlighted image.
Is there anyway i can keep the show the background lighter and just highlight the differences.
from PIL import Image, ImageChops
point_table = ([0] + ([255] * 255))
def black_or_b(a, b):
diff = ImageChops.difference(a, b)
diff = diff.convert('L')
# diff = diff.point(point_table)
h,w=diff.size
new = diff.convert('RGB')
new.paste(b, mask=diff)
return new
a = Image.open('i1.png')
b = Image.open('i2.png')
c = black_or_b(a, b)
c.save('diff.png')
!https://drive.google.com/file/d/0BylgVQ7RN4ZhTUtUU1hmc1FUVlE/view?usp=sharing
PIL does have some handy image manipulation methods,
but also a lot of shortcomings when one wants
to start doing serious image processing -
Most Python lterature will recomend you to switch
to use NumPy over your pixel data, wich will give
you full control -
Other imaging libraries such as leptonica, gegl and vips
all have Python bindings and a range of nice function
for image composition/segmentation.
In this case, the thing is to imagine how one would
get to the desired output in an image manipulation program:
You'd have a black (or other color) shade to place over
the original image, and over this, paste the second image,
but using a threshold (i.e. a pixel either is equal or
is different - all intermediate values should be rounded
to "different) of the differences as a mask to the second image.
I modified your function to create such a composition -
from PIL import Image, ImageChops, ImageDraw
point_table = ([0] + ([255] * 255))
def new_gray(size, color):
img = Image.new('L',size)
dr = ImageDraw.Draw(img)
dr.rectangle((0,0) + size, color)
return img
def black_or_b(a, b, opacity=0.85):
diff = ImageChops.difference(a, b)
diff = diff.convert('L')
# Hack: there is no threshold in PILL,
# so we add the difference with itself to do
# a poor man's thresholding of the mask:
#(the values for equal pixels- 0 - don't add up)
thresholded_diff = diff
for repeat in range(3):
thresholded_diff = ImageChops.add(thresholded_diff, thresholded_diff)
h,w = size = diff.size
mask = new_gray(size, int(255 * (opacity)))
shade = new_gray(size, 0)
new = a.copy()
new.paste(shade, mask=mask)
# To have the original image show partially
# on the final result, simply put "diff" instead of thresholded_diff bellow
new.paste(b, mask=thresholded_diff)
return new
a = Image.open('a.png')
b = Image.open('b.png')
c = black_or_b(a, b)
c.save('c.png')
Here's a solution using libvips:
import sys
from gi.repository import Vips
a = Vips.Image.new_from_file(sys.argv[1], access = Vips.Access.SEQUENTIAL)
b = Vips.Image.new_from_file(sys.argv[2], access = Vips.Access.SEQUENTIAL)
# a != b makes an N-band image with 0/255 for false/true ... we have to OR the
# bands together to get a 1-band mask image which is true for pixels which
# differ in any band
mask = (a != b).bandbool("or")
# now pick pixels from a or b with the mask ... dim false pixels down
diff = mask.ifthenelse(a, b * 0.2)
diff.write_to_file(sys.argv[3])
With PNG images, most CPU time is spent in PNG read and write, so vips is only a bit faster than the PIL solution.
libvips does use a lot less memory, especially for large images. libvips is a streaming library: it can load, process and save the result all at the same time, it does not need to have the whole image loaded into memory before it can start work.
For a 10,000 x 10,000 RGB tif, libvips is about twice as fast and needs about 1/10th the memory.
If you're not wedded to the idea of using Python, there are a few really simple solutions using ImageMagick:
“Diff” an image using ImageMagick