Is my topojson file structured properly in order to render a map on folium? - topojson

I've downloaded the us-counties shapefile from the US census bureau and converted it into topojson file using mapshaper.com. Unfortunately, I have to parse through the topojson quite a bit to get the FIPS county code. I'm using Folium to render the map but keep getting an error.
I've taken my dataframe and made it into a series of FIPS_codes and $amounts. Using the style_function, I call the FIPS_codes from the topojson file and compare that value to the series to render a map of us-counties.
import branca
colorscale = branca.colormap.linear.YlOrRd_09.scale(0, 50e3)
def style_function(feature):
county_dict = cms_2017_grouped_series.get(
features['objects']['tl_2017_us_county']['geometries']['properties']['GEOID'], None)
return {
'fillOpacity': 0.5,
'weight': 0,
'fillColor': '#black' if employed is None else colorscale(employed)
}
The error I'm getting is AttributeError: 'list' object has no attribute 'get'
The rest of code needed to render the map is below
m = folium.Map(
location=[48, -102],
tiles='cartodbpositron',
zoom_start=3
)
folium.TopoJson(
json.load(open(county_geo)),
'objects.tl_2017_us_county.geometries.properties.GEOID',
style_function=style_function
).add_to(m)

I followed your steps to create the topojson and good news, it checks out. Just need to change up a couple of things with your code
I created some mock user data first. I'm using geopandas and the topjson file to make it easy on myself, but you would just use your pandas dataframe that contains county and employment numbers
import geopandas as gpd
gdf = gpd.read_file('tl_2017_us_county.json')
gdf['employed'] = np.random.randint(low=1, high=100000, size= len(gdf))
Create a Series using your dataframe. This will be used in the style func to "bind" your data to the map
cms_2017_grouped_series = gdf.set_index('GEOID')['employed']
print(cms_2017_grouped_series.head())
GEOID
31039 54221
53069 68374
35011 8477
31109 2278
31129 40247
Name: employed, dtype: int64
This is pretty close to your style function. I've just changed the line with the .get() to use the corrected dict keys of feature. Oh and I'm using the return value(employed) in the fillColor below
import branca
colorscale = branca.colormap.linear.YlOrRd_09.scale(0, 50e3)
def style_function(feature):
employed = cms_2017_grouped_series.get(feature['properties']['GEOID'], None)
return {
'fillOpacity': 0.5,
'weight': 0,
'fillColor': '#black' if employed is None else colorscale(employed)
}
Slight mod of the object_path is next. I'm also saving the map and then opening it in Chrome since it wouldn't render in my notebook due to the size
m = folium.Map(
location=[48, -102],
tiles='cartodbpositron',
zoom_start=3
)
folium.TopoJson(open('tl_2017_us_county.json'), 'objects.tl_2017_us_county',
style_function=style_function).add_to(m)
m.save('map.html')

Related

Why Python stops to work on GPU when using SimpleITK library in MONAI transforms?

I'm using Python 3.9 with Spyder 5.2.2 (Anaconda) for a U-Net segmentation task with MONAI. After importing all the images in a dictionary, I create these lines to define pre-process steps:
import SimpleITK as sitk
from monai.inferers import SimpleInferer
from monai.transforms import (
AsDiscrete,
DataStatsd,
AddChanneld,
Compose,
Activations,
LoadImaged,
Resized,
RandFlipd,
ScaleIntensityRanged,
DataStats,
AsChannelFirstd,
AsDiscreted,
ToTensord,
EnsureTyped,
RepeatChanneld,
EnsureType
)
from monai.transforms import Transform
monai_load = [
LoadImaged(keys=["image","segmentation"],image_only=False,reader=PILReader()),
EnsureTyped(keys=["image", "segmentation"], data_type="numpy"),
AddChanneld(keys=["segmentation","image"]),
RepeatChanneld(keys=["image"],repeats=3),
AsChannelFirstd(keys=["image"], channel_dim = 0),
]
monai_transforms =[
AsDiscreted(keys=["segmentation"],threshold=0.5),
ToTensord(keys=["image","segmentation"]),
]
class N4ITKTransform(Transform):
def __call__(self,image):
filtered = []
for channel in image["image"]:
inputImage = sitk.GetImageFromArray(channel)
inputImage = sitk.Cast(inputImage, sitk.sitkFloat32)
corrector = sitk.N4BiasFieldCorrectionImageFilter()
outputImage = corrector.Execute(inputImage)
filtered.append(sitk.GetArrayFromImage(outputImage))
image["image"] = np.stack(filtered)
return image
train_transforms = Compose(monai_load + [N4ITKTransform()] + monai_transforms)
When i recall these transforms with Compose and apply them to the train images, python does not work on GPU despite
torch.cuda.is_available()
return True.
These are the lines where I apply the transforms:
train_ds = IterableDataset(data = train_data, transform = train_transforms)
train_loader = DataLoader(dataset = train_ds, batch_size = batch_size, num_workers = 0, pin_memory = True)
When I define the U-Net model, I send it to 'cuda'.
The problem is in the SimpleITK transform. If I don't use them, Python works on GPU as usual.
Thank you in advance for getting back to me.
Federico
The answer is simple: SimpleITK uses CPU for processing.
I am not sure whether it is possible to get it to use some of the GPU-accelerated filters from ITK (its base library). If you use ITK Python, you have the possibility to use GPU-filters. But only a few filters have GPU implementations. N4BiasFieldCorrection does NOT have a GPU implementation. So if you want to use this filter, it needs to be done on the CPU.

color bar issue with cartopy, 'GeoAxesSubplot' object has no attribute 'get_array'

I'm using pandas, geopandas, and cartopy to make spatial plots for data points.
Everything works fine, except for when I try to add color bar.
Below is my code and the error. Any help is appreciated.
fig = plt.figure()
ax = plt.axes(projection=ccrs.PlateCarree())
reader = shpreader.Reader('countyl010g.shp')
counties = list(reader.geometries())
COUNTIES = cfeature.ShapelyFeature(counties, ccrs.PlateCarree())
ax.add_feature(COUNTIES, facecolor='none', edgecolor='gray')
ax.coastlines()
ax.add_feature(cartopy.feature.STATES)
dp=pd.read_csv('Arash.csv',index_col=False)
def remove_minutes(state):
state=datetime.datetime.strptime(state, '%Y-%m-%d %H:%M:%S')
state= state.replace(minute=0)
return state
dp['TIMESTAMP']=dp['TIMESTAMP'].apply(remove_minutes)
dp.set_index(['TIMESTAMP'], inplace=True)
dp= dp[dp.index.day == 28]
dp['coordinates'] = dp[['Longitude', 'Latitude']].values.tolist()
dp['coordinates'] = dp['coordinates'].apply(Point)
dp = gpd.GeoDataFrame(dp, geometry='coordinates')
ac=dp.plot(ax=ax,column='CO_CMAQ',markersize=0.05,cmap='turbo')
ax.set_xlim(-119,-117)
ax.set_ylim(33.5,34.5)
fig.colorbar(ac,ax=ax)
And here is the error:
File "C:\Python-practice\GHG\spatial_plot_mobile.py", line 102, in
fig.colorbar(ac,ax=ax)
File
"C:\Users\akash\anaconda3\lib\site-packages\matplotlib\figure.py",
line 2343, in colorbar cb = cbar.colorbar_factory(cax, mappable,
**cb_kw)
File
"C:\Users\akash\anaconda3\lib\site-packages\matplotlib\colorbar.py",
line 1734, in colorbar_factory cb = Colorbar(cax, mappable, **kwargs)
File
"C:\Users\akash\anaconda3\lib\site-packages\matplotlib\colorbar.py",
line 1202, in init
if mappable.get_array() is not None: AttributeError: 'GeoAxesSubplot'
object has no attribute 'get_array'
Thank you again,
Old question, but I landed on it because I had an analogous error, so in case it helps anyone else in future:
The problem is that ac here is a GeoAxesSubplot object within a Matplotlib figure. When Matplotlib is given fig.colorbar(ac,ax=ax), it doesn't know what to do with ac.
As per the Geopandas documentation, the information about the colorbar needs to go into the Geopandas .plot() itself, so that the relevant line above would look something like:
ac=dp.plot(ax=ax,column='CO_CMAQ',markersize=0.05,cmap='turbo',
legend=True, legend_kwds={'label':"",'orientation':""})
with the legend_kwds filled in as desired, e.g. orientation="horizontal".
More advice on modifying Geopandas colorbars here.

Pytorch: load dataset of grayscale images

I want to load a dataset of grayscale images. I used ImageFolder but this doesn't load gray images by default as it converts images to RGB.
I found solutions that load images with ImageFolder and after convert images in grayscale, using:
transforms.Grayscale(num_output_channels=1)
or
ImageOps.grayscale(image)
Is it correct?
How can I load grayscale imaged without conversion? I try ImageDataBunch, but I have problems to import fastai.vision
Assuming the dataset is stored in the "Dataset" folder as given below, set the root directory as "Dataset":
Dataset
class_1
img1.png
img2.png
class_2
img1.png
img2.png
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader, random_split
from torchvision import transforms
root = 'Dataset/'
data_transform = transforms.Compose([transforms.Grayscale(num_output_channels=1),
transforms.ToTensor()])
dataset = ImageFolder(root, transform=data_transform)
For reference, train and test dataset are being split into 70% and 30% respectively.
# Split test and train dataset
train_size = int(0.7 * len(dataset))
test_size = len(dataset) - train_size
train_data, test_data = random_split(dataset, [train_size, test_size])
This dataset can be further divided into train and test data loaders as given below to perform operation in batches.
Usually you will see the dataset is assigned batch_size once to be used for both train and test loaders. But, I try to define it separately. The idea is to give the batch_size such that it is a factor of the train/test data loader's size, otherwise it will give an error.
# Set batch size of train data loader
batch_size_train = 20
# Set batch size of test data loader
batch_size_test = 22
# load the split train and test data into batches via DataLoader()
train_loader = DataLoader(train_data, batch_size=batch_size_train, shuffle=True)
test_loader = DataLoader(test_data, batch_size=batch_size_test, shuffle=True)
Yes, that is correct and AFAIK pillow by default loads images in RGB, see e.g. answers to this question. So conversion to grayscale is the only way, though takes time of course.
Pure pytorch solution (if ImageFolder isn't appropriate)
You can roll out your own data loading functionalities and If I were you I wouldn't go fastai route as it's pretty high level and takes away control from you (you might not need those functionalities anyway).
In principle, all you have to do is to create something like this below:
import pathlib
import torch
from PIL import Image
class ImageDataset(torch.utils.data.Dataset):
def __init__(self, path: pathlib.Path, images_class: int, regex="*.png"):
self.files = [file for file in path.glob(regex)]
self.images_class: int = images_class
def __getitem__(self, index):
return Image.open(self.files[index]).convert("LA"), self.images_class
# Assuming you have `png` images, can modify that with regex
final_dataset = (
ImageDataset(pathlib.Path("/path/to/dogs/images"), 0)
+ ImageDataset(pathlib.Path("/path/to/cats/images"), 1)
+ ImageDataset(pathlib.Path("/path/to/turtles/images"), 2)
)
Above would get you images from the paths provided above and each image would return appropriate provided class.
This gives you more flexibility (different folder setting than torchvision.datasets.ImageFolder) for a few more lines.
Ofc, you could add more of those or use loop or whatever else.
You could also apply torchvision.transforms, e.g. transforming images above to tensors, read
torchdata solution
Disclaimer, author here. If you are cocerned about loading times of your data and grayscale transformation you could use torchdata third party library for pytorch.
Using it one could create the same thing as above but use cache or map (to use torchvision.transforms or other transformations easily) and some other things known e.g. from tensorflow.data module, see below:
import pathlib
from PIL import Image
import torchdata
# Change inheritance
class ImageDataset(torchdata.Dataset):
def __init__(self, path: pathlib.Path, images_class: int, regex="*.png"):
super().__init__() # And add constructor call and that's it
self.files = [file for file in path.glob(regex)]
self.images_class: int = images_class
def __getitem__(self, index):
return Image.open(self.files[index]), self.images_class
final_dataset = (
ImageDataset(pathlib.Path("/path/to/dogs/images"), 0)
+ ImageDataset(pathlib.Path("/path/to/cats/images"), 1)
+ ImageDataset(pathlib.Path("/path/to/turtles/images"), 2)
).cache() # will cache data in-memory after first pass
# You could apply transformations after caching for possible speed-up
torchvision ImageFolder loader
As correctly pointed out by #jodag in the comments, one can use loader callable with single argument path to do customized data opening, e.g. for grayscale it could be:
from PIL import Image
import torchvision
dataset = torchvision.datasets.ImageFolder(
"/path/to/images", loader=lambda path: Image.open(path).convert("LA")
)
Please notice you could also use it for other types of files, those doesn't have to be images.
Make custom loader, feed it to ImageFolder:
import numpy as np
from PIL import Image, ImageOps
def gray_reader(image_path):
im = Image.open(image_path)
im2 = ImageOps.grayscale(im)
im.close()
return np.array(im2) # return np array
# return im2 # return PIL Image
some_dataset = ImageFolder(image_root_path, loader=gray_reader)
Edit:
Below code is much better than previous, get color image and convert to grayscale in transform()
def get_transformer(h, w):
valid_transform = transforms.Compose([
transforms.ToPILImage(),
transforms.Grayscale(num_output_channels=1),
transforms.Resize((h, w)),
transforms.ToTensor(),
transforms.Normalize([0.5], [0.5]) ])
return valid_transform

kivy: possible to use buffer as image source?

I've got code along the lines of the following which generates a new image out of some existing images.
from PIL import Image as pyImage
def create_compound_image(back_image_path, fore_image_path, fore_x_position):
back_image_size = get_image_size(back_image_path)
fore_image_size = get_image_size(fore_image_path)
new_image_width = (fore_image_size[0] / 2) + back_image_size[0]
new_image_height = fore_image_size[1] + back_image_size[1]
new_image = create_new_image_canvas(new_image_width, new_image_height)
back_image = pyImage.open(back_image_path)
fore_image = pyImage.open(fore_image_path)
new_image.paste(back_image, (0, 0), mask = None)
new_image.paste(fore_image, (fore_x_position, back_image_size[1]), mask = None)
return new_image
Later in the code, I've got something like this:
from kivy.uix.image import Image
img = Image(source = create_compound_image(...))
If I do the above, I get the message that Image.source only accepts string/unicode.
If I create a StringIO.StringIO() object from the new image, and try to use that as the source, the error message is the same as above. If I use the output of the StringIO object's getvalue() method as the source, the message is that the source must be encoded string without NULL bytes, not str.
What is the proper way to use the output of the create_compound_image() function as the source when creating a kivy Image object?
It seems you want to just combine two images into one, you can actually just create a texture using Texture.create and blit the data to a particular pos using Texture.blit_buffer .
from kivy.core.image import Image
from kivy.graphics import Texture
bkimg = Image(bk_img_path)
frimg = Image(fr_img_path)
new_size = ((frimg.texture.size[0]/2) + bkimg.texture.size[0],
frimg.texture.size[1] + bkimg.texture.size[1])
tex = Texture.create(size=new_size)
tex.blit_buffer(pbuffer=bkimg.texture.pixels, pos=(0, 0), size=bkimg.texture.size)
tex.blit_buffer(pbuffer=frimg.texture.pixels, pos=(fore_x_position, bkimg.texture.size[1]), size=frimg.texture.size)
Now you can use this texture anywhere directly like::
from kivy.uix.image import Image
image = Image()
image.texture = tex
source is a StringProperty and is expecting a path to file. That's why you got errors when you tried to pass PIL.Image object, StringIO object or string representation of image. It's not what framework wants. As for getting image from StringIO, it was discussed before here:
https://groups.google.com/forum/#!topic/kivy-users/l-3FJ2mA3qI
https://github.com/kivy/kivy/issues/684
You can also try much simpler, quick and dirty method - just save your image as a tmp file and read it normal way.

How to choose which figure on which to display an image with matplotlib

I have a python script with multiple figures that I would like to update during a loop. Some will be images, and others will be line/scatter plots. I am having trouble getting the image to display on the correct figure. (the line and scatter data are showing up on the right figures, but the image seems to always be going on the figure that was created last, eventually I'll be displaying more than one image figure, so I can't just create the image figure last)
Here is roughly the code I have so far, the 3D scatter plot is showing up on Figure 1, but both the image and the line plots are showing up on Figure 3, with Figure 2 blank:
import matplotlib.pyplot as plt
from collections import deque
class Bla():
def __init__( self ):
self.pc_fig = plt.figure(1)
self.pc_ax = self.pc_fig.add_subplot(111, projection='3d')
self.pc_ax.set_xlim3d([0, 50])
self.pc_ax.set_ylim3d([0, 50])
self.pc_ax.set_zlim3d([0, 20])
self.pc_ax.hold(False)
self.vts_fig = plt.figure(2)
self.vts_ax = self.vts_fig.add_subplot(111)
self.em_fig = plt.figure(3)
self.em_ax = self.em_fig.add_subplot(111)
self.em_ax.hold(True)
self.image_data = deque()
self.motion_data = deque()
plt.ion()
plt.show()
def run( self ):
em_prev_xy = ( 0, 0 )
while True:
if len( self.motion_data ) > 0:
data1 = self.motion_data.popleft()
em_xy = data1.get_em()
self.em_ax.plot( [ em_prev_xy[0], em_xy[0] ], [ em_prev_xy[1], em_xy[1] ],'b')
pc = self.get_pc()
pc_index = nonzero(pc>.002)
pc_value = pc[pc_index] * 100
self.pc_ax.scatter(pc_index[0],pc_index[1],pc_index[2],s=pc_value)
self.pc_ax.set_xlim3d([0, 50])
self.pc_ax.set_ylim3d([0, 50])
self.pc_ax.set_zlim3d([0, 20])
plt.pause( 0.0001 ) # This is needed for the display to update
if len( self.image_data ) > 0:
im = self.image_data.popleft()
plt.imshow( im, cmap=plt.cm.gray, axes=self.vts_ax )
plt.pause( 0.0001 )
def main():
bla = Bla()
bla.run()
if __name__ == "__main__":
main()
Basically I have some queues that get populated in a callback when new data arrives, and I want this data to be displayed as it arrives.
I am new to matplotlib, so any help with my image display issue or tips for better ways of using matplotlib to display figures in general will be much appreciated
You are mixing the OO and state machine interfaces. See this answer for an explanation of what is going on.
Replace this line:
plt.imshow( im, cmap=plt.cm.gray, axes=self.vts_ax )
with
the_axes_you_want.imshow(...)
which should fix your image issue.

Resources