Sorry, this might turn out to be a very basic problem but I'm stumped:
I'm trying to get the lengths of skeleton branches. I'm able to skeletonize my image and can see the skeleton with draw.overlay_skeleton_2d() but when I try to get any statistics on the skeleton back I get an error:
TypeError Traceback (most recent call last)
<ipython-input-3-5924e1b8ffc8> in <module>
4 # fig, ax = plt.subplots()
5 # draw.overlay_skeleton_2d(img, skeleton0, dilate=1, axes=ax)
----> 6 branch_data = summarize(Skeleton(skeleton0))
7 branch_data.head()
~\AppData\Roaming\Python\Python38\site-packages\skan\csr.py in __init__(self, skeleton_image, spacing, source_image, _buffer_size_offset, keep_images, junction_mode, unique_junctions)
383 self.nbgraph = csr_to_nbgraph(graph, pixel_values)
384 self.coordinates = coords
--> 385 self.paths = _build_skeleton_path_graph(
386 self.nbgraph, _buffer_size_offset=_buffer_size_offset
387 )
~\AppData\Roaming\Python\Python38\site-packages\skan\csr.py in _build_skeleton_path_graph(graph, _buffer_size_offset)
259 def _build_skeleton_path_graph(graph, *, _buffer_size_offset=None):
260 if _buffer_size_offset is None:
--> 261 max_num_cycles = graph.indices.size // 4
262 _buffer_size_offset = max_num_cycles
263 degrees = np.diff(graph.indptr)
TypeError: expected dtype object, got 'numpy.dtype[int32]'
Any tips would be great. I'm still a Python beginner so, like I said, the mistake might be a very basic one.
Rgds,
Olie
*Edits included
This is my code:
import cv2
from skan import draw, skeleton_to_csgraph, Skeleton, summarize
from skimage import morphology
# import numpy as np
from matplotlib import pyplot as plt
%matplotlib notebook
# load image and convert to bool
img = cv2.imread(save_p + "\\" + img_name + ".png")
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img = img.astype(bool)
# skeletonize
skeleton0 = morphology.skeletonize(img)
### check skeleton
# fig, ax = plt.subplots()
# draw.overlay_skeleton_2d(img, skeleton0, dilate=1, axes=ax)
### from Documentation (linked below)
branch_data = summarize(Skeleton(skeleton0)) # ignoring scaling for now
branch_data.head()
# Skel_obj = Skeleton(skeleton0) # testing this but same problem
# Skel_obj.path_lengths(0)
I have attempted to follow a couple of posts (e.g. this but for that particular case the module wont install and anyway... it seems like skan.summarize is exactly what I need - documentation
Name: numpy
Version: 1.23.4
Name: scikit-image
Version: 0.19.3
Name: pandas
Version: 1.3.4
Name: skan
Version: 0.10.0
Name: numba
Version: 0.50.1
Windows 10 PC
Related
There is a package based on the Catboost algorithm, [https://contrib.scikit-learn.org/category_encoders/_modules/category_encoders/cat_boost.html#CatBoostEncoder] that claims to use catboost algorithm to encode datasets. But it has not had all the features in the catboost package that allow refining the model training phase. After training, I'm trying to find a way to use the catboost to transform the categorical variables from a dataset. Could you help me with that?
# If you want to test this on your local notebook
# http://contrib.scikit-learn.org/categorical-encoding/
!pip install category-encoders
# import libraries
import pandas as pd
# Make dataset based on Josh Starmer video example
# https://www.youtube.com/watch?v=EzjtTh-WUWY
categorical = ["Blue","Red","Green","Blue","Green","Green","Blue"]
numerical = [1.72, 1.32, 1.81, 1.56, 1.64, 1.61, 1.73]
Label = [1 , 0 , 1 , 0 , 1 , 0 , 0]
df = pd.DataFrame({
'favorite_color':categorical,
'Hight(m)':numerical,
'LovesTroll2':Label,
})
feature_list = list(df.columns) #['favorite_color', 'Hight(m)', 'LovesTroll2']
%%time
# import libraries
from category_encoders.cat_boost import CatBoostEncoder
import category_encoders as ce
# Define catboost encoder
cbe_encoder = ce.cat_boost.CatBoostEncoder() #approach1
CBE_encoder = CatBoostEncoder() #approach2
# Fit encoder and transform the features
train_cbe = cbe_encoder.fit_transform(df[feature_list], df[feature_list[-1]]) #approach1
Train_cbe = CBE_encoder.fit_transform(df[feature_list], df[feature_list[-1]]) #approach2
#print(Train_cbe)
# favorite_color Hight(m) LovesTroll2
#0 0.428571 1.72 1
#1 0.428571 1.32 0
#2 0.428571 1.81 1
#3 0.714286 1.56 0
#4 0.714286 1.64 1
#5 0.809524 1.61 0
#6 0.476190 1.73 0
# plot the encoded results over target/label
#train_cbe.plot(style=['o','rx'])
import matplotlib.pyplot as plt
plt.scatter(Train_cbe['LovesTroll2'], Train_cbe['favorite_color'])
plt.show()
I am trying to do a simple custom training loop.
For some reason, the train_step() function gets ignored and a normal training loop is carried. I noticed when the word "Hello" is not printed when I run the script on my MacBook Pro with M1 chip. My old MacBook Pro (AMD) works perfectly and the code also worked perfectly on Google Colab.
The tensor flow version is 2.0.0 and Keras is 2.3.1. Thanks in advance for your help.
My Code is:
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import Model
# tf.config.run_functions_eagerly(True)
class CustomModel(Model):
# #tf.function
def train_step(self, data):
# Unpack the data. Its structure depends on your model and
# on what you pass to `fit()`.
x, y = data
print('Hello')
with tf.GradientTape() as tape:
y_pred = self(x, training=True) # Forward pass
# Compute the loss value
# (the loss function is configured in `compile()`)
loss = self.compiled_loss(y, y_pred, regularization_losses=self.losses)
# Compute gradients
trainable_vars = self.trainable_variables
gradients = tape.gradient(loss, trainable_vars)
# Update weights
self.optimizer.apply_gradients(zip(gradients, trainable_vars))
# Update metrics (includes the metric that tracks the loss)
self.compiled_metrics.update_state(y, y_pred)
# Return a dict mapping metric names to current value
return {m.name: m.result() for m in self.metrics}
# Construct and compile an instance of CustomModel
inputs = keras.Input(shape=(32,))
outputs = keras.layers.Dense(1)(inputs)
model = CustomModel(inputs, outputs)
model.compile(optimizer="adam", loss="mse", metrics=["mae"])
# Just use `fit` as usual
x = np.random.random((1000, 32))
y = np.random.random((1000, 1))
model.fit(x, y, epochs=3)
Hello is not printed:
Hello is printed during training:
I am practicing with this life expectancy dataset from Kaggle (https://www.kaggle.com/datasets/kumarajarshi/life-expectancy-who?select=Life+Expectancy+Data.csv) and I want to train and visualize a classification and regression tree model. however, I keep getting an error that says "InvocationException: GraphViz's executables not found". I am wondering if this is because of the nature of the continuous numerical target dataset type? how can I visualize the model?
code:
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
import numpy as np
import seaborn as sn
from sklearn import datasets
from sklearn import metrics
from sklearn import tree
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder
from sklearn.tree import export_graphviz
import matplotlib.pyplot as plt,pydotplus
from IPython.display import Image,display
data = pd.read_csv('Life Expectancy Data.csv')
data = data.dropna(how = 'any')
#feature selection
data = data.drop(columns=['infant deaths', ' thinness 5-9 years', 'Alcohol', 'percentage expenditure', 'Hepatitis B', 'Total expenditure', 'Population', ' thinness 5-9 years', 'Year', 'Country'])
# Creating a instance of label Encoder.
le = LabelEncoder()
# Using .fit_transform function to fit label
# encoder and return encoded label
label = le.fit_transform(data['Status'])
# removing the column 'Status' from df
data.drop('Status', axis=1, inplace=True)
# Appending the array to our dataFrame
# with column name 'Status'
data['Status'] = label
#training model
model_data = data
X = data.drop(columns=['Life expectancy '])
y = data['Life expectancy ']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
model = DecisionTreeRegressor()
model.fit(X_train, y_train)
#visualizing tree
LEtree = tree.export_graphviz(model,
feature_names = ['Adult Mortality', 'Measles', ' BMI', 'under-five deaths', 'Polio', 'Diphtheria', ' HIV/AIDS', 'GDP', ' thinness 1-19 years', 'Income composition of resources', 'Schooling', 'Status'],
class_names = y,
label = 'all',
rounded = True,
filled = True)
graph=pydotplus.graph_from_dot_data(LEtree)
display(Image(graph.create_png()))
full error message:
InvocationException Traceback (most recent call last)
Input In [27], in <cell line: 2>()
1 graph=pydotplus.graph_from_dot_data(LEtree)
----> 2 display(Image(graph.create_png()))
File ~\Anaconda3\lib\site-packages\pydotplus\graphviz.py:1797, in Dot.__init__.<locals>.<lambda>(f, prog)
1792 # Automatically creates all the methods enabling the creation
1793 # of output in any of the supported formats.
1794 for frmt in self.formats:
1795 self.__setattr__(
1796 'create_' + frmt,
-> 1797 lambda f=frmt, prog=self.prog: self.create(format=f, prog=prog)
1798 )
1799 f = self.__dict__['create_' + frmt]
1800 f.__doc__ = (
1801 '''Refer to the docstring accompanying the'''
1802 ''''create' method for more information.'''
1803 )
File ~\Anaconda3\lib\site-packages\pydotplus\graphviz.py:1959, in Dot.create(self, prog, format)
1957 self.progs = find_graphviz()
1958 if self.progs is None:
-> 1959 raise InvocationException(
1960 'GraphViz\'s executables not found')
1962 if prog not in self.progs:
1963 raise InvocationException(
1964 'GraphViz\'s executable "%s" not found' % prog)
InvocationException: GraphViz's executables not found
Try Installing the Graphviz in a proper directory
you can install in Anaconda from conda-command-prompt using the below command -
conda install -c conda-forge python-graphviz
and replace the previously installed graphviz directory this might help you with the problem
I used transfer learning to train the model. The fundamental model was efficientNet.
You can read more about it here
from tensorflow import keras
from keras.models import Sequential,Model
from keras.layers import Dense,Dropout,Conv2D,MaxPooling2D,
Flatten,BatchNormalization, Activation
from keras.optimizers import RMSprop , Adam ,SGD
from keras.backend import sigmoid
Activation function
class SwishActivation(Activation):
def __init__(self, activation, **kwargs):
super(SwishActivation, self).__init__(activation, **kwargs)
self.__name__ = 'swish_act'
def swish_act(x, beta = 1):
return (x * sigmoid(beta * x))
from keras.utils.generic_utils import get_custom_objects
from keras.layers import Activation
get_custom_objects().update({'swish_act': SwishActivation(swish_act)})
Model Definition
model = enet.EfficientNetB0(include_top=False, input_shape=(150,50,3), pooling='avg', weights='imagenet')
Adding 2 fully-connected layers to B0.
x = model.output
x = BatchNormalization()(x)
x = Dropout(0.7)(x)
x = Dense(512)(x)
x = BatchNormalization()(x)
x = Activation(swish_act)(x)
x = Dropout(0.5)(x)
x = Dense(128)(x)
x = BatchNormalization()(x)
x = Activation(swish_act)(x)
x = Dense(64)(x)
x = Dense(32)(x)
x = Dense(16)(x)
# Output layer
predictions = Dense(1, activation="sigmoid")(x)
model_final = Model(inputs = model.input, outputs = predictions)
model_final.summary()
I saved it using:
model.save('model.h5')
I get the following error trying to load it:
model=tf.keras.models.load_model('model.h5')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-12-e3bef1680e4f> in <module>()
1 # Recreate the exact same model, including its weights and the optimizer
----> 2 model = tf.keras.models.load_model('PhoneDetection-CNN_29_July.h5')
3
4 # Show the model architecture
5 model.summary()
10 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/utils/generic_utils.py in class_and_config_for_serialized_keras_object(config, module_objects, custom_objects, printable_module_name)
319 cls = get_registered_object(class_name, custom_objects, module_objects)
320 if cls is None:
--> 321 raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
322
323 cls_config = config['config']
ValueError: Unknown layer: FixedDropout
```python
I was getting the same error while trying to do the inference by loading my saved model.
Then i just imported the effiecientNet library in my inference notebook as well and the error was gone.
My import command looked like:
import efficientnet.keras as efn
(Note that if you havent installed effiecientNet already(which is unlikely), you can do so by using !pip install efficientnet command.)
I had this same issue with a recent model. Researching the source code you can find the FixedDropout Class. I added this to my inference code with import of backend and layers. The rate should also match the rate from your efficientnet model, so for the EfficientNetB0 the rate is .2 (others are different).
from tensorflow.keras import backend, layers
class FixedDropout(layers.Dropout):
def _get_noise_shape(self, inputs):
if self.noise_shape is None:
return self.noise_shape
symbolic_shape = backend.shape(inputs)
noise_shape = [symbolic_shape[axis] if shape is None else shape
for axis, shape in enumerate(self.noise_shape)]
return tuple(noise_shape)
model = keras.models.load_model('model.h5',
custom_objects={'FixedDropout':FixedDropout(rate=0.2)})
I was getting the same error. Then I import the below code. then it id working properly
import cv2
import matplotlib.pyplot as plt
import tensorflow as tf
from sklearn.metrics import confusion_matrix
import itertools
import os, glob
from tqdm import tqdm
from efficientnet.tfkeras import EfficientNetB4
if you don't have to install this. !pip install efficientnet. If you have any problem put here.
In my case, I had two files train.py and test.py.
I was saving my .h5 model inside train.py and was attempting to load it inside test.py and got the same error. To fix it, you need to add the import statements for your efficientnet models inside the file that is attempting to load it as well (in my case, test.py).
from efficientnet.tfkeras import EfficientNetB0
I am trying to classify two objects. I would like to get Accuracy and Cross Entropy from the evaluate.py script.
Here is the code I'm trying.
evaluate.py (by tensorflow for poets)
#!/usr/bin/python
#
# Copyright 2017 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import sys
import argparse
import numpy as np
import PIL.Image as Image
import tensorflow as tf
import scripts.retrain as retrain
from scripts.count_ops import load_graph
def evaluate_graph(graph_file_name):
with load_graph(graph_file_name).as_default() as graph:
ground_truth_input = tf.placeholder(
tf.float32, [None, 5], name='GroundTruthInput')
image_buffer_input = graph.get_tensor_by_name('input:0')
final_tensor = graph.get_tensor_by_name('final_result:0')
accuracy, _ = retrain.add_evaluation_step(final_tensor, ground_truth_input)
logits = graph.get_tensor_by_name("final_training_ops/Wx_plus_b/add:0")
xent = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
labels=ground_truth_input,
logits=logits))
#image_dir = 'tf_files/flower_photos'
image_dir = 'tf_files/test_images'
testing_percentage = 10
validation_percentage = 10
validation_batch_size = 100
category = 'testing'
image_lists = retrain.create_image_lists(
image_dir, testing_percentage,
validation_percentage)
class_count = len(image_lists.keys())
ground_truths = []
filenames = []
for label_index, label_name in enumerate(image_lists.keys()):
for image_index, image_name in enumerate(image_lists[label_name][category]):
image_name = retrain.get_image_path(
image_lists, label_name, image_index, image_dir, category)
ground_truth = np.zeros([1, class_count], dtype=np.float32)
ground_truth[0, label_index] = 1.0
ground_truths.append(ground_truth)
filenames.append(image_name)
accuracies = []
xents = []
with tf.Session(graph=graph) as sess:
for filename, ground_truth in zip(filenames, ground_truths):
image = Image.open(filename).resize((224, 224), Image.ANTIALIAS)
image = np.array(image, dtype=np.float32)[None, ...]
image = (image - 128) / 128.0
feed_dict = {
image_buffer_input: image,
ground_truth_input: ground_truth}
eval_accuracy, eval_xent = sess.run([accuracy, xent], feed_dict)
accuracies.append(eval_accuracy)
xents.append(eval_xent)
return np.mean(accuracies), np.mean(xents)
if __name__ == "__main__":
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
accuracy, xent = evaluate_graph(*sys.argv[1:])
print('Accuracy: %g' % accuracy)
print('Cross Entropy: %g' % xent)
However when I run the above script for prediction I get the following error:
ValueError: Cannot feed value of shape (1, 224, 224) for Tensor
u'input:0', which has shape '(1, 224, 224, 3)'
How can I solve this error?
It seems like you are feeding grayscale images into input placeholder. Grayscale images have only 1 channel, hence the shape (224, 224, ) (dimension of size 1 is omitted), while pretrained network you are trying requires RGB images with 3 channels and shape (224, 224, 3)
If your images are actualy RGB, you might have an error here:
image = np.array(image, dtype=np.float32)[None, ...]
this indexing: [None, ...] does not seem necessary.
If your images are actually grayscale, you may convert them into RGB format using PIL.convert() (one channel will be repeated 3 times):
image = image.convert("RGB")
Although with channel duplication running 3-channel CNN is inefficient (computation is performed for the same data 3 times) and likely will perform worse than with colored images, this should run the script and will get you on track quickly.