i have trouble with this error while using word2vec from gensim
MemoryError: Unable to allocate 380. MiB for an array with shape (331792, 300) and data type float32
and this is my code
from gensim.models import Word2Vec
#app.route('/admin/index')
def show_admin():
model_baru = Word2Vec.load('model/idwiki_word2vec_300.model')
tester = model_baru.most_similar(positive=['wanita'],topn=3)
hasil = print(tester)
return render_template('admin/index.html',hasil=hasil)
what happen with this error?
please help me
thanks for your attention
Related
I have a simple request but the result seems complicated to reach.
I have done a simple model in Tensorflow based on images divided in 3 labels (quite simple).
After data importation, the model is trained + tested up to 60% of good results.
Then, I'd like to test a single image using the existing model and see if the model guesses the correct label.
The whole process is quite simple up to the test process:
#%%
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
test_image = mpimg.imread(os.path.join("images","cnn","cnn_test.png"))[:, :, :channels]
plt.imshow(test_image)
plt.axis("off")
plt.show()
X_test = test_image.reshape(-1,height, width, channels)
# initialize the variables
#sess.run(tf.global_variables_initializer())
X_test = test_image.reshape(-1, height, width, channels)
tf.reset_default_graph()
# x is the input array, which will contain the data from an image
saver = tf.train.import_meta_graph('./02_CNN/data/my_model8.ckpt.meta')
print('meta graph imported')
X_test = test_image.reshape(-1, height, width, channels)
sess = tf.Session()
saver.restore(sess, './02_CNN/data/my_model8.ckpt')
print('model graph restored')
feed_dict = {x: X_test}
prediction = sess.run(feed_dict)
max_index = np.argmax(prediction)
print(max_index)
I have the following result:
TypeError: Fetch argument array([[[[ 0.03137255, 0.20392157, 0.36862746],
[ 0.02745098, 0.20784314, 0.37254903],
[ 0.02352941, 0.21176471, 0.3764706 ],
...,
....
[ 0.08627451, 0.08235294, 0.09019608],
[ 0.12941177, 0.12156863, 0.12941177],
[ 0.13725491, 0.1254902 , 0.13725491]]]], dtype=float32)
has invalid type <class 'numpy.ndarray'>, must be a string or Tensor.
(Can not convert a ndarray into a Tensor or Operation.)
Any idea about how to solve this issue?
Many thanks in advance,
Nicolas
I would like to train a simple graph in Tensorflow with labelled images.
Here is the code:
images_train = tf.convert_to_tensor(image_list_train, dtype=dtypes.string)
labels_train = tf.convert_to_tensor(label_list_train, dtype=dtypes.int32)
with tf.Session() as sess:
init.run()
sess.run(training_op, feed_dict={X: images_train.eval(), y: labels_train.eval()})
Unfortunally, it doesn't recognize the data and here is the error message:
ValueError: could not convert string to float:
b'C:/Users/Nicolas/Documents/Photos3/im_photo_224.jpg'
Any idea to solve this issue?
Many thanks in advance,
Nicolas
I recently asked how to convert Float32 or Uint8 arrays into images in the Images package. I got an answer for the Float32 case, but am still having trouble figuring out how to save a Uint8 array.
As an example, let's create a random Uint8 array using the traditional Matlab scheme where the dimensions are (m,n,3):
array = rand(Uint8, 50, 50, 3);
img = convert(Image, array);
Using the same approach as works for the Float32 case,
imwrite(img, "out.png")
fails with message
ERROR: method 'mapinfo' has no method matching mapinfo(::Type{ImageMagick}, ::Image{Uint8, 3, Image{Uint8, 3, Array{Uint8, 3}}}).
I checked the documentation, and it says
If data encodes color information along one of the dimensions of the array (as opposed to using a ColorValue array, from the Color.jl package), be sure to specify the "colordim" and "colorspace" in properties.
However, inspecting the img object previously created shows that it has colordim = 3 and colorspace = RGB already set up, so this can't be the problem.
I then searched the documentation for all instances of MapInfo. In core.md there is one occurrence:
scalei: a property that controls default contrast scaling upon display. This should be a MapInfo value, to be used for setting the contrast upon display. In the absence of this property, the range 0 to 1 will be used.
But there was no information on what exactly a MapInfo object is, so I looked further, and in function_reference.md it says:
Here is how to directly construct the major concrete MapInfo types:
MapNone(T), indicating that the only form of scaling is conversion to type T. This is not very safe, as values "wrap around": for example, converting 258 to a Uint8 results in 0x02, which would look dimmer than 255 = 0xff.
...
and some other examples. So I tried to specify scalei = MapNone(Uint8) as follows:
img2 = Image(img, colordim = 3, colorspace = "RGB", scalei = MapNone(Uint8));
imwrite(img, "out.png")
but got the same error again.
How do you encode Uint8 image data using Images in Julia?
You can convert back and forth between arrays of primitive types such as UInt8 and arrays of color types. These conversions are achieved in a unified way via two functions: colorview and channelview.
Example
Convert array of UInt8 to array of RGB:
arr = rand(UInt8, 3, 50, 50)
img = colorview(RGB, arr / 255)
Convert back to channel view:
channelview(img)
Notes
In this example the RGB color type requires that the entries of the array live in [0,1] as floating point. I manually converted UInt8 to Float64 using an explicit division by 255. There is probably a more generic way of achieving this result with reinterpret or some other function in Images.jl
The colorview and channelview functions assume that the channel dimension is the first dimension of the array. You can use permutedims in case your channels live in a different dimension, or use some function in Images.jl (maybe reinterpretc?) to do it efficiently without memory copies.
I am using sci-kit image to get the "regionprops" of a segmented image. I then wish to replace each of the segment labels with their corresponding statistic (e.g eccentricity).
from skimage import segmentation
from skimage.measure import regionprops
#a segmented image
labels = segmentation.slic(img1, compactness=10, n_segments=200)
propimage = labels
#props loop
for region in regionprops(labels1, properties ='eccentricity') :
eccentricity = region.eccentricity
propimage[propimage==region] = eccentricity
This runs, but the propimage values do not change from their original labels
I have also tried:
for i in range(0,max(labels)):
prop = regions[i].eccentricity #the way to cal a single prop
propimage[i]= prop
This delivers this error
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
I am a recent migrant from matlab where I have implemented this, but the data structures used are completely different.
Can any one help me with this?
Thanks
Use ndimage from scipy : the sum() function can operate using your label array.
from scipy import ndimage as nd
sizes = nd.sum(label_file[0]>0, labels=label_file[0], index=np.arange(0,label_file[1])
You can then evaluate the distribution with numpy.histogram and so on.
from scipy.misc import imread
from matplotlib import pyplot
import cv2
from cv2 import cv
from SRM import SRM ## Module for Statistical Regional Segmentation
im = imread("lena.png")
im2 = cv2.imread("lena.png")
print type(im), type(im2), im.shape, im2.shape
## Prints <type 'numpy.ndarray'> <type 'numpy.ndarray'> (120, 120, 3) (120, 120, 3)
srm = SRM(im, 256)
segmented = srm.run()
srm2 = SRM(im2, 256)
segmented2 = srm2.run()
pic = segmented/256
pic2 = segmented2/256
pyplot.imshow(pic)
pyplot.imsave("onePic.jpg", pic)
pic = pic.astype('uint8')
cv2.imwrite("onePic2.jpg", pic2)
pyplot.show()
onePic.jpg gives the correct segmented image but onePic2.jpg gives a complete black image.
Converting the datatype to uint8 using pic = pic.astype('uint8') did not help. I still gives a black image!
onePic.jpg using pyplot.imsave():
onePic2.jpg using cv2.imwrite():
Please help!
Before converting pic to uint8, you need to multiply it by 255 to get the correct range.
Although I agree with #sansuiso, in my case I found a possible edge case where my images were being shifted either one bit up in the scale or one bit down.
Since we're dealing with unsigned ints, a single shift means a possible underflow/overflow, and this can corrupt the whole image.
I found cv2's convertScaleAbs with an alpha value of 255.0 to yield better results.
def write_image(path, img):
# img = img*(2**16-1)
# img = img.astype(np.uint16)
# img = img.astype(np.uint8)
img = cv.convertScaleAbs(img, alpha=(255.0))
cv.imwrite(path, img)
This answer goes into more detail.
I encountered a similar situation with face detection, I wonder if there is a better way to execute this, here is my solution here as a reference.
from deepface import DeepFace
import cv2
import matplotlib.pyplot as plt
# import image and output
img_path = "image.jpg"
detected_face = DeepFace.detectFace(img_path, target_size = (128, 128))
plt.imshow(detected_face)
# image color scaling and saving
detected_face = cv2.cvtColor( detected_face,cv2.COLOR_BGR2RGB)
detected_face = cv2.convertScaleAbs(detected_face, alpha=(255.0))
cv2.imwrite("image_thumbnail.jpg", detected_face)