I am using flow_from_directory in ImageDataGenerator from keras to train my convolution neural network. I downloaded the ILSVRC2012 images from this link and I am trying to train and validate my network. The training folder has images categorized in corresponding folders, but the validation images are not categorized into folders, which is not letting me use flow_from_directory to test the validation accuracy. Is there any easy way of categorizing the validation data into corresponding folders?
I tried with the below link of git code.
- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh
- https://github.com/pytorch/examples/tree/master/imagenet
This shell script worked for me to convert validation directory into sub directories where images categorized into corresponding folders.
Try to convert your validation directory into respective images categorized directories and give that parent directory of subdirectories as input to your code.
Thanks
You can use the libraries mxnet and gluon to classify the imagenet validation data in respective folders.
See https://gluon-cv.mxnet.io/build/examples_datasets/imagenet.html.
Then yuo can use the keras validation data generator.
I'm doing a finetuning in two classes, first you need create something like that in your code
path_directory = "../images/"
path_classes_name = ['class_0','class_2',...,'class_999'] full list in order
image.ImageDataGenerator(rescale=0).flow_from_directory(path_directory, target_size=(244, 244), batch_size= 128, class_mode='categorical', shuffle=True,classes=path_classes_name)
after this the path_directory need by organized exactly the blog.keras say. Keras will make the magic for you!
Tip: Don't forget then create the folders in order and with synset name (something like n04111531)! Not the name literal of classes =]
Related
I'm looking to extract all images from new and legacy Word documents and spreadsheets to assist in a real time document classification system, and looking at the documentation, I seem to have run into a problem. I'm having no problems finding documentation within the hwpf module and packages for extracting images from the file, but when it comes to the other 3, it seems as though they don't support the same methods.
What I want to do is to have one block of code that is document type agnostic when it comes to the 4 above mentioned types, I just want fast, easy access to the pictures in the files so I can move on to my next task, but at this point it looks like only the hwpf module supports extraction of pictures or the methods in 'PicturesTable'.
I'm also somewhat concerned about the performance of the library: it looks like it loads the entire file when all I want to do is scrape the images out of it. Any suggestions on a library that operates directly on the 'Data' bytestream and the folder structure of the .***x zip files?
I've already tried using OLEtools to try to extract pictures from the streams, and I'm now moving on to this tool. I havn't tried any tools that operate on the lower levels of the documents yet though.
Im using a pre-trained image classifier to evaluate input data treatments. I downloaded the ImageNet ILSVRC2014 CLS-LOC validation dataset to use as base. I need to know the actual classes of the images to evaluate my treatments (need to detect correct classifications). In the 2014 toolkit there is ILSVRC2014_clsloc_validation_ground_truth.txt file that according to the readme is supposed to contain class labels (in form of ID:s) for the 50 000 images in the data set. There are 50 000 entries/lines in the file so this far all seems good but i also want the corresponding semantic class labels/names.
I found these in a couple of places online and they seem to be coherent (1000 classes). But then i looked at the first image which is a snake, the ground truth for the first pic is 490, the 490:th row in the semantic name list is "chain". That's weird but still kind of close. The second image is two people skiing, the derived class "polecat". I tried many more with similar results.
I must have misunderstood something. Isn't the ground truth supposed to be the "correct" answers for the validation set? Have i missed something in the translation between ID:s and semantic labels?
The readme in the 2014 imagenet dev-kit states:
" There are a total of 50,000 validation images. They are named as
ILSVRC2012_val_00000001.JPEG
ILSVRC2012_val_00000002.JPEG
...
ILSVRC2012_val_00049999.JPEG
ILSVRC2012_val_00050000.JPEG
There are 50 validation images for each synset.
The classification ground truth of the validation images is in
data/ILSVRC2014_clsloc_validation_ground_truth.txt,
where each line contains one ILSVRC2014_ID for one image, in the
ascending alphabetical order of the image file names.
The localization ground truth for the validation images can be downloaded
in xml format. "
Im doing this as part of my bachelor thesis and really want to get it right.
Thanks in advance
This problem is now solved. In the ILSVRC2017 development kit there is a map_clsloc.txt file with the correct mappings.
I trained my model in Nvidia Digits 5 and I would now like to extract the accuracy and loss plots that were generated during training for a report. Is this data saved somewhere so that it would possible to extract the data for these plots so that I could plot it in Python and perhaps ultimately modify the plots to compare different models etc?
The best solution I have found is to either look at the HTML file or to scan the text file caffe_output.log that is produced by Caffe. The text file is usually stored in /var/digits/jobs/insert_your_job_id/ but you can also just run on linux systems:
locate caffe_output.log
Go to your DIGITS job folder and locate your job's subfolder. Inside you'll find a file status.pickle, which is a pickled object containing all your job's information.
You can load it in python like so:
import digits
import pickle
data = pickle.load(open('status.pickle','rb'))
This object is somewhat generic and may contain multiple tasks. For a typical classification task it will likely be just one, but you will still need to access it via data.tasks[0]. From there you can grab the plots:
data.tasks[0].combined_graph_data()
which returns a somewhat convoluted dict (unfortunately - since your network can produce many accuracy/loss outputs, as well as even custom ones). It contains everything you need though - I managed to plot accuracy with:
plt.plot( data.tasks[0].combined_graph_data()['columns'][2][1:] )
but it's likely that you'll have to write a bit of custom code. As always, dir() is your friend.
I've done the various steps to adapt CMU Sphinx using some recorded WAV files. Now where to the resulting files go?
I want to let each user on the system adapt CMU Sphinx to his own voice, so I don't want to store the files in /usr/local.
Is there a default directory or do I just have to specify them and their locations every time I run pocketSphinx?
Depending on the type of adaption you use, the resulting files can be either a transformation matrix which you specify with -mllr or a completely new model which is saved where you pointed out to. Adapted model has the same structure as the default one, so yes, you specify them and their locations every time your run the application. See the official tutorial for more information.
I’m new to EE and trying to learn the basics. Some questions about the File Manager:
I upload a photo and put “cat, kitten” in the description. When I do a search for “kitten”, it finds the photo. But when I do a search for “cat”, I get nothing. Any ideas what’s going on?
The file metadata are: file title, file name, description, credit, and location. What if I wanted to add custom fields? How do I do that?
In the template files, how do I access a particular manipulation (I call this “rendition”) of an image? Say I define a rendition “thumbnail” to be 100x100. How do I access that particular rendition in a template?
Is there a way to randomize the file names of the files being uploaded?
After uploading an image and testing it against PageSpeed, it turns out that the image can still be optimized via losslessly compressing it. How can this problem be addressed?
Ah, the file manager. Not EE's brightest spot.
It would not surprise me if the search in the File Manager was not
very robust. I'd try more variations to narrow it down (what kind of
characters affect the results - commas, dashes, spaces, etc ... do
partial terms match?)
You cannot currently add custom metadata to files in the file manager.
Use this syntax: {field_name:rendition}, e.g.,
{my_image:thumbnail} (docs).
Nope.
EE just uses the GD library available in your PHP install to resize
images. If you want the highest possible optimization, you'll have
to do your image manipulations yourself.
Given your queries, I would suggest you have a look at Assets by Pixel and Tonic. It offers a far superior file management experience on most of these fronts.