Related
I am trying to run the code given in the link
https://github.com/bd622/DiscretHashing
Discrete Hashing is a method for dimensionality reduction that is used on approximate nearest neighbor search. I want to load in the implementation on the MNIST database that is available in http://yann.lecun.com/exdb/mnist/. I have extracted the files from their compressed gz format.
PROBLEM 1 :
Using the solution to read MNIST database provided in Reading MNIST Image Database binary file in MATLAB
I am getting the following error:
Error using fread
Invalid file identifier. Use fopen to generate a valid file identifier.
Error in Reading (line 7)
A = fread(fid, 1, 'uint32');
Here is the code:
clear all;
close all;
%//Open file
fid = fopen('t10k-images-idx3-ubyte', 'r');
A = fread(fid, 1, 'uint32');
magicNumber = swapbytes(uint32(A));
%//For each image, store into an individual cell
imageCellArray = cell(1, totalImages);
for k = 1 : totalImages
%//Read in numRows*numCols pixels at a time
A = fread(fid, numRows*numCols, 'uint8');
%//Reshape so that it becomes a matrix
%//We are actually reading this in column major format
%//so we need to transpose this at the end
imageCellArray{k} = reshape(uint8(A), numCols, numRows)';
end
%//Close the file
fclose(fid);
UPDATE : Problem 1 solved and the revised code is
clear all;
close all;
%//Open file
fid = fopen('t10k-images.idx3-ubyte', 'r');
A = fread(fid, 1, 'uint32');
magicNumber = swapbytes(uint32(A));
%//Read in total number of images
%//A = fread(fid, 4, 'uint8');
%//totalImages = sum(bitshift(A', [24 16 8 0]));
%//OR
A = fread(fid, 1, 'uint32');
totalImages = swapbytes(uint32(A));
%//Read in number of rows
%//A = fread(fid, 4, 'uint8');
%//numRows = sum(bitshift(A', [24 16 8 0]));
%//OR
A = fread(fid, 1, 'uint32');
numRows = swapbytes(uint32(A));
%//Read in number of columns
%//A = fread(fid, 4, 'uint8');
%//numCols = sum(bitshift(A', [24 16 8 0]));
%// OR
A = fread(fid, 1, 'uint32');
numCols = swapbytes(uint32(A));
for k = 1 : totalImages
%//Read in numRows*numCols pixels at a time
A = fread(fid, numRows*numCols, 'uint8');
%//Reshape so that it becomes a matrix
%//We are actually reading this in column major format
%//so we need to transpose this at the end
imageCellArray{k} = reshape(uint8(A), numCols, numRows)';
end
%//Close the file
fclose(fid);
PROBLEM 2:
I cannot understand how to apply the 4 files of MNIST in the code. The code contains variables
traindata = double(traindata);
testdata = double(testdata);
How do I prepare the MNIST database so that I can apply to the implementation?
UPDATE : I implemented the solution but I keep getting this error
Error using fread
Invalid file identifier. Use fopen to generate a valid file identifier.
Error in mnist_parse (line 11)
A = fread(fid1, 1, 'uint32');
These are the files
demo.m % this is the main file that calls the function to read in the MNIST data
clear all
clc
[Trainimages, Trainlabels] = mnist_parse('C:\Users\Desktop\MNIST\train-images-idx3-ubyte', 'C:\Users\Desktop\MNIST\train-labels-idx1-ubyte');
[Testimages, Testlabels] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte');
k=5;
digit = images(:,:,k);
lbl = label(k);
function [images, labels] = mnist_parse(path_to_digits, path_to_labels)
% Open files
fid1 = fopen(path_to_digits, 'r');
% The labels file
fid2 = fopen(path_to_labels, 'r');
% Read in magic numbers for both files
A = fread(fid1, 1, 'uint32');
magicNumber1 = swapbytes(uint32(A)); % Should be 2051
fprintf('Magic Number - Images: %d\n', magicNumber1);
A = fread(fid2, 1, 'uint32');
magicNumber2 = swapbytes(uint32(A)); % Should be 2049
fprintf('Magic Number - Labels: %d\n', magicNumber2);
% Read in total number of images
% Ensure that this number matches with the labels file
A = fread(fid1, 1, 'uint32');
totalImages = swapbytes(uint32(A));
A = fread(fid2, 1, 'uint32');
if totalImages ~= swapbytes(uint32(A))
error('Total number of images read from images and labels files are not the same');
end
fprintf('Total number of images: %d\n', totalImages);
% Read in number of rows
A = fread(fid1, 1, 'uint32');
numRows = swapbytes(uint32(A));
% Read in number of columns
A = fread(fid1, 1, 'uint32');
numCols = swapbytes(uint32(A));
fprintf('Dimensions of each digit: %d x %d\n', numRows, numCols);
% For each image, store into an individual slice
images = zeros(numRows, numCols, totalImages, 'uint8');
for k = 1 : totalImages
% Read in numRows*numCols pixels at a time
A = fread(fid1, numRows*numCols, 'uint8');
% Reshape so that it becomes a matrix
% We are actually reading this in column major format
% so we need to transpose this at the end
images(:,:,k) = reshape(uint8(A), numCols, numRows).';
end
% Read in the labels
labels = fread(fid2, totalImages, 'uint8');
% Close the files
fclose(fid1);
fclose(fid2);
end
I am the original author of Method #1 that you spoke of. The process to read in the training data and test labels is quite simple. In terms of reading in images, the code that you showed above reads the files perfectly and is in a cell array format. However, you are missing reading in the number of images, rows and columns inside the file. Take note that the MNIST format for this file is in the following fashion. The left column is the offset in bytes you are referencing with respect to the beginning:
[offset] [type] [value] [description]
0000 32 bit integer 0x00000803(2051) magic number
0004 32 bit integer 60000 number of images
0008 32 bit integer 28 number of rows
0012 32 bit integer 28 number of columns
0016 unsigned byte ?? pixel
0017 unsigned byte ?? pixel
........
xxxx unsigned byte ?? pixel
The first four bytes are a magic number: 2051 to ensure that you're reading in the file properly. The next four bytes denote the total number of images, then the next four bytes are the rows and finally the next four bytes are the columns. There should be 60000 images of size 28 rows by 28 columns. After this, the pixels are interleaved in row major format so you have to loop over series of 28 x 28 pixels and store them. In this case, I've stored them in a cell array and each element in this cell array would be one digit. The same format is for the test data as well, but there are 10000 images instead.
As for the actual labels, it's roughly the same format but there are some slight differences:
[offset] [type] [value] [description]
0000 32 bit integer 0x00000801(2049) magic number (MSB first)
0004 32 bit integer 60000 number of items
0008 unsigned byte ?? label
0009 unsigned byte ?? label
........
xxxx unsigned byte ?? label
The first four bytes are a magic number: 2049, then the second set of four bytes tells you how many labels there are and finally there is exactly 1 byte for each corresponding digit in the dataset. The test data is also the same format but there are 10000 labels. As such, once you read in the necessary data in the label set, you just need one fread call and ensure that the data is unsigned 8-bit integer to read in the rest of the labels.
Now the reason why you have to use swapbytes is because MATLAB will read in the data in little-endian format, meaning that the least significant byte from a set of bytes is read in first. You use swapbytes to rearrange this order when you're done.
As such, I have modified this code for you so that it's an actual function that takes in a set of two strings: The full path to the image file of digits and the full path to the digits. I have also changed the code so that the images are a 3D numeric matrix as opposed to a cell array for faster processing. Take note that when you start reading in the actual image data, each pixel is unsigned 8-bit integer, so there's no need to do any swapping of bytes. This was only required when reading in multiple bytes in one fread call:
function [images, labels] = mnist_parse(path_to_digits, path_to_labels)
% Open files
fid1 = fopen(path_to_digits, 'r');
% The labels file
fid2 = fopen(path_to_labels, 'r');
% Read in magic numbers for both files
A = fread(fid1, 1, 'uint32');
magicNumber1 = swapbytes(uint32(A)); % Should be 2051
fprintf('Magic Number - Images: %d\n', magicNumber1);
A = fread(fid2, 1, 'uint32');
magicNumber2 = swapbytes(uint32(A)); % Should be 2049
fprintf('Magic Number - Labels: %d\n', magicNumber2);
% Read in total number of images
% Ensure that this number matches with the labels file
A = fread(fid1, 1, 'uint32');
totalImages = swapbytes(uint32(A));
A = fread(fid2, 1, 'uint32');
if totalImages ~= swapbytes(uint32(A))
error('Total number of images read from images and labels files are not the same');
end
fprintf('Total number of images: %d\n', totalImages);
% Read in number of rows
A = fread(fid1, 1, 'uint32');
numRows = swapbytes(uint32(A));
% Read in number of columns
A = fread(fid1, 1, 'uint32');
numCols = swapbytes(uint32(A));
fprintf('Dimensions of each digit: %d x %d\n', numRows, numCols);
% For each image, store into an individual slice
images = zeros(numRows, numCols, totalImages, 'uint8');
for k = 1 : totalImages
% Read in numRows*numCols pixels at a time
A = fread(fid1, numRows*numCols, 'uint8');
% Reshape so that it becomes a matrix
% We are actually reading this in column major format
% so we need to transpose this at the end
images(:,:,k) = reshape(uint8(A), numCols, numRows).';
end
% Read in the labels
labels = fread(fid2, totalImages, 'uint8');
% Close the files
fclose(fid1);
fclose(fid2);
end
To call this function, simply specify the path to both the image file and the labels file. Assuming you are running this file in the same directory where the files are located, you would do one of the following for the training images:
[images, labels] = mnist_parse('train-images-idx3-ubyte', 'train-labels-idx1-ubyte');
Also, you would do the following for the test images:
[images, labels] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte');
To access the kth digit, you would simply do:
digit = images(:,:,k);
The corresponding label for the kth digit would be:
lbl = label(k);
To finally get this data into a format that is acceptable for that code that I have seen on Github, they assume that the rows correspond to training examples and the columns correspond to features. If you wish to have this format, simply reshape the data so that the image pixels are spread out over the columns.
Therefore, just do this:
[trainingdata, traingnd] = mnist_parse('train-images-idx3-ubyte', 'train-labels-idx1-ubyte');
trainingdata = double(reshape(trainingdata, size(trainingdata,1)*size(trainingdata,2), []).');
traingnd = double(traingnd);
[testdata, testgnd] = mnist_parse('t10k-images-idx3-ubyte', 't10k-labels-idx1-ubyte');
testdata = double(reshape(testdata, size(testdata,1)*size(testdata_data,2), []).');
testgnd = double(testgnd);
The above uses the same variables as in the script so you should be able to plug this in and it should work. The second line reshapes the matrix so that each digit is in a column, but we need to transpose this so that each digit is in a column. We also need to cast to double as that is what the Github code is doing. The same logic is applied to the test data. Also take note that I've explicitly cast the training and test labels to double to ensure maximum compatibility in whatever algorithms you decide to use on this data.
Happy digit hacking!
Given the following three dimensional hankel matrix:
hankel_matrix = hankel(1:21, 21:999);
hankel_matrix = repmat(hankel_matrix, [1 1 no_of_weighted_composites]);
And the matrix:
composites_collection = permute(reshape(data_composites.', size(data_composites, 2),1,[]),[2 3 1]);
where data_composites is a 999 x 56 matrix.
Essentially what I want to achieve is use the hankel matrix to index the 56 different pages of single rows found in the composites collection.
I had assumed this would be as easy as data_composites_collection(hankel_matrix) but that simply uses the first page of data_composites_collection and repeats this for all 56 pages.
My current implementation is:
function subsequent_hankel_index = createMovingSnapshotsOfValues(~, matrix_of_numbers, data_composites, windows, no_of_weighted_composites)
data_composites_collection = permute(reshape(data_composites.', size(data_composites, 2),1,[]),[2 3 1]);
hankel_index = NaN(260, 999, no_of_weighted_composites, size(windows, 2));
for window_size = 1:size(windows, 2);
for composite = 1:no_of_weighted_composites;
hankel_matrix = hankel(1:windows(window_size), windows(window_size):length(matrix_of_numbers));
desired_row = data_composites_collection(:,:,composite);
[rows, columns] = size(desired_row(hankel_matrix));
hankel_index(1:rows,1:columns,composite,window_size) = desired_row(hankel_matrix);
subsequent_hankel_index = hankel_index;
end
end
end
However, this is incredibly slow for the amount of data I have. I have assumed vectorising the above would greatly help the speed of my programme. Any tips or approaches to the indexing would be greatly appreciated.
Many thanks in advance!
for an input matrix
in = [1 1;
1 2;
1 3;
1 4;
2 5;
2 6;
2 7;
3 8;
3 9;
3 10;
3 11];
i want to get the output matrix
out = [1 5 8;
2 6 9;
3 7 10;
4 0 11];
meaning i want to reshape the second input column into an output matrix, where all values corresponding to one value in the first input column are written into one column of the output matrix.
As there can be different numbers of entries for each value in the first input column (here 4 values for "1" and "3", but only 3 for "2"), the normal reshape function is not applicable. I need to pad all columns to the maximum number of rows.
Do you have an idea how to do this matlab-ish?
The second input column can only contain positive numbers, so the padding values can be 0, -x, NaN, ...
The best i could come up with is this (loop-based):
maxNumElem = 0;
for i=in(1,1):in(end,1)
maxNumElem = max(maxNumElem,numel(find(in(:,1)==i)));
end
out = zeros(maxNumElem,in(end,1)-in(1,1));
for i=in(1,1):in(end,1)
tmp = in(in(:,1)==i,2);
out(1:length(tmp),i) = tmp;
end
Either of the following approaches assumes that column 1 of in is sorted, as in the example. If that's not the case, apply this initially to sort in according to that criterion:
in = sortrows(in,1);
Approach 1 (using accumarray)
Compute the required number of rows, using mode;
Use accumarray to gather the values corresponding to each column, filled with zeros at the end. The result is a cell;
Concatenate horizontally the contents of all cells.
Code:
[~, n] = mode(in(:,1)); %//step 1
out = accumarray(in(:,1), in(:,2), [], #(x){[x; zeros(n-numel(x),1)]}); %//step 2
out = [out{:}]; %//step 3
Alternatively, step 1 could be done with histc
n = max(histc(in(:,1), unique(in(:,1)))); %//step 1
or with accumarray:
n = max(accumarray(in(:,1), in(:,2), [], #(x) numel(x))); %//step 1
Approach 2 (using sparse)
Generate a row-index vector using this answer by #Dan, and then build your matrix with sparse:
a = arrayfun(#(x)(1:x), diff(find([1,diff(in(:,1).'),1])), 'uni', 0); %//'
out = full(sparse([a{:}], in(:,1), in(:,2)));
Introduction to proposed solution and Code
Proposed here is a bsxfun based masking approach that uses the binary operators available as builtins for use with bsxfun and as such I would consider this very appropriate for problems like this. Of course, you must also be aware that bsxfun is a memory hungry tool. So, it could pose a threat if you are dealing with maybe billions of elements depending also on the memory available for MATLAB's usage.
Getting into the details of the proposed approach, we get the counts of each ID from column-1 of the input with histc. Then, the magic happens with bsxfun + #le to create a mask of positions in the output array (initialized by zeros) that are to be filled by the column-2 elements from input. That's all you need to tackle the problem with this approach.
Solution Code
counts = histc(in(:,1),1:max(in(:,1)))'; %//' counts of each ID from column1
max_counts = max(counts); %// Maximum counts for each ID
mask = bsxfun(#le,[1:max_counts]',counts); %//'# mask of locations where
%// column2 elements are to be placed
out = zeros(max_counts,numel(counts)); %// Initialize the output array
out(mask) = in(:,2); %// place the column2 elements in the output array
Benchmarking (for performance)
The benchmarking presented here compares the proposed solution in this post against the various methods presented in Luis's solution. This skips the original loopy approach presented in the problem as it appeared to be very slow for the input generated in the benchmarking code.
Benchmarking Code
num_ids = 5000;
counts_each_id = randi([10 100],num_ids,1);
num_runs = 20; %// number of iterations each approach is run for
%// Generate random input array
in = [];
for k = 1:num_ids
in = [in ; [repmat(k,counts_each_id(k),1) rand(counts_each_id(k),1)]];
end
%// Warm up tic/toc.
for k = 1:50000
tic(); elapsed = toc();
end
disp('------------- With HISTC + BSXFUN Masking approach')
tic
for iter = 1:num_runs
counts = histc(in(:,1),1:max(in(:,1)))';
max_counts = max(counts);
out = zeros(max_counts,numel(counts));
out(bsxfun(#le,[1:max_counts]',counts)) = in(:,2);
end
toc
clear counts max_counts out
disp('------------- With MODE + ACCUMARRAY approach')
tic
for iter = 1:num_runs
[~, n] = mode(in(:,1)); %//step 1
out = accumarray(in(:,1), in(:,2), [], #(x){[x; zeros(n-numel(x),1)]}); %//step 2
out = [out{:}];
end
toc
clear n out
disp('------------- With HISTC + ACCUMARRAY approach')
tic
for iter = 1:num_runs
n = max(histc(in(:,1), unique(in(:,1))));
out = accumarray(in(:,1), in(:,2), [], #(x){[x; zeros(n-numel(x),1)]}); %//step 2
out = [out{:}];
end
toc
clear n out
disp('------------- With ARRAYFUN + Sparse approach')
tic
for iter = 1:num_runs
a = arrayfun(#(x)(1:x), diff(find([1,diff(in(:,1).'),1])), 'uni', 0); %//'
out = full(sparse([a{:}], in(:,1), in(:,2)));
end
toc
clear a out
Results
------------- With HISTC + BSXFUN Masking approach
Elapsed time is 0.598359 seconds.
------------- With MODE + ACCUMARRAY approach
Elapsed time is 2.452778 seconds.
------------- With HISTC + ACCUMARRAY approach
Elapsed time is 2.579482 seconds.
------------- With ARRAYFUN + Sparse approach
Elapsed time is 1.455362 seconds.
slightly better, but still uses a loop :(
out=zeros(4,3);%set to zero matrix
for i = 1:max(in(:,1)); %find max in column 1, and loop for that number
ind = find(in(:,1)==i); %
out(1: size(in(ind,2),1),i)= in(ind,2);
end
don't know if you can avoid the loop...
I wish to ask if anybody out there knows how to partition an image into 8 different rows and 1 column? I have tried using mat2cell() and using the demo on their wiki as a reference, I tried partitioning the image into 8 rows, however not all image partition rows are displayed.
If you see the image below, 2, 4, 6, 8 is not displayed. I am also not sure why is it of 16 blocks.
Can somebody help me check my code? I am not really used to the MatLab syntax and language. I trying my best to understand now.
My code for splitting the blocks are as follows:
blockSizeR = 50; % Rows in block.
blockSizeC = 512; % Columns in block.
wholeBlockRows = floor(rows / blockSizeR);
blockVectorR = [blockSizeR * ones(1, wholeBlockRows), rem(rows, blockSizeR)];
wholeBlockCols = floor(columns / blockSizeC);
blockVectorC = [blockSizeC * ones(1, wholeBlockCols), rem(columns, blockSizeC)];
if numberOfColorBands > 1
% It's a color image.
ca = mat2cell(rgbImage, blockVectorR, blockVectorC, numberOfColorBands);
else
ca = mat2cell(rgbImage, blockVectorR, blockVectorC);
end
% Now display all the blocks.
plotIndex = 1;
numPlotsR = size(ca, 1);
numPlotsC = size(ca, 2);
for r = 1 : numPlotsR
for c = 1 : numPlotsC
fprintf('plotindex = %d, c=%d, r=%d\n', plotIndex, c, r);
% Specify the location for display of the image.
subplot(numPlotsR, 1, plotIndex);
% Extract the numerical array out of the cell
% just for tutorial purposes.
rgbBlock = ca{r,c};
imshow(rgbBlock); % Could call imshow(ca{r,c}) if you wanted to.
[rowsB columnsB numberOfColorBandsB] = size(rgbBlock);
% Make the caption the block number.
caption = sprintf('Block #%d of %d\n%d rows by %d columns', ...
plotIndex, numPlotsR*numPlotsC, rowsB, columnsB);
title(caption);
drawnow;
% Increment the subplot to the next location.
plotIndex = plotIndex + 1;
end
end
I am new to MatLab, so is there is a simpler method to do this that I missed out, please do suggest or better still, if there are references that I can refer to. Many thanks (:
If you know the dimensions of your matrix, you can do the math to figure out how to divide the number of rows into 4 equal parts:
e.g. If: size(rockinsMatrix) == [10 20] (a 10row x 20column) matrix,
then you could split it into a set of 4 sub-matrices, two with 3 rows, and 2 with 2 columns.
If you want the matrices in a cell array then you can do that at that time.
I managed to solve already, the error lies in the for loop. I changed the for r = 1 : numPlotsR into r = 1 : (number of rows I want) for c = 1 : numPlotsC into c= 1: 1(as I only want one column), and used subplot(8,1,k) or (8,2,k) where k is the plot index. Just answering this in case anybody encounter such problem in future and want to use my code as a reference. Cheers!
as a part of my homework i need to implement this pattern matching.
the goal is to "Detect as many of the 0's (zeros) as you can in image coins4.tif."
i was given the NGC function. and i need to use it
this is my main.m file
Image = readImage('coins4.tif');
Pattern = readImage('zero.tif');
showImage(Image);
showImage(Pattern);
message = sprintf('Pattern matching Normalized Correlation');
PatternMatching(Image , Pattern);
uiwait(msgbox(message,'Done', 'help'));
close all
this is my PatternMatching function.
function [ output_args ] = PatternMatching( Image , Pattern )
% Pattern matching – Normalized Correlation
% Detect as many of the 0's (zeros) as you can in image coins4.tif.
% Use the 0 of the 10 coin as pattern.
% Use NGC_pm and find good threshold. Display original image with? detected regions marked using drawRect.
% NGCpm(im,pattern);
% drawRect(rectCoors,color);
% rectCoors = [r0,c0,rsize,csize] - r0,c0 = top-left corner of rect.
% rsize = number of rows, csize = number of cols
%
% color = an integer >=1 representing a color in the color wheel
% (curerntly cycles through 8 different colors
showImage(Image);
hold on
res = NGCpm(Image, Pattern);
for i = 1:size(res,1)
for j = 1:size(res,2)
if res(i,j) > 0.9999
drawRect([i j size(Pattern,1) size(Pattern,2)], 5)
end
end
end
end
this is the Given NGCpm.m file
function res=NGC_PM(im,pattern)
[n m]=size(pattern);
[im_row,im_col]=size(im);
if ~(var(pattern(:))==0)
res = normxcorr2(pattern, im);
res=res(n:im_row,m:im_col);
else
res=zeros(size(im)-size(pattern)+1);
end;
res = 1-abs(res); % res = abs(res);
this is the pattern i'm trying to find and the results, i'm getting
i'm trying to find as many "Zeros" as possiable using the zero pattern of the coin 10.
i'm tryingto understand if there is something wrong with my algorithm in the PatternMatching function. since the NGCpm function is already given to me, all i need to do is just loop of the best threshold ,correct?
or do i need to blur the image or the pattern?
this is the fixed version of this function.
function [ output_args ] = patternMatching( Image , Pattern )
% Pattern matching – Normalized Correlation
% Detect as many of the 0's (zeros) as you can in image coins4.tif.
% Use the 0 of the 10 coin as pattern.
% Use NGC_pm and find good threshold. Display original image with? detected regions marked using drawRect.
% NGCpm(im,pattern);
% drawRect(rectCoors,color);
% rectCoors = [r0,c0,rsize,csize] - r0,c0 = top-left corner of rect.
% rsize = number of rows, csize = number of cols
%
% color = an integer >=1 representing a color in the color wheel
% (curerntly cycles through 8 different colors
showImage(Image);
hold on
res = 1-NGCpm(Image, Pattern);
normalized_corellation = uint8(255*res/max(max(res)));
res_thresh = thresholdImage(normalized_corellation,100);
for i = 1:size(res_thresh,1)
for j = 1:size(res_thresh,2)
if res_thresh(i,j) > 0
drawRect([i j size(Pattern,1) size(Pattern,2)], 5)
end
end
end
end