Vectorized formula for output layer in a neural network - performance

Question

I have a neural network and want to use the trained neural network to solve for a set of test data. What I am struggling with is writing the formula for the hidden layer and for the output layer. I aim to have a vectorized formula but I will also be happy to implement a loop variation.
Now I believe I have the correct formula for the hidden layer and only need one for the output layer, but would appreciate it anyone confirm that it is the vectorized formula.
% Variables
% Xtest test training data
% thetah - trained weights for inputs to hidden layer
% thetao - trained weights for hidden layer to outputs
% ytest - output
htest = (1 ./ (1 + exp(-(thetah * Xtest'))))' ; % FORMULA FOR HIDDEN LAYER
ytest = ones(mtest, num_outputs) ; % FORMULA FOR OUTPUT LAYER

Answer 1

Below you can find both vectorized and loop implementations of the forward propagation. It is possible, that your input data have to be adapted to the code below, because of different notations and the way you store data in your matrices.
You need to add a bias unit to both input and hidden layer.
In order to simplify the work on the implementation and debugging I took some data from the open source machine learning repository and trained the network for the wine classification task.
Xtest - input data [178x13]
y - output class [178x1]
thetah - parameters of the hidden layer [15x14]
thetao -
parameters of the output layer [3x16]
The network separates the input data with rate 97.7%
Here is the code:
function [] = nn_fp()
load('Xtest.mat'); %input data 178x13
load('y.mat'); %output data 178x1
load('thetah.mat'); %Parameters of the hidden layer 15x14
load('thetao.mat'); %Parameters of the output layer 3x16
predict_simple(Xtest, y, thetah, thetao);
predict_vectorized(Xtest, y, thetah, thetao);
end
function predict_simple(Xtest, y, thetah, thetao)
mtest = size(Xtest, 1); %number of input examples
n = size(Xtest, 2); %number of features
hl_size = size(thetah, 1); %size of the hidden layer (without the bias unit)
num_outputs = size(thetao, 1); %size of the output layer
%add a bias unit to the input layer
a1 = [ones(mtest, 1) Xtest]; %[mtest x (n+1)]
%compute activations of the hidden layer
z2 = zeros(mtest, hl_size); %[mtest x hl_size]
a2 = zeros(mtest, hl_size); %[mtest x hl_size]
for i=1:mtest
for j=1:hl_size
for k=1:n+1
z2(i, j) = z2(i, j) + a1(i, k)*thetah(j, k);
end
a2(i, j) = sigmoid_simple(z2(i, j));
end
end
%add a bias unit to the hidden layer
a2 = [ones(mtest, 1) a2]; %[mtest x (hl_size+1)]
%compute activations of the output layer
z3 = zeros(mtest, num_outputs); %[mtest x num_outputs]
h = zeros(mtest, num_outputs); %[mtest x num_outputs]
for i=1:mtest
for j=1:num_outputs
for k=1:hl_size+1
z3(i, j) = z3(i, j) + a2(i, k)*thetao(j, k);
end
h(i, j) = sigmoid_simple(z3(i, j)); %the hypothesis
end
end
%calculate predictions for each input example based on the maximum term
%of the hypothesis h
p = zeros(size(y));
for i=1:mtest
max_ind = 1;
max_value = h(i, 1);
for j=2:num_outputs
if (h(i, j) > max_value)
max_ind = j;
max_value = h(i, j);
end
end
p(i) = max_ind;
end
%calculate the success rate of the prediction
correct_count = 0;
for i=1:mtest
if (p(i) == y(i))
correct_count = correct_count + 1;
end
end
rate = correct_count/mtest*100;
display(['simple version rate:', num2str(rate)]);
end
function predict_vectorized(Xtest, y, thetah, thetao)
mtest = size(Xtest, 1); %number of input examples
%add a bias unit to the input layer
a1 = [ones(mtest, 1) Xtest];
%compute activations of the hidden layer
z2 = a1*thetah';
a2 = sigmoid_universal(z2);
%add a bias unit to the hidden layer
a2 = [ones(mtest, 1) a2];
%compute activations of the output layer
z3 = a2*thetao';
h = sigmoid_universal(z3); %the hypothesis
%calculate predictions for each input example based on the maximum term
%of the hypothesis h
[~,p] = max(h, [], 2);
%calculate the success rate of the prediction
rate = mean(double((p == y))) * 100;
display(['vectorized version rate:', num2str(rate)]);
end
function [ s ] = sigmoid_simple( z )
s = 1/(1+exp(-z));
end
function [ s ] = sigmoid_universal( z )
s = 1./(1+exp(-z));
end

Answer 2

Assuming that your Xtest has dimensions N by M where N is the number of examples and M is the number of features, thetah is a M by H1 matrix where H1 is the number of hidden layers in the first layer and thetao is a H1 by O matrix where O is the number of output classes you do the following:
a1 = Xtest * thetah;
z1 = 1 / (1 + exp(-a1)); %Assuming you are using sigmoid units
a2 = z1 * thetao;
z2 = softmax(a2);
Read more about softmax here.