Problem description
I have a 3D numpy array, denoted as data, of shape N x R x C, i.e. N samples, R rows and C columns. I would like to obtain histograms along column for each combination of sample and row. However bin edges (see argument bins in numpy.histogram), of fixed length S, will be different at different rows but are shared across samples. Consider this example for illustration, for the 1st sample (data[0]), bin edge sequence for its 1st row is different from that for its 2nd row, but is the same as that for the 1st row from the 2nd sample (data[1]). Thus all the bin edge sequences are stored in a 2D numpy array of shape R x S, denoted as bin_edges.
My question is how to efficiently calculate the histograms?
A working but slow solution
Using numpy.histogram, I was able to come up with a working but fairly slow solution as shown in the below code snippet
```
Get dummy data
N: number of samples
R: number of rows (or kernels)
C: number of columns (or pixels)
S: number of bins
```
import numpy as np
N, R, C, S = 100, 50, 1000, 10
data = np.random.randn(N, R, C)
# for each row/kernel, pool pixels of all samples
poolsamples = np.swapaxes(data, 0, 1).reshape(R, -1)
# use quantiles as bin edges
percentiles = np.linspace(0, 100, num=(S + 1))
bin_edges = np.transpose(np.percentile(poolsamples, percentiles, axis=1))
```
A working but slow solution of getting histograms along column
```
hist = np.empty((N, R, S))
for idx in np.arange(R):
bin_edges_i = bin_edges[idx, :]
counts = np.apply_along_axis(
lambda a: np.histogram(a, bins=bin_edges_i)[0],
1, data[:, idx, :])
hist[:, idx, :] = counts
Possible directions
Fancy numpy reshape to avoid using for loop at all
This problem arises from extracting low-end characteristics for each image forwarded through a trained neural network. Therefore, if the histogram extraction can be embedded in TensorFlow graph and ultimately be carried out on GPU, that would be ideal!
I noticed a python package fast-histogram which claims to be 7-15x faster than numpy.histogram. However 1d histogram function can only takes number of bins instead of actual bin positions
numexpr?
I would love to hear any inputs! Thanks in advance!
Making use of 2D version of np.searchsorted : searchsorted2d -
def vectorized_app(data, bin_edges):
N, R, C = data.shape
a = np.sort(data.reshape(-1,C),1)
b = np.repeat(bin_edges[None],N,axis=0).reshape(-1,bin_edges.shape[-1])
idx = searchsorted2d(a,b)
idx[:,0] = 0
idx[:,-1] = a.shape[1]
out = (idx[:,1:] - idx[:,:-1]).reshape(N,R,-1)
return out
Runtime test -
In [591]: N, R, C, S = 100, 50, 1000, 10
...: data = np.random.randn(N, R, C)
...:
...: # for each row/kernel, pool pixels of all samples
...: poolsamples = np.swapaxes(data, 0, 1).reshape(R, -1)
...: # use quantiles as bin edges
...: percentiles = np.linspace(0, 100, num=(S + 1))
...: bin_edges = np.transpose(np.percentile(poolsamples, percentiles, axis=1))
...:
In [592]: %timeit org_app(data, bin_edges)
1 loop, best of 3: 481 ms per loop
In [593]: %timeit vectorized_app(data, bin_edges)
1 loop, best of 3: 224 ms per loop
In [595]: np.allclose(org_app(data, bin_edges), vectorized_app(data, bin_edges))
Out[595]: True
More than 2x speedup there.
Closer look reveals that the bottleneck with the proposed vectorized method is the sorting itself -
In [594]: %timeit np.sort(data.reshape(-1,C),1)
1 loop, best of 3: 194 ms per loop
We need this sorting to use searchsorted.
I have a CSR matrix of counts (X_ngrams). I would like to build a sparse log-odds matrix by taking the log of the quotient of each entry and the sum across the row. Here is my best shot:
log_odds = X_ngrams.asfptype() # convert the counts to floats
row_sums = log_odds.sum(axis=1) # sum up each row
log_odds.log1p() # take log of each element
for ii in xrange(row_sums.shape[0]):
log_odds[ii,:].__add__(math.log(row_sums[ii,0]))
But that gives an error:
NotImplementedError: adding a nonzero scalar to a sparse matrix is not supported
So, my question is: how do I modify the contents of a CSR? I only want to modify the elements that are present.
Other approaches would also be welcome. The basic problem is to modify a CSR based on the sum across the columns for each row for the elements that exist.
So far as I can tell, one cannot apply an arbitrary function for elementwise calculation on a CSR sparse matrix. Instead, you can create a new sparse matrix with the same structure and just run the calculation across the sparse data. Here is sample code that shows how to calculate the log() of the ratio of each element to the sum across the columns on each row:
X_ngrams.sort_indices() # *MUST* have indices sorted for this to work!
row_sums = np.squeeze(np.asarray(X_ngrams.sum(axis=1),dtype=np.float64))
rows,cols = X_ngrams.nonzero()
data = np.array( [ math.log(x/row_sums[rows[ii]]) for ii,x in enumerate(X_ngrams.data)] )
new_odds = csr_matrix((data,X_ngrams.indices,X_ngrams.indptr),shape=X_ngrams.shape)
Here is a sample of the results, printing the first element of each row in both matrices:
row_sum Xngrams new_odds
[ 0][1439] 1063 20 -3.973118
[ 1][ 13] 1677 18 -4.534390
[ 2][1439] 5323 68 -4.360285
[ 3][1439] 983 15 -4.182559
This is not fast, but I suppose it is good enough. The sample X_ngrams data set has 2,596,855 non-zero elements with a shape = (2257, 202262) and the creation of the new matrix takes 10.5s on my 5 year old macbook pro.
You can use csr_matrix.nonzero method to get the arrays of indices of nonzero elements.
a =
1
2
3
b =
1 2 3
a.*b
ans =
1 2 3
2 4 6
3 6 9
I used the .* operator to multiply a row vector and a column vector in Octave to see the results. I dont understand how the answer is obtained.
This is because Octave (in a notable difference from Matlab) automatically broadcasts.
The * operator in Octave is the matrix multiplication operator. So in your case a*b would output (in Matlab as well)
a*b
ans =
1 2 3
2 4 6
3 6 9
which should be expected. The product of a 3-by-1 matrix with a 1-by-3 matrix would have dimensions 3-by-3 (inner dimensions must match, the result takes the outer dimensions).
However the .* operator is the element-wise multiplication operation. That means that instead of matrix multiplication, this would multiply each corresponding elements of the two inputs independent from the rest of the matrix. So [1,2,3].*[1,2,3] (or a'.*b) results in [1,4,9]. Again this is in Matlab and Octave.
When using element-wise operations, it is important that the dimensions of the inputs exactly match. So [1,2,3].*[1,2] will through an error because the dimensions do not match. In Matlab, your a.*b will through an error as well. HOWEVER in Octave it won't, instead it will automatically broadcast. You can imagine this is as if it takes one of your inputs and replicates it on a singleton dimension (so in a column vector, the second dimension is a singleton dimension because it's size is 1) and then applies the operator element-wise. In your case you have two matrices with singleton dimensions (i.e. a columan vector and a row vector) so it actually broadcasts twice and you effectively (but note that it does not actually expand the matrices in memory and is often far faster than using repmat) get
[1,2,3;1,2,3;1,2,3].*[1,1,1;2,2,2;3,3,3]
which produces the result you see.
In matlab, to achieve the same result you would have to explicitly call the bsxfun function (binary singleton expansion function) like so:
bsxfun(#times, a, b)
I have the following two 2x2 matrices:
A = [0.0087 0.0368;
-0.0368 0.0087]
B = [1.5653 2.0499;
-2.0499 1.5653]
I have checked that the Hermitian part of each of the matrices are positive definite:
eig(0.5*(A+A'))>0
so I don't know why using chol(A) returns an error saying that the matrix must be positive definite.
Am I missing something?
For matrix (say A) using the formula skewness(A(:)) we can easily get the skewness of the whole matrix. But doing same thing for an image (which is also a matrix) its not working.
Say I'm running the following code:
>> I=imread('lenna.jpg');
>> s=skewness(I(:))
The error coming is
Integers can only be combined with integers of the same class, or scalar doubles.
Error in ==> skewness at 39
x0 = x - repmat(nanmean(x,dim), tile);
I is of type uint8 after imread(), you can convert it to double first by using im2double().
Try
>> I=imread('lenna.jpg');
>> I2 = im2double(I);
>> s=skewness(I2(:))