If I have constructed a sparse matrix using the sparse(i, j, k) constructor, how can I then normalize the columns of the matrix (so that each column sums to 1)? I cannot efficiently normalize the entries before I create the matrix, so any help is appreciated. Thanks!
The easiest way would be a broadcasting division by the sum of the columns:
julia> A = sprand(4,5,.5)
A./sum(A,1)
4x5 Array{Float64,2}:
0.0 0.0989976 0.0 0.0 0.0795486
0.420754 0.458653 0.0986313 0.0 0.0
0.0785525 0.442349 0.0 0.856136 0.920451
0.500693 0.0 0.901369 0.143864 0.0
… but it looks like that hasn't been optimized for sparse matrices yet, and falls back to a full matrix. So a simple loop to iterate over the columns does the trick:
julia> for (col,s) in enumerate(sum(A,1))
s == 0 && continue # What does a "normalized" column with a sum of zero look like?
A[:,col] = A[:,col]/s
end
A
4x5 sparse matrix with 12 Float64 entries:
[2, 1] = 0.420754
[3, 1] = 0.0785525
[4, 1] = 0.500693
[1, 2] = 0.0989976
[2, 2] = 0.458653
[3, 2] = 0.442349
[2, 3] = 0.0986313
[4, 3] = 0.901369
[3, 4] = 0.856136
[4, 4] = 0.143864
[1, 5] = 0.0795486
[3, 5] = 0.920451
julia> sum(A,1)
1x5 Array{Float64,2}:
1.0 1.0 1.0 1.0 1.0
This works entirely within sparse matrices and is done in-place (although it is still allocating new sparse matrices for each column slice).
Given a Matrix A (does not matter whether or not it is sparse) normalize by any dimension
A ./ sum(A,1) or A ./ sum(A,2)
to show that it works:
A = sprand(10,10,0.3)
println(sum(A,1))
println(A ./ sum(A,1))
only caveat
A[1,:] = 0
println(A ./ sum(A,1))
as you can see the column 1 now only contains NaNs because we divide by zero. Also we end up with a Matrix and not a sparse Matrix.
On the other hand one can quickly come up with an efficient specialized solution for your problem.
function normalize_columns(A :: SparseMatrixCSC)
sums = sum(A,1)
I,J,V = findnz(A)
for idx in 1:length(V)
V[idx] /= sums[J[idx]]
end
sparse(I,J,V)
end
#Matt B came up with a very similar answer while I was typing this up :)
Remember that sparse matrices in Julia are in compressed column form. So you can access the data directly:
for col = 1 : size(A, 2)
i = A.colptr[col]
k = A.colptr[col+1] - 1
n = i <= k ? norm(A.nzval[i:k]) : 0.0 # or whatever you like
n > 0.0 && (A.nzval[i:k] ./= n)
end
# get the column sums of A
S = vec(sum(A,1))
# get the nonzero entries in A. ei is row index, ej is col index, ev is the value in A
ei,ej,ev = findnz(A)
# get the number or rows and columns in A
m,n = size(A)
# create a new normalized matrix. For each nonzero index (ei,ej), its new value will be
# the old value divided by the sum of that column, which can be obtained by S[ej]
A_normalized = sparse(ei,ej,ev./S[ej],m,n)
the following gives what you want:
A = sprand(4,5,0.5)
B = A./sparse(sum(A,1))
The problem is that sum(A,1) gives a 1x5 dense array so combining with the sparse matrix A through the ./ operator gives a dense array. So you need to force it to be of sparse type. Or you can type
sparse(A ./ sum(A,1)).
Related
I am doing some stuff on leetcode and came up with solution it works fine but some cases.
Here is the problem itself:
But in case like this it doesn't:
It doesn't make sense how can I rotate elements if k is bigger than length of array.
If you have any idea how to improve this solution I would be grateful
class Solution:
def rotate(self, nums: List[int], k: int) -> None:
"""
Do not return anything, modify nums in-place instead.
"""
if len(nums) > k:
self.swap(nums, 0, len(nums)-1)
self.swap(nums, 0,k-1)
self.swap(nums, k, len(nums)-1)
def swap(self, nums, start, end):
while start < end:
nums[start], nums[end] = nums[end], nums[start]
start+=1
end-=1
In order to understand why this doesn't work for the cases where k is larger than the array length, let me try to explain some of the logic behind rotating by such values of k.
The modulo operator, % will be useful. For example, if an array is 5 long, and you want to rotate by 5, you end up with the same array. So technically, you'd optimally want to rotate by 0. This is where the % operator comes into play. 5 % 5 = 0. If we want to rotate an array length 5 by 7 spots, we would end up with the same thing as rotating the array by 2, and it turns out that 7 % 5 = 2. Do you see where I am going with this?
This also holds true if the value of k is less than the length of the array. Say we want to rotate an array length 5 by 3, we do 3 % 5 = 3.
So for any rotation of amount k and array length L, optimization rotation amount n is equivalent to n = k % L.
You should modify your code at the beginning of your rotate method to adjust the rotation amount:
k = k % L
and use this value to rotate the correct amount.
The fastest and cleanest solution by far and large is:
def rotate_right(items, shift):
shift = -shift % len(items)
return items[shift:] + items[:shift]
ll = [i + 1 for i in range(7)]
# [1, 2, 3, 4, 5, 6, 7]
rotate_right(ll, 3)
# [5, 6, 7, 1, 2, 3, 4]
rotate_right([1, 2], 3)
# [2, 1]
of course, short of using numpy.roll() or itertools.cycle().
How can we efficiently calculate pairwise cosine distances in a matrix using TensorFlow? Given an MxN matrix, the result should be an MxM matrix, where the element at position [i][j] is the cosine distance between i-th and j-th rows/vectors in the input matrix.
This can be done with Scikit-Learn fairly easily as follows:
from sklearn.metrics.pairwise import pairwise_distances
pairwise_distances(input_matrix, metric='cosine')
Is there an equivalent method in TensorFlow?
There is an answer for getting a single cosine distance here: https://stackoverflow.com/a/46057597/288875 . This is based on tf.losses.cosine_distance .
Here is a solution which does this for matrices:
import tensorflow as tf
import numpy as np
with tf.Session() as sess:
M = 3
# input
input = tf.placeholder(tf.float32, shape = (M, M))
# normalize each row
normalized = tf.nn.l2_normalize(input, dim = 1)
# multiply row i with row j using transpose
# element wise product
prod = tf.matmul(normalized, normalized,
adjoint_b = True # transpose second matrix
)
dist = 1 - prod
input_matrix = np.array(
[[ 1, 1, 1 ],
[ 0, 1, 1 ],
[ 0, 0, 1 ],
],
dtype = 'float32')
print "input_matrix:"
print input_matrix
from sklearn.metrics.pairwise import pairwise_distances
print "sklearn:"
print pairwise_distances(input_matrix, metric='cosine')
print "tensorflow:"
print sess.run(dist, feed_dict = { input : input_matrix })
which gives me:
input_matrix:
[[ 1. 1. 1.]
[ 0. 1. 1.]
[ 0. 0. 1.]]
sklearn:
[[ 0. 0.18350345 0.42264974]
[ 0.18350345 0. 0.29289323]
[ 0.42264974 0.29289323 0. ]]
tensorflow:
[[ 5.96046448e-08 1.83503449e-01 4.22649741e-01]
[ 1.83503449e-01 5.96046448e-08 2.92893231e-01]
[ 4.22649741e-01 2.92893231e-01 0.00000000e+00]]
Note that this solution may not be the optimal one as it calculates all entries of the (symmetric) result matrix, i.e. does almost twice of the calculations. This is likely not a problem for small matrices, for large matrices a combination of loops may be faster.
Note also that this does not have a minibatch dimension so works for a single matrix only.
Elegant solution (output is the same as from scikit-learn pairwise_distances function):
def compute_cosine_distances(a, b):
# x shape is n_a * dim
# y shape is n_b * dim
# results shape is n_a * n_b
normalize_a = tf.nn.l2_normalize(a,1)
normalize_b = tf.nn.l2_normalize(b,1)
distance = 1 - tf.matmul(normalize_a, normalize_b, transpose_b=True)
return distance
test
input_matrix = np.array([[1, 1, 1],
[0, 1, 1],
[0, 0, 1]], dtype = 'float32')
compute_cosine_distances(input_matrix, input_matrix)
output:
<tf.Tensor: id=442, shape=(3, 3), dtype=float32, numpy=
array([[5.9604645e-08, 1.8350345e-01, 4.2264974e-01],
[1.8350345e-01, 5.9604645e-08, 2.9289323e-01],
[4.2264974e-01, 2.9289323e-01, 0.0000000e+00]], dtype=float32)>
When using tf.boolean_mask(), a Value Error is raised. It reads "Number of mask dimensions must be specified, even if some dimensions are None. E.g. shape=[None] is ok, but shape=None is not.
I suspect that something is going wrong when I create my boolean mask s, because when I just create a boolean mask by hand, all works fine. However, I've checked the shape and the dtype of s so far, and couldn't notice anything suspicious. Both seemed to be identical to the shape and type of the boolean mask I created by hand.
Please see a screenshot of the problem.
The following should allow you to reproduce the error on your machine. You need tensorflow, numpy and scipy.
with tf.Session() as sess:
# receive five embedded vectors
v0 = tf.constant([[3.0,1.0,2.,4.,2.]])
v1 = tf.constant([[4.0,0,1.0,4,1.]])
v2 = tf.constant([[1.0,1.0,0.0,4.,8.]])
v3 = tf.constant([[1.,4,2.,5.,2.]])
v4 = tf.constant([[3.,2.,3.,2.,5.]])
# concatenate the five embedded vectors into a matrix
VT = tf.concat([v0,v1,v2,v3,v4],axis=0)
# perform SVD on the concatenated matrix
s, u1, u2 = tf.svd(VT)
e = tf.square(s) # list of eigenvalues
v = u1 # eigenvectors as column vectors
# sample a set
s = tf.py_func(sample_dpp_bin,[e,v],tf.bool)
X = tf.boolean_mask(VT,s)
print(X.eval())
This is the code to generate s. s is a sample from a determinantal point process (for the mathematically interested).
Note that I'm using tf.py_func to wrap this python function:
import tensorflow as tf
import numpy as np
from scipy.linalg import orth
def sample_dpp_bin(e_val,e_vec):
# e_val = np.array of eigenvalues
# e_vec = array of eigenvectors (= column vectors)
eps = 0.01
# sample a set of eigenvectors
ind = (np.random.rand(len(e_val)) <= (e_val)/(1+e_val))
k = sum(ind)
if k == e_val.size:
return np.ones(e_val.size,dtype=bool) # check for full set
if k == 0:
return np.zeros(e_val.size,dtype=bool)
V = e_vec[:,np.array(ind)]
# sample a set of k items
sample = np.zeros(e_val.size,dtype=bool)
for l in range(k-1,-1,-1):
p = np.sum(V**2,axis=1)
p = np.cumsum(p / np.sum(p)) # item cumulative probabilities
i = int((np.random.rand() <= p).argmax()) # choose random item
sample[i] = True
j = (np.abs(V[i,:])>eps).argmax() # pick an eigenvector not orthogonal to e_i
Vj = V[:,j]
V = orth(V - (np.outer(Vj,(V[i,:]/Vj[i]))))
return sample
The output if I print s and tf.reshape(s) is
[False True True True True]
[5]
The output if I print VT and tf.reshape(VT) is
[[ 3. 1. 2. 4. 2.]
[ 4. 0. 1. 4. 1.]
[ 1. 1. 0. 4. 8.]
[ 1. 4. 2. 5. 2.]
[ 3. 2. 3. 2. 5.]]
[5 5]
Any help much appreciated.
Following example works for me.
import tensorflow as tf
import numpy as np
tensor = [[1, 2], [3, 4], [5, 6]]
mask = np.array([True, False, True])
t_m = tf.boolean_mask(tensor, mask)
sess = tf.Session()
print(sess.run(t_m))
Output:
[[1 2]
[5 6]]
Provide your runnable code snippet to reproduce the error. I think you might be doing something wrong in s.
Update:
s = tf.py_func(sample_dpp_bin,[e,v],tf.bool)
s_v = (s.eval())
X = tf.boolean_mask(VT,s_v)
print(X.eval())
mask should be a np array not TF tensor. You don't have to use tf.pyfunc.
The error message states that the shape of the mask is not defined. What do you get if you print tf.shape(s)? I'd bet the problem with your code is that the shape of s is completely unknown, and you could fix that with a simple call like s.set_shape((None)) (to simply specify that s is a 1-dimensional tensor). Consider this code snippet:
X = np.random.randint(0, 2, (100, 100, 3))
with tf.Session() as sess:
X_tf = tf.placeholder(tf.int8)
# X_tf.set_shape((None, None, None))
y = tf.greater(tf.reduce_max(X_tf, axis=(0, 1)), 0)
print(tf.shape(y))
z = tf.boolean_mask(X_tf, y, axis=2)
print(sess.run(z, feed_dict={X_tf: X}))
This prints a shape of Tensor("Shape_3:0", shape=(?,), dtype=int32) (i.e., even the dimensions of y are unknown) and returns the same error as you have. However, if you uncomment the set_shape line, then X_tf is known to be 3-dimensional and so s is 1-dimensional. The code then works. So, I think all you need to do is add a s.set_shape((None)) call after the py_func call.
I have a square matrix of indeterminate row & column length (assume rows and columns are equal as befits a square).
I've plotted out an example matrix as follows:
matrix = [
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
]
My goal is to get a sum from top-left to bottom-right of the diagonal values.
Obviously in this example, this is all i'll need:
diagsum = matrix[0][0]+matrix[1][1]+matrix[2][2]
#=> 15
I see the pattern where it's a +1 incremental for each row & column argument in the matrix, so the code i've developed for my matrix of indeterminate length (supplied as the argument to my method diagsum would preferably need to implement some sort of row_count method on my matrix argument.
If
arr = [[1,2,3],
[4,5,6],
[7,8,9]]
then:
require 'matrix'
Matrix[*arr].trace
#=> 15
This will sum the diagonal values.
matrix = []
matrix[0] = [1,2,3]
matrix[1] = [4,5,6]
matrix[2] = [7,8,9]
def diagsum(mat)
sum = 0
mat.each_with_index { |row,i| sum += row[i] }
sum
end
puts (diagsum matrix) # 15
Not clear what x is.
But assuming that it is the number of columns/rows, you have 0..x, while the index only goes up to x - 1. You should change it to 0...x.
You are assigning to variable i, whose scope is only in the block.
You are only using i once, perhaps intended to correspond to either row or column, but not both.
You are adding the indices instead of the corresponding elements.
each will return the receiver regardless of whatever you get in the blocks.
puts will return nil regardless of whatever you get.
I need help optimizing this loop. matrix_1 is a (nx 2) int matrix and matrix_2 is a (m x 2), m & n very.
index_j = 1;
for index_k = 1:size(Matrix_1,1)
for index_l = 1:size(Matrix_2,1)
M2_Index_Dist(index_j,:) = [index_l, sqrt(bsxfun(#plus,sum(Matrix_1(index_k,:).^2,2),sum(Matrix_2(index_l,:).^2,2)')-2*(Matrix_1(index_k,:)*Matrix_2(index_l,:)'))];
index_j = index_j + 1;
end
end
I need M2_Index_Dist to provide a ((n*m) x 2) matrix with the index of matrix_2 in the first column and the distance in the second column.
Output example:
M2_Index_Dist = [ 1, 5.465
2, 56.52
3, 6.21
1, 35.3
2, 56.52
3, 0
1, 43.5
2, 9.3
3, 236.1
1, 8.2
2, 56.52
3, 5.582]
Here's how to apply bsxfun with your formula (||A-B|| = sqrt(||A||^2 + ||B||^2 - 2*A*B)):
d = real(sqrt(bsxfun(#plus, dot(Matrix_1,Matrix_1,2), ...
bsxfun(#minus, dot(Matrix_2,Matrix_2,2).', 2 * Matrix_1*Matrix_2.')))).';
You can avoid the final transpose if you change your interpretation of the matrix.
Note: There shouldn't be any complex values to handle with real but it's there in case of very small differences that may lead to tiny negative numbers.
Edit: It may be faster without dot:
d = sqrt(bsxfun(#plus, sum(Matrix_1.*Matrix_1,2), ...
bsxfun(#minus, sum(Matrix_2.*Matrix_2,2)', 2 * Matrix_1*Matrix_2.'))).';
Or with just one call to bsxfun:
d = sqrt(bsxfun(#plus, sum(Matrix_1.*Matrix_1,2), sum(Matrix_2.*Matrix_2,2)') ...
- 2 * Matrix_1*Matrix_2.').';
Note: This last order of operations gives identical results to you, rather than with an error ~1e-14.
Edit 2: To replicate M2_Index_Dist:
II = ndgrid(1:size(Matrix_2,1),1:size(Matrix_2,1));
M2_Index_Dist = [II(:) d(:)];
If I understand correctly, this does what you want:
ind = repmat((1:size(Matrix_2,1)).',size(Matrix_1,1),1); %'// first column: index
d = pdist2(Matrix_2,Matrix_1); %// compute distance between each pair of rows
d = d(:); %// second column: distance
result = [ind d]; %// build result from first column and second column
As you see, this code calls pdist2 to compute the distance between every pair of rows of your matrices. By default this function uses Euclidean distance.
If you don't have pdist2 (which is part of the the Statistics Toolbox), you can replace line 2 above with bsxfun:
d = squeeze(sqrt(sum(bsxfun(#minus,Matrix_2,permute(Matrix_1, [3 2 1])).^2,2)));