Related
How can we efficiently calculate pairwise cosine distances in a matrix using TensorFlow? Given an MxN matrix, the result should be an MxM matrix, where the element at position [i][j] is the cosine distance between i-th and j-th rows/vectors in the input matrix.
This can be done with Scikit-Learn fairly easily as follows:
from sklearn.metrics.pairwise import pairwise_distances
pairwise_distances(input_matrix, metric='cosine')
Is there an equivalent method in TensorFlow?
There is an answer for getting a single cosine distance here: https://stackoverflow.com/a/46057597/288875 . This is based on tf.losses.cosine_distance .
Here is a solution which does this for matrices:
import tensorflow as tf
import numpy as np
with tf.Session() as sess:
M = 3
# input
input = tf.placeholder(tf.float32, shape = (M, M))
# normalize each row
normalized = tf.nn.l2_normalize(input, dim = 1)
# multiply row i with row j using transpose
# element wise product
prod = tf.matmul(normalized, normalized,
adjoint_b = True # transpose second matrix
)
dist = 1 - prod
input_matrix = np.array(
[[ 1, 1, 1 ],
[ 0, 1, 1 ],
[ 0, 0, 1 ],
],
dtype = 'float32')
print "input_matrix:"
print input_matrix
from sklearn.metrics.pairwise import pairwise_distances
print "sklearn:"
print pairwise_distances(input_matrix, metric='cosine')
print "tensorflow:"
print sess.run(dist, feed_dict = { input : input_matrix })
which gives me:
input_matrix:
[[ 1. 1. 1.]
[ 0. 1. 1.]
[ 0. 0. 1.]]
sklearn:
[[ 0. 0.18350345 0.42264974]
[ 0.18350345 0. 0.29289323]
[ 0.42264974 0.29289323 0. ]]
tensorflow:
[[ 5.96046448e-08 1.83503449e-01 4.22649741e-01]
[ 1.83503449e-01 5.96046448e-08 2.92893231e-01]
[ 4.22649741e-01 2.92893231e-01 0.00000000e+00]]
Note that this solution may not be the optimal one as it calculates all entries of the (symmetric) result matrix, i.e. does almost twice of the calculations. This is likely not a problem for small matrices, for large matrices a combination of loops may be faster.
Note also that this does not have a minibatch dimension so works for a single matrix only.
Elegant solution (output is the same as from scikit-learn pairwise_distances function):
def compute_cosine_distances(a, b):
# x shape is n_a * dim
# y shape is n_b * dim
# results shape is n_a * n_b
normalize_a = tf.nn.l2_normalize(a,1)
normalize_b = tf.nn.l2_normalize(b,1)
distance = 1 - tf.matmul(normalize_a, normalize_b, transpose_b=True)
return distance
test
input_matrix = np.array([[1, 1, 1],
[0, 1, 1],
[0, 0, 1]], dtype = 'float32')
compute_cosine_distances(input_matrix, input_matrix)
output:
<tf.Tensor: id=442, shape=(3, 3), dtype=float32, numpy=
array([[5.9604645e-08, 1.8350345e-01, 4.2264974e-01],
[1.8350345e-01, 5.9604645e-08, 2.9289323e-01],
[4.2264974e-01, 2.9289323e-01, 0.0000000e+00]], dtype=float32)>
I have a 3d tensor and would like to ensure that all values fall within a given range (0-1 in this case). In order to do this I have already written the following code:
function capTo1or0 (Tensor3d)
tensor_width=Tensor3d:size()[2]
tensor_height=Tensor3d:size()[3]
tensor_depth=Tensor3d:size()[1]
for i=1,tensor_width,1 do
for j=1,tensor_height,1 do
for k=1,tensor_depth,1 do
if(Tensor3d[k][i][j])>1 then
Tensor3d[k][i][j]=1
end
if(Tensor3d[k][i][j]<0.0) then
Tensor3d[k][i][j]=0.0
end
end
end
end
return Tensor3d
end
and it works, there is just one problem: performance is terrible, I know that there has to be some better way of doing this then looping over the entire array given that most tensor operations that do not involve manually looping over an array are much faster. Anybody know how to make this faster?
An example in this is say that I have a `2-3-3` array with the values
[1, 2, 0.5][0.5,0.2,-0.2]
[0.1,0.2,0.3][1, 1, 1 ]
[-2, -1, 2 ][0.2,-5,-1 ]
then I expect an outcome of
[1, 1, 0.5][0.5,0.2,0]
[0.1,0.2,0.3][1, 1, 1 ]
[0, 0, 1 ] [0.2,0,-1 ]
replacing every value under the lower bound of 0 with 0 and every value over the upper bound of 1 with 1.
Anybody know how to do this fast?
I have never used Torch, but it's documentation says:
http://torch7.readthedocs.io/en/rtd/maths/#torch.clamp
[res] torch.clamp([res,] tensor1, min_value, max_value)
Clamp all elements in the tensor into the range [min_value,
max_value]. ie:
y_i = x_i, if x_i >= min_value or x_i <= max_value
= min_value, if x_i < min_value
= max_value, if x_i > max_value
z=torch.clamp(x,0,1) will return a new tensor with the result of x
bounded between 0 and 1.
torch.clamp(z,x,0,1) will put the result in z.
x:clamp(0,1) will perform the clamp operation in place (putting the
result in x).
z:clamp(x,0,1) will put the result in z.
I guess that is what you are looking for?
I have implemented several graph coloring algorithms as a class project and I would like to test and compare their performance. For this I need test graphs with known chromatic number.
I have expected this to be quite common requirement so that there will be algorithm and/or library readily available. It turned out that I was wrong. After quite extensive search the best I found is a paper from 1979.¹ In section 6 there is a method for generation of graphs with known chromatic number described. Before endeavoring to translate it from math to programmers I would like to know whether there really isn’t implementation available to avoid reinventing a wheel.
¹ Leighton, F. T. (1979). A graph coloring algorithm for large scheduling problems. In Journal of Research of the National Bureau of Standards (pp. 489-506). http://doi.org/10.6028/jres.084.024
So I don't know ruby, but came across this question when I was looking at the same paper and also confused. I've implemented the algorithm in Rust (link to playground) using the petgraph library to represent the graphs.
But the basic idea is that you want to choose some values:
a: multiplier (also called factor)
c: increment
m: modulus
x[0]: some seed or starting value
That define a pseudo-random number generator (specifically a Linear congruential generator wikipedia) which is defined by the recursive relation:
x[i] = (x[i - 1] * a + c) % m
For some nice properties, we also need the parameters to meet the criteria (which Leighton mentions in the paper as the first step):
m much greater than n (the number of nodes)
gcd(m, n) == k
gcd(m, c) == 1 or m and c are coprime
for any prime number p: if m % p == 0 then (a-1) % p == 0
if m % 4 == 0 then (a-1) % 4 == 0
I believe these criteria enable Leighton to prove that the procedure works, but I am not sure. Luckily, he provides some values for m, c, a, and x[0] in Appendix C. For example:
// Values taken from Leighton
let chromatic_number = 5; // k
let modulus = 84_035; // m
let increment = 6_859; // c
let multiplier = 8_401; // a
let mut x_i = vec![33_289]; // x[0]
After that, we generate a vector of random numbers x_i according to the recurrence relation:
let num_rand_numbers = 100;
for i in (1..num_rand_numbers) {
// x[i] = (x[i-1] * a + c) % m
x_i.push((x_i[i - 1] * multiplier + increment) % modulus);
}
So we've got random numbers x_i, but they're in the interval [0, m] and our nodes are in the range [0, n-1] where m is a lot greater than n. So create a vector y_i where:
// Construct y_i in [0, n-1] such that y_i = x_i % n
let y_i = x_i.iter() // Convert x_i to an iterator
.map(|x| x % num_nodes) // y_i = x_i % n
.collect::<Vec<_>>(); // convert iterator to vector
So there's a lot of work just to efficiently choose some random vectors but once they're chosen, the algorithm goes like:
Create a new undirected graph with num_nodes nodes. In my testing, Leighton's method resulted in lots of islands with some nodes completely unreachable from the others. So here I connect them all in a line so that we don't have any island nodes.
let mut graph = UnGraph::new_undirected();
let mut prev;
let mut curr = graph.add_node(Node {});
for n in 1..num_nodes {
prev = curr;
curr = graph.add_node(Node {});
graph.add_edge(prev, curr, Edge {});
}
Cache the node indices so we don't have to look them up every time
let node_indices = graph.node_indices().collect::<Vec<_>>();
Now define an h-clique as a set of nodes which are all fully connected to
each other, such that each node has h-1 edges coming from it.
This corresponds to h colours being required to colour that clique.
We create multiple cliques of decreasing degrees: start with the high-degree cliques and looping until you link up the cliques of degree 2.
let mut y_idx = 0;
// This for-loop takes the b-vector (which contains how many 2-cliques, 3-cliques, 4-cliques, ..., k-cliques there should be and converts it into multiple tuples like (number of cliques, size of those cliques)
'make_cliques: for (num_cliques, clique_degree) in b_vec.into_iter().zip((1..(k + 1)).into_iter()).rev() {
if clique_degree <= 1 {
break;
}
For each clique of size clique_degree that we want to create:
for clique_idx in 0..num_cliques {
// Create a clique with nodes indexed y_idx to y_idx+clique_degree
for i in 0..clique_degree {
for j in (i + 1)..clique_degree {
Go through the randomly-selected nodes, and attempt to connect them together
let a = node_indices[y_i[y_idx + i]];
let b = node_indices[y_i[y_idx + j]];
If we've already added this edge => don't add it again
if graph.contains_edge(a, b) {
continue;
}
Note that in my testing, Leighton's method would add edges to a node even if it already had the maximum number of edges. So here I check if either of the candidate nodes already have the maximum number of neighbours, and if it does, I don't add this edge.
if graph.neighbors(b).count() == k || graph.neighbors(a).count() == k {
continue;
}
Now that we know it's a valid edge, actually add it:
graph.add_edge(a, b, Edge {});
If we've reached the maximum number of edges => stop adding edges
if graph.edge_count() == num_edges {
print!("Have enough edges, exiting");
break 'make_cliques;
}
}
}
y_idx += clique_degree;
}
}
Hope that helps! Maybe someone can convert the rust to ruby. The full, running code is available on the playground.
Example output with m=84035, n=10, k=5, c=6859, a=8401, b_vec: [0, 1, 1, 3, 6]
Link to Dotgraph visualisation
Values: m=84035, n=10, k=5, c=6859, a=8401, b_vec: [0, 1, 1, 3, 6]
Making 6 cliques of degree 5
Making 3 cliques of degree 4
Making 1 cliques of degree 3
Making 1 cliques of degree 2
Making 0 cliques of degree 1
Dotstring of graph:
graph {
0 [ label = "Node" ]
1 [ label = "Node" ]
2 [ label = "Node" ]
3 [ label = "Node" ]
4 [ label = "Node" ]
5 [ label = "Node" ]
6 [ label = "Node" ]
7 [ label = "Node" ]
8 [ label = "Node" ]
9 [ label = "Node" ]
0 -- 1 [ ]
1 -- 2 [ ]
2 -- 3 [ ]
3 -- 4 [ ]
4 -- 5 [ ]
5 -- 6 [ ]
6 -- 7 [ ]
7 -- 8 [ ]
8 -- 9 [ ]
9 -- 3 [ ]
9 -- 7 [ ]
9 -- 1 [ ]
9 -- 0 [ ]
3 -- 7 [ ]
3 -- 1 [ ]
7 -- 1 [ ]
8 -- 2 [ ]
8 -- 0 [ ]
2 -- 0 [ ]
4 -- 6 [ ]
4 -- 0 [ ]
5 -- 2 [ ]
4 -- 8 [ ]
}
Ruby yields results in fraction for matrix inverse operation but not for matrix multiplication operation. E.g., below code:
require 'matrix'
(Matrix[ [-1/2] ] * Matrix[ [1/2]])
yields Matrix[[0]] instead of Matrix[[-1/4]]. Why this behavior?
Your problem has been identified, but your real question has not been answered, namely, when multiplying matrix objects, when is an element of the product an integer, a rational number or a float?
If a and b are matrix objects, each element of a*b will be:
an integer if all elements of a and b used in its calculation are integers;
a rational number if, among the elements of a and b used in its calculation there is at least one rational number and the remainder are integers or rationals; and
a float if at least one of the elements of a and b used in its calculation is a float.
I will give a few examples, but first let's consider how Ruby expresses rational numbers. A rational number is a number that can be expressed as the ratio of two integers. For example, 1.5 is a rational number since it can be expressed 3/2. We can't write it that way in Ruby, however, since 3/2 will be replaced by 1, the result of integer division. Instead, we create an instance of the class Rational:
r = Rational(3,2)
#=> (3/2)
and use that in the calculations. (Note the parentheses in the return value.) We can extract its numerator and denominator, or convert it to an integer (rounding down or up) or a float:
r.numerator
#=> 3
r.denominator
#=> 2
r.to_i
#=> 1
r.ceil
#=> 2
r.to_f
#=> 1.5
Now let's look at some examples.
require 'matrix'
Matrix[[Rational(-1,2)]] * Matrix[[Rational(1,2)]]
#=> Matrix[[(-1/4)]]
Matrix[[-1]] * Matrix[[Rational(1,2)]]
# => Matrix[[(-1/2)]]
Matrix[[-0.5]] * Matrix[[Rational(1,2)]]
#=> Matrix[[-0.25]]
Matrix[[Rational(-1,2), Rational(1,2)]].transpose * Matrix[[Rational(1,2), 0.5]]
#=> Matrix[[(-1/4), -0.25], [(1/4), 0.25]]
Now let's consider the inverse of a matrix:
Matrix[[Rational(-1,2), 1],[2, Rational(2,3)]].inverse
#=> Matrix[[(-2/7), (3/7)], [(6/7), (3/14)]]
Matrix[[Rational(-1,2), 1.0],[2, Rational(2,3)]].inverse
#=> Matrix[[-0.2857142857142857, 0.4285714285714286],
# [0.8571428571428571, 0.21428571428571427]]
In calculating the inverse of a matrix with n rows and columns there are n "pivoting" steps. If, as in my latter example, the matrix contains a mix of integers, rationals and floats, when each pivot operation is performed:
each integer will be converted to a float if at least one float is used in its calculation, else it will be converted to a rational if at least one rational is used in its calculation; else it will remain an integer;
each rational will be converted to a float if there is at least one float used in its calculation, else it will remain a rational; and
each float will remain a float.
As "once a float always a float", it won't be long before all elements of the computed matrix are floats. (I believe it can be proven that the inverse will contain all floats if the original matrix contains at least one float.)
you need to choose correct type of data. Int divided by int is int.
1/2 -> 0
1.0/2 -> 0.5
1/2.0 -> 0.5
0.5 -> 0.5
1/2r -> (1/2)
(1/2r).to_f -> 0.5
Matrix[ [-1/2r] ] * Matrix[ [1/2r]]
so basically Matrix[ [-1/2] ] * Matrix[ [1/2] ] is the same as Matrix[ [-1] ] * Matrix[ [0] ]
There is no fractional result for integer division. For example, 5 divided by 3 is 1 with remainder 2. In your code, -1/2 is -1, and 1/2 is 0. And -1 times 0 is 0.
I have an unsorted set of n-dimensional vectors and, for each of the n dimensions in turn, I am looking for the subsets of vectors that differ in only this dimension's component. How can I do this efficiently?
Example:
[ (1,2,3), (1,3,3), (2,3,3), (1,2,5), (2,2,5), (2,3,4) ]
dim 3 variable: [ (1,2,3), (1,2,5) ] & [ (2,3,3), (2,3,4) ]
dim 2 variable: [ (1,2,3), (1,3,3) ]
dim 1 variable: [ (1,3,3), (2,3,3) ] & [ (1,2,5), (2,2,5) ]
Thanks very much for your help!
EDIT
As requested in a comment I am now posting my buggy code:
recursive subroutine get_peaks_on_same_axis(indices, result, current_dim, look_at, last_dim, mode, upper, &
num_groups, num_dim)
! Group the indices that denote the location of peaks within PEAK_INDICES which have n-1 dimensions in common.
! Eventually, RESULT will hold the groups of these peaks.
! e.g.: result(1,:) == (3,7,9) <= peak_indices(3), peak_indices(7), and peak_indices(9) belong together
integer, intent(in) :: indices(:), current_dim, look_at, last_dim, mode, num_dim
integer, intent(inout) :: upper(:), num_groups, result(:,:) ! in RESULT: each line holds a group of peaks
integer :: i, pos_on_axis, next_dim, aux(0:num_dim-1), stat
integer, allocatable :: num_peaks(:), groups(:,:)
integer, save :: slot
if (mode.eq.0) slot = 1
! we're only writing to RESULT once group determination has been completed
if (current_dim.eq.last_dim) then
! saving each column of 'groups' of the instance of the subroutine called one level further up
! = > those are the peaks which have n-1 dimensions in common
upper(slot) = ubound(indices,1)
result(slot,1:upper(slot)) = indices
num_groups = slot ! after the final call it will contain the actual number of peak groups
slot = slot + 1
return
end if
aux(0:num_dim-2) = (/ (i,i = 2,num_dim) /)
aux(num_dim-1) = 1
associate(peak_indices => public_spectra%intensity(look_at)%peak_indices, &
ndp => public_spectra%axes(look_at)%ax_set(current_dim)%num_data_points)
! potentially as many peaks as there are points in this dimension
allocate(num_peaks(ndp), groups(ndp,ubound(indices,1)), stat=stat)
if (stat.ne.0) call aloerr('spectrum_paraphernalia.f90',763)
num_peaks(:) = 0
! POS_ON_AXIS: ppm value of the peak in dimension DIM, converted to an index on the axis
! GROUPS: peaks that have the same axis index in dimension DIM; line: index on axis;
do i=1,ubound(indices,1)
pos_on_axis = peak_indices(current_dim,indices(i))
num_peaks(pos_on_axis) = num_peaks(pos_on_axis) + 1 ! num. of peaks that have this coordinate
groups(pos_on_axis,num_peaks(pos_on_axis)) = indices(i)
end do
next_dim = aux(mod(current_dim+(num_dim-1),num_dim))
do pos_on_axis=1,ubound(num_peaks,1)
if (num_peaks(pos_on_axis).gt.0) then
call get_peaks_on_same_axis(groups(pos_on_axis,1:num_peaks(pos_on_axis)), result, next_dim, look_at, last_dim, &
1, upper, num_groups, num_dim)
end if
end do
end associate
end subroutine
What about the naive way?
Let's assume, you have m vectors with length n.
Then you have to compare all vectors with each other which results in 1/2*(m^2+m-) = O(m^2) comparisons.
In each comparison you check your vectors element wise. If you find one difference you have to make sure, that there is no other difference. In best case, all vectors differ in the first 2 elements which is then 2 comparisons. The worst case is one or no difference which leads to n comparisons for the appropriate vectors.
If there is only one difference you can store its dimension, otherwise store a value like 0 or -1.