I have a tensor A (M x N x C) and mask(M x N) for tensor A.
Due to memory issue for my transformer network, I want to make a small tensor by taking only the part defined by the mask from tensor A.
For example, Figure 1 is my tensor A. I paint gray for masked query-key pair.
Figure 1. example for tensor A
I don't need gray colored value for further calculation. So I want to make smaller tensor including all required value.
From Figure 1 tensor, I hope to make tensor like Figure 2. In Figure 2, gray colored value is just dummy value, and whether the index of corresponding value is a dummy value can be known through the mask.(Figure 3)
Figure 2. smaller tensor
Figure 3. Mask indicating index of dummy value filled
How can I do this with efficient torch operation?
I think you are looking for sparse tensors. Sparse representation does not give you a "packed" matrix with "dummy" values, but rather a different way of storing only those entires you care about.
Pytorch also support some operations on sparse matrices.
Related
I have two sparse matrices "Matrix1" and "Matrix2" of the same size p x n.
By sparse matrix I mean that it contains a lot of exactly zero elements.
I want to show the two matrices under the same colormap and a unique colorbar. Doing this in MATLAB is straightforward:
bottom = min(min(min(Matrix1)),min(min(Matrix2)));
top = max(max(max(Matrix1)),max(max(Matrix2)));
subplot(1,2,1)
imagesc(Matrix1)
colormap(gray)
caxis manual
caxis([bottom top]);
subplot(1,2,2)
imagesc(Matrix2)
colormap(gray)
caxis manual
caxis([bottom top]);
colorbar;
My problem:
In fact, when I show the matrix using imagesc(Matrix), it can ignore the noises (or backgrounds) that always appear with using imagesc(10*log10(Matrix)).
That is why, I want to show the 10*log10 of the matrices. But in this case, the minimum value will be -Inf since the matrices are sparse. In this case caxis will give an error because bottom is equal to -Inf.
What do you suggest me? How can I modify the above code?
Any help will be very appreciated!
A very important point is that the minimum value in your matrix will always be 0. Leveraging this, a very simple way to address your problem is to add 1 inside the log operation so that values that map to 0 in the original matrix also map to 0 in the log operation. This avoids the -Inf error that you're encountering. In fact, this is a very common way of visualizing the Fourier Transform if you will. Adding 1 to the logarithm ensures that the transform has no negative values in the output, yet the derivative or its rate of change remains intact as the effect is simply a translation of the curve by 1 unit to the left.
Therefore, simply do imagesc(10*log10(1 + Matrix));, then the minimum is always bounded at 0 while the maximum is unbounded but subject to the largest value that is seen in Matrix.
consider the following code-
a=tf.convert_to_tensor(np.array([[1001,1002],[3,4]]), dtype=tf.float32)
b=tf.reduce_max(a,reduction_indices=[1], keep_dims=True)
with tf.Session():
print b.eval()
What exactly is the purpose of keep_dims here? I tested quite a bit, and saw that the above is equivalent to-
b=tf.reduce_max(a,reduction_indices=[1], keep_dims=False)
b=tf.expand_dims(b,1)
I maybe wrong, but my guess is that if keep_dims is False, we get a 2D coloumn vector. And if keep_dims=True, we have a 2x1 matrix. But how are they different?
If you reduce over one or more indices (i.e. dimensions of the tensor), you effectively reduce the rank of the tensor (i.e. its number of dimensions or, in other words, the number of indices you need in order to access an element of the tensor). By setting keep_dims=True, you are telling tensorflow to keep the dimensions over which you reduce. They will then have size 1, but they are still there. While a column vector and a nx1 matrix are conceptually the same thing, in tensorflow, these are tensors of rank 1 (you need a single index to access an element) and rank 2 (you need two indices to access an element), respectively.
I am wondering if I understood the mean normalization of images correctly.
As far as I know, you calculate the mean value over all pixels (lets assume it is in grayscale). Then, for each pixel, you subtract this mean value.
But how should one deal with negative values which could arise? For example, the whole image has a mean value of 100, but one specific pixel has an intensity of 90. After this normalization, the pixel's value would be -10.
This may not be quite what you're looking for but one option that avoids negative numbers in your output would be to normalize to the range of values present rather than to the image mean.
The equation would be: X' = (X - Xmin)/(Xmax - Xmin). This rescales the image to be between 0 and 1 (no negative values involved). If you'd like to save it as an easily view-able greyscale you could multiply values by 255 to rescale it.
It may also be worth noting that unless the entire image has a constant intensity, there is guaranteed to be some negative values after subtracting the mean (not simply a possibility that they could arise).
You don't have to deal with negative inputs, the model can handle them. It is good practice, for a Neural Network for example, to have inputs in the range [-1, 1]
I'm kind of ashamed to even ask this but here goes. In every Matlab help file where the input matrix is a NxD matrix X Matlab describes the matrix arrangement as
Data, specified as a numeric matrix. The rows of X correspond to
observations, and the columns correspond to variables.
Above taken from help of kmeans
I'm kind of confused as to what does Matlab mean by observations and variables.
Suppose I have a data matrix composed of 100 images. Each image is represented by a feature vector of size 128 x 1. So here is 100 my observations and 128 the variables or is it the other way around?
Will my data matrix be of the size 128 x 100 or 100 x 128
Eugene's explanation in a statistical and probability construct is great, but I would like to explain it more in the viewpoint of data analysis and image processing.
Think of an observation as one sample from your data set. In this case, one observation is one image. For each sample, it has some dimensionality associated to it or a number of variables used to represent such a sample.
For example, if we had a set of 100 2D Cartesian points, the amount of observations is 100, while the dimensionality or the total number of variables used to describe the point is 2: We have a x point and a y point. As such, in the MATLAB universe, we'd place all of these data points into a single matrix. Each row of the matrix denotes one point in your data set. Therefore, the matrix you would create here is 100 x 2.
Now, go back to your problem. We have 100 images and each image can be expressed by 128 features. This suspiciously looks like you are trying to use SIFT or SURF to represent an image so think of this situation where each image can be described by a 128-dimensional vector, or a histogram with bins of 128 elements. Each feature is part of the dimensionality makeup that makes up the image. Therefore, you would have a 100 x 128 matrix. Each row represents one image, where each image is represented as a 1 x 128 feature vector.
In general, MATLAB's machine learning and data analysis algorithms assume that your matrix is M x N, where M is the total number of points that make up your data set while N is the dimensionality of one such point in your data set. In MATLAB's universe, the total number of observations is equal to the total number of points in your data set, while the total number of features / distinct attributes to represent one sample is the total number of variables.
tl:dr
Observation: One sample from your data set
Variable: One feature / attribute that helps describe an observation or sample in your data set.
Number of observations: Total number of points in your data set
Number of variables: Total number of features / attributes that make up an observation or sample in your data set.
It looks like you are talking about some specific statistical/probabilistic functions. In statistics or probability theory there are some random variables that are results of some kind of measurements/observations over time (or some other dimension). So such a matrix is just a collection of N measurements of D different random variables.
I am a quite newbie on matlab and i have a simple issue that is disturbing me,
I want to know if its possible to covert all the subscripts of a matrix to linear indices.
when using SUB2IND i must inform de x and y coordinates, but i want to convert all at the same time.
i can use function FIND which returns two vectors x and y and this way i can use SUB2IND succesfully, but FIND only returns the x and y coordinates of nonzero elements.
is there a smart way to do this?
If you want all of the elements of an array A as linear subscripts, this can be done simply via:
IND = 1:numel(A);
This works for any size or dimension array.
More on array indexing in Matlab, including the difference between linear indexing and logical indexing. When you use find you're essentially using logical indexing to obtain linear indexing. The find function can be used to reliably obtain all of your linear indices, via IND = find(A==A);, but this is horrendously inefficient.
you don't need to convert, just use a single number \ 1-D vector when accessing elements of your matrix. For example, given a 5x5 matrix M
M=magic(5);
you can access the last element using M(5,5) or using M(25) ...
similarly M(21:25) will give you the info of M(1,5),M(2,5),...M(5,5).