I am trying to assign a the same random number to each observation within a group. Thus in the dataset below, the value of the variable "random" would be equal for each observation where gp=B, and would take another value for each observation where gp=A, and so on.
data test ;
input gp $ a b c ;
datalines;
B 2 2 3
B 2 2 3
A 1 2 3
A 1 2 3
C 3 3 4
C 3 3 4
;
Stupidly I tried to create a different seed for each group based upon common unique values to each group:
data test2 ;
set test ;
seed = a*b*c ;
random = ranuni(seed) ;
run ;
This creates a common starting point per group, but which obviously changes for each observation.
How can I obtain a random number equivalent for each observation in the group? Due to the very large size of the real dataset I would like to avoid any sorting or other time consuming processes.
The required datset would thus look something like:
data want ;
input gp $ a b c random ;
datalines;
B 2 2 3 0.123
B 2 2 3 0.123
A 1 2 3 0.456
A 1 2 3 0.456
C 3 3 4 0.789
C 3 3 4 0.789
;
this should do the trick, ask me if you have any questions:
proc sort data=test;
by gp;
run;
data test2;
drop seed;
set test;
by gp;
retain random;
if first.gp then do;
seed = a*b*c ;
random = ranuni(seed) ;
end;
run;
basically, each time you call ranuni you get a new random number, so you only want to call it when the id (gp) changes.
Related
I'm trying to extract a matrix with two columns. The first column is the data that I want to group into a vector, while the second column is information about the group.
A =
1 1
2 1
7 2
9 2
7 3
10 3
13 3
1 4
5 4
17 4
1 5
6 5
the result that i seek are
A1 =
1
2
A2 =
7
9
A3 =
7
10
13
A4=
1
5
17
A5 =
1
6
as an illustration, I used the eval function but it didn't give the results I wanted
Assuming that you don't actually need individually named separated variables, the following will put the values into separate cells of a cell array, each of which can be an arbitrary size and which can be then retrieved using cell index syntax. It makes used of logical indexing so that each iteration of the for loop assigns to that cell in B just the values from the first column of A that have the correct number in the second column of A.
num_cells = max (A(:,2));
B = cell (num_cells,1);
for idx = 1:max(A(:,2))
B(idx) = A((A(:,2)==idx),1);
end
B =
{
[1,1] =
1
2
[2,1] =
7
9
[3,1] =
7
10
13
[4,1] =
1
5
17
[5,1] =
1
6
}
Cell arrays are accessed a bit differently than normal numeric arrays. Array indexing (with ()) will return another cell, e.g.:
>> B(1)
ans =
{
[1,1] =
1
2
}
To get the contents of the cell so that you can work with them like any other variable, index them using {}.
>> B{1}
ans =
1
2
How it works:
Use max(A(:,2)) to find out how many array elements are going to be needed. A(:,2) uses subscript notation to indicate every value of A in column 2.
Create an empty cell array B with the right number of cells to contain the separated parts of A. This isn't strictly necessary, but with large amounts of data, things can slow down a lot if you keep adding on to the end of an array. Pre-allocating is usually better.
For each iteration of the for loop, it determines which elements in the 2nd column of A have the value matching the value of idx. This returns a logical array. For example, for the third time through the for loop, idx = 3, and:
>> A_index3 = A(:,2)==3
A_index3 =
0
0
0
0
1
1
1
0
0
0
0
0
That is a logical array of trues/falses indicating which elements equal 3. You are allowed to mix both logical and subscripts when indexing. So using this we can retrieve just those values from the first column:
A(A_index3, 1)
ans =
7
10
13
we get the same result if we do it in a single line without the A_index3 intermediate placeholder:
>> A(A(:,2)==3, 1)
ans =
7
10
13
Putting it in a for loop where 3 is replaced by the loop variable idx, and we assign the answer to the idx location in B, we get all of the values separated into different cells.
I have the following mock up table
#n a b group
1 1 1 1
2 1 2 1
3 2 2 1
4 2 3 1
5 3 4 2
6 3 5 2
7 4 5 2
I am using SAS for this problem. In column group, the rows that are interconnected through a and b are grouped. I will try to explain why these rows are in the same group
row 1 to 2 are in group 2 since they both have a = 1
row 3 is in group 2 since b = 2 in row 2 and 3 and row 2 is in group 1
row 3 and 4 are in group 1 since a = 2 in both rows and row 3 is in group 1
The overall logic is that if a row x contains the same value of a or b as row y, row x also belongs to the same group as y is a part of.
Following the same logic, row 5,6 and 7 are in group 2.
Is there any way to make an algorithm to find these groups?
Case I:
Grouping defined as to be item linkage within contiguous rows.
Use the LAG function to examine both variables prior values. Increase the group value if both have changed. For example
group + ( a ne lag(a) and b ne lag(b) );
Case II:
Grouping determined from pair item slot value linkages over all data.
From grouping pairs by either key
General statement of problem:
-----------------------------
Given: P = p{i} = (p{i,1},p{i,2}), a set of pairs (key1, key2).
Find: The distinct groups, G = g{x}, of P,
such that each pair p in a group g has this property:
key1 matches key1 of any other pair in g.
-or-
key2 matches key2 of any other pair in g.
Demonstrates
… an iterative way using hashes.
Two hashes maintain the groupId assigned to each key value.
Two additional hashes are used to maintain group mapping paths.
When the data can be passed without causing a mapping, then the groups have
been fully determined.
A final pass is done, at which point the groupIds are assigned to each
pair and the data is output to a table.
In Pandas 0.19 I have a large dataframe with a Multiindex of the following form
C0 C1 C2
A B
bar one 4 2 4
two 1 3 2
foo one 9 7 1
two 2 1 3
I want to sort bar and foo (and many more double lines as them) according to "two" to get the following:
C0 C1 C2
A B
bar one 4 4 2
two 1 2 3
foo one 7 9 1
two 1 2 3
I am interested in speed (as I have many columns and many pairs of rows). I am also happy with re-arranging the data if it speeds up the sorting. Many thanks
Here is a mostly numpy solution that should yield good performance. It first selects only the 'two' rows and argsorts them. It then sets this order for each row of the original dataframe. It then unravels this order (after adding a constant to offset each row) and the original dataframe values. It then reorders all the original values based on this unraveled, offset and argsorted array before creating a new dataframe with the intended sort order.
rows, cols = df.shape
df_a = np.argsort(df.xs('two', level=1))
order = df_a.reindex(df.index.droplevel(-1)).values
offset = np.arange(len(df)) * cols
order_final = order + offset[:, np.newaxis]
pd.DataFrame(df.values.ravel()[order_final.ravel()].reshape(rows, cols), index=df.index, columns=df.columns)
Output
C0 C1 C2
A B
bar one 4 4 2
two 1 2 3
foo one 7 9 1
two 1 2 3
Some Speed tests
# create much larger frame
import string
idx = pd.MultiIndex.from_product((list(string.ascii_letters), list(string.ascii_letters) + ['two']))
df1 = pd.DataFrame(index=idx, data=np.random.rand(len(idx), 3), columns=['C0', 'C1', 'C2'])
#scott boston
%timeit df1.groupby(level=0).apply(sortit)
10 loops, best of 3: 199 ms per loop
#Ted
1000 loops, best of 3: 5 ms per loop
Here is a solution, albeit klugdy:
Input dataframe:
C0 C1 C2
A B
bar one 4 2 4
two 1 3 2
foo one 9 7 1
two 2 1 3
Custom sorting function:
def sortit(x):
xcolumns = x.columns.values
x.index = x.index.droplevel()
x.sort_values(by='two',axis=1,inplace=True)
x.columns = xcolumns
return x
df.groupby(level=0).apply(sortit)
Output:
C0 C1 C2
A B
bar one 4 4 2
two 1 2 3
foo one 7 9 1
two 1 2 3
There are two vectors:
a = 1:5;
b = 1:2;
in order to find all combinations of these two vectors, I am using the following piece of code:
[A,B] = meshgrid(a,b);
C = cat(2,A',B');
D = reshape(C,[],2);
the result includes all the combinations:
D =
1 1
2 1
3 1
4 1
5 1
1 2
2 2
3 2
4 2
5 2
now the questions:
1- I want to decrease the number of operations to improve the performance for vectors with bigger size. Is there any single function in MATLAB that is doing this?
2- In the case that the number of vectors is more than 2, the meshgrid function cannot be used and has to be replaced with for loops. What is a better solution?
For greater than 2 dimensions, use ndgrid:
>> a = 1:2; b = 1:3; c = 1:2;
>> [A,B,C] = ndgrid(a,b,c);
>> D = [A(:) B(:) C(:)]
D =
1 1 1
2 1 1
1 2 1
2 2 1
1 3 1
2 3 1
1 1 2
2 1 2
1 2 2
2 2 2
1 3 2
2 3 2
Note that ndgrid expects (rows,cols,...) rather than (x,y).
This can be generalized to N dimensions (see here and here):
params = {a,b,c};
vecs = cell(numel(params),1);
[vecs{:}] = ndgrid(params{:});
D = reshape(cat(numel(vecs)+1,vecs{:}),[],numel(vecs));
Also, as described in Robert P.'s answer and here too, kron can also be useful for replicating values (indexes) in this way.
If you have the neural network toolbox, also have a look at combvec, as demonstrated here.
One way would be to combine repmat and the Kronecker tensor product like this:
[repmat(a,size(b)); kron(b,ones(size(a)))]'
ans =
1 1
2 1
3 1
4 1
5 1
1 2
2 2
3 2
4 2
5 2
This can be scaled to more dimensions this way:
a = 1:3;
b = 1:3;
c = 1:3;
x = [repmat(a,1,numel(b)*numel(c)); ...
repmat(kron(b,ones(1,numel(a))),1,numel(c)); ...
kron(c,ones(1,numel(a)*numel(b)))]'
There is a logic! First: simply repeat the first vector. Secondly: Use the tensor product with the dimension of the first vector and repeat it. Third: Use the tensor product with the dimension of (first x second) and repeat (in this case there is not fourth, so no repeat.
i want to create 1D vector in matlab from given matrix,for this i have implemented following algorithm ,which use trivial way
% create one dimensional vector from 2D matrix
function [x]=one_dimensional(b,m,n)
k=1;
for i=1:m
for t=1:n
x(k)=b(i,t);
k=k+1;
end
end
x;
end
when i run it using following example,it seems to do it's task fine
b=[2 1 3;4 2 3;1 5 4]
b =
2 1 3
4 2 3
1 5 4
>> one_dimensional(b,3,3)
ans =
2 1 3 4 2 3 1 5 4
but generally i know that,arrays are not good way to use in matlab,because it's performance,so what should be effective way for transformation matrix into row/column vector?i am just care about performance.thanks very much
You can use the (:) operator...But it works on columns not rows so you need to transpose using the 'operator before , for example:
b=b.';
b(:)'
ans=
2 1 3 4 2 3 1 5 4
and I transposed again to get a row output (otherwise it'll the same vector only in column form)
or also, this is an option (probably a slower one):
reshape(b.',1,[])