I have a structure called s in Matlab. This is a structure with two fields a and b. The structure size is 1 x 1,620,000.
It is a very large structure (that probably takes half of the ram of my machine). This is what the structure looks like:
I am looking for an efficient way to concatenate each of the fields a and b into two separate arrays that I can then export to csv. I built the code below, to do so, but even after 12 hours running it has not even reached a quarter of the loop. Any more efficient way of doing this?
a = [];
b =[];
total_n = size(s,2);
count = 1;
while size(s,2)>0
if size(s(1).a,1)
a = [a; s(1).a];
end
if size(s(1).b,1)
b = [b; s(1).b];
end
s(1) = []; %to save memory
if mod(count,1000) == 0
fprintf('Done %2f \n', [count/total_n])
end
count = count+1;
end
s(1) = []; %to save memory
ah, but such huge misunderstanding that comment is.
if size(s) is 1 x 1,620,000, you just suddenly forced the loop to do (under the hood, you dont see it)
snew=zeros(1,size(s,2)-1) # now you use double memory
snew=s(2:end) # now you force an unnecesary copy
So not only does that line make your code require double the memory, but also in each loop, you make an unnecesary copy of a large array.
Just replace your while for a normal for loop of for ii=1:size(s,2) and then index s!
Now, you can see hopefully then why the following is equally a big mistake (not only that, but any modern MATLAB version is currently telling you this is a bad idea in your editor)
a=[]
a=[a;s(1).a]
In here in each loop you are forcing MATLAB to make a new a that is 1 bigger than before, and copy the contents of the old a there.
instead, preallocate the size of a.
As you don't know what you are going to put there, I suggest using a cell array, as each s(ii).a has a different length.
You can then, after the loop, remove all empty (isempty) cells if you want.
Managed to do it efficiently:
s= struct2cell(s);
s= squeeze(s);
a = a(1,:);
a = a';
a = vertcat(a{:});
b = a(2,:);
b = b';
b = vertcat(b{:});
Related
I have some MWE below. What I want is to have a subsection of a range, interact with the rest of the range, but not itself.
For instance if the range is 1:100, I want to have a for loop that will have each index in 4:6, interact with all values of 1:100 BUT NOT 4:6.
I want to do this using ranges/filters to avoid generating temporary arrays.
In my case the total range is the number of atoms in the system. The sub-range, is the atoms in a specific molecule. I need to do calculations where each atom in a molecule interacts with all other atoms, but not the atoms in the same molecule.
Further
I am trying to avoid using if statements because that messes up parallel codes. Doing this with an if statement would be
for i=4:6
for j = 1:100
if j == 4 || j==5 || j==6
continue
end
println(i, " ", j)
end
end
I have actual indexing in my code, I would never hardcode values like the above... But I want to avoid that if statement.
Trials
The following does what I want, but I now realize that using filter is bad when it comes to memory and the amount used scales linearly with b.
a = 4:6
b = 1:100
for i in a
for j in filter((b) -> !(b in a),b)
print(i, " ", j)
end
end
Is there a way to get the double for loop I want where the outer is a sub-range of the inner, but the inner does not include the outer sub-range and most importantly is fast and does not create alot of memory usage like filter does?
If memory usage is really a concern, consider two for loops using the range components:
systemrange = 1:50
moleculerange = 4:12
for i in systemrange[1]:moleculerange[1]-1
println(i)
end
for i in moleculerange[end]+1:systemrange[end]
println(i)
end
You might be able to do each loop in its own thread.
What about creating a custom iterator?
Note that example below needs some adjustments depending on how you define the exception lists (for example for long list with non continues indices you should use binary search).
struct RangeExcept
start::Int
stop::Int
except::UnitRange{Int}
end
function Base.iterate(it::RangeExcept, (el, stop, except)=(it.except.start > 1 ? it.start : it.except.stop+1, it.stop, it.except))
new_el = el+1
if new_el in except
new_el = except.stop+1
end
el > stop && return nothing
return (el, (new_el, stop,except))
end
Now let us test the code:
julia> for i in RangeExcept(1,10,3:7)
println(i)
end
1
2
8
9
10
I've been using MATLAB to read through a bunch of output files and have noticed that it was reading the files fairly slowly in comparison to a reader that I wrote in Python for the same files (on the order of 120s for MATLAB, 4s for Python on the same set). The files have a combination of letters and numbers, where the numbers I actually want each have a unique string on the same line, but there is no real pattern to the rest of the file. Is there a faster way to read in non-uniformly formatted text files in MATLAB?
I tried using the code profiler in MATLAB to see what takes the most time, and it seemed to be the strfind and strsplit functions. Deeper down, the strfun\private\strescape seems to be the culprit which takes up around 50% of the time, which is called by strsplit function.
I am currently using a combination of strfind and strsplit in order to search through a file for 5 specific strings, then convert the string after it into a double.
lots of text before this
#### unique identifying text here
lots of text before this
sometext X = #####
Y = #####
Z = #####
more text = ######
I am iterating through the file with approximately the following code, repeated for each number that is being found.
fid=fopen(filename)
tline=fgets(fid)
while ischar(tline)
if ~isempty(strfind(tline('X =')))
tempstring=strsplit(tline(13:length(tline)),' ');
result=str2double(char(tempstring(2)));
end
tline=fgets(fid);
end
I'm guessing this will be a bit faster, but maybe not by much.
s = fileread('texto');
[X,s] = strtok(strsplit(s, "X = "){2}); X = str2num(X);
[Y,s] = strtok(strsplit(s, "Y = "){2}); Y = str2num(Y);
[Z,s] = strtok(strsplit(s, "Z = "){2}); Z = str2num(Z);
Obviously this is highly specific to your text example. You haven't given me any more info on how the variables might change etc so presumably you'll have to implement try/catch blocks if files are not consistent etc.
PS. This is octave syntax which allows chaining operations. For matlab, split them into separate operations as appropriate.
EDIT: ach, nevermind, here's the matlab compatible one too. :)
s = fileread('texto');
C = strsplit(s, 'X = '); [X,s] = strtok(C{2}); X = str2num(X);
C = strsplit(s, 'Y = '); [Y,s] = strtok(C{2}); Y = str2num(Y);
C = strsplit(s, 'Z = '); [Z,s] = strtok(C{2}); Z = str2num(Z);
I am working with large matrices of data (Nrow x Ncol) that are too large to be stored in memory. Instead, it is standard in my field of work to save the data into a binary file. Due to the nature of the work, I only need to access 1 column of the matrix at a time. I also need to be able to modify a column and then save the updated column back into the binary file. So far I have managed to figure out how to save a matrix as a binary file and how to read 1 'column' of the matrix from the binary file into memory. However, after I edit the contents of a column I cannot figure out how to save that column back into the binary file.
As an example, suppose the data file is a 32-bit identity matrix that has been saved to disk.
Nrow = 500
Ncol = 325
data = eye(Float32,Nrow,Ncol)
stream_data = open("data","w")
write(stream_data,data[:])
close(stream_data)
Reading the entire file from disk and then reshaping back into the matrix is straightforward:
stream_data = open("data","r")
data_matrix = read(stream_data,Float32,Nrow*Ncol)
data_matrix = reshape(data_matrix,Nrow,Ncol)
close(stream_data)
As I said before, the data-matrices I am working with are too large to read into memory and as a result the code written above would normally not be possible to execute. Instead, I need to work with 1 column at a time. The following is a solution to read 1 column (e.g. the 7th column) of the matrix into memory:
icol = 7
stream_data = open("data","r")
position_data = 4*Nrow*(icol-1)
seek(stream_data,position_data)
data_col = read(stream_data,Float32,Nrow)
close(stream_data)
Note that the coefficient '4' in the 'position_data' variable is because I am working with Float32. Also, I don't fully understand what the seek command is doing here, but it seems to be giving me the correct output based on the following tests:
data == data_matrix # true
data[:,7] == data_col # true
For the sake of this problem, lets say I have determined that the column I loaded (i.e. the 7th column) needs to be replaced with zeros:
data_col = zeros(Float32,size(data_col))
The problem now, is to figure out how to save this column back into the binary file without affecting any of the other data. Naturally I intend to use 'write' to perform this task. However, I am not entirely sure how to proceed. I know I need to start by opening up a stream to the data; however I am not sure what 'mode' I need to use: "w", "w+", "a", or "a+"? Here is a failed attempt using "w":
icol = 7
stream_data = open("data","w")
position_data = 4*Nrow*(icol-1)
seek(stream_data,position_data)
write(stream_data,data_col)
close(stream_data)
The original binary file (before my failed attempt to edit the binary file) occupied 650000 bytes on disk. This is consistent with the fact that the matrix is size 500x325 and Float32 numbers occupy 4 bytes (i.e. 4*500*325 = 650000). However, after my attempt to edit the binary file I have observed that the binary file now occupies only 14000 bytes of space. Some quick mental math shows that 14000 bytes corresponds to 7 columns of data (4*500*7 = 14000). A quick check confirms that the binary file has replaced all of the original data with a new matrix with size 500x7, and whose elements are all zeros.
stream_data = open("data","r")
data_new_matrix = read(stream_data,Float32,Nrow*7)
data_new_matrix = reshape(data_new_matrix,Nrow,7)
sum(abs(data_new_matrix)) # 0.0f0
What do I need to do/change in order to only modify only the 7th 'column' in the binary file?
Instead of
icol = 7
stream_data = open("data","w")
position_data = 4*Nrow*(icol-1)
seek(stream_data,position_data)
write(stream_data,data_col)
close(stream_data)
in the OP, write
icol = 7
stream_data = open("data","r+")
position_data = 4*Nrow*(icol-1)
seek(stream_data,position_data)
write(stream_data,data_col)
close(stream_data)
i.e. replace "w" with "r+" and everything works.
The reference to open is http://docs.julialang.org/en/release-0.4/stdlib/io-network/#Base.open and it explains the various modes. Preferably open shouldn't be used with the original somewhat confusing but definitely slower string parameter.
You can use SharedArrays for the need you describe:
data=SharedArray("/some/absolute/path/to/a/file", Float32,(Nrow,Ncols))
# do something with data
data[:,1]=a[:,1].+1
exit()
# restart julia
data=SharedArray("/some/absolute/path/to/a/file", Float32,(Nrow,Ncols))
#show data[1,1]
# prints 1
Now, be mindful that you're supposed to handle synchronisation to read/write from/to this file (if you have async workers) and that you're not supposed to change the size of the array (unless you know what you're doing).
I'm attempting to fill a table of 26 values randomly. That is, I have a table called rndmalpha, and I want to randomly insert the values throughout the table. This is the code I have:
rndmalpha = {}
for i= 1, 26 do
rndmalpha[i] = 0
end
valueadded = 0
while valueadded = 0 do
a = math.random(1,26)
if rndmalpha[a] == 0 then
rndmalpha[a] = "a"
valueadded = 1
end
end
while valueadded = 0 do
a = math.random(1,26)
if rndmalpha[a] == 0 then
rndmalpha[a] = "b"
valueadded = 1
end
end
...
The code repeats itself until "z", so this is just a general idea. The problem I'm running into, however, is as the table gets filled, the random hits less. This has potential to freeze up the program, especially in the final letters because there are only 2-3 numbers that have 0 as a value. So, what happens if the while loop goes through a million calls before it finally hits that last number? Is there an efficient way to say, "Hey, disregard positions 6, 13, 17, 24, and 25, and focus on filling the others."? For that matter, is there a much more efficient way to do what I'm doing overall?
The algorithm you are using seems pretty non-efficient, it seems to me that all you need is to initialize a table with all alphabet:
math.randomseed(os.time())
local t = {"a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z"}
and Then shuffle the elements:
for i = 1, #t*2 do
local a = math.random(#t)
local b = math.random(#t)
t[a],t[b] = t[b],t[a]
end
Swapping the elements for #t*2 times gives randomness pretty well. If you need more randomness, increase the number of shuffling, and use a better random number generator. The random() function provided by the C library is usually not that good.
Instead of randoming for each letter, go through the table once and get something random per position. The method you're using could take forever because you might never hit it.
Never repeat yourself. Never repeat yourself! If you're copy and pasting too often, it's a sure sign something has gone wrong. Use a second table to contain all the possible letters you can choose, and then randomly pick from that.
letters = {"a","b","c","d","e"}
numberOfLetters = 5
rndmalpha = {}
for i in 1,26 do
rndmalpha[i] = letters[math.random(1,numberOfLetters)]
end
This is similar to a question I asked before, but is slightly different:
So I have a very large structure array in matlab. Suppose, for argument's sake, to simplify the situation, suppose I have something like:
structure(1).name, structure(2).name, structure(3).name structure(1).returns, structure(2).returns, structure(3).returns (in my real program I have 647 structures)
Suppose further that structure(i).returns is a vector (very large vector, approximately 2,000,000 entries) and that a condition comes along where I want to delete the jth entry from structure(i).returns for all i. How do you do this? or rather, how do you do this reasonably fast? I have tried some things, but they are all insanely slow (I will show them in a second) so I was wondering if the community knew of faster ways to do this.
I have parsed my data two different ways; the first way had everything saved as cell arrays, but because things hadn't been working well for me I parsed the data again and placed everything as vectors.
What I'm actually doing is trying to delete NaN data, as well as all data in the same corresponding row of my data file, and then doing the very same thing after applying the Hampel filter. The relevant part of my code in this attempt is:
for i=numStock+1:-1:1
for j=length(stock(i).return):-1:1
if(isnan(stock(i).return(j)))
for k=numStock+1:-1:1
stock(k).return(j) = [];
end
end
end
stock(i).return = sort(stock(i).return);
stock(i).returnLength = length(stock(i).return);
stock(i).medianReturn = median(stock(i).return);
stock(i).madReturn = mad(stock(i).return,1);
end;
for i=numStock:-1:1
for j = length(stock(i+1).volume):-1:1
if(isnan(stock(i+1).volume(j)))
for k=numStock:-1:1
stock(k+1).volume(j) = [];
end
end
end
stock(i+1).volume = sort(stock(i+1).volume);
stock(i+1).volumeLength = length(stock(i+1).volume);
stock(i+1).medianVolume = median(stock(i+1).volume);
stock(i+1).madVolume = mad(stock(i+1).volume,1);
end;
for i=numStock+1:-1:1
for j=stock(i).returnLength:-1:1
if (abs(stock(i).return(j) - stock(i).medianReturn) > 3*stock(i).madReturn)
for k=numStock+1:-1:1
stock(k).return(j) = [];
end
end;
end;
end;
for i=numStock:-1:1
for j=stock(i+1).volumeLength:-1:1
if (abs(stock(i+1).volume(j) - stock(i+1).medianVolume) > 3*stock(i+1).madVolume)
for k=numStock:-1:1
stock(k+1).volume(j) = [];
end
end;
end;
end;
However, this returns an error:
"Matrix index is out of range for deletion.
Error in Failure (line 110)
stock(k).return(j) = [];"
So instead I tried by parsing everything in as vectors. Then I decided to try and delete the appropriate entries in the vectors prior to building the structure array. This isn't returning an error, but it is very slow:
%% Delete bad data, Hampel Filter
% Delete bad entries
id=strcmp(returns,'');
returns(id)=[];
volume(id)=[];
date(id)=[];
ticker(id)=[];
name(id)=[];
permno(id)=[];
sp500(id) = [];
id=strcmp(returns,'C');
returns(id)=[];
volume(id)=[];
date(id)=[];
ticker(id)=[];
name(id)=[];
permno(id)=[];
sp500(id) = [];
% Convert returns from string to double
returns=cellfun(#str2double,returns);
sp500=cellfun(#str2double,sp500);
% Delete all data for which a return is not a number
nanid=isnan(returns);
returns(nanid)=[];
volume(nanid)=[];
date(nanid)=[];
ticker(nanid)=[];
name(nanid)=[];
permno(nanid)=[];
% Delete all data for which a volume is not a number
nanid=isnan(volume);
returns(nanid)=[];
volume(nanid)=[];
date(nanid)=[];
ticker(nanid)=[];
name(nanid)=[];
permno(nanid)=[];
% Apply the Hampel filter, and delete all data corresponding to
% observations deleted by the filter.
medianReturn = median(returns);
madReturn = mad(returns,1);
for i=length(returns):-1:1
if (abs(returns(i) - medianReturn) > 3*madReturn)
returns(i) = [];
volume(i)=[];
date(i)=[];
ticker(i)=[];
name(i)=[];
permno(i)=[];
end;
end
medianVolume = median(volume);
madVolume = mad(volume,1);
for i=length(volume):-1:1
if (abs(volume(i) - medianVolume) > 3*madVolume)
returns(i) = [];
volume(i)=[];
date(i)=[];
ticker(i)=[];
name(i)=[];
permno(i)=[];
end;
end
As I said, this is very slow, probably because I'm using a for loop on a very large data set; however, I'm not sure how else one would do this. Sorry for the gigantic post, but does anyone have a suggestion as to how I might go about doing what I'm asking in a reasonable way?
EDIT: I should add that getting the vector method to work is probably preferable, since my aim is to put all of the return vectors into a matrix and get all of the volume vectors into a matrix and perform PCA on them, and I'm not sure how I would do that using cell arrays (or even if princomp would work on cell arrays).
EDIT2: I have altered the code to match your suggestion (although I did decide to give up speed and keep with the for-loops to keep with the structure array, since reparsing this data will be way worse time-wise). The new code snipet is:
stock_return = zeros(numStock+1,length(stock(1).return));
for i=1:numStock+1
for j=1:length(stock(i).return)
stock_return(i,j) = stock(i).return(j);
end
end
stock_return = stock_return(~any(isnan(stock_return)), : );
This returns an Index exceeds matrix dimensions error, and I'm not sure why. Any suggestions?
I could not find a convenient way to handle structures, therefore I would restructure the code so that instead of structures it uses just arrays.
For example instead of stock(i).return(j) I would do stock_returns(i,j).
I show you on a part of your code how to get rid of for-loops.
Say we deal with this code:
for j=length(stock(i).return):-1:1
if(isnan(stock(i).return(j)))
for k=numStock+1:-1:1
stock(k).return(j) = [];
end
end
end
Now, the deletion of columns with any NaN data goes like this:
stock_return = stock_return(:, ~any(isnan(stock_return)) );
As for the absolute difference from medianVolume, you can write a similar code:
% stock_return_length is a scalar
% stock_median_return is a column vector (eg. [1;2;3])
% stock_mad_return is also a column vector.
median_return = repmat(stock_median_return, stock_return_length, 1);
is_bad = abs(stock_return - median_return) > 3.* stock_mad_return;
stock_return = stock_return(:, ~any(is_bad));
Using a scalar for stock_return_length means of course that the return lengths are the same, but you implicitly assume it in your original code anyway.
The important point in my answer is using any. Logical indexing is not sufficient in itself, since in your original code you delete all the values if any of them is bad.
Reference to any: http://www.mathworks.co.uk/help/matlab/ref/any.html.
If you want to preserve the original structure, so you stick to stock(i).return, you can speed-up your code using essentially the same scheme but you can only get rid of one less for-loop, meaning that your program will be substantially slower.