I'm trying to implement a 8 puzzle program using the manhattan heuristic using OOP in matlab, with each node represented by an object "state". However, i'm having lost of speed problems implementing an object array for queue. After lots of trial and error, this is the fastest i could go (i hate using global objects but it seems that any attempts at passing the array or using an object for it slows it down tremendously):
.classdef state < handle
. properties
. position; %3x3 array
. moves;
. blank;
. parent;
. prevmove;
. mdist;
. end
....
queue is preallocated
queue = repmat(state,1,5000);
.function newfront = q(item,front)
.global queue;
.if isempty(front)
. newfront = 1;
. queue(1) = item;
.else
. for i = front:-1:0
. if i == 0
. break;
. end
. a = item.mdist;
. b = queue(i).mdist;
. if (a < b)
. break;
. else
. queue(i+1) = queue(i);
. end
. end
. newfront = front + 1;
. queue(i+1) = item;
.end
.end
Using MATLAB profiler:
eightpuzzle 1 67.941 s 2.550 s
eightpuzzle>q 4722 59.657 s 59.657 s
....
118 queue(i+1) = queue(i); 2583916 29.064 s 48.7%
120 end 2583916 16.202 s 27.2%
114 b = queue(i).mdist; 2587318 4.357 s 7.3%
113 a = item.mdist; 2587318 2.783 s 4.7%
115 if (a < b) 2587318 2.721 s 4.6%
is there anyway to make it run faster? interesting thing to note was that by using
a = item.mdist;
b = queue(i).mdist;
if (a < b)
...
instead of
if (item.mdist < queue(i).mdist)
overall runtime was already halved.
Thanks!
I don't know if it might related to this topic about general OOP slowness in MATLAB:
Is MATLAB OOP slow or am I doing something wrong?
And here's yet another link about slow behaviour of handle classes, but it's likely fixed so far since the bug was reported years ago:
http://www.mathworks.com/matlabcentral/newsreader/view_thread/288746
There is a large amount of memory overhead when using object arrays in MATLAB, just as in structure arrays. Each field of each array element is its own mxArray and that adds significant memory overhead. Consider using a 1x1 structure with non-scalar fields, rather than a 1xN structure with scalar fields - it will be substantially smaller in memory, and the access to each field will be much faster.
An example:
A 1x100 structure array with two scalar fields "a" and "b" requires (100*2) + 1 mxArray objects in memory.
A 1x1 structure array with two non-scalar fields "a" and "b" each of size 1x100 requires 3 mxArray objects in memory.
Your array of objects is very similar to a structure array, where the fields of each array element can be thought of as pointers. Let's say you have an object array "obj" that is 1xN with fields "a" and "b". Referencing the Kth object field's "a":
obj(K).a
This is an index operation and also an mxArray pointer dereference to the field "a".
I concede that using a scalar object would make your code less intuitive, but that's a trade-off you'll have to weigh.
Related
I have a structure called s in Matlab. This is a structure with two fields a and b. The structure size is 1 x 1,620,000.
It is a very large structure (that probably takes half of the ram of my machine). This is what the structure looks like:
I am looking for an efficient way to concatenate each of the fields a and b into two separate arrays that I can then export to csv. I built the code below, to do so, but even after 12 hours running it has not even reached a quarter of the loop. Any more efficient way of doing this?
a = [];
b =[];
total_n = size(s,2);
count = 1;
while size(s,2)>0
if size(s(1).a,1)
a = [a; s(1).a];
end
if size(s(1).b,1)
b = [b; s(1).b];
end
s(1) = []; %to save memory
if mod(count,1000) == 0
fprintf('Done %2f \n', [count/total_n])
end
count = count+1;
end
s(1) = []; %to save memory
ah, but such huge misunderstanding that comment is.
if size(s) is 1 x 1,620,000, you just suddenly forced the loop to do (under the hood, you dont see it)
snew=zeros(1,size(s,2)-1) # now you use double memory
snew=s(2:end) # now you force an unnecesary copy
So not only does that line make your code require double the memory, but also in each loop, you make an unnecesary copy of a large array.
Just replace your while for a normal for loop of for ii=1:size(s,2) and then index s!
Now, you can see hopefully then why the following is equally a big mistake (not only that, but any modern MATLAB version is currently telling you this is a bad idea in your editor)
a=[]
a=[a;s(1).a]
In here in each loop you are forcing MATLAB to make a new a that is 1 bigger than before, and copy the contents of the old a there.
instead, preallocate the size of a.
As you don't know what you are going to put there, I suggest using a cell array, as each s(ii).a has a different length.
You can then, after the loop, remove all empty (isempty) cells if you want.
Managed to do it efficiently:
s= struct2cell(s);
s= squeeze(s);
a = a(1,:);
a = a';
a = vertcat(a{:});
b = a(2,:);
b = b';
b = vertcat(b{:});
A simplified example:
nsize = 100
vsize = 10000
varray = [rand(vsize) for i in 1:nsize] #say, I have a set of vectors.
for k in 1:nsize
varray[k] = rand(vsize, vsize) * varray[k]
end
Obviously, the above for loop can be parallelized.
According to Parallel Map and Loops in Julia manual,
I need to used SharedArray. However, ShardArray cannot have Array{Float64,1} as element type.
julia> a = SharedArray(Array{Float64,1}, nsize)
ERROR: ArgumentError: type of SharedArray elements must be bits types, got Array{Float64,1}
in __SharedArray#138__ at sharedarray.jl:45
in SharedArray at sharedarray.jl:116
How can I solve this problem?
Currently, you can't, because a SharedArray requires a contiguous block of memory, which means that its elements must be "bits types," and this isn't true for Array. (Array is implemented in C and has some header information, which makes them not densely-packable.)
However, if all of your "element" arrays have the same size, and you don't absolutely require the ability to modify single elements of the "inner" arrays, you could try using StaticArrays as elements. (Thanks to #Wouter in the comments below for pointing out that this needed updating.)
As a personal challenge I'm trying to implement the SIMON block cipher in Ruby. I'm running into some issues finding the best way to work with the data. The full code related to this question is located at: https://github.com/Rami114/Personal/blob/master/Simon/Simon.rb
SIMON requires both XOR, shift and circular shift operations, the last of which is forcing me to work with BigNums so I can perform the left circular shift with math rather than a more complex/slower double loop on byte arrays.
Is there a better way to convert a string to a BigNum and back again.
String -> BigNum (where N is 64 and pt is a string of plaintext)
pt = pt.chars.each_slice(N/8).map {|x| x.join.unpack('b*')[0].to_i(2)}.to_a
So I break the string into individual characters, slice into N-sized arrays (the word size in SIMON) and unpack each set into a BigNum. That appears to work fine and I can convert it back.
Now my SIMON code is currently broken, but that's more the math I think/hope and not the code. The conversion back is (where ct is an array of bignums representing the ciphertext):
ct.map { |x| [x.to_s(2).rjust(128,'0')].pack('b*') }.join
I seem to have to right-justify pad the string as bignums are of undefined width so I have no leading 0s. Unfortunately the pack requires the defined with to have sensible output.
Is this a valid method of conversion? Is there a better way? I'm not sure on either count and hoping someone here can help out.
E: For #torimus, the circular shift implementation I'm using (From link above)
def self.lcs (bytes, block_size, shift)
((bytes << shift) | (bytes >> (block_size - shift))) & ((1<< block_size)-1)
end
If you would be equally happy with unpack('B*') with msb first binary numbers (which you could well be if all your processing is circular), then you could also use .unpack('Q>') instead of .unpack('B*')[0].to_i(2) for generating pt:
pt = "qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM1234567890!#"
# Your version (with 'B' == msb first) for comparison:
pt_nums = pt.chars.each_slice(N/8).map {|x| x.join.unpack('B*')[0].to_i(2)}.to_a
=> [8176115190769218921, 8030025283835160424, 7668342063789995618, 7957105551900562521,
6145530372635706438, 5136437062280042563, 6215616529169527604, 3834312847369707840]
# unpack to 64-bit unsigned integers directly
pt_nums = pt.unpack('Q>8')
=> [8176115190769218921, 8030025283835160424, 7668342063789995618, 7957105551900562521,
6145530372635706438, 5136437062280042563, 6215616529169527604, 3834312847369707840]
There are no native 128-bit pack/unpacks to return in the other direction, but you can use Fixnum to solve this too:
split128 = 1 << 64
ct = pt # Just to show round-trip
ct.map { |x| [ x / split128, x % split128 ].pack('Q>2') }.join
=> "\x00\x00\x00\x00\x00\x00\x00\x00qwertyui . . . " # truncated
This avoids a lot of the temporary stages on your code, but at the expense of using a different byte coding - I don't know enough about SIMON to comment whether this is adaptable to your needs.
I am quite new to image processing and would like to produce an array that stores 10 images. After which I would like to run a for loop through some code that identifies some properties of the images, specifically the surface area of a biological specimen, which then spits out an array containing 10 areas.
Below is what I have managed to scrap up so far, and this is the ensuing error message:
??? Index exceeds matrix dimensions.
Error in ==> Testing1 at 14
nova(i).img = imread([myDir B(i).name]);
Below is the code I've been working on so far:
my_Dir = 'AC04/';
ext_img='*.jpg';
B = dir([my_Dir ext_img]);
nfile = max(size(B));
nova = zeros(1,nfile);
for i = 1:nfile
nova(i).img = imread([myDir B(i).name]);
end
areaarray = zeros(1,nfile);
for k = 1:nfile
[nova(k), threshold] = edge(nova(k), 'sobel');
.
.
.
.%code in this area is irrelevant to the problem I think%
.
.
.
areaarray(k) = bwarea(BWfinal);
end
areaarray
There are few ways you could store an image in a kind of an array structure in Matlab. You could use array of structs. In that case you could do as you did:
nova(i).img = imread([myDir B(i).name]);
You access first image with nova(1).img, second one with nova(2).img etc.
Other way to do it is to use cell array (similar to arrays but are more flexible in the sense that members could be of the different type):
nova{i} = imread([myDir B(i).name]);
You access first image with nova{1}, second one with nova{2} etc.
[ IMPORTANT ] In both cases you should remove this line from code:
nova = zeros(1,nfile);
I suppose you've tried to pre-allocate memory for images, and since you're beginner I advise you not to be concerned with it. It is an optimization concern to be addressed if you come across some performance issues - and if you don't come across them, take advantage of Matlab's automatic memory (re)allocation.
Given a string array of variable length, print the lengths of each element in the array.
For example, given:
string[] ex = {"abc", "adf", "df", "ergd", "adfdfd");
The output should be:
2 3 4 6
One possibility I'm considering is to use a linked list to save each string length, and sort while inserting and finally display the results.
Any other suggestions for efficient solutions to this problem?
Whenever you want to maintain a collection of distinct things (ie: filter out duplicates), you probably want a set.
There are many different data structures for storing sets. Some of these, like search trees, will also "sort" the values for you. You could try using one of the many forms of binary search trees.
What you are doing now (or the given answer) is called the insertion sort. It basically compare the length of the string-to-insert from the inserted strings. After then, when printing, teh length of string-to-print (at current pointer) will be compared to the length of the string before it and after it, if has the same length, do not print!
Another approach is, the bubble sort, it will sort two strings at a time, sort them, then move to next string...
The printing is the most important part in your program, regardless of what sorting algorithm you use, it doesn't matter.
Here's an algorithm for bubble sort and printing process, it's VB so just convert it...
Dim YourString(4) As String
YourString(0) = "12345" 'Will not be printed
YourString(1) = "12345" 'Will not be printed
YourString(2) = "123" 'Will be printed
YourString(3) = "1234" 'Will be printed
Dim RoundLimit As Integer = YourString.Length - 2
'Outer loop for how many times we will sort the whole array...
For CycleCounter = 0 To RoundLimit
Dim CompareCounter As Integer
'Inner loop to compare strings...
For CompareCounter = 0 To RoundLimit - CycleCounter - 1
'Compare lengths... If the first is greater, sort! Note: this is ascending
If YourString(CompareCounter).Length > YourString(CompareCounter + 1).Length Then
'Sorting process...
Dim TempString = YourString(CompareCounter)
YourString(CompareCounter) = YourString(CompareCounter + 1)
YourString(CompareCounter + 1) = TempString
End If
Next
Next
'Cycles = Array length - 2 , so we have 2 cycles here
'First Cycle!!!
'"12345","12345","123","1234" Compare 1: index 0 and 1 no changes
'"12345","123","12345","1234" Compare 2: index 1 and 2 changed
'"12345","123","1234","12345" Compare 3: index 2 and 3 changed
'Second Cycle!!!
'"123","12345","1234","12345" Compare 1: index 0 and 1 changed
'"123","1234","12345","12345" Compare 2: index 1 and 2 changed
'"123","1234","12345","12345" Compare 3: index 2 and 3 no changes
'No more cycle!
'Now print it! Or use messagebox...
Dim CompareLimit As Integer = YourString.Length - 2
For CycleCounter = 0 To CompareLimit
'If length is equal to next string or the preceeding string, do not print...
If ((CycleCounter - 1) <> -1) Then 'Check if index exist
If YourString(CycleCounter).Length = YourString(CycleCounter - 1).Length Then
Continue For 'The length is not unique, exit compare, go to next iteration...
End If
End If
If ((CycleCounter + 1) <> YourString.Length - 1) Then 'Check if index exist
If YourString(CycleCounter).Length = YourString(CycleCounter + 1).Length Then
Continue For 'The length is not unique, exit compare, go to next iteration...
End If
End If
'All test passed, the length is unique, show a dialog!
MsgBox(YourString(CycleCounter))
Next
The question as stated doesn't say anything about sorting or removing duplicates from the results. It is only the given output that implies the sorting and duplicate removal. It doesn't say anything about optimisation for speed or space or writing for maintainability.
So there really isn't enough information for a "best" solution.
If you want a solution that will work in most languages you probably should stick with an array. Put the lengths in a new array, sort it, then print in a loop that remembers that last value to skip duplicates. I wouldn't want to use a language that couldn't cope with that.
If a language is specified you might be able to take advantage of set or associate array type data structures to handle the duplicates and/or sorting automatically. E.g., in Java you could pick a collection class that automatically ignores duplicates and sorts, and you could structure your code such that a one line change to use a different class would let you keep duplicates, or not sort. If you are using C# you could probably write the whole thing as a one-line LINQ statement...
Here is a C++ solution:
#include <set>
#include <vector>
#include <string>
#include <iostream>
using namespace std;
int main()
{
string strarr[] = {"abc", "adf", "df", "ergd", "adfsgf"};
vector< string > vstr(strarr, strarr + 5);
set< size_t > s;
for (size_t i = 0; i < vstr.size(); i++)
{
s.insert( vstr[i].size() );
}
for (set<size_t>::iterator ii = s.begin(); ii != s.end(); ii++)
cout << *ii << " ";
cout << endl;
return 0;
}
Output:
$ g++ -o set-str set-str.cpp
$ ./set-str
2 3 4 6
A set is used because (quoting from here):
Sets are a kind of associative container that stores unique elements,
and in which the elements themselves are the keys.
Associative containers are containers especially designed to be
efficient accessing its elements by their key (unlike sequence
containers, which are more efficient accessing elements by their
relative or absolute position).
Internally, the elements in a set are always sorted from lower to
higher following a specific strict weak ordering criterion set on
container construction.
Sets are typically implemented as binary search trees.
And for details on vector see here and here for string.
Depending on the language, the easiest way might be to iterate through the array using a for loop
for (i=0;i<array.length;i++){
print array[i].length;
}
do you need to print them in order?