I'm writing the code for found the solution using chi-square fitting.
It includes many parameters and massive equations, thus, the time cost is so high.
However, some equations and parameters don't need to be calculated(or read) for every iteration. Just one calculation at the first iteration is enough.
In order to reduce the time cost, I want to fix the values, on the other hand, not swap the value.
Is there any method and/or trick for this purpose?
I don't think there is any other way other than to have two loops. This may be tricky, but you can first calculate your equations outside the main (second) loop. However since it's only on the first iteration, technically you only have 1 loop as a loop with 1 iteration is basically useless. Afterwards, you can use them in the loop.
#Calculate here
#loop here
...
...
I think you are asking about avoiding re-computation by buffering known results. This technique is called memoization.
This can be achieved by using Memoization:
using Memoization
#memoize function myfunction(x)
sleep(0.5)
x .^ 2
end
Let us have a look how this works:
julia> #time myfunction(1:3)
0.513210 seconds (6 allocations: 224 bytes)
3-element Vector{Int64}:
1
4
9
julia> #time myfunction(1:3)
0.000004 seconds
3-element Vector{Int64}:
1
4
9
You can see that subsequent calls to myfunction use the memoized buffer.
Related
I can't seem to rationalize the Big-O notation for a while loop that has an indeterminate number of iterations.
In my code for a personal project, I have a list containing all 0s. I then implement a while loop that will generate a random integer between 0 and 9. If the value in the list at the index of the random number is a 0, then the value is written to a 1 and the while loop exits. Otherwise, a random number is generated again and the process repeats.
I'm not entirely sure what the time complexity of this would be, however. For example, if after 9 iterations of the algorithm, every single value in the list except index 9 is 1, and if the random number generator just happens to not generate the number 9 for, say, 99 iterations, then it would exit after 99 + 9 iterations. Wouldn't the worst-case be O(infinity)? I don't think this is possible, but I figured I'd ask since I wasn't sure.
My textbooks and online resources don't seem to provide much insight on examples such as this. I'm sure that the best-case would be O(1), but the average and worst cases I'm a bit unsure about.
I found a similar problem that has the same premise. Here's the pseudocode, where n is some integer of arbitrary size:
sample_found = false
while(!sample_found) {
if (rand(0,n) == 0) {
sample_found = true
}
}
In the worst case, this would run infinitely, right? I'm not sure about average case, either.
It sounds like you're using IID Bernoulli trials to control looping, with a probability p=0.1 of continuing. Assuming that's the case, you can just use the Geometric distribution.
The mean of this distribution is just 1/p, so 10, and I'd use quantiles to further understand how many draws would be needed to finish. For example:
10% of the time you'd expect to finish immediately
50% of the runs you'd need to loop 6 times or less
90% of runs finish by 21 loops
99% of runs finish by 43 loops
calculated in R, using qgeom(c(0.1, 0.5, 0.9, 0.99), 0.1).
The worst case obviously goes out to infinity, but in practice you'd be unlikely to loop 200 times. 1-pgeom(200, 0.1) gives 6e-10, so you can expect to iterate your loop more than a billion times before needing to wait this many iterations.
Good morning,
I wrote the following code which works with small-is numbers, to find the biggest prime factor of a number. I can't use Prime and I need to come-up with a manual solution.
def is_prime?(number)
list = (2..(Math.sqrt(number))).to_a
list.each do |i|
return false if number % i == 0
end
true
end
def biggest_prime(number)
list = (2..((number))).to_a
divisors = list.select{|i| is_prime?(i)}
divisors.select{|i| number % i == 0 }.max
end
The prime factors of 13195 are 5, 7, 13 and 29.
biggest_prime(13195) => 29
However when I try with the edge case of biggest_prime(600851475143) the system freezes.
Anybody could tell me how to refactor my code to make it more efficient?
Many thanks!
There are many problems with your code that make it massively inefficient:
In your biggest_prime method, you construct an Array in memory of every number less than your target number, instead of just iterating through the range. For 600851475143, the size of that Array in memory is ~4 TiByte. Do you have 4 TiByte of RAM? If not, your system will create a huge swapfile and be swapping constantly. Use a Range instead! In fact, you are already using a Range, but then convert it to an Array for absolutely no reason at all.
The same happens in your is_prime? method. This Array is a lot smaller (only about 6 MiByte at its largest), but you create it over and over and over again, 600 billion times! Creating an Array of n numbers takes O(n) amount of time, so creating an Array of sqrt(n) numbers n times takes O(n * sqrt(n)) time. In total, you need SUM(sqrt(n), n = 1 TO 600851475143) steps, which is about 310498000000000000 steps.
That is also the number of steps you need for your entire algorithm. Even if you had a 10 GHz CPU, and this CPU had 100 cores, and your problem were perfectly parallelizable, and you could perform one entire iteration of your algorithm within a single CPU instruction, you would still need 3.5 days to get the result. Since you are not using parallelism, probably don't have a 10 GHz CPU, and an iteration is going to take 10s or hundreds of CPU cycles, a more realistic number would be at least 100 years.
So, what you need to do, is to massively cut down the number of iterations, and the amount of memory.
The latter is easiest: use Ranges instead of Arrays.
The former requires a bit of thinking. Here is one idea: You check the same numbers over and over again. That's not necessary. Once you have determined that a number is a divisor, you already know it is a divisor, you don' need to check it again.
Likewise, you check the same number over and over again for primality. But once you have determined that a number is prime, you don't need to check that again, it's not very likely to change, after all.
But first, let's look at what the most efficient solution would look like:
require 'prime'
def biggest_prime(number)
number.prime_division.last.first
end
I would personally just use the built-in functions for this instead of trying to reinvent the wheel.
require 'prime'
number = 13195
Prime.each(number).select { |n| number % n == 0 }
This will result in the expected output of [5, 7, 13, 29].
The last value will always be the largest, so...
Prime.each(number).select { |n| number % n == 0 }.last
Gives the 29 you are looking for. This could obviously be cleaned up a little bit, but gives you the idea.
This "freezes" up with the exact same number, which is likely do to going into a Bignum and out of the 32-bit integer range (actually 31 bits for Ruby). I would have to dig into the C code to see what the freeze is all about, though, or test with some other 64-bit and higher numbers to see if the results are repeatable.
When we look at the problem like it should be solved in the paper, we can use prime_division. It gives you a set of prime pairs your number divided by. It works fast with big numbers.
p (Prime.prime_division(number))[-1].max
You can do it by changing the max number dividing it until you have no more prime numbers to divide against. That way you don't care about the intermediate prime numbers and reduce the iterations greatly :)
def top_prime(n)
max = n
lower = 2
while lower < max
while max % lower == 0 && max != lower
max = max / lower
end
lower = lower+1
end
max
end
puts top_prime(600851475143)
Usually if in every iteration you need to generate a random number, it's better to generate all the random numbers once at the beginning and then in each iteration, you retrieve the number you generated from the table.
However, I encounter a problem today which against this intuition. I have a numeric vector of length N, in each iteration, I have to reshuffle the order of this vector and do something.
I implement 2 ways: the first one is exactly as I just described, i.e., in each iteration, I call function
V = randperm(N)
In the 2nd method, before the loop, I call function
W = arrayfun(#(x)randperm(N),(1:T-1)','UniformOutput',0);
W = cell2mat(W);
Here T is the total number of iterations. Then in each iteration, I just retrieve the row from the matrix W. Then the rest of the code for 2 methods are exactly the same.
However, I find the 2nd code is much slower than the 1st one (I checked that the command of: W = arrayfun(#(x)randperm(N),(1:T-1)','UniformOutput',0); does not take much time). I run the profiler, I find that in the 2nd code call the function ismember>ismemberR2012a 3 times more than the 1st code. I couldn't figure out why.
first of all you should note that both functions randperm and arrayfun are implemented as mex files. So my first guess would be that the arrayfun is a process that is calling the randperm and storing its result in a cell and repeating this process. That makes it a bit slower. If you try to do the same in the first solution it will cost more.
I performed some tests as well.
I do not know the scale of N and T. But still I tested the functions on my machine
N=10000;
T = 1000;
tic;
for ii=1:T-1
V=randperm(N);
end
toc;
tic;
W = arrayfun(#(x)randperm(N),(1:T-1)','UniformOutput',0);
W = cell2mat(W);
toc;
And the result is:
For N=10000 and T=1000:
Elapsed time is 0.849627 seconds.
Elapsed time is 0.996933 seconds.
I think the reason is that the first solution is called T-1 times, whereas the second one is called only once. And it has to manage the big cell assignment.
Even doing this does not improve the performance of the code that much:
tic;
for ii=1:T-1
Z = arrayfun(#(x)randperm(N),ii,'UniformOutput',0);
end
toc;
Elapsed time is 0.827387 seconds.
Elapsed time is 0.966013 seconds.
Elapsed time is 0.951093 seconds.
Then running the profiler:
If you can generate random numbers on the fly, what is the point of generating and storing them and then loading them again when needed?
the numbers are collected in different runs and obviously due to the stochastic nature of the execution time they are not consistent.
Just making a note that if N is smaller than 10, you might want to use perms:
Before the loop:
% get all available permutations:
W = perms(1:N);
% get a list of T random permutations from W:
p = randi(length(W),T,1);
and then use p and W inside the loop:
V = W(p(k),:); % for iteration k
I'm currently doing some shearSorting and cannot figure out when this operation is supposed to be done with an n x n matrix.
What I'm doing currently is I'm copying the matrix at the start of each iteration of the loop to a temp matrix and then at the end of each iteration of the loop I'm comparing both the original and the temp matrices and if they are the same then I break out of the loop and exit. I do not like this approach as we always end up going through one extra iteration after the matrix in sorted and done which is a waste of CPU time and cycles.
There has to be a better way to do this checking. I keep finding references to log(n) to signify how many iteration we need but I don't believe they mean actual log(n) as log(5) for a 5x5 matrix in 0.69 which is impossible for number of iterations.
Any suggestions?
SO I know shearSort takes log(n) run iterations to complete so for a case of 5x5 matrix we will have 3 runs for rows and 3 runs for columns. But what if the 5x5 matrix I was given is kinda almost sorted and only needs one or 2 more iterations to be completed, in that case I do not see the point in iterating 6 time through it as this would be considered a waste of CPU power and cycles.
Also we have the following solution: if we copy the matrix at start of each iteration of the shearSort function to a temporary matrix and at the end of each iteration we compare the 2 matrices together and they are the same then we know that we are done (Note here an iteration would mean both a row and a column sort as a matrix might not need a row sort at first but would need a column sort after ). In this case we would be preserving CPU cycles in case the matrix doesn't need N + 1 iterations, but this solution would provide an issue which is when N + 1 iterations are needed then we would be doing N + 3 iterations to finish ( the extra 2 iterations would be one to check if 2 matrices are same for row and one for column).
To solve this we would have to use a combination of both solutions:
we would still be copying the matrix at start and comparing it to temp matrix at the end and if they are equal before we get to the N + 1 iterations then we are done and do not need to go on any further, and if they are not then we go to the N + 1 iteration and stop after since we know at this point the matrix should be sorted after N + 1 iterations.
Is there any way to generate a function F that, given a sequence, such as:
seq = [1 2 4 3 0 5 4 2 6]
Then F(seq) will return a function that generates that sequence? That is,
F(seq)(0) = 1
F(seq)(1) = 2
F(seq)(2) = 4
... and so on
Also, if it is, what is the function of lowest complexity that does so, and what is the complexity of the generated functions?
EDIT
It seems like I'm not clear, so I'll try to exemplify:
F(seq([1 3 5 7 9])}
# returns something like:
F(x) = 1 + 2*x
# limited to the domain x ∈ [1 2 3 4 5]
In other words, I want to compute a function that can be used to algebraically, using mathematical functions such as +, *, etc, restore a sequence of integers, even if you cleaned it from memory. I don't know if it is possible, but, as one could easily code an approximation for such function for trivial cases, I'm wondering how far it goes and if there is some actual research concerning that.
EDIT 2 Answering another question, I'm only interested in sequences of integers - if that is important.
Please let me know if it is still not clear!
Well, if you just want to know a function with "+ and *", that is to say, a polynomial, you can go and check Wikipedia for Lagrange Polynomial (https://en.wikipedia.org/wiki/Lagrange_polynomial).
It gives you the lowest degree polynomial that encodes your sequence.
Unfortenately, you probably won't be able to store less than before, as the probability of the polynom being of degree d=n-1 where n is the size of the array is very high with random integers.
Furthermore, you will have to store rational numbers instead of integers.
And finally, the access to any number of the array will be in O(d) (using Horner algorithm for polynomial evaluation), in comparison to O(1) with the array.
Nevertheless, if you know that your sequences may be very simple and very long, it might be an option.
If the sequence comes from a polynomial with a low degree, an easy way to find the unique polynomial that generates it is using Newton's series. Constructing the polynomial for a n numbers has O(n²) time complexity, and evaluating it has O(n).
In Newton's series the polynomial is expressed in terms of x, x(x-1), x(x-1)(x-2) etc instead of the more familiar x, x², x³. To get the coefficients, basically you compute the differences between subsequent items in the sequence, then the differences between the differences, until only one is left or you get a sequence of all zeros. The numbers you get along the bottom, divided by factorial of the degree of the term, give you the coefficients. For example with the first sequence you get these differences:
1 2 4 3 0 5 4 2 6
1 2 -1 -3 5 -1 -2 4
1 -3 -2 8 -6 -1 6
-4 1 10 -14 5 7
5 9 -24 19 2
4 -33 43 -17
-37 76 -60
113 -136
-249
The polynomial that generates this sequence is therefore:
f(x) = 1 + x(1 + (x-1)(1/2 + (x-2)(-4/6 + (x-3)(5/24 + (x-4)(4/120
+ (x-5)(-37/720 + (x-6)(113/5040 + (x-7)(-249/40320))))))))
It's the same polynomial you get using other techniques, like Lagrange interpolation; this is just the easiest way to generate it as you get the coefficients for a polynomial form that can be evaluated with Horner's method, unlike the Lagrange form for example.
There is no magic if you say that the sequence could be completely random. And yet, it is always possible, but won't save you memory. Any interpolation method requires the same amount of memory in the worst case. Because, if it didn't, it would be possible to compress everything to a single bit.
On the other hand, it is sometimes possible to use a brute force, some heuristics (like genetic algorithms), or numerical methods to reproduce some kind of mathematical expression having a specified type, but good luck with that :)
Just use some archiving tools instead in order to save memory usage.
I think it will be useful for you to read about this: http://en.wikipedia.org/wiki/Entropy_(information_theory)