Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
In general, in computational complexity, we talk about time and space complexity. That is, we think about how much time or space that is necessary for solving some problem.
I would like to know if there is another kind of resource (beyond time and space) that we could use a reference for discussing computacional complexity.
People have considered the number of references to external memory (https://www.ittc.ku.edu/~jsv/Papers/Vit.IO_book.pdf) and the use of cache memory (https://en.wikipedia.org/wiki/Cache-oblivious_algorithm). Where the computations is split between two or more nodes, the complexity of communication between those nodes is of interest (https://en.wikipedia.org/wiki/Communication_complexity) and there are some neat proofs around here.
There are also links between these measures. Most obviously, using almost any resource takes time, so anything that takes no more than T units of time is likely to take no more than O(T) units of any other resource. There is a paper "An Overview of the Theory of Computational Complexity" by Hartmanis and Hopcroft, which puts computational complexity on a firm mathematical footing. This defines a very general notion of computational complexity measures and (Theorem 4) proves that (their summary) "a function which is "easy" to compute in one measure is "easy" to compute in other measures". However this result (like most of the rest of the paper) is in mathematically abstract terms which don't necessarily have any practical consequence in the real world. The connection between the two complexities used here is loose enough that it is entirely possible that polynomial complexity in one measure could be exponential complexity (or worse) in the other measure.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
So far in my learning of algorithms, I have assumed that asymptotic boundings are directly related to patterns in control structures.
So if we have n^2 time complexity, I was thinking that this automatically means that I have to use nested loops. But I see that this is not always correct (and for other time complexities, not just quadratic).
How to approach this relationship between time complexity and control structure?
Thank you
Rice's theorem is a significant obstacle to making general statements about analyzing running time.
In practice there's a repertoire of techniques that get applied. A lot of algorithms have nested loop structure that's easy to analyze. When the bounds of one of those loops is data dependent, you might need to do an amortized analysis. Divide and conquer algorithms can often be analyzed with the Master Theorem or Akra–Bazzi.
In some cases, though, the running time analysis can be very subtle. Take union-find, for example: getting the inverse Ackermann running time bound requires pages of proof. And then for things like the Collatz conjecture we have no idea how to even get a finite bound.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
So from my understading, we can only evaluate algorithm with asymptotic analysis but when executing a algorithm, it can only return amount of time.
My question is how can we compare those two ?
They are comparable but not in the way you want.
If you have an implementation that you evaluate asymptotically to be say O(N^2) and you measure to run in 60 seconds for an input of N=1000, then if you change the input to N=2000 I would expect the run-time to be on the order of 60*(2^2) 4 minutes (I increase the input by a factor of two, the runtime increases by a factor of 2 square).
Now, if you have another algorithm also O(N^2) you can observe it to run for N=1000 in 10 seconds (the compiler creates faster instructions, or the CPU is better). Now when you move to N=2000 the runtime I would expect to be around 40 seconds (same logic). If you actually measure it you might still see some differences from the expected value because of system load or optimizations but they become less significant as N grows.
So you can't really say which algorithm will be faster based on asymptotic complexity alone. The asymptotic complexity guarantees that there will be an input sufficiently large where the lower complexity is going to be faster, but there's no promise what "sufficiently large" means.
Another example is search. You can do linear search O(N) or binary search O(logN). If your input is small (<128 ints) the compiler and processor make linear search faster than binary search. However grow N to say 1 million items and the binary search will be much faster than linear.
As a rule, for large inputs optimize complexity first and for small inputs optimize run-time first. As always if you care about performance do benchmarks.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I have a small confusion with the nature of heuristics.
We know that heuristics need not give correct outputs for all input instances.
But then, why are heuristics proposed??
Heuristics are used to trade off performance (usually execution speed, but also memory consumption) with potential accuracy or generality. For example, your anti virus software uses heuristics to characterize what a virus might look like, and can take advantage of that piece of information to determine which files it should spend more time analyzing. A good heuristic has the property that it can save substantial time with minimal cost.
In graph traversal theory, a heuristic for an A* search algorithm need not be perfect. It just needs to have a predicted cost function h(x) that is less than or equal to the true cost to the goal state in order to guarantee an optimal solution. The closer h(x) equals the true cost, the quicker an optimal solution will be found.
Let me give you an example which might help you understand the importance of heuristics.
In Artificial Intelligence, search problems are mainly classified as blind search and directed search. Blind search is where you make use of algorithms such as BFS and DFS and there is a reason they are called blind search, they don't have any knowledge about the direction you should go, you just have to explore and explore until you reach the goal node, imagine the time and space complexity for those algorithms.
Now if you look at the directed search algorithm such as A*, where you have some kind of heuristic function or in simple terms an assumption about which direction you should take the next step.
Although heuristics does not guarantee the best result but rather will try to give you a better solution and sometimes even the best. There are so many classes of problems (Ex. games you play) where a better solution does the task rather than wasting so much time and space in finding the best solution.
I hope it helps.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I'm not sure if this is a problem with my understanding but this aspect of Big Oh notation seems strange to me. Say you have two algorithms - the first preforms n^2 operations and the second performs n^2-n operations. Because of the dominance of the quadratic term, both algorithms would have complexity O(n^2), yet the second algorithm will always be better than the first. That seems weird to me, Big Oh notation makes it seem like they are same. I dunno...
Big O is not about the time it takes to execute your algorithm, it is about how well it will scale when presented with large data sets (large values of n).
When presented with a large data set, the n^2 term will quickly overshadow any linear term. So the linear term becomes insignificant.
When n grows towards infinity n^2 will be much greater then n so the -n won't have any significant difference on the outcome.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
For a class, a question that was posed by my teacher was the algorithmic cost of multiplying a matrix times its transpose. With the standard 3 loop matrix multiplication algorithm, the efficiency is O(N^3), and I wonder if there was a way to manipulate or take advantage of matrix * matrix transpose to get a faster algorithm. I understand that when you multiply a matrix by its transpose you have to calculate less of the matrix because its symmetrical, but I can't think of how to manipulate an algorithm that could take less than O(n^3).
i know there's algorithms like Coppensmith and Straussen that are faster general matrix multiplication algorithms but could anyone give any hints or insights on how to computationally take advantage of the transpose?
Thanks
As of right now there aren't any aymptotic barrier-breaking properties of this particular multiplication.
The obvious optimization is to take advantage of the symmetry of the product. That is to say, the [i][j]th entry is equal to the [j][i]th entry.
For implementation-specific optimizations, there is a significant amount of caching that you can do. A very significant amount of time in the multiplication of large matrices is spent transferring data to and from memory and CPU. So CPU designers implemented a smart caching system whereby recently used memory is stored in a small memory section called the cache. In addition to that, they also made it so that nearby memory is also cached. This is because a lot of the memory IO is due to reading/writing from/to arrays, which are stored sequentially.
Since the transpose of a matrix is simply the same matrix with the indices swapped, caching a value in the matrix can have over twice the impact.