I've been reading things here and there for a while now about using an "ant colony" model as a heuristic approach to optimizing various types of algorithms. However, I have yet to find an article or book that discusses ant colony optimizations in an introductory manner, or even in a lot of detail. Can anyone point me at some resources where I can learn more about this idea?
On the off chance that you know German (yes, sorry …), a friend and I have written an introduction with code about this subject which I myself find quite passable. The text and code uses the example of TSP to introduce the concept.
Even if you don't know German, take a look at the code and the formulas in the text, this might still serve.
link Wikipedia actually got me started. I read the article and got to coding. I was solving a wicked variation of the traveling salesman problem. It's an amazing meta-heuristic. Basically, any type of search problem that can be put into a graph (nodes & edges, symmetric or not) can be solved with an ACO.
Look out for the difference between global and local pheromone trails. Local pheromones discourage one generation of ants from traversing the same path. They keep the model from converging. Global pheromones are attractors and should snag at least one ant per generation. They encourage optimum paths over several generations.
The best suggestion I have, is simply to play with the algorithm. Setup a basic TSP solver and some basic colony visualization. Then have some fun. Working with ants, conceptually, is way cool. You program their basic behaviors and then set them loose. I even grow fond of them. :)
ACOs are a greedier form of genetic algorithms. Play with them. Alter their communicative behaviors and pack behavior. You'll rapidly begin to see network / graph programming in an entirely different way. That's their biggest benefit, not the recipe that most people see it as.
You just gotta play with it to really understand it. Books & research papers only give a general sky-high understanding. Like a bike, you just gotta start riding. :)
ACOs, by far, are my favorite abstraction for graph problems.
National Geographic wrote an interesting article awhile back talking about some of the theories.
The best resource for these topics is Google scholar. Ive been working on Ant Colony Optimization algorithms for a while, here are some good papers:
Ant Colony Optimization - A New Metaheuristic
Ant Colony Optimization - Artificial Ants as a Computational Intelligence Technique
Just search for "Ant Colony" on google scholar.
Also, search for papers published by Marco Dorigo.
I am surprised nobody has mentioned the bible of ACO:
Marco Dorigo & Thomas Stützle: Ant Colony Optimization
This book is written by the author of ACO and it is highly readable. You can take it to the beach and have fun reading it. But it is also the most complete resource of all, great as a reference when implementing the thing.
You can read some excerpts on Google Books
Another great source of wisdom is the ACO Homepage
See for example this article on scholarpedia.
There is also discussion here in the What is the most efficient way of finding a path through a small world graph? question.
At first glance this seems to be closely related to (or prehaps a special case of) the Metropolis algorithm. So that's another possible direction for searching.
Addition: This PDF file includes a reference to the original Metropolis paper from 1953.
Well, i found the Homepage of Eric Rollins and his different Implementations (Haskell, Scala, Erlang,...) of a ACO Algorithm helpfull.
And also the Book from Enrique Alba, titled "Parallel Metaheuristics: A New Class of Algorithms" where you can find a whole chapter of explanation about ACO Algorithms and their different usages.
Hth
Related
For my research project in biology for my final year I need to present a project in the field of Biotechnology. Being passionate about programming I immediately thought of Evolutionary Algorithms! However I am not sure if Evolutionary Algorithms would fall into the category of Biotechnology, hence I would rather confirm with the best and most passionate programming experts on the world.
Unfortunately no, a genetic algorithm (ga) is just an optimization technique that is inspired from various evolutionary processes like mutation or crossover. They belong to the area of evolutionary computing and artificial intelligence and not biotechnology.
Please follow this link for a brief introduction to genetic algorithms.
Biotechnology from the other hand has to do with actual organisms that are used in some way to make a product or an application. It sounds kind of broad but that is only because the particular field is in itself very broad. We use forms of biotechnology for thousands of years now in many common and not so common ways. This is not bad though as it gives you a lot of freedom regarding your project. Choose anything from food production to medicine and you will still be relevant to the subject.
Maybe the links provided will give you some inspiration.
Link one
Link two
Until you're implementing your evolutionary algorithms with organic material, no.
They are, of course, inspired from the way modern organisms have come to exist. But there's no biology in what you're doing.
No. It's just an example of a biological algorithm adapted for computational purposes.
Other examples include Ant-Colony Optimization, Flocking behavior, etc.
IIRC, Biotechnology requires the use of actual biology (i.e., living things or parts of them) adapted for technological purposes, not just an algorithmic emulation or modelling of their processes.
"Designing the right algorithm for a given application is a difficult job. It requires a major creative act, taking a problem and pulling a solution out of the ether. This is much more difficult than taking someone else's idea and modifying it or tweaking it to make it a little better. The space of choices you can make in algorithm design is enormous, enough to leave you plenty of freedom to hang yourself".
I have studied several basic design techniques of algorithms like Divide and Conquer, Dynamic Programming, greedy, backtracking etc.
But i always fail to recognize what principles to apply when i come across certain programming problems. I want to master the designing of algorithms.
So can any one suggest a best place to understand the principles of algorithm design in depth.....
I suggest Programming Pearls, 2nd edition, by Jon Bentley. He talks a lot about algorithm design techniques and provides examples of real world problems, how they were solved, and how different algorithms affected the runtime.
Throughout the book, you learn algorithm design techniques, program verification methods to ensure your algorithms are correct, and you also learn a little bit about data structures. It's a very good book and I recommend it to anyone who wants to master algorithms. Go read the reviews in amazon: http://www.amazon.com/Programming-Pearls-2nd-Edition-Bentley/dp/0201657880
You can have a look at some of the book's contents here: http://netlib.bell-labs.com/cm/cs/pearls/
Enjoy!
You can't learn algorithm design just from reading books. Certainly, books can help. Books like Programming Pearls as suggested in another answer are great because they give you problems to work. Each problem forces you to think about how to solve a particular type of problem.
The idea is that you expose yourself to many different types of problems and their solutions. In doing so, you learn how to examine a problem and see if it shares anything in common with problems you've already seen. In that regard, it's not a whole lot different than the way you learned how to solve "word problems" in math class. Granted, most algorithm problems are more complex than having to figure out where on the tracks the two trains will collide, but the way you learn how to solve the problems is the same. You learn common techniques used to solve simple problems, then combine those techniques to solve more complex problems, etc.
Read, practice, lather, rinse, repeat.
In addition to books like Programming Pearls, there are sites online that post different programming challenges that you can test yourself on. It helps if you have friends or co-workers who also are interested in algorithms, because you can bounce ideas off each other and pose interesting challenges, or work together to come up with solutions to problems.
Did I mention that it takes practice?
"Mastering" anything takes time. A long time. A popular theory is that it takes 10,000 hours of practice to become an expert at anything. There's some dispute about that for particular endeavors, but in general it's true. You don't master anything overnight. You have to study. And practice. And read what others have done. Study some more and practice some more.
A good book about algorithm design is Kleinbeg Tardos. Every design technique depends on the problem that you are going to tackle. It is very important to do the exercises in the algorithm books and have feedback from teachers about that.
If there exist a locally optimal choice taht brings the globally optimal solution you can use a greedy algorithm.
If the problem has optimal substructure, you can use dynamic programming.
I am planning to implement spam filter using Naive Bayesian classification model.
Online I see a lot of info on Naive Bayesian classification, but the problem is its a lot of mathematical stuff, than clearly stating how its done. And the problem is I am more of a programmer than a mathematician (yes I had learnt Probability and Bayesian theorem back in school, but out of touch for a long long time, and I don't have luxury of learning it now (Have nearly 3 weeks to come-up with a working prototype)).
So if someone can explain or point me to location where its explained for programmers than a mathematician, it would be a great help.
PS: By the way I have to implement it in C, if you want to know. :(
Regards,
Microkernel
The book Programming Collective Intelligence has chapter that covers this and other methods. The chapter (#6) can be understood without reference to previous chapters, is written clearly, and discusses only the minimal mathematics necessary to get the job done.
You could try this website. It's got some source code.
I would highly recommend Andrew Moore's tutorials and I think you should start with this one.
You could also take a look at POPFile, an open source spam filter engine.
Have you looked at dspam?
http://dspam.irontec.com/faq.shtml#1.0
http://www.nuclearelephant.com/
When faced with a problem in software I usually see a solution right away. Of course, what I see is usually somewhat off, and I always need to sit down and design (admittedly, I usually don't design enough), but I get a certain intuition right away.
My problem is I don't get that same intuition when it comes to advanced algorithms. I feel much more up to the task of building another Facebook then building another Google search, or a Music Genom project. It's probably because I've been building software for quite some time, but I have little experience with composing algorithms.
I would like the community's advice on what to read and what projects to undertake to be better at composing algorithms.
(This question has nothing to do with Algorithmic composition. Well, almost nothing)
+1 To whoever said experience is the best teacher.
There are several online portals which have a lot of programming problems, that you can submit your own solutions to, and get an automated pass/fail indication.
http://www.spoj.pl/
http://uva.onlinejudge.org/
http://www.topcoder.com/tc
http://code.google.com/codejam/contests.html
http://projecteuler.net/
https://codeforces.com
https://leetcode.com
The USACO training site is the training program that all USA computing olympiad participants go through. It goes step by step, introducing more and more complex algorithms as you go.
You might find it helpful to perform algorithms physically. For example, when you're studying sorting algorithms, practice doing each one with a deck of cards. That will activate different parts of your brain than reading or programming alone will.
Steve Yegge referred to "The Algorithm Design Manual" in one of his rants. I haven't seen it myself, but it sounds like it's just the ticket from his description.
My absolute favorite for this kind of interview preparation is Steven Skiena's The Algorithm Design Manual. More than any other book it helped me understand just how astonishingly commonplace (and important) graph problems are – they should be part of every working programmer's toolkit. The book also covers basic data structures and sorting algorithms, which is a nice bonus. But the gold mine is the second half of the book, which is a sort of encyclopedia of 1-pagers on zillions of useful problems and various ways to solve them, without too much detail. Almost every 1-pager has a simple picture, making it easy to remember. This is a great way to learn how to identify hundreds of problem types.
problem domain
First you must understand the problem domain. An elegant solution to the wrong problem is no good, nor is an inefficient solution to the right problem in most cases. Solution quality, in other words, is often relative. A simple scheduling problem that has a deterministic solution that takes ten minutes to run may be fine if schedules are realculated once per week, but if schedules change several times a day then a genetic algorithm solution that converges in a few seconds may be required.
decomposition and mapping
Second, decompose the problem into sub-problems and known/unknown elements that correspond to elements of the solution. Sometimes this is obvious, e.g. to count widgets you need a way of identifying widgets, an incrementable counter, and a way of storing the count. Sometimes it is not so obvious. Sometimes you have to decompose the problem, the domain, and possible solutions at the same time and try several different mappings between them to find one that leads to the correct results [this is the general method].
model
Model the solution, in your head at least, and walk through it to see if it works correctly. Adjust as necessary (See decomposition and mapping, above).
composition/interfaces
Many times you can find elements of the problem and elements of the solution that map to each other and produce partial results that are useful. This composition and interface construction provides the kernal of the solution, and also serves to reduce the scope of the problem remaining. So then you just loop back to the top with a smaller initial problem, and go through it again.
experience
Experience is the best teacher, of course, but reading about different kinds of problems and solutions will also be helpful. Studying some of the well-known algorithms and their applications is likewise very helpful, e.g. Dijkstra, Bresenham, Unification, and of course, graph theory.
I am not sure intuition can be cultivated, but I think I know what you are asking. The more problems you solve, the more information and experience you have at your disposal for future problems. So, I say just practice. Practice programming real world applications and you run into plenty of problems. Sometimes, solving puzzles can be very educational as well.
I try to find physical analogues when I'm looking at a complex problem.
The other day I thought I'd attempt creating the Fibonacci algorithm in my code, but I've never been good at maths.
I ended up writing my own method with a loop but it seemed inefficient or not 'the proper way'.
Does anyone have any recommendations/reading material on implementing algorithms in code?
I find Project Euler useful for this kind of thing. It forces you to think about an algorithm and then implement it. Many of the questions then have extensive discussions on how to solve the problem (from the naive solutions to some pretty ingenious ones) that you can use to see what you did right and wrong.
In the discussion threads you'll find various implementations from other people in many different languages too. Coming up with a solution yourself and then comparing it to that from other people is (imho) a good way to learn.
Both of these introductory books have good information about this sort of thing:
How To Design Programs and moreso Structure and Interpretation of Computer Programs
Both are somewhat funcitonal (and scheme) oriented, but that's a natural fit for these sorts of problems.
On top of that, you might get quite a bit out of Project Euler
Derive your algorithm test-driven. I've been able to write much more complex algorithms correctly by using TDD than I was before.
Go on youtube and look at some of the lectures on Introduction to Algorithms. There are some really, really good lectures that break down some of the most common algorithms such as the Fibonacci series and how to optimize them.
Start reading about O notation so you can understand how your algorithm grows with variable size input and how to classifiy the run-time of the algorithm you have.
Start with this video series which I found excellent material on the subject:
Algorithms Lecture
If you can't translate pseudo code for a fibonacci function to your language, then you should go and find a basic tutorial for your language, since it seems that you have not yet grasped its basic idioms.
If you have a working solution, but feel insecure about it, show it to others for review.