I am sure you all know Quantopian (I love it!!).
Even though I am pretty good with Python, I am still having trouble writing a full algorithm (I am trying to write one using Fundamentals data).
I have successfully used pipeline to get the stocks that I want, but am having trouble with writing the buys and the sells. Specifically, how to tell the program to buy this stock with some logic behind it and how to sell (the same or some other) stock.
I have gone through tons of resources, checked out other users' algorithms, went through both Quantopian and non Quantopian resources but still having trouble. Do you have any suggestions for other resources?
P.S. the link above is this guy 'sentdex'. His tutorials are the best!! It would have been fine if Quantopian didn't upgrade their systems, so unfortunately his tutorials are out of date now.
Background Knowledge:
Building up intuition can be tough. I found the books written by Ernie Chan to be most helpful when looking to complement my cs + math background with domain knowledge. Going through the process of translating portions of his books from MATLAB to Python was especially helpful.
Quantopian Specific Material:
Following strategies written by James Christopher will be worth your while. This algo in particular should serve as a great reference
https://www.quantopian.com/posts/long-short-pipeline-multi-factor
The lecture series also has a number of gems that will aid in building up the fundamentals necessary to do meaningful work on the platform.
Last But Not Least:
Use the community section as a learning tool and take advantage of the fact that it is the only place where you can engage with almost 100k quants in one coherent place. The "algotrading" subreddit isn't bad - that said it can occasionally feel like chewing on sand as many posts are misinformed.
Disclaimer: I interned at Quantopian in the Fall.
y can try think global about
initialize --> shedule_function --> rebalance --> if-elif logic: order(context.your_ticker_variable, amount)
and what about filters, I try custom - but still having trouble with it!
Related
I am trying to build some smart home devices by myself. And I am very interested in building IoT algorithms like Nest Thermostat does which is able to learn the characteristics of the house and the behavior of the family members.
Though I have some machine learning basics, I barely know about thermal model which is all the researches and methods of Nest based on.
So if I want do some study and create similar algorithms like Nest do by myself, how should I get started? Any suggested references?
You said it yourself - thermal modelling. So read up on thermodynamics. If you don't read on thermodynamics you won't know which part of thermodynamics to read on to model heat distribution in a house.
One of the most important thing about being a programmer is not programming. Programming is almost the least important thing a programmer does (slightly lower than debugging). The most important thing about being a programmer is to understand the requirements of the program.
So someone writing an accounting program should know a bit about accounting. He doesn't need to be an expert but he should at least be able to spot a bug.
Working for big companies you'll find that usually you'll have project managers and systems analysts helping you figure out the requirements. But coding your own project you have to be your own project manager and architect. So you have to do the reading-up.
Now, apart from the general advice above, when writing software to control real-world objects and phenomena you can't get away from knowing about the PID loop (Proportional, Integral, Differential). It's how software thermostats control the temperature of industrial ovens. It's how quadcopters can hover without becoming unstable. It's how Segways balance themselves.
The theory behind PID is more than a hundred years old. It was developed to govern steam engines. But it is so useful and important that we generally still depend on it in electronics.
There's a lot of math-heavy theory out there about PIDs. There are also a lot of less complicated rule-of-thumb guides about PIDs aimed at technicians and mechanics. I suggest reading the simpler less theory-heavy guides first then work your way up if you need to know something.
I have to take a small introductory talk on ruby tomorrow, I want to avoid going the boring power point presentation way and have a hands on session.
The goal would be to introduce ruby to people, just the basic concepts really.
I'm planning to take an example from Why's Poignant Guide, do you know of any interesting example that would captivate the attention of the audience and make it an interesting talk.
EDIT : I'm done with the talk, it went reasonably well, there were about 50 people who turned up, About 10 of them picked up ruby really well. Some complained that I went too fast. All in all I covered the basics of ruby, didn't touch the OO Stuff. As for the examples I gave one in which we could scrape data from our college website using watir-webdriver. Thanks to all for your valuable answers and comments.
Ok, so your audience are not programmers, so there's no point of pointing out Ruby's advantages over other languages. Also, there's no place for advanced topics such as metaprogramming or more serious OO or functional programming.
What I would try to show them first is irb, how they could evaluate simple mathematical expressions, and show them the concept of variables.
Strings and string interpolation.
Loops (10.times{ puts 'Hello world!' }) and branches (if-then-else).
If you have time, show them arrays (not sure about hashes)
I would also try to squeeze in a simple program in Sinatra as well. Students already know web and you can show "how the web really works" using couple lines of Sinatra code.
Maybe they'd be interested in a bit of web scraping.
My project Easy Roommate parser visits a flat-mate sharing web site, and parses profiles to see which ones are compatible with what you want. My main warning is that my code isn't very good, and if lots of people used it, the web site owners may complain. However, it would be solving a problem that's common to many students.
Another project I did ages ago was Get to philosophy, which tried to work out why, when you click on the first link in each Wikipedia article you come across, you usually end up at Philosophy. Warning: this project is abandoned.
Why's Guide is a little nuts for the average audience, more of a work of art than an educational tool. If you like whimsy, you might want to check out Rails for Zombies.
You could show them how to write programs that can be useful for college students in general. For example, let's say they have to study some chapters of a book for an exam. I have a Python scripts, 30 lines or so, that takes a number of days and a list of tuples representing page ranges, and prints how many pages you need to study each day, and at what page you need to get by the end of each day. For example, if you have to study pages [10,19] and [30,33] over two days, you'd need to get to the end of page 16 on the first day, and to the end of page 33 by the second.
You could show them how to implement this sort of thing with Ruby. It's a bit dry, but very practical.
To show oo stuff (if you want to do) consider showing some automation of the Browser with watir. -> Real oo with real objects all in simple Ruby. Perfect for interesting and lively demos to motivate young people (maybe everybody wants to try it out after the presentation (my experience)).
http://watir.com/examples/
Sonic Pi is a Ruby based SDK and DSL. Its typical usage is in music creation and sound generation (samples and synthesizer sounds). The Sonic Pi website http://sonic-pi.net contains tutorials, which are specifically designed to teach programming from the start. There is even a guide for teachers! Sonic Pi is really entertaining and it gives you immediate sense of achievement.
This is a generic question - i know a bit of perl and python and i am looking in to learn programming so that once i get a hang of it i can start developing apps and then websites.
I request you to give me algorithm(steps :)) that what should be my approach towards learning it.
I have posted small questions on perl/python and i have recieved great help from everyone.
Note:- i am not in a hurry to learn i know it takes time and thats fine.
Please give any suggestions you think are valid(Please dun push me to learn Lisp,Haskell - i am a beginner)
Step 1. Read
Step 2. Implement
Step 3. Repeat Step 1 until ur frustrated.
Simple Algo
Start with a good book that covers control structures, etc. (two I would recommend include Head First Programming, or learning to program (which is a Ruby book).
After that, try out some basic stuff to learn your syntax, control structures, logic, etc. - some good sites for this are:
Project Euler
Coding Dojo
Code Kata (pragpub)
Then, move up from there (at that point you may want more framework specific stuff - MVC, Rails, etc.).
On a side note - language is largely irrelevant. I'm mentoring an apprentice developer at work now (in C# but he's also looking at some Python), and in our last coding session, we wrote very little code, but spent a ton of time chewing through edge cases in a code kata.
(An addendum)
Once you're past basic control structures and syntax, you're going to find the toughest parts are design, abstraction, problem solving, etc. - so for those some good ways to pick up those skills include:
Getting a good mentor (even a virtual one).
Looking at LOTS of code. If you are not in a dev shop, look at an opensource project you're interested in to see how other people code.
Contribute to OSS
Start small, and build something for yourself (I'm always partial to building your own blog site - like a Jedi has to build their own lightsaber ;)
Have fun!
I am a one-man shop at the place where I work, and when I started there I had zero experience and a BS degree from a below-par school of Computer Science. On top of that, my first project at the company involved not just figuring out good design principles, it also involved learning a new language. Needless to say, my code was crappy in the beginning, and all the new features I've added since then have been hacked on top of all that crappy code. It's amazing to me that my software works as well as it does.
I have learned a TON during my employment, and I am dying to refactor my code to make it more readable so future new hires can dive in and help me with it. I also REALLY want to make it easier to add new features without having to hack stuff together. I think it would be useful to learn a framework like Prism for WPF/Silverlight, but I have a huge to-do list (since I am a one-man shop), and it looks like it will take a pretty decent amount of time just to learn how to use it.
Now I have read up a little bit on Prism to where I know the basic principles behind it. Furthermore, it wouldn't be hard to write my own code that accomplishes some of the same things that Prism is used for. I've actually done that already to some degree and I'm making good progress on making things more modular.
My question is this: should I go on writing more infrastructure code that gives me exactly what I need and no more, or should I take the time to learn something like Prism? Or maybe it could be asked like this: Should I spend time writing my own simple custom solutions, or should I spend time trying to grasp a rich, vast framework that may possibly be more complex than is necessary? And what factors should I take into account when making the decision?
I wrote my own PHP MVC framework for a recent project with exactly what I needed. It was loads of fun, taught me a lot, and an overall good experience, and I will never, ever do it again. While an excellent secondary distraction project, it highly detracted from my productivity on the main project.
Really, a lot of it depends on just how much infrastructure you will have to develop. If it's just a tiny bit that won't take more than an hour or two, go for it. If it will take significant amounts of time, use someone else's work, move on, and get your project done.
With the background you give - mostly self-educated and no peers to discuss your current development - you should absolutely check out other libraries and tools. At the very least, get new input how code can be designed and problems can be solved. You may feel that you have achieved something - and you have, congratulations - but that's a plateau, not the peak.
"I have no time to learn something new because I have so much to do"
- that's what I read in your rationale for more code.
This is a warning sign - you are moving yourself into a dangerous position. No time to learn? No time to document? No time to think of all the implications? No time to do it right? No time to train a new employee? No time to call it a day?
You won't solve this problem by learning prism, or any other library, but it's the wrong rationale.
Third, code bogs you down. Having more code to maintain makes you slower. One-man-startups can crank out hundreds, even thousands of LOC per day for days and weeks. As projects and organizations get larger, you end up with an average of a few dozen.
As a recommendation from personal experience: write bulding blocks, not frameworks. Frameworks are great when you have to make the same application with different company logos over and over. Or, as TDWTF's Alex says, the key is in the differences not the similarities.
I don't want you to stop writing code, far from it. But you are discussing a tradeoff, and from the information you've given, I would recommend to put most weight on learning new things.
If the app your writing is going to be around for a while and have to be maintained, particularly by other developers, then any time spent to learn and integrate a standard framework will be worthwhile.
It'll provide documentation for how the app is written and any developer familiar with that framework will be able to pick it up faster. It should reduce the amount of code you have to write and help you concentrate on your specific business problem and not the plumbing of writing an application.
The core issue is, how many times will you re-use the framework, saving you each time the work of re-implementing similar stuff instead? Remember, the stuff you write from scratch, if it's supposed to be any good at all, will have to be tested, validated against different environments (clients &c), and maintained -- all stuff that would come to you "for free" by using a good, actively maintained framework.
If you're going to use that framework only a couple of times, maybe the net returns are still in favor of rewriting from scratch -- but if the framework covers a field that you need in more than just a couple cases, the returns on the investment of learning to use the framework (assuming it's any good!-) vs redoing things from scratch are going to be vastly positive!
I was in a similar situation when I graduated college. I received an offer from a large company about 1.5 years into my stay at the small company. What I learned was this (may be different for you and others):
It was an awesome idea to work at a small company right out of school. I say this because you have to wear many different hats. For example you would write the code, test the code, deploy the code, write stored procs, etc. The end result is that you are familiar with the entire process from conception to whatever. That experience is critical I think.
I loved writing code. I remember the days that I would be driving home and thinking of my day spent dealing with production support issues. I was spending more time supporting customers and writing "one offs" that I wasn't mostly writing code.
Working for a big company is a bad idea right of college. When you're working for a big company they have a specific role for you, and you have specific boundaries. If you're a developer at a big company odds are that you are not deploying the application(s) to production, or tuning stored procs.
Working for a big company is a great idea after working for a small one. That's because if you work at the small company it will force you to learn about a lot more than just coding. And if you understand that you will be a better developer.
Working with good developers makes you better. When you work with a group of guys that are good you will get better. This is because each dev has a specific history that they bring to the group, and you all learn from one another. On the group that I work with mainly right now there is: an MSBuild expert, a Silverlight expert and an F# expert, and other good guys. So some of the guys learn MSBuild from me, and I learn from them. Just talking to guys who are good can make you better.
So if I was you, don't spend too much time there. Maybe 1 or 2 years, after that find a job somewhere that has some talented developers. You will be a much better dev in 5 years. I know that I am because of my move.
I'll play contrarian: YAGNI (You Aren't Going to Need It).
What if the framework
Is badly designed?
Is buggy?
Is too slow?
Will be different in two years, and the old version won't be supported?
Discussions of frameworks often assume that frameworks are great, where the reality is that frameworks vary just like anything else, and many frameworks are larded with stuff you are never going to need.
Here's some advice that I hope bears on your more specific questions:
Continue to make incremental improvements. It sounds like you are being productive with this approach and that it is paying off for you.
Learn more about the framework. Or multiple frameworks. Maybe you can try a small pilot project in, say, 1 to 4 days.
It is an honorable strategy to learn about a framework not in order to use the framework, but in order to cherry-pick the best ideas and adopt them into your own designs.
If you decide for the time being not to adopt a framework, this is an easy decision to revisit later. If you decide to adopt a framework, backing away from it later can be very expensive. It may be worth paying some extra costs up front to reduce the probability of making a very expensive mistake.
I think where I have been burned the most is by depending on somebody else's code base that changed out from under me. I call this the "every Perl script I ever wrote was broken a year later" problem. But I work on a lot of different small projects that tend to get bursty attention and have a very long lifetime compared to the time it took to create one. If you have one big project that you're working with daily for years, you can adapt more easily to changes in external frameworks.
Who has that tool?
This is what I ask to myself everytime I need to solve every problem. This is the main factor to evaluate the effort necessary for develop the tool.
When starting a large project, everyone define well known (at project scope) usefull statements; thinking to enlarge that abstraction layer depends on the frequency of the problem, the importance of the the problem solution, the effort to develop the solution.
Having been a hobbyist programmer for 3 years (mainly Python and C) and never having written an application longer than 500 lines of code, I find myself faced with two choices :
(1) Learn the essentials of data structures and algorithm design so I can become a l33t computer scientist.
(2) Learn Qt, which would help me build projects I have been itching to build for a long time.
For learning (1), everyone seems to recommend reading CLRS. Unfortunately, reading CLRS would take me at least an year of study (or more, I'm not Peter Krumins). I also understand that to accomplish any moderately complex task using (2), I will need to understand at least the fundamentals of (1), which brings me to my question : assuming I use C++ as the programming language of choice, which parts of CLRS would give me sufficient knowledge of algorithms and data structures to work on large projects using (2)?
In other words, I need a list of theoretical CompSci topics absolutely essential for everyday application programming tasks. Also, I want to use CLRS as a handy reference, so I don't want to skip any material critical to understanding the later sections of the book.
Don't get me wrong here. Discrete math and the theoretical underpinnings of CompSci have been on my "TODO: URGENT" list for about 6 months now, but I just don't have enough time owing to college work. After a long time, I have 15 days off to do whatever the hell I like, and I want to spend these 15 days building applications I really want to build rather than sitting at my desk, pen and paper in hand, trying to write down the solution to a textbook problem.
(BTW, a less-math-more-code resource on algorithms will be highly appreciated. I'm just out of high school and my math is not at the level it should be.)
Thanks :)
This could be considered heresy, but the vast majority of application code does not require much understanding of algorithms and data structures. Most languages provide libraries which contain collection classes, searching and sorting algorithms, etc. You generally don't need to understand the theory behind how these work, just use them!
However, if you've never written anything longer than 500 lines, then there are a lot of things you DO need to learn, such as how to write your application's code so that it's flexible, maintainable, etc.
For a less-math, more code resource on algorithms than CLRS, check out Algorithms in a Nutshell. If you're going to be writing desktop applications, I don't consider CLRS to be required reading. If you're using C++ I think Sedgewick is a more appropriate choice.
Try some online comp sci courses. Berkeley has some, as does MIT. Software engineering radio is a great podcast also.
See these questions as well:
What are some good computer science resources for a blind programmer?
https://stackoverflow.com/questions/360542/plumber-programmers-vs-computer-scientists#360554
Heed the wisdom of Don and just do it. Can you define the features that you want your application to have? Can you break those features down into smaller tasks? Can you organize the code produced by those tasks into a coherent structure?
Of course you can. Identify any 'risky' areas (areas that you do not understand, e.g. something that requires more math than you know, or special algorithms you would have to research) and either find another solution, prototype a solution, or come back to SO and ask specific questions.
Moving from 500 loc to a real (eve if small) application it's not that easy.
As Don was pointing out, you'll need to learn a lot of things about code (flexibility, reuse, etc), you need to learn some very basic of configuration management as well (visual source safe, svn?)
But the main issue is that you need a way to don't be overwhelmed by your functiononalities/code pair. That it's not easy. What I can suggest you is to put in place something to 'automatically' test your code (even in a very basic way) via some regression tests. Otherwise it's going to be hard.
As you can see I think it's no related at all to data structure, algorithms or whatever.
Good luck and let us know
I must say that sitting down with a dry old textbook and reading it through is not the way to learn how to do anything effectively, even if you are making notes. Doing it is the best way to learn, using the textbooks as a reference. Indeed, using sites like this as a reference.
As for data structures - learn which one is good for whatever situation you envision: Sets (sorted and unsorted), Lists (ArrayList, LinkedList), Maps (HashMap, TreeMap). Complexity of doing basic operations - adding, removing, searching, sorting, etc. That will help you to select an appropriate library data structure to use in your application.
And also make sure you're reasonably warm with MVC - i.e., ensure your model is separate from your view (the QT front-end) as best as possible. Best would be to have the model and algorithms working on their own, and then put the GUI on top. Or a unit test on top. Etc...
Good luck!
It's like saying you want to move to France, so should you learn french from a book, and what are the essential words - or should you just go to France and find out which words you need to know from experience and from copying the locals.
Writing code is part of learning computer science. I was writing code long before I'd even heard of the term, and lots of people were writing code before the term was invented.
Besides, you say you're itching to write certain applications. That can't be taught, so just go ahead and do it. Some things you only learn by doing.
(The theoretical foundations will just give you a deeper understanding of what you wind up doing anyway, which will mainly be copying other people's approaches. The only caveat is that in some cases the theoretical stuff will tell you what's futile to attempt - e.g. if one of your itches is to solve an NP complete problem, you probably won't succeed :-)
I would say the practical aspects of coding are more important. In particular, source control is vital if you don't use that already. I like bzr as an easy to set up and use system, though GUI support isn't as mature as it could be.
I'd then move on to one or both of the classics about the craft of coding, namely
The Pragmatic Programmer
Code Complete 2
You could also check out the list of recommended books on Stack Overflow.