After getting used to the Q-Learning algorithm in discrete action-state-space I would like to expand this now to continous spaces. To do this I read the chapter On-Policy Control with Approximation of Sutton´s introduction. Here, the usage of differentiable functions like a linear function or an ANN are recommended to solve the problem of continous action-state-space. Nevertheless Sutton then discribes the tiling method which maps the continous variables onto a discrete presentation. Is this always necessary?
Trying to understand this methods I tried to implement the example of the Hill Climbing Car in the book without the tiling method and a linear base function q. As my state space is 2 dimensional, and my action is one dimensional I used a three dimensional weight vector w in this equation:
When I now try to choose the action which will maximize the output, the obvious answer will be a=1, if w_2 > 0. Therefore, the weight will slowly converge to positive zero and the agent will not learn anything useful. As Sutton is able to solve the problem using the tiling I am wondering if my problem is caused by the absence of the tiling method or if I am doing anything else wrong.
So: Is the tiling always necessary?
Regarding your main question about tiling, the answer is no, not always it is necessary using tiling.
As you tried, it's a good idea to implement some easy example as the Hill Climbing Car in order to fully understand the concepts. Here, however, you are misundertanding something important. When the book talks about linear methods, it is refering to linear in the parameters, which means that you can extract a set of (non linear) features and combine them linearly. This kind of approximators can represent functions much more complex than a standard linear regression.
The parametrization you have proposed it's not able to represent a non-linear Q function. Taking into account that in the Hill Climbing problem you want to learn Q-functions of this style:
You will need something more powefull than . An easy solution for your problem could be to use a Radial Basis Function (RBF) network. In this case, you use a set of features (or BF, like for example Gaussians functions) to map your state space:
Additionally, if your action space is discrete and small, the easiest solution is to maintain an independent RBF network for each action. For selecting the action, simply compute the Q value for each action and select the one with higher value. In this way you avoid the (complex) optimization problem of selecting the best action in a continuous function.
You can find a more detailed explanation on the Busoniu et al. book Reinforcement Learning and Dynamic Programming Using Function Approximators, pages 49-51. It's available for free here.
I was reading through a paper published by the University of Washington with tips about Machine Learning algorithms.
The way they break down an ML algorithm is into 3 parts. A representation, evaluation and optimization. I would like to understand how these three parts work together, so what is the process like throughout a typical machine learning algorithm.
I understand that my question is very abstract and each algorithm will be different, but if you know of a way to explain it abstractly please do. If not feel free to use a specific algorithm to explain.
Paper: http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf ~
See Table 1.
Sebastian Raschka has provided a very nice flowchart of the (supervised) machine learning process.
(I am familiar with text classification, so I will use this as an example)
In short :
Representation: Your data needs to be brought into a suitable algorithmic form. For Text classification you would for example extract features from your full text documents (input data) and bring them into a bag-of-words representation.
Application of the respective algorithm: Here your algorithm is executed. After training you have to do some kind of evaluation to measure your success (the criteria for evaluation is depending on the task). (The Evaluation is the final part of this step)
Optimization: The learning algorithms usually have one or more parameters to tune. Having reference pairs of parameter configuration / evaluation (from the previous step) you can now tweak the paramaters and evaluate the effect on your results.
(In the flowchart this is the edge, where we go back to the step "Learning algorithm training")
From information in internet, particular this link: http://forums.wolfram.com/mathgroup/archive/1999/Nov/msg00213.html
It was interseting problem of " How to do Lie algebra in Mathematica".
Now,our my small seminar(project) we have been problem Poincaré–Birkhoff–Witt theorem and apply to calculus universal enveloping lie algebra sl(3). My teacher asked us to write program in Mathematica to calculus cofficient and ingredient of 2 monomial of su(3) .Example :
Product of x^15*y^10*z^7 and x^4*y^5*z^6
Reamark: x,y,z be generate of universl enveloping algebra su(3) i.e:
lie bracket[x,z]=2x , [y,z]= -2y and [x,y]=z.
I just know this is very important problem for quantum mechanics and actually we are stuck at the moment, we have no ideal solve this problem. I feel in mathematica NonCommutativeMultiply function be very rudimentary.
Hope you could suggest or help us.
How can I do symbolic computations in mathematica? I do not want to give a value to any Variable. Can I use Mathematica to create a set of equations, consisting only of symbolic variables, and then combine them, or solve by one specific variable?
If this were good, it could pretty much replace a paper sheet for doing mathematics.
It would be awesome if you could provide some examples. I hope the question is clear
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Are they the same thing?
No.
A function is a block of code in a computer program.
An algorithm is an abstract concept that describes how to solve a problem.
In mathematics, a function is "a mathematical relation such that each element of a given set (the domain of the function) is associated with an element of another set (the range of the function)" (source - google.com, define:function).
In computer science, a function is a piece of code that optionally takes parameters, optionally gives a result, and optionally has a side effect (depending on the language - some languages forbid side-effects). It must have a specific machine implementation in order to execute.
The computer science term came out of the mathematical term, being the machine implementation of the mathematical concept.
An algorithm is "a precise rule (or set of rules) specifying how to solve some problem" (source - google.com, define:algorithm). An algorithm can be defined outside of computer science, and does not have a definitive machine implementation. You can "implement" it by writing it out by hand :)
The key difference here is, in computer science, an algorithm is abstract, and doesn't have a definitive machine implementation. A function is concrete, and does have a machine implementation.
An algorithm is a set of instructions.
In computer programming, a function is an implementation of an algorithm.
An algorithm is a series of steps (a process) for performing a calculation, whereas a function is the mathematical relationship between parameters and results.
A function in programming is different than the typical, mathematical meaning of function because it's a set of instructions implementing an algorithm for calculating a function.
An algorithm describes the general idea, whereas a function is an actual working implementation of that idea.
It might be almost a philosophical question, but I'de say an algorithm is the answer(or how-to) to a problem at hand where as a function does not necerally answer one problem in itself.
What you want to do normally is split your algorithm into severals function that each has their own goal, which, in the end, will achieve the problem at hand, when used together.
Ex : You want to Sort a list of numbers. The algorightm used would be for example the Merge-sort algorithm. That specific algorithm is actually composed of more than one functions, one that will split your array, another to check for equality, another to merge everything back together, and so on.
A mathematical function is the interface, or specification of the inputs and outputs of an algorithm.
An algorithm is the precise recipe that defines the steps that may implement a function.
Confusingly, computer language designers diffuse this distinction by using the concept function, func, method, etc, to talk about both concepts.
So the distinction is one of specification vs. definition.
There is also a semantic distinction: an algorithm seeks to provide a solution to a problem. It is goal-oriented. A function simply is - there is no essential teleological component.
An algorithm is the implementation of a function.
In some cases, the algorithm is trivial:
Function: Sum of two numbers.
Algorithm:
int sum(int x, int y){
return x+y;
}
In other cases, it is not:
Function: Best chess move.
Algorithm:
Move bestChessMove(State gameState){
//I don't know the algorithm.
}
Algorithm is a (possibly informal but necessarily precise) sequence of instructions. Function is a formal rule that associates some input w/ a specific output. Functions implement and formalize algorithms. E.g. we can formalize "go from a to b" as go(a)=b or go(x,a)=b (w/ x the one who goes), etc. According to Wikipedia,
An algorithm is an effective method that can be expressed within a
finite amount of space and time and in a well-defined formal
language for calculating a function.
So you can say that "go(ing) from a to b" is an effective method for calculating go(a)=b (if you want)
An Algorithm usually refers to the method or process used to end up with the result after mathematical processing. A Function is a subroutine used to avoid writing the same code over and over again. They are different in their uses. For instance, there may be an Algorithm that is used for encrypting data, and a function for posting code to a webpage.
Here is some further reference:
http://en.wikipedia.org/wiki/Algorithm
http://en.wikipedia.org/wiki/Function_(computer_science)
A function is a symbolic representation whereare a method is the mechanical steps needed to get the answer.
Suppose this function:
f(x) = x^ 2
Now if I tell you to count f(5000) you have to do things that this function doesn't say. Like for example how to multiply. So really these are just symbols.
But if I have a python method for example:
x = math.pow(500, 2) # or whatever it is
Then in this case the steps themselves for each operation are totally defined (in the libraries ;) ).