Using pyparsing i try to parse some text with a compound expression like
a = pp.Word(pp.alphas).setResultsName('A')
b = pp.Word(pp.nums).setResultsName('B')
c = pp.Word(pp.alphas).setResultsName('C')
expr = a + b + c
and parseString fails with the Exception
ParseException: Expected W:(0123...) (at char 7), (line:1, col:8)
So far so good. However, to understand better what's going on, is it possible to ask pyparsing/parseString to tell me directly what character from the input string didn't match? (I can, of course, calculate this myself from the information in the Exception text.)
Additionally is it possible to see in which sub-expression (a,b or c) the exception was raised?
Pyparsing exceptions include a method markInputline() that will print the last line of the input string and a marker where the exception occurred:
import pyparsing as pp
a = pp.Word(pp.alphas).setResultsName('A')
b = pp.Word(pp.nums).setResultsName('B')
c = pp.Word(pp.alphas).setResultsName('C')
expr = a + b + c
try:
expr.parseString("lskdjf lskdjf sdlkfj")
except ParseException as pe:
print(pe.markInputline())
lskdjf >!<lskdjf sdlkfj
(You can specify a different marker if you don't like '>!<'.)
Here is another method I've used that makes use of the col and line attributes of the ParseException:
alphaword = pp.Word(pp.alphas).setName('alphaword')
numword = pp.Word(pp.nums).setName('numword')
expr = alphaword('A') + numword('B') + alphaword('C')
try:
expr.parseString('sldkj slkdj sldkj')
except ParseException as pe:
print(pe.line)
print(' '*(pe.col-1) + '^')
print(pe)
sldkj slkdj sldkj
^
Expected numword (at char 6), (line:1, col:7)
A couple of other points:
I've used setName() to give the expressions themselves names, so that the exception messages are a little more readable. Please note the distinction between setName and setResultsName.
I've used the call syntax for defining results names. In practice (or just out of laziness) I found the '.setResultsName' method call to really detract from the grammar definition portion of the code. So in place of expr.setResultsName('xyz'), you can just write expr('xyz').
Related
When I try to filter a DataFrame by a Date the following way:
#linq df |> #where(:a .> (myDate + Dates.Month(18)))
I get the error message:
LoadError: MethodError: `convert` has no method matching convert(::Type{DataFramesMeta.SymbolParameter{T}}, ::Expr)
This may have arisen from a call to the constructor DataFramesMeta.SymbolParameter{T}(...),
since type constructors fall back to convert methods.
Closest candidates are:
call{T}(::Type{T}, ::Any)
convert{T}(::Type{T}, !Matched::T)
DataFramesMeta.SymbolParameter(!Matched::Symbol)
while loading In[79], in expression starting on line 2
while if I define a variable t it works:
t = (myDate + Dates.Month(18))
#linq df |> #where(:a .> t)
Is the first case a fundamentally wrong way of using the #where clause? If so, why? (myDate + Dates.Month(18)) should be just a Date just as t is, so why is this different behavior; also how to make the two cases behave the same way?
def success?
return #fhosts.empty? and #khosts.empty? and #shosts.any?
end
When I run that instance method, I get an error:
/home/fandingo/code/management/lib/ht.rb:37: void value expression
return #fhosts.empty? and #khosts.empty? and #shosts.any?
I'm confused by what's happening since this works
def success?
#fhosts.empty? and #khosts.empty? and #shosts.any?
# This also works
# r = #fhosts.empty? and #khosts.empty? and #shosts.any?
# return r
end
I'm coming from a Python background, and I don't want anything to do with implicit returns. Programming has plenty of landmines as it is.
If we have an arbitrary expression, E, that consists of boolean operations and and or together, here are some operations we could perform:
if E -- works
E -- works
* v = E -- works
return E -- broken
Why doesn't the last case work?
Edit: Actually v = E doesn't work. Only
v = Ei
is evaluated. Ei+1...k are ignored.
This is likely due to the very weak binding of and which causes it to parse out differently than you expect:
return x and y
This actually means:
(return x) and y
Since you're returning immediately it doesn't have a chance to evaluate the remainder of the expression.
Your version without return is correct:
x and y
This doesn't have a binding issue and is more idiomatic Ruby. Remember you only need to put an explicit return if you're trying to force an exit before the last line of the method. Being opposed to implicit returns is going to make your code look heavily non-Ruby. They're one of the reasons Ruby is so clean and simple, and how things like a.map { |v| v * 2 } works.
The When in Rome principle applies here. If you want to write Python-style Ruby you're going to be going against the grain. It's like saying "I don't like how you say X in your spoken language, so I'll just ignore that and do it my way."
This should also work:
return x && y
The && method is very strongly bound so return is the last thing evaluated here.
Or if you really want to use and for whatever reason:
return (x and y)
I am looking for a general-purpose way of defining textual expressions which allow a value to be validated.
For example, I have a value which should only be set to 1, 2, 3, 10, 11, or 12.
Its constraint might be defined as: (value >= 1 && value <= 3) || (value >= 10 && value <= 12)
Or another value which can be 1, 3, 5, 7, 9 etc... would have a constraint like value % 2 == 1 or IsOdd(value).
(To help the user correct invalid values, I'd like to show the constraint - so something descriptive like IsOdd is preferable.)
These constraints would be evaluated both on client-side (after user input) and server-side.
Therefore a multi-platform solution would be ideal (specifically Win C#/Linux C++).
Is there an existing language/project which allows evaluation or parsing of similar simple expressions?
If not, where might I start creating my own?
I realise this question is somewhat vague as I am not entirely sure what I am after. Searching turned up no results, so even some terms as a starting point would be helpful. I can then update/tag the question accordingly.
You may want to investigate dependently typed languages like Idris or Agda.
The type system of such languages allows encoding of value constraints in types. Programs that cannot guarantee the constraints will simply not compile. The usual example is that of matrix multiplication, where the dimensions must match. But this is so to speak the "hello world" of dependently typed languages, the type system can do much more for you.
If you end up starting your own language I'd try to stay implementation-independent as long as possible. Look for the formal expression grammars of a suitable programming language (e.g. C) and add special keywords/functions as required. Once you have a formal definition of your language, implement a parser using your favourite parser generator.
That way, even if your parser is not portable to a certain platform you at least have a formal standard from where to start a separate parser implementation.
You may also want to look at creating a Domain Specific Language (DSL) in Ruby. (Here's a good article on what that means and what it would look like: http://jroller.com/rolsen/entry/building_a_dsl_in_ruby)
This would definitely give you the portability you're looking for, including maybe using IronRuby in your C# environment, and you'd be able to leverage the existing logic and mathematical operations of Ruby. You could then have constraint definition files that looked like this:
constrain 'wakeup_time' do
6 <= value && value <= 10
end
constrain 'something_else' do
check (value % 2 == 1), MustBeOdd
end
# constrain is a method that takes one argument and a code block
# check is a function you've defined that takes a two arguments
# MustBeOdd is the name of an exception type you've created in your standard set
But really, the great thing about a DSL is that you have a lot of control over what the constraint files look like.
there are a number of ways to verify a list of values across multiple languages. My preferred method is to make a list of the permitted values and load them into a dictionary/hashmap/list/vector (dependant on the language and your preference) and write a simple isIn() or isValid() function, that will check that the value supplied is valid based on its presence in the data structure. The beauty of this is that the code is trivial and can be implemented in just about any language very easily. for odd-only or even-only numeric validity again, a small library of different language isOdd() functions will suffice: if it isn't odd it must by definition be even (apart from 0 but then a simple exception can be set up to handle that, or you can simply specify in your code documentation that for logical purposes your code evaluates 0 as odd/even (your choice)).
I normally cart around a set of c++ and c# functions to evaluate isOdd() for similar reasons to what you have alluded to, and the code is as follows:
C++
bool isOdd( int integer ){ return (integer%2==0)?false:true; }
you can also add inline and/or fastcall to the function depending on need or preference; I tend to use it as an inline and fastcall unless there is a need to do otherwise (huge performance boost on xeon processors).
C#
Beautifully the same line works in C# just add static to the front if it is not going to be part of another class:
static bool isOdd( int integer ){ return (integer%2==0)?false:true; }
Hope this helps, in any event let me know if you need any further info:)
Not sure if it's what you looking for, but judging from your starting conditions (Win C#/Linux C++) you may not need it to be totally language agnostic. You can implement such a parser yourself in C++ with all the desired features and then just use it in both C++ and C# projects - thus also bypassing the need to add external libraries.
On application design level, it would be (relatively) simple - you create a library which is buildable cross-platform and use it in both projects. The interface may be something simple like:
bool VerifyConstraint_int(int value, const char* constraint);
bool VerifyConstraint_double(double value, const char* constraint);
// etc
Such interface will be usable both in Linux C++ (by static or dynamic linking) and in Windows C# (using P/Invoke). You can have same codebase compiling on both platforms.
The parser (again, judging from what you've described in the question) may be pretty simple - a tree holding elements of types Variable and Expression which can be Evaluated with a given Variable value.
Example class definitions:
class Entity {public: virtual VARIANT Evaluate() = 0;} // boost::variant may be used typedef'd as VARIANT
class BinaryOperation: public Entity {
private:
Entity& left;
Entity& right;
enum Operation {PLUS,MINUS,EQUALS,AND,OR,GREATER_OR_EQUALS,LESS_OR_EQUALS};
public:
virtual VARIANT Evaluate() override; // Evaluates left and right operands and combines them
}
class Variable: public Entity {
private:
VARIANT value;
public:
virtual VARIANT Evaluate() override {return value;};
}
Or, you can just write validation code in C++ and use it both in C# and C++ applications :)
My personal choice would be Lua. The downside to any DSL is the learning curve of a new language and how to glue the code with the scripts but I've found Lua has lots of support from the user base and several good books to help you learn.
If you are after making somewhat generic code that a non programmer can inject rules for allowable input it's going to take some upfront work regardless of the route you take. I highly suggest not rolling your own because you'll likely find people wanting more features that an already made DSL will have.
If you are using Java then you can use the Object Graph Navigation Library.
It enables you to write java applications that can parse,compile and evaluate OGNL expressions.
OGNL expressions include basic java,C,C++,C# expressions.
You can compile an expression that uses some variables, and then evaluate that expression
for some given variables.
An easy way to achieve validation of expressions is to use Python's eval method. It can be used to evaluate expressions just like the one you wrote. Python's syntax is easy enough to learn for simple expressions and english-like. Your expression example is translated to:
(value >= 1 and value <= 3) or (value >= 10 and value <= 12)
Code evaluation provided by users might pose a security risk though as certain functions could be used to be executed on the host machine (such as the open function, to open a file). But the eval function takes extra arguments to restrict the allowed functions. Hence you can create a safe evaluation environment.
# Import math functions, and we'll use a few of them to create
# a list of safe functions from the math module to be used by eval.
from math import *
# A user-defined method won't be reachable in the evaluation, as long
# as we provide the list of allowed functions and vars to eval.
def dangerous_function(filename):
print open(filename).read()
# We're building the list of safe functions to use by eval:
safe_list = ['math','acos', 'asin', 'atan', 'atan2', 'ceil', 'cos', 'cosh', 'degrees', 'e', 'exp', 'fabs', 'floor', 'fmod', 'frexp', 'hypot', 'ldexp', 'log', 'log10', 'modf', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh']
safe_dict = dict([ (k, locals().get(k, None)) for k in safe_list ])
# Let's test the eval method with your example:
exp = "(value >= 1 and value <= 3) or (value >= 10 and value <= 12)"
safe_dict['value'] = 2
print "expression evaluation: ", eval(exp, {"__builtins__":None},safe_dict)
-> expression evaluation: True
# Test with a forbidden method, such as 'abs'
exp = raw_input("type an expression: ")
-> type an expression: (abs(-2) >= 1 and abs(-2) <= 3) or (abs(-2) >= 10 and abs(-2) <= 12)
print "expression evaluation: ", eval(exp, {"__builtins__":None},safe_dict)
-> expression evaluation:
-> Traceback (most recent call last):
-> File "<stdin>", line 1, in <module>
-> File "<string>", line 1, in <module>
-> NameError: name 'abs' is not defined
# Let's test it again, without any extra parameters to the eval method
# that would prevent its execution
print "expression evaluation: ", eval(exp)
-> expression evaluation: True
# Works fine without the safe dict! So the restrictions were active
# in the previous example..
# is odd?
def isodd(x): return bool(x & 1)
safe_dict['isodd'] = isodd
print "expression evaluation: ", eval("isodd(7)", {"__builtins__":None},safe_dict)
-> expression evaluation: True
print "expression evaluation: ", eval("isodd(42)", {"__builtins__":None},safe_dict)
-> expression evaluation: False
# A bit more complex this time, let's ask the user a function:
user_func = raw_input("type a function: y = ")
-> type a function: y = exp(x)
# Let's test it:
for x in range(1,10):
# add x in the safe dict
safe_dict['x']=x
print "x = ", x , ", y = ", eval(user_func,{"__builtins__":None},safe_dict)
-> x = 1 , y = 2.71828182846
-> x = 2 , y = 7.38905609893
-> x = 3 , y = 20.0855369232
-> x = 4 , y = 54.5981500331
-> x = 5 , y = 148.413159103
-> x = 6 , y = 403.428793493
-> x = 7 , y = 1096.63315843
-> x = 8 , y = 2980.95798704
-> x = 9 , y = 8103.08392758
So you can control the allowed functions that should be used by the eval method, and have a sandbox environment that can evaluate expressions.
This is what we used in a previous project I worked in. We used Python expressions in custom Eclipse IDE plug-ins, using Jython to run in the JVM. You could do the same with IronPython to run in the CLR.
The examples I used in part inspired / copied from the Lybniz project explanation on how to run a safe Python eval environment. Read it for more details!
You might want to look at Regular-Expressions or RegEx. It's proven and been around for a long time. There's a regex library all the major programming/script languages out there.
Libraries:
C++: what regex library should I use?
C# Regex Class
Usage
Regex Email validation
Regex to validate date format dd/mm/yyyy
I am trying to run a granger causality test on two currency pairs but I seem to get this error message in Shell whenever I try and test it. Can anyone please advise?
I am very new to programming and need this to run an analysis for my project. In shell, I am putting -
import ats15
ats15.grangertest('EURUSD', 'EURGBP', 8)
What is going wrong? I have copied the script below.
Thanks in advance.
Heading ##def grangertest(Y,X,maxlag):
"""
Performs a Granger causality test on variables (vectors) Y and X.
The null hypothese is: Does X cause Y ?
Returned value: pvalue, F, df1, df2
"""
# Create linear model involving Y lags only.
n = len(Y)
if n != len(X):
raise ValueError, "grangertest: incompatible Y,X vectors"
M = [ [0] * maxlag for i in range(n-maxlag)]
for i in range(maxlag, n):
for j in range(1, maxlag+1):
M[i-maxlag][j-1] = Y[i-j]
fit = ols(M, Y[maxlag:])
RSSr = fit.RSS
# Create linear model including X lags.
for i in range(maxlag, n):
xlagged = [X[i-j] for j in range(1, maxlag+1)]
M[i-maxlag].extend(xlagged)
fit = ols(M, Y[maxlag:])
RSSu = fit.RSS
df1 = maxlag
df2 = n - 2 * maxlag - 1
F = ((RSSr - RSSu)/df1)/(RSSu/df2)
pvalue = 1.0 - stats.f.cdf(F,df1,df2)
return pvalue, F, df1, df2, RSSr, RSSu
You didn't post the full traceback, but this error message:
TypeError: unsupported operand type(s) for -: 'str' and 'int'
means what it says. There's an operand - -- the subtraction operator -- and it doesn't know how to handle subtracting an integer from a string. Why would strings be involved? Well, you're calling the function with:
ats15.grangertest('EURUSD', 'EURGBP', 8)
and so you're giving grangertest two strings and an integer. But it seems like grangertest expects
def grangertest(Y,X,maxlag):
two sequences (lists, arrays, whatever) of numbers to use as Y and X, not strings. If EURUSD and EURGBP are names you've given to lists beforehand, then you don't need the quotes:
ats15.grangertest(EURUSD, EURGBP, 8)
but if not, then you should pass grangertest the lists under whatever name you've called them.
The input to the grangertest function must be two lists of numbers. grangertest doesn't know anything about currencies, so passing it currency strings won't work.
You have to fetch the exchange rate data somehow so that you can pass it to grangertest. If EURUSD and EURGBP are variables, then you don't put quotes around them when you pass them to a function (e.g. ats15.grangertest(EURUSD, EURGBP, 8)).
Where can I find a list of Scala's "magic" functions, such as apply, unapply, update, +=, etc.?
By magic-functions I mean functions which are used by some syntactic sugar of the compiler, for example
o.update(x,y) <=> o(x) = y
I googled for some combination of scala magic and synonyms of functions, but I didn't find anything.
I'm not interested with the usage of magic functions in the standard library, but in which magic functions exists.
As far as I know:
Getters/setters related:
apply
update
identifier_=
Pattern matching:
unapply
unapplySeq
For-comprehensions:
map
flatMap
filter
withFilter
foreach
Prefixed operators:
unary_+
unary_-
unary_!
unary_~
Beyond that, any implicit from A to B. Scala will also convert A <op>= B into A = A <op> B, if the former operator isn't defined, "op" is not alphanumeric, and <op>= isn't !=, ==, <= or >=.
And I don't believe there's any single place where all of Scala's syntactic sugars are listed.
In addition to update and apply, there are also a number of unary operators which (I believe) qualify as magical:
unary_+
unary_-
unary_!
unary_~
Add to that the regular infix/suffix operators (which can be almost anything) and you've got yourself the complete package.
You really should take a look at the Scala Language Specification. It is the only authoritative source on this stuff. It's not that hard to read (as long as you're comfortable with context-free grammars), and very easily searchable. The only thing it doesn't specify well is the XML support.
Sorry if it's not exactly answering your question, but my favorite WTF moment so far is # as assignment operator inside pattern match. Thanks to soft copy of "Programming in Scala" I found out what it was pretty quickly.
Using # we can bind any part of a pattern to a variable, and if the pattern match succeeds, the variable will capture the value of the sub-pattern. Here's the example from Programming in Scala (Section 15.2 - Variable Binding):
expr match {
case UnOp("abs", e # UnOp("abs", _)) => e
case _ =>
}
If the entire pattern match succeeds,
then the portion that matched the
UnOp("abs", _) part is made available
as variable e.
And here's what Programming Scala says about it.
That link no longer works. Here is one that does.
I'll also add _* for pattern matching on an arbitrary number of parameters like
case x: A(_*)
And operator associativity rule, from Odersky-Spoon-Venners book:
The associativity of an operator in Scala is determined by its last
character. As mentioned on <...>, any method that ends
in a ‘:’ character is invoked on its right operand, passing in the
left operand. Methods that end in any other character are the other
way around. They are invoked on their left operand, passing in the
right operand. So a * b yields a.*(b), but a ::: b yields b.:::(a).
Maybe we should also mention syntactic desugaring of for expressions which can be found here
And (of course!), alternative syntax for pairs
a -> b //converted to (a, b), where a and b are instances
(as correctly pointed out, this one is just an implicit conversion done through a library, so it's probably not eligible, but I find it's a common puzzler for newcomers)
I'd like to add that there is also a "magic" trait - scala.Dynamic:
A marker trait that enables dynamic invocations. Instances x of this trait allow method invocations x.meth(args) for arbitrary method names meth and argument lists args as well as field accesses x.field for arbitrary field names field.
If a call is not natively supported by x (i.e. if type checking fails), it is rewritten according to the following rules:
foo.method("blah") ~~> foo.applyDynamic("method")("blah")
foo.method(x = "blah") ~~> foo.applyDynamicNamed("method")(("x", "blah"))
foo.method(x = 1, 2) ~~> foo.applyDynamicNamed("method")(("x", 1), ("", 2))
foo.field ~~> foo.selectDynamic("field")
foo.varia = 10 ~~> foo.updateDynamic("varia")(10)
foo.arr(10) = 13 ~~> foo.selectDynamic("arr").update(10, 13)
foo.arr(10) ~~> foo.applyDynamic("arr")(10)
As of Scala 2.10, defining direct or indirect subclasses of this trait is only possible if the language feature dynamics is enabled.
So you can do stuff like
import scala.language.dynamics
object Dyn extends Dynamic {
def applyDynamic(name: String)(a1: Int, a2: String) {
println("Invoked " + name + " on (" + a1 + "," + a2 + ")");
}
}
Dyn.foo(3, "x");
Dyn.bar(3, "y");
They are defined in the Scala Language Specification.
As far as I know, there are just three "magic" functions as you mentioned.
Scalas Getter and Setter may also relate to your "magic":
scala> class Magic {
| private var x :Int = _
| override def toString = "Magic(%d)".format(x)
| def member = x
| def member_=(m :Int){ x = m }
| }
defined class Magic
scala> val m = new Magic
m: Magic = Magic(0)
scala> m.member
res14: Int = 0
scala> m.member = 100
scala> m
res15: Magic = Magic(100)
scala> m.member += 99
scala> m
res17: Magic = Magic(199)