Eliminate hyphen from AppleScript's text item delimiters - applescript

Question

I am having problems separating words from each other when it comes to equations, because I can't separate the equation into two parts if there is a negative variable involved.
set function to "-3x"
return word 1 of function
that would return "3x", because a hyphen is a text item delimiter, but I want it to return "-3x". Is there any way to remove the hyphen from the text item delimiters or any other way to include the hyphen into the string?

Answer 1

To give an idea, here's a simple tokenizer for a very simple Lisp-like language:
-- token types
property StartList : "START"
property EndList : "END"
property ANumber : "NUMBER"
property AWord : "WORD"
-- recognized token chars
property _startlist : "("
property _endlist : ")"
property _number : "+-.1234567890"
property _word : "abcdefghijklmnopqrstuvwxyz"
property _whitespace : space & tab & linefeed & return
to tokenizeCode(theCode)
considering diacriticals, hyphens, punctuation and white space but ignoring case and numeric strings
set i to 1
set l to theCode's length
set tokensList to {}
repeat while i ≤ l
set c to character i of theCode
if c is _startlist then
set end of tokensList to {tokenType:StartList, tokenText:c}
set i to i + 1
else if c is _endlist then
set end of tokensList to {tokenType:EndList, tokenText:c}
set i to i + 1
else if c is in _number then
set tokenText to ""
repeat while character i of theCode is in _number and i ≤ l
set tokenText to tokenText & character i of theCode
set i to i + 1
end repeat
set end of tokensList to {tokenType:ANumber, tokenText:tokenText}
else if c is in _word then
set tokenText to ""
repeat while character i of theCode is in _word and i ≤ l
set tokenText to tokenText & character i of theCode
set i to i + 1
end repeat
set end of tokensList to {tokenType:AWord, tokenText:tokenText}
else if c is in _whitespace then -- skip over white space
repeat while character i of theCode is in _whitespace and i ≤ l
set i to i + 1
end repeat
else
error "Unknown character: '" & c & "'"
end if
end repeat
return tokensList
end considering
end tokenizeCode
The syntax rules for this language are as follows:
A number expression contains one or more digits, "+" or "-" signs, and/or decimal point. (The above code currently doesn't check that the token is a valid number, e.g. it'll happily accept nonsensical input like "0.1.2-3+", but that's easy enough to add.)
A word expression contains one or more characters (a-z).
A list expression begins with a "(" and ends with a ")". The first token in a list expression must be the name of the operator to apply; this may be followed by zero or more additional expressions representing its operands.
Any unrecognized characters are treated as an error.
For example, let's use it to tokenize the mathematical expression "3 + (2.5 * -2)", which in prefix notation is written like this:
set programText to "(add 3 (multiply 2.5 -2))"
set programTokens to tokenizeCode(programText)
--> {{tokenType:"START", tokenText:"("},
{tokenType:"WORD", tokenText:"add"},
{tokenType:"NUMBER", tokenText:"3"},
{tokenType:"START", tokenText:"("},
{tokenType:"WORD", tokenText:"multiply"},
{tokenType:"NUMBER", tokenText:"2.5"},
{tokenType:"NUMBER", tokenText:"-2"},
{tokenType:"END", tokenText:")"},
{tokenType:"END", tokenText:")"}}
Once the text is split up into a list of tokens, the next step is to feed that list into a parser which assembles it into an abstract syntax tree which fully describes the structure of the program.
Like I say, there's a bit of a learning curve to this stuff, but you can write it in your sleep once you've grasped the basic principles. Ask and I'll add an example of how to parse these tokens into usable form later.

Answer 2

Following on from before, here's a parser that turns the tokenizer's output into a tree-based data structure that describes the program's logic.
-- token types
property StartList : "START"
property EndList : "END"
property ANumber : "NUMBER"
property AWord : "WORD"
-------
-- handlers called by Parser to construct Abstract Syntax Tree nodes,
-- simplified here for demonstration purposes
to makeOperation(operatorName, operandsList)
return {operatorName:operatorName, operandsList:operandsList}
end makeOperation
to makeWord(wordText)
return wordText
end makeWord
to makeNumber(numberText)
return numberText as number
end makeNumber
-------
-- Parser
to makeParser(programTokens)
script ProgramParser
property currentToken : missing value
to advanceToNextToken()
if programTokens is {} then error "Found unexpected end of program after '" & currentToken & "'."
set currentToken to first item of programTokens
set programTokens to rest of programTokens
return
end advanceToNextToken
--
to parseOperation() -- parses an '(OPERATOR [OPERANDS ...])' list expression
advanceToNextToken()
if currentToken's tokenType is AWord then -- parse 'OPERATOR'
set operatorName to currentToken's tokenText
set operandsList to {}
advanceToNextToken()
repeat while currentToken's tokenType is not EndList -- parse 'OPERAND(S)'
if currentToken's tokenType is StartList then
set end of operandsList to parseOperation()
else if currentToken's tokenType is AWord then
set end of operandsList to makeWord(currentToken's tokenText)
else if currentToken's tokenType is ANumber then
set end of operandsList to makeNumber(currentToken's tokenText)
else
error "Expected word, number, or list expression but found '" & currentToken's tokenText & "' instead."
end if
advanceToNextToken()
end repeat
return makeOperation(operatorName, operandsList)
else
error "Expected operator name but found '" & currentToken's tokenText & "' instead."
end if
end parseOperation
to parseProgram() -- parses the entire program
advanceToNextToken()
if currentToken's tokenType is StartList then
return parseOperation()
else
error "Found unexpected '" & currentToken's tokenText & "' at start of program."
end if
end parseProgram
end script
end makeParser
-------
-- parse the tokens list produced by the tokenizer into an Abstract Syntax Tree
set programTokens to {{tokenType:"START", tokenText:"("}, ¬
{tokenType:"WORD", tokenText:"add"}, ¬
{tokenType:"NUMBER", tokenText:"3"}, ¬
{tokenType:"START", tokenText:"("}, ¬
{tokenType:"WORD", tokenText:"multiply"}, ¬
{tokenType:"NUMBER", tokenText:"2.5"}, ¬
{tokenType:"NUMBER", tokenText:"-2"}, ¬
{tokenType:"END", tokenText:")"}, ¬
{tokenType:"END", tokenText:")"}}
set parserObject to makeParser(programTokens)
set abstractSyntaxTree to parserObject's parseProgram()
--> {operatorName:"add", operandsList:{3, {operatorName:"multiply", operandsList:{2.5, -2}}}}
The ProgramParser object is a very, very simple recursive descent parser, a collection of handlers, each of which knows how to turn a sequence of tokens into a specific data structure. In fact, the Lisp-y syntax used here is so simple it really only requires two handlers: parseProgram, which gets everything underway, and parseOperation, which knows how to read the tokens that make up a (OPERATOR_NAME [OPERAND1 OPERAND2 ...]) list and turn it into a record that describes a single operation (add, multiply, etc) to be performed.
The nice thing about an AST, especially a very simple regular one like this, is you can manipulate it as data in its own right. For instance, given the program (multiply x y) and a definition of y = (add x 1), you could walk the AST and replace any mention of y with its definition, in this case giving (multiply x (add x 1)). i.e. You can not only do arithmetic calculations (algorithmic programming), but algebraic manipulations (symbolic programming) too. That's a bit heady for here, but I'll see about knocking together a simple arithmetical evaluator for later.

Answer 3

To finish off, here's a simple evaluator for the parser's output:
to makeOperation(operatorName, operandsList)
if operatorName is "add" then
script AddOperationNode
to eval(env)
if operandsList's length ≠ 2 then error "Wrong number of operands."
return ((operandsList's item 1)'s eval(env)) + ((operandsList's item 2)'s eval(env))
end eval
end script
else if operatorName is "multiply" then
script MultiplyOperationNode
to eval(env)
if operandsList's length ≠ 2 then error "Wrong number of operands."
return ((operandsList's item 1)'s eval(env)) * ((operandsList's item 2)'s eval(env))
end eval
end script
-- define more operations here as needed...
else
error "Unknown operator: '" & operatorName & "'"
end if
end makeOperation
to makeWord(wordText)
script WordNode
to eval(env)
return env's getValue(wordText)'s eval(env)
end eval
end script
end makeWord
to makeNumber(numberText)
script NumberNode
to eval(env)
return numberText as number
end eval
end script
end makeNumber
to makeEnvironment()
script EnvironmentObject
property _storedValues : {}
--
to setValue(theKey, theValue)
-- theKey : text
-- theValue : script
repeat with aRef in _storedValues
if aRef's k is theKey then
set aRef's v to theValue
return
end if
end repeat
set end of _storedValues to {k:theKey, v:theValue}
return
end setValue
--
to getValue(theKey)
repeat with aRef in _storedValues
if aRef's k is theKey then return aRef's v
end repeat
error "'" & theKey & "' is undefined." number -1728
end getValue
--
end script
end makeEnvironment
to runProgram(programText, theEnvironment)
set programTokens to tokenizeCode(programText)
set abstractSyntaxTree to makeParser(programTokens)'s parseProgram()
return abstractSyntaxTree's eval(theEnvironment)
end runProgram
This replaces the make... handlers used to test the parser with new handlers that construct full-blown objects representing each type of structure that can make up an Abstract Syntax Tree: numbers, words, and operations. Each object defines an eval handler that knows how to evaluate that particular structure: in a NumberNode it simply returns the number, in a WordNode it retrieves and evaluates the structure stored under that name, in an AddOperationNode it evaluates each operand then sums them, and so on.
For example, to evaluate our original 3 + 2.5 * -2 program:
set theEnvironment to makeEnvironment()
runProgram("(add 3 (multiply 2.5 -2))", theEnvironment)
--> -2.0
In addition, an EnvironmentObject is used to store named values. For example, to store a value named "x" for use by a program:
set theEnvironment to makeEnvironment()
theEnvironment's setValue("x", makeNumber(5))
runProgram("(add 3 x)", theEnvironment)
--> 8
Obviously this will need a bit more work to make it into a proper calculator: a full set of operator definitions, better error reporting, and so on. Plus you'll probably want to replace the parenthesized prefix syntax with a more familiar infix syntax, for which you'll need something like a Pratt parser that can handle precedence, association, etc. But once you've got the basics working it's just a matter of reading up on the various techniques and making changes and improvements one by one until you arrive at the desired solution. HTH.

Answer 4

You can write a calculator in AppleScript if you wish to, but you need to do it as you would in any other language: 1. using a tokenizer to split the input text into a list of tokens, 2. feeding those tokens to a parser which assembles them into an abstract syntax tree, and 3. evaluating that tree to produce a result.
For what you're doing, you could probably write your tokenizer as a regular expression (assuming you don't mind dipping down to NSRegularExpression via the AppleScript-ObjC bridge). For parsing, I recommend reading up on Pratt parsers, which are easy to implement yet powerful enough to support prefix, infix, and posfix operators and operator precedence. For evaluation, a simple recursive AST walking algorithm may well be sufficient, but one step at a time.
These are all well-solved problems, so you won't have any trouble finding tutorials and other online information on how to do it. (Lots of crap, of course, so be prepared to spend some time figuring out how tell the good from the bad.)
Your one problem is that you none will be written specifically for AppleScript, so be prepared to spelunk material written around other languages (Python, Java, etc, etc) and translate from that to AS yourself. That'll require some effort and patience wading through all the programmer-speak, but is eminently doable (I originally cut my teeth on AppleScript and now write my own automation scripting languages) and a great learning exercise for developing your skills.