How to define /#-like operator - syntax

I would like to define a new operator of the form x /==> y, where
the operator /==> is treated as e.g. the /# operator of Map, and
is translated to MyFunction[x, y]. There is one important aspect: I
want the resulting operator to behave in the frontend like any two-bit
operator does, that is, the two characters (a Divide and a
DoubleLongRightArrow) should be connected together, no syntax
coloration should appear, and they are to be selected together when
clicked, so precedence must be set. Also, I'd rather avoid using the
Notation` package. As a result, I'd like to see something like this:
In[11]:= FullForm[x/\[DoubleLongRightArrow]y]
Out[11]//FullForm= MyFunction[x,y]
Does anyone have an idea how to achieve this?

The Notation Package is perhaps the closest to doing this kind of thing, but according to the response to my own question of a similar nature, what you want is unfortunately not practical.
Don't let this stop you from trying however, as you will probably learn new things in the process. The Notation Package and the the functions that underpin it are far from useless.
You may also find the replies to this question informative.
There are a number of functions that are useful for manual implementation of syntax changes. Rather than try to write my own help file for these, I will direct you to the official pages on these functions. After reading them, please ask any focused questions you have, or for help with implementing specific ideas. I or others here should be able to either answer your question, show you how to do something, or explain why it is not readily possible.
The index page on Textual Input and Output.
MakeBoxes, and MakeExpression, and an example of their use.
PreRead
More drastically, one might use CellEvaluationFunction which can be used to do unusual things.
There are more, and I will try to extend this list later. (others are welcome to edit this post)

Thanks to Mr.Wizard's links, I've found the only example in the documentation on how to parse new operators (the gplus example in Low-Level Input). According to this example, here is my version for the new operator PerArrow. Please comment/critize on the code below:
In[1]:= PerArrow /: MakeBoxes[PerArrow[x_, y_], StandardForm] :=
RowBox[{MakeBoxes[x, StandardForm],
RowBox[{AdjustmentBox["/", BoxMargins -> -.2],
AdjustmentBox["\[DoubleLongRightArrow]", BoxMargins -> -.1]}],
MakeBoxes[y, StandardForm]}];
MakeExpression[
RowBox[{x_, "/", RowBox[{"\[DoubleLongRightArrow]", y_}]}],
StandardForm] :=
MakeExpression[RowBox[{"PerArrow", "[", x, ",", y, "]"}],
StandardForm];
In[3]:= PerArrow[x, y]
Out[3]= x /\[DoubleLongRightArrow] y
In[4]:= x /\[DoubleLongRightArrow]y
Out[4]= x /\[DoubleLongRightArrow] y
In[5]:= FullForm[x /\[DoubleLongRightArrow]y]
Out[5]//FullForm= \!\(\*
TagBox[
StyleBox[
RowBox[{"PerArrow", "[",
RowBox[{"x", ",", "y"}], "]"}],
ShowSpecialCharacters->False,
ShowStringCharacters->True,
NumberMarks->True],
FullForm]\)
For sake of clarity, here is a screenshot as well:
Since the operator is not fully integrated, further concerns are:
the operator is selected weird when clicked (DoubleLongRightArrow with y instead of with /).
accordingly, the parsing part requires the DoubleLongRightArrow to be RowBox-ed with y, otherwise it yields syntax error
syntax coloration (at In[4] and In[5])
it prints weird if inputted directly (notice the large gaps at In[4] and In[5])
Now, I can live with these, though it would be nice to have some means to iron out all the minor issues. I guess all these boil down to basically some even lower-level syntax handler, which does not now how to group the new operator. Any idea on how to tackle these? I understand that Cell has a multitude of options which might come handy (like CellEvaluationFunction, ShowAutoStyles and InputAutoReplacements) though I'm again clueless here.

Related

What is the best way to read the code of already defined function (especially from the System` context)?

Occasionally we like to look into how certain System` functions are defined (when they're written in Mathematica). This question is about the best way to do that.
Points to keep in mind:
Of couse ReadProtected needs to be removed first.
Builtins usually need to be used at least once before they get loaded into the kernel. Is a single simple invocation usually sufficient for this when they have extended functionality (e.g. through options)?
Information (??) gives the definition in a hard-to-read format (no indentation, and all private context names prepended). What is the best way to get rid of the context names, and get formatted code?
One idea for getting rid of certain contexts is Block[{$ContextPath = Append[$ContextPath, "SomeContext`Private`"], Information[symbol]]. Code can be auto-formatted using Workbench. Some issues remain, e.g. Information doesn't quote strings, preventing the code from being able to be copied into Workbench.
Generally, I'm interested in how people do this, what methods they use to make the code of builtins as easy to read as possible.
Use case: For example, recently I digged into the code of RunThrough when I found out that it simply doesn't work on Windows XP (turns out it fails to quote the names of temp files when the path to them contains spaces).
Update: It appears that there used to be a function for printing definitions without context prepended, Developer`ContextFreeForm, but it's not working any more in newer versions.
Regarding the pretty-printing: the following is a very schematic code which builds on the answer of #Mr.Wizard to show that a few simple rules can go a long way towards improving the readability of the code:
Internal`InheritedBlock[{RunThrough},
Unprotect[RunThrough];
ClearAttributes[RunThrough, ReadProtected];
Block[{$ContextPath = Append[$ContextPath, "System`Dump`"]},
With[{boxes = ToBoxes# DownValues[RunThrough]},
CellPrint[Cell[BoxData[#], "Input"]] &[
boxes /.
f_[left___, "\[RuleDelayed]", right___] :>
f[left, "\[RuleDelayed]", "\n", right] //.
{
RowBox[{left___, ";", next : Except["\n"], right___}] :>
RowBox[{left, ";", "\n", "\t", next, right}],
RowBox[{sc : ("Block" | "Module" | "With"), "[",
RowBox[{vars_, ",", body_}], "]"}] :>
RowBox[{sc, "[", RowBox[{vars, ",", "\n\t", body}], "]"}]
}]]]]
This is for sure not a general solution (in particular it won't work well on deeply nested functional code without many separate statements), but I am sure it can be improved and generalized without too much trouble to cover many cases of interest.
Good question, because I don't think I have seen this discussed yet.
I do essentially the same thing you outlined. You can get a somewhat different print-out with Definition, and more information with FullDefinition:
Unprotect[RunThrough];
ClearAttributes[RunThrough, ReadProtected]
Block[{$ContextPath = Append[$ContextPath, "System`Dump`"]},
Print # FullDefinition # RunThrough
]

Times and NonCommutativeMultiply, handing the difference automatically

I've got some symbols which should are non-commutative, but I don't want to have to remember which expressions have this behaviour whilst constructing equations.
I've had the thought to use MakeExpression to act on the raw boxes, and automatically uplift multiply to non-commutative multiply when appropriate (for instance when some of the symbols are non-commutative objects).
I was wondering whether anyone had any experience with this kind of configuration.
Here's what I've got so far:
(* Detect whether a set of row boxes represents a multiplication *)
Clear[isRowBoxMultiply];
isRowBoxMultiply[x_RowBox] := (Print["rowbox: ", x];
Head[ToExpression[x]] === Times)
isRowBoxMultiply[x___] := (Print["non-rowbox: ", x]; False)
(* Hook into the expression maker, so that we can capture any \
expression of the form F[x___], to see how it is composed of boxes, \
and return true or false on that basis *)
MakeExpression[
RowBox[List["F", "[", x___, "]"]], _] := (HoldComplete[
isRowBoxMultiply[x]])
(* Test a number of expressions to see whether they are automatically \
detected as multiplies or not. *)
F[a]
F[a b]
F[a*b]
F[a - b]
F[3 x]
F[x^2]
F[e f*g ** h*i j]
Clear[MakeExpression]
This appears to correctly identify expressions that are multiplication statements:
During evaluation of In[561]:= non-rowbox: a
Out[565]= False
During evaluation of In[561]:= rowbox: RowBox[{a,b}]
Out[566]= True
During evaluation of In[561]:= rowbox: RowBox[{a,*,b}]
Out[567]= True
During evaluation of In[561]:= rowbox: RowBox[{a,-,b}]
Out[568]= False
During evaluation of In[561]:= rowbox: RowBox[{3,x}]
Out[569]= True
During evaluation of In[561]:= non-rowbox: SuperscriptBox[x,2]
Out[570]= False
During evaluation of In[561]:= rowbox: RowBox[{e,f,*,RowBox[{g,**,h}],*,i,j}]
Out[571]= True
So, it looks like it's not out of the questions that I might be able to conditionally rewrite the boxes of the underlying expression; but how to do this reliably?
Take the expression RowBox[{"e","f","*",RowBox[{"g","**","h"}],"*","i","j"}], this would need to be rewritten as RowBox[{"e","**","f","**",RowBox[{"g","**","h"}],"**","i","**","j"}] which seems like a non trivial operation to do with the pattern matcher and a rule set.
I'd be grateful for any suggestions from those more experienced with me.
I'm trying to find a way of doing this without altering the default behaviour and ordering of multiply.
Thanks! :)
Joe
This is not a most direct answer to your question, but for many purposes working as low-level as directly with the boxes might be an overkill. Here is an alternative: let the Mathematica parser parse your code, and make a change then. Here is a possibility:
ClearAll[withNoncommutativeMultiply];
SetAttributes[withNoncommutativeMultiply, HoldAll];
withNoncommutativeMultiply[code_] :=
Internal`InheritedBlock[{Times},
Unprotect[Times];
Times = NonCommutativeMultiply;
Protect[Times];
code];
This replaces Times dynamically with NonCommutativeMultiply, and avoids the intricacies you mentioned. By using Internal`InheritedBlock, I make modifications to Times local to the code executed inside withNoncommutativeMultiply.
You now can automate the application of this function with $Pre:
$Pre = withNoncommutativeMultiply;
Now, for example:
In[36]:=
F[a]
F[a b]
F[a*b]
F[a-b]
F[3 x]
F[x^2]
F[e f*g**h*i j]
Out[36]= F[a]
Out[37]= F[a**b]
Out[38]= F[a**b]
Out[39]= F[a+(-1)**b]
Out[40]= F[3**x]
Out[41]= F[x^2]
Out[42]= F[e**f**g**h**i**j]
Surely, using $Pre in such manner is hardly appropriate, since in all your code multiplication will be replaced with noncommutative multiplication - I used this as an illustration. You could make a more complicated redefinition of Times, so that this would only work for certain symbols.
Here is a safer alternative based on lexical, rather than dynamic, scoping:
ClearAll[withNoncommutativeMultiplyLex];
SetAttributes[withNoncommutativeMultiplyLex, HoldAll];
withNoncommutativeMultiplyLex[code_] :=
With ## Append[
Hold[{Times = NonCommutativeMultiply}],
Unevaluated[code]]
you can use this in the same way, but only those instances of Times which are explicitly present in the code would be replaced. Again, this is just an illustration of the principles, one can extend or specialize this as needed. Instead of With, which is rather limited in its ability to specialize / add special cases, one can use replacement rules which have similar semantics.
If I understand correctly, you want to input
a b and a*b
and have MMA understand automatically that Times is really a non commutative operator (which has its own -separate - commutation rules).
Well, my suggestion is that you use the Notation package.
It is very powerful and (relatively) easy to use (especially for a sophisticated user like you seem to be).
It can be used programmatically and it can reinterpret predefined symbols like Times.
Basically it can intercept Times and change it to MyTimes. You then write code for MyTimes deciding for example which symbols are non commuting and then the output can be pretty formatted again as times or whatever else you wish.
The input and output processing are 2 lines of code. That’s it!
You have to read the documentation carefully and do some experimentation, if what you want is not more or less “standard hacking” of the input-output jobs.
Your case seems to me pretty much standard (again: If I understood well what you want to achieve) and you should find useful to read the “advanced” pages of the Notation package.
To give you an idea of how powerful and flexible the package is, I am using it to write the input-output formatting of a sizable package of Category Theory where noncommutative operations abound. But wait! I am not just defining ONE noncommutative operation, I am defining an unlimited number of noncommutative operations.
Another thing I did was to reinterpret Power when the arguments are categories, without overloading Power. This allows me to treat functorial categories using standard mathematics notation.
Now my “infinite” operations and "super Power" have the same look and feel of standard MMA symbols, including copy-paste functionality.
So, this doesn't directly answer the question, but it's does provide the sort of implementation that I was thinking about.
So, after a bit of investigation and taking on board some of #LeonidShifrin's suggestions, I've managed to implement most of what I was thinking of. The idea is that it's possible to define patterns that should be considered to be non-commuting quantities, using commutingQ[form] := False. Then any multiplicative expression (actually any expression) can be wrapped with withCommutativeSensitivity[expr] and the expression will be manipulated to separate the quantities into Times[] and NonCommutativeMultiply[] sub-expressions as appropriate,
In[1]:= commutingQ[b] ^:= False;
In[2]:= withCommutativeSensitivity[ a (a + b + 4) b (3 + a) b ]
Out[1]:= a (3 + a) (a + b + 4) ** b ** b
Of course it's possible to use $Pre = withCommutativeSensitivity to have this behaviour become default (come on Wolfram! Make it default already ;) ). It would, however, be nice to have it a more fundamental behaviour though. I'd really like to make a module and Needs[NonCommutativeQuantities] at the beginning of any note book that is needs it, and not have all the facilities that use $Pre break on me (doesn't tracing use it?).
Intuitively I feel that there must be a natural way to hook this functionality into Mathematica on at the level of box parsing and wire it up using MakeExpression[]. Am I over extending here? I'd appreciate any thoughts as to whether I'm chasing up a blind alley. (I've had a few experiments in this direction, but always get caught in a recursive definition that I can't work out how to break).
Any thoughts would be gladly received,
Joe.
Code
Unprotect[NonCommutativeMultiply];
ClearAll[NonCommutativeMultiply]
NonCommutativeMultiply[a_] := a
Protect[NonCommutativeMultiply];
ClearAll[commutingQ]
commutingQ::usage = "commutingQ[\!\(\*
StyleBox[\"expr\", \"InlineFormula\",\nFontSlant->\"Italic\"]\)] \
returns True if expr doesn't contain any constituent parts that fail \
the commutingQ test. By default all objects return True to \
commutingQ.";
commutingQ[x_] := If[Length[x] == 0, True, And ## (commutingQ /# List ## x)]
ClearAll[times2, withCommutativeSensitivity]
SetAttributes[times2, {Flat, OneIdentity, HoldAll}]
SetAttributes[withCommutativeSensitivity, HoldAll];
gatherByCriteria[list_List, crit_] :=
With[{gathered =
Gather[{#, crit[#1]} & /# list, #1[[2]] == #2[[2]] &]},
(Identity ## Union[#[[2]]] -> #[[1]] &)[Transpose[#]] & /# gathered]
times2[x__] := Module[{a, b, y = List[x]},
Times ## (gatherByCriteria[y, commutingQ] //.
{True -> Times, False -> NonCommutativeMultiply,
HoldPattern[a_ -> b_] :> a ## b})]
withCommutativeSensitivity[code_] := With ## Append[
Hold[{Times = times2, NonCommutativeMultiply = times2}],
Unevaluated[code]]
This answer does not address your question but rather the problem that leads you to ask that question. Mathematica is pretty useless when dealing with non-commuting objects but since such objects abound in, e.g., particle physics, there are some usefull packages around to deal with the situation.
Look at the grassmanOps package. They have a method to define symbols as either commuting or anti-commuting and overload the standard NonCommutativeMultiply to handle, i.e. pass through, commuting symbols. They also define several other operators, such as Derivative, to handle anti-commuting symbols. It is probably easily adapted to cover arbitrary commutation rules and it should at the very least give you an insigt into what things need to be changed if you want to roll your own.

May I write {x,a,b}//Do[...,#]& instead of Do[...,{x,a,b}]?

I'm in love with Ruby. In this language all core functions are actually methods. That's why I prefer postfix notation – when the data, which I want to process is placed left from the body of anonymous processing function, for example: array.map{...}. I believe, that it has advantages in how easy is this code to read.
But Mathetica, being functional (yeah, it can be procedural if you want) dictates a style, where Function name is placed left from the data. As we can see in its manuals, // is used only when it's some simple Function, without arguments, like list // MatrixForm. When Function needs a lot of arguments, people who wrote manuals, use syntax F[data].
It would be okay, but my problem is the case F[f,data], for example Do[function, {x, a, b}]. Most of Mathematica functions (if not all) have arguments in exactly this order – [function, data], not [data, function]. As I prefer to use pure functions to keep namespace clean instead of creating a lot of named functions in my notebook, the argument function can be too big – so big, that argument data would be placed on the 5-20th line of code after the line with Function call.
This is why sometimes, when evil Ruby nature takes me under control, I rewrite such functions in postfix way:
Because it's important for me, that pure function (potentially big code) is placed right from processing data. Yeah I do it and I'm happy. But there are two things:
this causes Mathematica's highlighting parser problem: the x in postfix notation is highlighted with blue color, not turquoise;
everytime when I look into Mathematica manuals, I see examples like this one: Do[x[[i]] = (v[[i]] - U[[i, i + 1 ;; n]].x[[i + 1 ;; n]])/ U[[i, i]], {i, n, 1, -1}];, which means... hell, they think it's easy to read/support/etc.?!
So these two things made me ask this question here: am I so bad boy, that use my Ruby-style, and should I write code like these guys do, or is it OK, and I don't have to worry, and should write as I like to?
The style you propose is frequently possible, but is inadvisable in the case of Do. The problem is that Do has the attribute HoldAll. This is important because the loop variable (x in the example) must remain unevaluated and be treated as a local variable. To see this, try evaluating these expressions:
x = 123;
Do[Print[x], {x, 1, 2}]
(* prints 1 and 2 *)
{x, 1, 2} // Do[Print[x], #]&
(* error: Do::itraw: Raw object 123 cannot be used as an iterator.
Do[Print[x], {123, 1, 2}]
*)
The error occurs because the pure function Do[Print[x], #]& lacks the HoldAll attribute, causing {x, 1, 2} to be evaluated. You could solve the problem by explicitly defining a pure function with the HoldAll attribute, thus:
{x, 1, 2} // Function[Null, Do[Print[x], #], HoldAll]
... but I suspect that the cure is worse than the disease :)
Thus, when one is using "binding" expressions like Do, Table, Module and so on, it is safest to conform with the herd.
I think you need to learn to use the styles that Mathematica most naturally supports. Certainly there is more than one way, and my code does not look like everyone else's. Nevertheless, if you continue to try to beat Mathematica syntax into your own preconceived style, based on a different language, I foresee nothing but continued frustration for you.
Whitespace is not evil, and you can easily add line breaks to separate long arguments:
Do[
x[[i]] = (v[[i]] - U[[i, i + 1 ;; n]].x[[i + 1 ;; n]]) / U[[i, i]]
, {i, n, 1, -1}
];
This said, I like to write using more prefix (f # x) and infix (x ~ f ~ y) notation that I usually see, and I find this valuable because it is easy to determine that such functions are receiving one and two arguments respectively. This is somewhat nonstandard, but I do not think it is kicking over the traces of Mathematica syntax. Rather, I see it as using the syntax to advantage. Sometimes this causes syntax highlighting to fail, but I can live with that:
f[x] ~Do~ {x, 2, 5}
When using anything besides the standard form of f[x, y, z] (with line breaks as needed), you must be more careful of evaluation order, and IMHO, readability can suffer. Consider this contrived example:
{x, y} // # + 1 & ## # &
I do not find this intuitive. Yes, for someone intimate with Mathematica's order of operations, it is readable, but I believe it does not improve clarity. I tend to reserve // postfix for named functions where reading is natural:
Do[f[x], {x, 10000}] //Timing //First
I'd say it is one of the biggest mistakes to try program in a language B in ways idiomatic for a language A, only because you happen to know the latter well and like it. There is nothing wrong in borrowing idioms, but you have to make sure to understand the second language well enough so that you know why other people use it the way they do.
In the particular case of your example, and generally, I want to draw attention to a few things others did not mention. First, Do is a scoping construct which uses dynamic scoping to localize its iterator symbols. Therefore, you have:
In[4]:=
x=1;
{x,1,5}//Do[f[x],#]&
During evaluation of In[4]:= Do::itraw: Raw object
1 cannot be used as an iterator. >>
Out[5]= Do[f[x],{1,1,5}]
What a surprise, isn't it. This won't happen when you use Do in a standard fashion.
Second, note that, while this fact is largely ignored, f[#]&[arg] is NOT always the same as f[arg]. Example:
ClearAll[f];
SetAttributes[f, HoldAll];
f[x_] := Print[Unevaluated[x]]
f[5^2]
5^2
f[#] &[5^2]
25
This does not affect your example, but your usage is close enough to those cases affected by this, since you manipulate the scopes.
Mathematica supports 4 ways of applying a function to its arguments:
standard function form: f[x]
prefix: f#x or g##{x,y}
postfix: x // f, and
infix: x~g~y which is equivalent to g[x,y].
What form you choose to use is up to you, and is often an aesthetic choice, more than anything else. Internally, f#x is interpreted as f[x]. Personally, I primarily use postfix, like you, because I view each function in the chain as a transformation, and it is easier to string multiple transformations together like that. That said, my code will be littered with both the standard form and prefix form mostly depending on whim, but I tend to use standard form more as it evokes a feeling of containment with regards to the functions parameters.
I took a little liberty with the prefix form, as I included the shorthand form of Apply (##) alongside Prefix (#). Of the built in commands, only the standard form, infix form, and Apply allow you easily pass more than one variable to your function without additional work. Apply (e.g. g ## {x,y}) works by replacing the Head of the expression ({x,y}) with the function, in effect evaluating the function with multiple variables (g##{x,y} == g[x,y]).
The method I use to pass multiple variables to my functions using the postfix form is via lists. This necessitates a little more work as I have to write
{x,y} // f[ #[[1]], #[[2]] ]&
to specify which element of the List corresponds to the appropriate parameter. I tend to do this, but you could combine this with Apply like
{x,y} // f ## #&
which involves less typing, but could be more difficult to interpret when you read it later.
Edit: I should point out that f and g above are just placeholders, they can, and often are, replaced with pure functions, e.g. #+1& # x is mostly equivalent to #+1&[x], see Leonid's answer.
To clarify, per Leonid's answer, the equivalence between f#expr and f[expr] is true if f does not posses an attribute that would prevent the expression, expr, from being evaluated before being passed to f. For instance, one of the Attributes of Do is HoldAll which allows it to act as a scoping construct which allows its parameters to be evaluated internally without undo outside influence. The point is expr will be evaluated prior to it being passed to f, so if you need it to remain unevaluated, extra care must be taken, like creating a pure function with a Hold style attribute.
You can certainly do it, as you evidently know. Personally, I would not worry about how the manuals write code, and just write it the way I find natural and memorable.
However, I have noticed that I usually fall into definite patterns. For instance, if I produce a list after some computation and incidentally plot it to make sure it's what I expected, I usually do
prodListAfterLongComputation[
args,
]//ListPlot[#,PlotRange->Full]&
If I have a list, say lst, and I am now focusing on producing a complicated plot, I'll do
ListPlot[
lst,
Option1->Setting1,
Option2->Setting2
]
So basically, anything that is incidental and perhaps not important to be readable (I don't need to be able to instantaneously parse the first ListPlot as it's not the point of that bit of code) ends up being postfix, to avoid disrupting the already-written complicated code it is applied to. Conversely, complicated code I tend to write in the way I find easiest to parse later, which, in my case, is something like
f[
g[
a,
b,
c
]
]
even though it takes more typing and, if one does not use the Workbench/Eclipse plugin, makes it more work to reorganize code.
So I suppose I'd answer your question with "do whatever is most convenient after taking into account the possible need for readability and the possible loss of convenience such as code highlighting, extra work to refactor code etc".
Of course all this applies if you're the only one working with some piece of code; if there are others, it is a different question alltogether.
But this is just an opinion. I doubt it's possible for anybody to offer more than this.
For one-argument functions (f#(arg)), ((arg)//f) and f[arg] are completely equivalent even in the sense of applying of attributes of f. In the case of multi-argument functions one may write f#Sequence[args] or Sequence[args]//f with the same effect:
In[1]:= SetAttributes[f,HoldAll];
In[2]:= arg1:=Print[];
In[3]:= f#arg1
Out[3]= f[arg1]
In[4]:= f#Sequence[arg1,arg1]
Out[4]= f[arg1,arg1]
So it seems that the solution for anyone who likes postfix notation is to use Sequence:
x=123;
Sequence[Print[x],{x,1,2}]//Do
(* prints 1 and 2 *)
Some difficulties can potentially appear with functions having attribute SequenceHold or HoldAllComplete:
In[18]:= Select[{#, ToExpression[#, InputForm, Attributes]} & /#
Names["System`*"],
MemberQ[#[[2]], SequenceHold | HoldAllComplete] &][[All, 1]]
Out[18]= {"AbsoluteTiming", "DebugTag", "EvaluationObject", \
"HoldComplete", "InterpretationBox", "MakeBoxes", "ParallelEvaluate", \
"ParallelSubmit", "Parenthesize", "PreemptProtect", "Rule", \
"RuleDelayed", "Set", "SetDelayed", "SystemException", "TagSet", \
"TagSetDelayed", "Timing", "Unevaluated", "UpSet", "UpSetDelayed"}

Unwanted evaluation in assignments in Mathematica: why it happens and how to debug it during the package-loading?

I am developing a (large) package which does not load properly anymore.
This happened after I changed a single line of code.
When I attempt to load the package (with Needs), the package starts loading and then one of the setdelayed definitions “comes alive” (ie. Is somehow evaluated), gets trapped in an error trapping routine loaded a few lines before and the package loading aborts.
The error trapping routine with abort is doing its job, except that it should not have been called in the first place, during the package loading phase.
The error message reveals that the wrong argument is in fact a pattern expression which I use on the lhs of a setdelayed definition a few lines later.
Something like this:
……Some code lines
Changed line of code
g[x_?NotGoodQ]:=(Message[g::nogood, x];Abort[])
……..some other code lines
g/: cccQ[g[x0_]]:=True
When I attempt to load the package, I get:
g::nogood: Argument x0_ is not good
As you see the passed argument is a pattern and it can only come from the code line above.
I tried to find the reason for this behavior, but I have been unsuccessful so far.
So I decided to use the powerful Workbench debugging tools .
I would like to see step by step (or with breakpoints) what happens when I load the package.
I am not yet too familiar with WB, but it seems that ,using Debug as…, the package is first loaded and then eventually debugged with breakpoints, ect.
My problem is that the package does not even load completely! And any breakpoint set before loading the package does not seem to be effective.
So…2 questions:
can anybody please explain why these code lines "come alive" during package loading? (there are no obvious syntax errors or code fragments left in the package as far as I can see)
can anybody please explain how (if) is possible to examine/debug
package code while being loaded in WB?
Thank you for any help.
Edit
In light of Leonid's answer and using his EvenQ example:
We can avoid using Holdpattern simply by definying upvalues for g BEFORE downvalues for g
notGoodQ[x_] := EvenQ[x];
Clear[g];
g /: cccQ[g[x0_]] := True
g[x_?notGoodQ] := (Message[g::nogood, x]; Abort[])
Now
?g
Global`g
cccQ[g[x0_]]^:=True
g[x_?notGoodQ]:=(Message[g::nogood,x];Abort[])
In[6]:= cccQ[g[1]]
Out[6]= True
while
In[7]:= cccQ[g[2]]
During evaluation of In[7]:= g::nogood: -- Message text not found -- (2)
Out[7]= $Aborted
So...general rule:
When writing a function g, first define upvalues for g, then define downvalues for g, otherwise use Holdpattern
Can you subscribe to this rule?
Leonid says that using Holdpattern might indicate improvable design. Besides the solution indicated above, how could one improve the design of the little code above or, better, in general when dealing with upvalues?
Thank you for your help
Leaving aside the WB (which is not really needed to answer your question) - the problem seems to have a straightforward answer based only on how expressions are evaluated during assignments. Here is an example:
In[1505]:=
notGoodQ[x_]:=True;
Clear[g];
g[x_?notGoodQ]:=(Message[g::nogood,x];Abort[])
In[1509]:= g/:cccQ[g[x0_]]:=True
During evaluation of In[1509]:= g::nogood: -- Message text not found -- (x0_)
Out[1509]= $Aborted
To make it work, I deliberately made a definition for notGoodQ to always return True. Now, why was g[x0_] evaluated during the assignment through TagSetDelayed? The answer is that, while TagSetDelayed (as well as SetDelayed) in an assignment h/:f[h[elem1,...,elemn]]:=... does not apply any rules that f may have, it will evaluate h[elem1,...,elem2], as well as f. Here is an example:
In[1513]:=
ClearAll[h,f];
h[___]:=Print["Evaluated"];
In[1515]:= h/:f[h[1,2]]:=3
During evaluation of In[1515]:= Evaluated
During evaluation of In[1515]:= TagSetDelayed::tagnf: Tag h not found in f[Null]. >>
Out[1515]= $Failed
The fact that TagSetDelayed is HoldAll does not mean that it does not evaluate its arguments - it only means that the arguments arrive to it unevaluated, and whether or not they will be evaluated depends on the semantics of TagSetDelayed (which I briefly described above). The same holds for SetDelayed, so the commonly used statement that it "does not evaluate its arguments" is not literally correct. A more correct statement is that it receives the arguments unevaluated and does evaluate them in a special way - not evaluate the r.h.s, while for l.h.s., evaluate head and elements but not apply rules for the head. To avoid that, you may wrap things in HoldPattern, like this:
Clear[g,notGoodQ];
notGoodQ[x_]:=EvenQ[x];
g[x_?notGoodQ]:=(Message[g::nogood,x];Abort[])
g/:cccQ[HoldPattern[g[x0_]]]:=True;
This goes through. Here is some usage:
In[1527]:= cccQ[g[1]]
Out[1527]= True
In[1528]:= cccQ[g[2]]
During evaluation of In[1528]:= g::nogood: -- Message text not found -- (2)
Out[1528]= $Aborted
Note however that the need for HoldPattern inside your left-hand side when making a definition is often a sign that the expression inside your head may also evaluate during the function call, which may break your code. Here is an example of what I mean:
In[1532]:=
ClearAll[f,h];
f[x_]:=x^2;
f/:h[HoldPattern[f[y_]]]:=y^4;
This code attempts to catch cases like h[f[something]], but it will obviously fail since f[something] will evaluate before the evaluation comes to h:
In[1535]:= h[f[5]]
Out[1535]= h[25]
For me, the need for HoldPattern on the l.h.s. is a sign that I need to reconsider my design.
EDIT
Regarding debugging during loading in WB, one thing you can do (IIRC, can not check right now) is to use good old print statements, the output of which will appear in the WB's console. Personally, I rarely feel a need for debugger for this purpose (debugging package when loading)
EDIT 2
In response to the edit in the question:
Regarding the order of definitions: yes, you can do this, and it solves this particular problem. But, generally, this isn't robust, and I would not consider it a good general method. It is hard to give a definite advice for a case at hand, since it is a bit out of its context, but it seems to me that the use of UpValues here is unjustified. If this is done for error - handling, there are other ways to do it without using UpValues.
Generally, UpValues are used most commonly to overload some function in a safe way, without adding any rule to the function being overloaded. One advice is to avoid associating UpValues with heads which also have DownValues and may evaluate -by doing this you start playing a game with evaluator, and will eventually lose. The safest is to attach UpValues to inert symbols (heads, containers), which often represent a "type" of objects on which you want to overload a given function.
Regarding my comment on the presence of HoldPattern indicating a bad design. There certainly are legitimate uses for HoldPattern, such as this (somewhat artificial) one:
In[25]:=
Clear[ff,a,b,c];
ff[HoldPattern[Plus[x__]]]:={x};
ff[a+b+c]
Out[27]= {a,b,c}
Here it is justified because in many cases Plus remains unevaluated, and is useful in its unevaluated form - since one can deduce that it represents a sum. We need HoldPattern here because of the way Plus is defined on a single argument, and because a pattern happens to be a single argument (even though it describes generally multiple arguments) during the definition. So, we use HoldPattern here to prevent treating the pattern as normal argument, but this is mostly different from the intended use cases for Plus. Whenever this is the case (we are sure that the definition will work all right for intended use cases), HoldPattern is fine. Note b.t.w., that this example is also fragile:
In[28]:= ff[Plus[a]]
Out[28]= ff[a]
The reason why it is still mostly OK is that normally we don't use Plus on a single argument.
But, there is a second group of cases, where the structure of usually supplied arguments is the same as the structure of patterns used for the definition. In this case, pattern evaluation during the assignment indicates that the same evaluation will happen with actual arguments during the function calls. Your usage falls into this category. My comment for a design flaw was for such cases - you can prevent the pattern from evaluating, but you will have to prevent the arguments from evaluating as well, to make this work. And pattern-matching against not completely evaluated expression is fragile. Also, the function should never assume some extra conditions (beyond what it can type-check) for the arguments.

controlling order of variables in an expression

In Mathematica, how do you change the order of importance of variables? for example: if i enter b+c+a+d, i get a+b+c+d but i want b and d to preceed other variables. so that i get b+d+a+c
note, i'd like to use it where + is non-commutative
First you need to define an ordering function like:
In[1]:= CPOrdering[a]=3;
CPOrdering[b]=1;
CPOrdering[d]=2;
CPOrdering[c]=4;
Although, for more complicated examples, you should probably be smarter about it than this - ie use pattern matching.
Then you can sort expressions using
In[5]:= CirclePlus[a,b,c,d]
SortBy[%,CPOrdering]
Out[5]= a\[CirclePlus]b\[CirclePlus]c\[CirclePlus]d
Out[6]= b\[CirclePlus]d\[CirclePlus]a\[CirclePlus]c
This can then be automated using something like
CPOrdering[a_, b_] := CPOrdering[a] < CPOrdering[b]
CirclePlus[a__] /; (!OrderedQ[{a}, CPOrdering]) := CirclePlus##SortBy[{a}, CPOrdering]
The underlying reason b+c+a+d becomes a+b+c+d in Mathematica is because Plus has the attribute Orderless. In general, a symbol f with attribute Orderless means that the elements of f in an expession f[e1, e2, e3], the elements ei should be sorted into canonical order, and in particular, Mathematica's canonical order equivalent to that of OrderedQ and Ordering.
Orderless is even accounted for during pattern matching:
In[47]:= a+b+c+d /. a+c -> e
Out[47]= b+d+e
It's highly, highly recommended that you do NOT remove the Orderless attribute from Plus because the consequences could be dire for lots of functionality in Mathematica.
As other posters have noted, your best bet is to simply define your own function that is NOT Orderless, and will therefore preserve argument order. You might also find HoldForm useful in very limited circumstances.
Also note that nothing stops you from typesetting symbols in an expression in whatever order you want in a notebook, as long as you don't evaluate-in-place, etc.
So, don't use "+", because Plus[] IS commutative.
Define your own myPlus[x_,y_]:= .... whatever.
If you have an idea of what your new Plus[] should do, post it and we may try to help you with the definition/
HTH!
PS> You may change the definition of Plus[] ... but :)

Resources