Related
I have the following situation:
On[t::shdw]
Message[t::shdw,"t","Foobar","Dork"]
-> t::shdw: Symbol t appears in multiple contexts Foobar; definitions in context Dork may shadow or be shadowed by other definitions. >>
Off[t::shdw]
Message[t::shdw,"t","Foobar","Dork"]
-> **No Warning**
<<MASStoolbox`
-> t::shdw: Symbol t appears in multiple contexts {MASStoolbox`MASS`,Global`}; definitions in context MASStoolbox`MASS` may shadow or be shadowed by other definitions. >>
How can that be? Does On and Off only affect the current Notebook and have no effect when loading packages? If so, how can one get rid of messages like these? How should shadowing errors be treated?
The reason for what you observed is that the statement Off[t::shdw] disables a shadowing message for the symbol t that is currently on the $ContextPath, but not for another t, created during the loading of another package. You should disable shadowing message for that second t, if you don't want to see it. Here is an example:
In[1]:= Off[Test1`a::shdw]
In[2]:=
BeginPackage["Test`"]
a
EndPackage[]
Out[2]= Test`
Out[3]= a
In[5]:= BeginPackage["Test1`"]
a
EndPackage[]
Out[5]= Test1`
Out[6]= a
Here, no message was generated.
Generally, however, I would not want to disable shadowing messages, since shadowing is a serious problem. Shadowing simply means that there is more than one context currently on the symbol search path ($ContextPath), containing symbol(s) with the same (short) name. Basically, if symbol is shadowed, and you refer to it by its short name (that is, symbol without its containing package), you have no guarantee which of the several symbols with the same short name will actually be used. So, the correct way to deal with this is to avoid shadowing altogether, which is always possible, if not always convenient.
The simplest way to avoid shadowing is to load the package you need into the private section of your package, which starts after the line Begin["`Private`"]. You do that by calling Needs[your-package], as always. In this way, the context of the loaded package does not stay on the $ContextPath together with your package's context. The worst case is when you need to load two packages with conflicting symbols. One way out is to construct a wrapper package for one of them, which would load that one privately and rename the conflicting symbols. This may be inconvenient though, since you'd have to rename all functions in a package, even those that don't conflict. Here is an example of a more flexible variant:
First package:
BeginPackage["Test`"];
g[x_?NumericQ] := x^2;
EndPackage[];
Second package:
BeginPackage["Test1`"];
g[x_?NumericQ] := x^3;
EndPackage[]
Main package:
BeginPackage["Main`"];
f::usage = "A test function of a single argument";
Begin["`Private`"];
Block[{$ContextPath = $ContextPath},
Needs["Test`"];
(* Define first delegate private function *)
g1 = Symbol["g"]];
Block[{$ContextPath = $ContextPath},
Needs["Test1`"];
(* Define second delegate private function *)
g2 = Symbol["g"];
];
f[x_] := g1[x]*g2[x]
End[]
EndPackage[]
Now, provided that the system knows where to find your packages:
In[2]:= Needs["Main`"]
In[3]:= f[x]
Out[3]= Test`g[x] Test1`g[x]
In[4]:= f[2]
Out[4]= 32
And we used two conflicting symbols for the same public function. Using Block above served to localize the code where the first or second definition of g is used. Note that there is some inconvenience with this method in that we need to use Symbol["your-symbol-name"], to delay the parsing until run-time.
A much simple method is to just refer to symbols by their long names. In the above, we could just use Test`g and Test1`g and then there is no need for the heavy machinery. This is somewhat less flexible however, since you have to "hard-code" the symbol's context into the code.
You can overload Message to turn it off:
Unprotect#Message;
Message[MessageName[s_, "shdw"],___] := Null;
Protect#Message;
This code also emits a more helpful and correct message, and it verifies its assertions with Assert:
Unprotect#Message;
Message[MessageName[s_, "shdw"],
rest : PatternSequence[snHeld_, pathsHeld_, newpath_]] :=
With[{paths = ReleaseHold#pathsHeld},
{memberpaths = Select[$ContextPath, MemberQ[paths, #] &],
sn = ReleaseHold#snHeld},
Print#StringTemplate[
"Note: Symbol `` added in context ``, so that it now appears in \
multiple contexts: ``. On the context path, it is currently found in: \
``. Definitions in the first context of the context path, ``, will \
shadow any other definitions."][sn, newpath, paths, memberpaths,
First#memberpaths];
(*symbol name*)
Assert[SymbolName#s === sn];
(*demonstrate that First#
memberpaths is the context used for short names*)
Assert[ToExpression#StringTemplate[
"Unevaluated#`` === Unevaluated#````"
][sn, First#memberpaths, sn
]
];
(*and not any other context*)
Assert#AllTrue[TrueQ][ToExpression#StringTemplate[
"Unevaluated#`` =!= Unevaluated#````"
][sn, #, sn
] & /# DeleteCases[paths, First#memberpaths]
];
];
Protect#Message;
Try:
Global`Image
Note: Symbol Image added in context Global`, so that it now appears in multiple contexts: {Global`, System`}. On the context path, it is currently found in: {System`, Global`}. Definitions in the first context of the context path, System`, will shadow any other definitions.
c.f. https://mathematica.stackexchange.com/questions/43381/result-about-context-is-inconsistent-with-the-description-of-power-programming/119887#119887
I was playing around with $NewSymbol trying to find something to answer this question with. The docs say that
$NewSymbol is applied before the
symbol is actually created. If the
action of $NewSymbol causes the symbol
to be created, perhaps in a different
context, then the symbol as created
will be the one used.
So I tried to automatically move a particular new symbol into a test context which should prevent its creation in the Global`* context, but the symbol gets created in both contexts.
In[1]:= Remove["Global`*"]
In[2]:= $NewSymbol=(Print[#1," : ",#2];
If[#1==="aLongTestSymbolName"&==="Global`",
Symbol["TestContext`"<>#1]])&;
In[3]:= x
During evaluation of In[3]:= x : Global`
Out[3]= x
In[4]:= aLongTestSymbolName
During evaluation of In[4]:= aLongTestSymbolName : Global`
During evaluation of In[4]:= aLongTestSymbolName : TestContext`
Out[4]= aLongTestSymbolName
In[5]:= Names["Global`*"]
Out[5]= {aLongTestSymbolName,x}
In[6]:= Names["TestContext`*"]
Out[6]= {TestContext`aLongTestSymbolName}
I believe that "aLongTestSymbolName" should not be in the Global` context.
Can anyone see what I've done wrong or if I've misinterpreted the documentation?
Note: Having the symbol created in both contexts is not an option for the automatic highlighting in the above linked to question. The idea is to reserve certain symbol names such as "x"~~___ for variables and "f"~~___ for functions and then use string patterns in $NewSymbol to move the symbols to the appropriate highlighted context.
That is because you passed the symbol name to Print, which immediately made the symbol in Global`. :-)
Or not. I should really try things before answering, but I thought I knew this one. Oops.
It seems to me now that $NewSymbol does not intercept the creation of a symbol, or if it does, how to make use of that is not clear.
If one uses:
$NewSymbol = Print["Name: ", #2, #] &;
then:
In[10]:= aNewSymbol
During evaluation of In[10]:= Name: Global`aNewSymbol
Out[10]= aNewSymbol
We see that $NewSymbol does not work like $PrePrint in that its output does not become the expression.
Therefore, if we use:
$NewSymbol = Symbol["TestContext`" <> #] &;
aSecondSymbol
aSecondSymbol is merrily created in Global` as though nothing had changed.
If $NewSymbol can be used to direct the context in which the symbol is created, as the documentation states, it is not clear to me how this may be done.
In this Mathematica Journal article about context issues you may find that just the presence of a new symbol in the parsing phase of the evaluation will add this symbol to the current context. In this case aLongTestSymbolName is handed to the Print and If as #1 and is therefore created in the current context Global`. I don't think there is anything you can do in the $NewSymbol function to prevent this.
The article mentions that even if you use Begin["Context1`"]; someSymbol; End[] someSymbol is not placed in the Context1` context unless Begin["Context1`"] is evaluated on a separate line.
Old habits die hard, and I realise that I have been using opts___Rule pattern-matching and constructs like thisoption /. {opts} /. Options[myfunction] in the very large package that I'm currently developing. Sal Manango's "Mathematica Cookbook" reminds me that the post-version-6 way of doing this is opts:OptionsPattern[] and OptionValue[thisoption]. The package requires version 8 anyway, but I had just never changed the way I wrote this kind of code over the years.
Is it worth refactoring all that from my pre-version-6 way of doing things? Are there performance or other benefits?
Regards
Verbeia
EDIT: Summary
A lot of good points were made in response to this question, so thank you (and plus one, of course) all. To summarise, yes, I should refactor to use OptionsPattern and OptionValue. (NB: OptionsPattern not OptionPattern as I had it before!) There are a number of reasons why:
It's a touch faster (#Sasha)
It better handles functions where the arguments must be in HoldForm (#Leonid)
OptionsPattern automatically checks that you are passing a valid option to that function (FilterRules would still be needed if you are passing to a different function (#Leonid)
It handles RuleDelayed (:>) much better (#rcollyer)
It handles nested lists of rules without using Flatten (#Andrew)
It is a bit easier to assign multiple local variables using OptionValue /# list instead of having multiple calls to someoptions /. {opts} /. Options[thisfunction] (came up in comments between #rcollyer and me)
EDIT: 25 July I initially thought that the one time using the /. syntax might still make sense is if you are deliberately extracting a default option from another function, not the one actually being called. It turns out that this is handled by using the form of OptionsPattern[] with a list of heads inside it, for example: OptionsPattern[{myLineGraph, DateListPlot, myDateTicks, GraphNotesGrid}] (see the "More information" section in the documentation). I only worked this out recently.
It seems like relying on pattern-matcher yields faster execution than by using PatternTest as the latter entails invocation of the evaluator. Anyway, my timings indicate that some speed-ups can be achieved, but I do not think they are so critical as to prompt re-factoring.
In[7]:= f[x__, opts : OptionsPattern[NIntegrate]] := {x,
OptionValue[WorkingPrecision]}
In[8]:= f2[x__, opts___?OptionQ] := {x,
WorkingPrecision /. {opts} /. Options[NIntegrate]}
In[9]:= AbsoluteTiming[Do[f[1, 2, PrecisionGoal -> 17], {10^6}];]
Out[9]= {5.0885088, Null}
In[10]:= AbsoluteTiming[Do[f2[1, 2, PrecisionGoal -> 17], {10^6}];]
Out[10]= {8.0908090, Null}
In[11]:= f[1, 2, PrecisionGoal -> 17]
Out[11]= {1, 2, MachinePrecision}
In[12]:= f2[1, 2, PrecisionGoal -> 17]
Out[12]= {1, 2, MachinePrecision}
While several answers have stressed different aspects of old vs. new way of using options, I'd like to make a few additional observations. The newer constructs OptionValue - OptionsPattern provide more safety than OptionQ, since OptionValue inspects a list of global Options to make sure that the passed option is known to the function. The older OptionQ seems however easier to understand since it is based only on the standard pattern-matching and isn't directly related to any of the global properties. Whether or not you want this extra safety provided by any of these constructs is up to you, but my guess is that most people find it useful, especially for larger projects.
One reason why these type checks are really useful is that often options are passed as parameters by functions in a chain-like manner, filtered, etc., so without such checks some of the pattern-matching errors would be very hard to catch since they would be causing harm "far away" from the place of their origin.
In terms of the core language, the OptionValue - OptionsPattern constructs are an addition to the pattern-matcher, and perhaps the most "magical" of all its features. It was not necessary semantically, as long as one is willing to consider options as a special case of rules. Moreover, OptionValue connects the pattern-matching to Options[symbol] - a global property. So, if one insists on language purity, rules as in opts___?OptionQ seem easier to understand - one does not need anything except the standard rule-substitution semantics to understand this:
f[a_, b_, opts___?OptionQ] := Print[someOption/.Flatten[{opts}]/.Options[f]]
(I remind that the OptionQ predicate was designed specifically to recognize options in the older versions of Mathematica), while this:
f[a_, b_, opts:OptionsPattern[]] := Print[OptionValue[someOption]]
looks quite magical. It becomes a bit clearer when you use Trace and see that the short form of OptionValue evaluates to a longer form, but the fact that it automaticaly determines the enclosing function name is still remarkable.
There are a few more consequences of OptionsPattern being a part of the pattern language. One is the speed improvements discussed by #Sasha. However, speed issues are often over-emphasized (this is not to detract from his observations), and I expect this to be especially true for functions with options, since these tend to be the higher-level functions, which will likely have non-trivial body, where most of the computation time will be spent.
Another rather interesting difference is when one needs to pass options to a function which holds its arguments. Consider a following example:
ClearAll[f, ff, fff, a, b, c, d];
Options[f] = Options[ff] = {a -> 0, c -> 0};
SetAttributes[{f, ff}, HoldAll];
f[x_, y_, opts___?OptionQ] :=
{{"Parameters:", {HoldForm[x], HoldForm[y]}}, {" options: ", {opts}}};
ff[x_, y_, opts : OptionsPattern[]] :=
{{"Parameters:", {HoldForm[x], HoldForm[y]}}, {" options: ", {opts}}};
This is ok:
In[199]:= f[Print["*"],Print["**"],a->b,c->d]
Out[199]= {{Parameters:,{Print[*],Print[**]}},{ options: ,{a->b,c->d}}}
But here our OptionQ-based function leaks evaluation as a part of pattern-matching process:
In[200]:= f[Print["*"],Print["**"],Print["***"],a->b,c->d]
During evaluation of In[200]:= ***
Out[200]= f[Print[*],Print[**],Print[***],a->b,c->d]
This is not completely trivial. What happens is that the pattern-matcher, to establish a fact of match or non-match, must evaluate the third Print, as a part of evaluation of OptionQ, since OptionQ does not hold arguments. To avoid the evaluation leak, one needs to use Function[opt,OptionQ[Unevaluated[opt]],HoldAll] in place of OptionQ. With OptionsPattern we don't have this problem, since the fact of the match can be established purely syntactically:
In[201]:= ff[Print["*"],Print["**"],a->b,c->d]
Out[201]= {{Parameters:,{Print[*],Print[**]}},{ options: ,{a->b,c->d}}}
In[202]:= ff[Print["*"],Print["**"],Print["***"],a->b,c->d]
Out[202]= ff[Print[*],Print[**],Print[***],a->b,c->d]
So, to summarize: I think choosing one method over another is largely a matter of taste - each one can be used productively, and also each one can be abused. I am more inclined to use the newer way, since it provides more safety, but I do not exclude that there exist some corner cases when it will surprise you - while the older method is semantically easier to understand. This is something similar to C-C++ comparison (if this is an appropriate one): automation and (possibly) safety vs. simplicity and purity. My two cents.
A little known (but frequently useful) fact is that options are allowed to appear in nested lists:
In[1]:= MatchQ[{{a -> b}, c -> d}, OptionsPattern[]]
Out[1]= True
The options handling functions such as FilterRules know about this:
In[2]:= FilterRules[{{PlotRange -> 3}, PlotStyle -> Blue,
MaxIterations -> 5}, Options[Plot]]
Out[2]= {PlotRange -> 3, PlotStyle -> RGBColor[0, 0, 1]}
OptionValue takes it into account:
In[3]:= OptionValue[{{a -> b}, c -> d}, a]
Out[3]= b
But ReplaceAll (/.) doesn't take this into account of course:
In[4]:= a /. {{a -> b}, c -> d}
During evaluation of In[4]:= ReplaceAll::rmix: Elements of {{a->b},c->d} are a mixture of lists and nonlists. >>
Out[4]= a /. {{a -> b}, c -> d}
So, if you use OptionsPattern, you should probably also use OptionValue to ensure that you can consume the set of options the user passes in.
On the other hand, if you use ReplaceAll (/.), you should stick to opts___Rule for the same reason.
Note that opts___Rule is also a little bit too forgiving in certain (admittedly obscure) cases:
Not a valid option:
In[5]:= MatchQ[Unevaluated[Rule[a]], OptionsPattern[]]
Out[5]= False
But ___Rule lets it through:
In[6]:= MatchQ[Unevaluated[Rule[a]], ___Rule]
Out[6]= True
Update: As rcollyer pointed out, another more serious problem with ___Rule is that it misses options specified with RuleDelayed (:>). You can work around it (see rcollyer's answer), but it's another good reason to use OptionValue.
Your code itself has a subtle, but fixable flaw. The pattern opts___Rule will not match options of the form a :> b, so if you ever need to use it, you'll have to update your code. The immediate fix is to replace opts___Rule with opts:(___Rule | ___RuleDelayed) which requires more typing than OptionsPattern[]. But, for the lazy among us, OptionValue[...] requires more typing than the short form of ReplaceAll. However, I think it makes for cleaner reading code.
I find the use of OptionsPattern[] and OptionValue to be easier to read and instantly comprehend what is being done. The older form of opts___ ... and ReplaceAll was much more difficult to comprehend on a first pass read through. Add to that, the clear timing advantages, and I'd go with updating your code.
I would like to overload Mathematica's Set function (=), which turns out to be too tricky for me (see following code example). I successfully overloaded other functions (e.g. Reverse in the code example). Any suggestions?
In[17]:= ClearAll[struct];
In[18]:= var1=struct[{1,2}]
Out[18]= struct[{1,2}]
In[19]:= Reverse#var1
Out[19]= struct[{1,2}]
In[20]:= Head[var1]
Out[20]= struct
In[21]:= struct/:Reverse[stuff_struct]:=struct[Reverse#stuff[[1]]]
In[22]:= Reverse#var1
Out[22]= struct[{2,1}]
In[23]:= struct/:Set[stuff_struct,rhs_]:=Set[struct[[1]],rhs]
In[24]:= var1="Success!"
Out[24]= Success!
In[25]:= var1
Out[25]= Success!
In[26]:= Head[var1]
Out[26]= String
In[27]:= ??struct
Global`struct
Reverse[stuff_struct]^:=struct[Reverse[stuff[[1]]]]
(stuff_struct=rhs_)^:=struct[[1]]=rhs
I don't think that what you want can be done with UpValues (alas), since the symbol (tag) must be not deeper than level one for definition to work. Also, the semantics you want is somewhat unusual in Mathematica, since most Mathematica expressions are immutable (not L-values), and their parts can not be assigned values. I believe that this code will do something similar to what you want:
Unprotect[Set];
Set[var_Symbol, rhs_] /;
MatchQ[Hold[var] /. OwnValues[var], Hold[_struct]] := Set[var[[1]], rhs];
Protect[Set];
For example:
In[33]:= var1 = struct[{1, 2}]
Out[33]= struct[{1, 2}]
In[34]:= var1 = "Success!"
Out[34]= "Success!"
In[35]:= var1
Out[35]= struct["Success!"]
But generally, adding DownValues to such important commands as Set is not recommended since this may corrupt the system in subtle ways.
EDIT
Expanding a bit on why your attempt failed: Mathematica implements flow control and assignment operators using the mechanism of argument holding (Hold* - attributes, described here). This mechanism allows it to, in particular, imitate pass-by-reference semantics needed for assignments. But then, at the moment when you assign to var1, Set does not know what is stored in var1 already, since it only has the symbol var1, not its value. The pattern _struct does not match because, even if the variable already stores some struct, Set only has the variable name. For the match to be successful, the variable inside Set would have to evaluate to its value. But then, the value is immutable and you can not assign to it. The code I suggested tests whether the variable has an assigned value that is of the form struct[something], and if so, modifies the first part (the Part command is an exception, it can modify parts of an L-value expression provided that those parts already exist).
You can read more on the topics of Hold* - attributes and related issues in many places, for example here and here
I also do not believe that this can be done with TagSet, because the first argument of Set must be held.
It seems to me that if modifying Set, it can be done with:
Unprotect[Set]
Set[s_, x_] /; Head[s] === struct := s[[1]] = x
However, Leonid knows Mathematica better than I, and he probably has a good reason for the longer definition.
What is the symbol $ used for internally?
I do not mean the compound forms x$388 or $5 etc., just the $ by itself.
I am wondering if this is a valid object to use in notation, or what I will break if I do.
It is unwise to have user variables that end in an odd number of $ characters (not counting the first character). x$, y$$$, and $$ are all poor choices for variable names.
This is because appending an odd number of $ to an identifier is a technique called "lexical renaming," which the Mathematica kernel uses to avoid conflicts in variable names when higher-order functions return functions that use the same variable names as their parents. This technique is used in a variety of scoping constructs, including Function, Module, With, and Rule; here is an example with Function:
In[1]:= f = Function[{x, y}, Function[{x}, x+y]]
Out[1]= Function[{x, y}, Function[{x}, x + y]]
In[2]:= f[2,3]
Out[2]= Function[{x$}, x$ + 3]
In[3]:= ?*`x$
Global`x$
Attributes[x$] = {Temporary}
In short, appending $ characters is a system-internal renaming mechanism, and identifiers of this form are recognized by Mathematica as "lexically renamed" versions of the $-less forms, with Temporary attribute. It is not recommended to use variables of this form in your own code.
Mathematica is a term-rewriting language that can behave like a lexically scoped functional language by use of internal rewriting mechanisms such as "lexical renaming."
In version 7, symbol System`$
used to be already created in a fresh kernel, but not used for anything as far as I know. In version 8, symbol $ is not pre-created:
In[1]:= Context["$"]
During evaluation of In[1]:= Context::notfound: Symbol $ not found. >>
Out[1]= Context["$"]
I would agree with Szabolcs that code using $ in System context might break in future versions, as well as any other code that modifies System symbols.