Symbol Contexts

What They Are and How to Use Them

John M. Novak

All Mathematica symbols exist in namespaces called contexts. There are a number of subtleties in understanding and managing contexts that any Mathematica programmer needs to be aware of. This talk describes the notion of contexts, the tools to manage them, and some of the “gotcha’s” involved in their use.

What is a Context?

Context Syntax

Every symbol in Mathematica has full name and a short name. The full name consist of the short name plus a context. The short name is given by a sequence of alphanumeric characters, starting with a letter — usual symbol naming conventions. E.g., Sin, foo, Integrate, mysym1, etc. The context is given by a sequence of names using the same convention as the short name, with each name followed by a backquote ( `, aka the context mark character — not to be confused with a single quote or reverse prime character!).

The significance of the short name vs. the full name is that symbols are commonly referred to by their shortname. The context provides a namespace, or a mechanism for allowing many symbols with the same short name to exist simultaneously. The context is also used for organizational purposes; the syntax of multiple symbol-like names separated by the context mark allows a hierarchical naming convention, where symbols with a common purpose can be placed in a common context.

Short Name to Context

So, if multiple symbols exist with the same short name, how does the user know which namespace is being employed in any given instance of a short name? This is determined by two features: the current context, and the context path.

$Context

$ContextPath

When a user employs a short name, the system first checks the current context, then the contexts on the context path for that symbol; if it exists in one or more of those contexts, then the first instance encountered is used; if it doesn't exist, then a new symbol corresponding to the short name is created in the current context.

Contexts and Parsing

A very important point to realize about contexts is that a symbol in short-name form is assigned a context during parsing, not during evaluation. This is a source of much confusion when using contexts.

We can use Begin to change the current context, and Context to determine the context of a given symbol.

Begin["NewCont`"]

$Context

newSym

Context[newSym]

End[]

If the context change occurs in the same evaluation as the symbol declaration, the symbol does not pick up the new context.

(Begin["NewCont2`"];  newSym2)

$Context

Context[newSym2]

End[]

It is possible in some cases to work around this issue by putting off the symbol parsing until the evaluation.

(Begin["NewCont3`"];  Symbol["newSym3"])

$Context

Context[newSym3]

End[]

As can be seen above, the canonical tool for changing the current context is the function Begin. This maintains a stack of contexts that can be restored by successive calls to End[]. It is possible to assign a value directly to $Context, though this is not generally used.

Another key point to remember about contexts is that you can always refer to a symbol by its full context name. If the symbol or context doesn't exist, it will automatically be created.

Managing the Context Path

Properties of the Context Path

The context path is used to manage what are termed user-visible symbols. There are generally many more contexts used by symbols in the system than are on the context path at any given time; the context path manages the use of short names, not the organization of full symbol names.

$ContextPath

Contexts[]

So, the system only pays attention to the current context and the context path when determining what context a symbol in short name form belongs to. However, a short name may appear in multiple contexts on the context path; when this happens, instances corresponding to later contexts in the path are called shadowed.

Shadowed Symbols

Typical examples of this occur during package loading. Contexts in packages will be described in more detail later; here we just use packages to load some symbols while modifying the context path.

Needs["Graphics`Colors`"]; 
    Needs["Miscellaneous`ChemicalElements`"]

$ContextPath

?Gold

Thus, we see that a context earlier on the context path shadows the later symbols. We can rearrange the context path:

$ContextPath = ($ContextPath /. 
{a___, 
              b : "Graphics`Colors`", c___} :> {b, a, 
              c})

?Gold

Inadvertent Shadowing

A common problem encountered by new users is using a symbol before its definition is loaded, then encountering problems with its further use. For example:

The following tries to use BarChart; but because the definition hasn't been loaded, this simply creates a new symbol.

BarChart[{1, 2, 3, 2, 3}]

<< Graphics`Graphics`

BarChart[{1, 2, 3, 2, 3}]

Why doesn't the reinvocation of BarChart work? Graphics`Graphics` is before Global` on the context path…

$ContextPath

…but remember that the current context is searched for a matching symbol before the context path.

Begin["NewCont`"]

BarChart[{1, 2, 3, 2, 3}]

End[]

The most common way of overcoming this problem involves removing the Global` symbol, which unshadows the symbol.

Remove[Global`BarChart]

Symbols that are Hidden

Recall that the path only governs visibility of short names, not the existence of symbols. This means that a symbol may exist in a hidden (non-user-visible) context, but won't be accessible by its short name unless the context is put on the context path.

?ZeroQ

$ContextPath  =  
        AppendTo[$ContextPath,  "Developer`"];

?ZeroQ

However, note that it is becoming increasingly common to have symbols that are deliberately placed in a hidden context. This is used for grouping more obscure functionality with a descriptive context name, which serves the purpose of making advanced functionality less intrusive for the casual user, and providing a descriptive mnemonic for the occasional user. The advanced user who makes extensive use of the functionality can, in turn, place the context on the context path himself.

The Developer` and Experimental` contexts are like this to a degree; but they are intended for more extensive use, as demonstrated by the presence of the convenient <<Experimental` mechanism for setting the context path. (Note that this is not actually loading a package; it simply appends the context to the context path.)

Another variant of this is the use of things like the FrontEnd` context, which contains inert versions of System` context functions.

Quit

Package Contexts

Basic Package Syntax

Packages are a good demonstration of why contexts are desirable. A package is a collection of Mathematica functions. Naturally, we want the definitions we write to be able to coexist with functions other people write. By placing all the definitions in a context, we can prevent one package from overriding another. Furthermore, we can use the hierarchical nature of context names to group functions with a common purpose. Furthermore, for convenient handling of namespaces, a correspondence is assumed to exist between a package context and its filename; the hierarchical context structure corresponds to a directory structure.

Generally, a package contains two sorts of symbols: symbols that we wish to be user-visible, and symbols that should be completely private to the package. The canonical form of a package achieves this goal with the following syntax:

BeginPackage["MyPackage`"]
userVisibleFunc;
Begin["`Private`"]
userVisibleFunc[x_] := x^2 (* x not user-visible *)
End[]
EndPackage[]

The BeginPackage function is used to manipulate both the current context and the context path, while storing their settings so that they may be reset later. In particular:

BeginPackage["MyPackage`"]

$Context

$ContextPath

EndPackage[]

$ContextPath

First, notice that the context path has been reset to just the System` context (which contains system-defined symbols, usually) and the package context. Furthermore, the current context is set to the package context. This means that any symbol given in short-name form will be checked against the System` context, and if it doesn't already exist, it will be created in the package context. Thus, it won't interfere with any other packages, but won't shadow a System` function.

Next, note that when the EndPackage evaluates, the context path is restored, with the exception that the package context is added to the beginning of the context path. This makes all symbols defined in the package context user-visible.

The user-visibility granted by the use of BeginPackage/EndPackage explains the use of a Begin/End pair within the package. After the symbols to be made user-visible are declared (usually by giving them a usage message), the current context is changed to something that won't be user-visible, hiding any further new symbols. As an aside, note the use of a preceding backquote in Begin["`Private`"]. This causes the new current context to be a sub-context of the previous current context; this is another example of the hierarchical grouping of contexts. The use of the name 'Private' for the hidden sub-context is a convention, not a requirement of syntax.

Dependent Packages

Now that we've effectively created a namespace for our use of short-names in a package, what do we do if we want to make use of symbols from another package? It happens that BeginPackage has an extended syntax to handle this issue.

BeginPackage["NewPack`",  {"Graphics`Shapes`"}]

$ContextPath

EndPackage[]

$ContextPath

The second argument of BeginPackage is a list of contexts; these are assumed to be packages, and Needs is performed on those contexts. Furthermore, the context path is modified to include these contexts, allowing you to call the functions in these packages by their short names.

One important point to notice is that while loading a package such as Graphics`Shapes` may in turn load other packages (in this case, Geometry`Rotations`), those other contexts do not appear on the context path. This means that if you want to work with the functions in Geometry`Rotations` by their short names in NewPack`, you must also list it in the BeginPackage second argument. However, the restored context path after the EndPackage includes all loaded user-visible contexts.

Needs after BeginPackage

This restoration of the context path is why it is inadvisable to call Needs directly from within a BeginPackage/EndPackage pair, but rather use the BeginPackage second argument for extra package loading.

BeginPackage["NewPack`"]

Needs["Graphics`SurfaceOfRevolution`"]

$ContextPath

EndPackage[]

$ContextPath

Notice that the restored context path does not contain Graphics`SurfaceOfRevolution`, even though that package has been loaded; in fact, I can call its functions by their full names, even though the short name is not visible:

?SurfaceOfRevolution

Graphics`SurfaceOfRevolution`SurfaceOfRevolution[
      Sin[x], {x, 0, 2  Pi}]

Since this is generally not a desired effect, it is not recommended… But!

Private Common Packages

Good programming practice often requires one to modularize common functionality into a separate package file. However, the BeginPackage dependency mechanism is specifically designed around user-visible contexts. Suppose you want a package of functions that is not user-visible? What's the best approach?

One technique is to simply use Needs as above. This is fairly concise, and should be effective; its only drawback is if you want the private common functionality to remain invisible if a user specifically loads the common package. (The embedded Needs technique assumes that the common package uses the standard BeginPackage mechanism, making the context visible after the Needs evaluates.)

A variant of the above can be created that makes all the private functions completely private. Suppose we write the common package using the following sort of internal structure:

BeginPackage["MyApp`Common`"]
MyApp`Internal`Func1::usage = "..."
MyApp`Internal`Func2::usage = "..."
Begin["`Private`"]
PrependTo[$ContextPath, "MyApp`Internal`"];
Func1[x_] := ...
Func2[x_] := ...
End[]
EndPackage[]

Then, in any package where you want to use the dependent functions, evaluate both a hidden Needs and the PrependTo[$ContextPath, "MyApp`Internal`"]. The net effect is that the MyApp`Common` context is used for Needs-based package loading management; but the private functions are in a separate context MyApp`Internal`, placed on the context path by hand. The EndPackage restores the context path, hiding the use of MyApp`Internal` from the user.

Quit

Interdependent Packages

A peculiar situation can arise as a result of the package dependency handling of BeginPackage. Consider the following interdependent packages:

BeginPackage["Pack1`", {"Pack2`"}]
Func1::usage = "First Function.";
Begin["`Private`"]
Func1[x_] := 2 Func2[x]
End[]
EndPackage[]

BeginPackage["Pack2`", {"Pack1`"}]
Func2::usage = "Second Function.";
Func3::usage = "Third Function -- just to motivate loading Pack1.";
Begin["`Private`"]
Func2[x_] := x^2
Func3[x_] := 2 Func1[x]
End[]
EndPackage[]

Needs["Pack2`"]; Needs["Pack1`"];

Func1[2]

Func2[2]

Why does this happen? It turns out that this is tied to the order of evaluation; If you load Pack2` before Pack1`, the first thing you find in Pack2` is a command to load Pack1`, before any of the functionality in Pack2` has been defined. Thus, although the BeginPackage puts Pack2` on the path when loading Pack1`, it is empty, so when Func2 is encountered in the definition for Func1, it is assumed to be a local symbol. The lesson is that adding a context to the context path does not automatically make available functions that will be defined in that context.

This sort of situation is best avoided by keeping mutually interdependent functions in a single package, or abstracting dependent functionality out into a third package which the other packages load.

Binary Files and context paths

DumpSave is a useful function for dumping out the internal representation of Mathematica functions to a file for fast loading and/or encoded storage. It can accept the names of contexts for symbols to be dumped out; and will dump all the symbols in the named context and its subcontexts. So, if you have a package "MyApp`Pack1`", with a private subcontext "MyApp`Pack1`Private`", the command DumpSave["app.mx", {"MyApp`Pack1`"}] will dump all symbols in the package's context with their definitions.

Suppose your application has multiple packages, "MyApp`Pack1`" and "MyApp`Pack2`", for instance. In this case, you would think that DumpSave["app.mx", {"MyApp`"}] would dump all the functionality, taking advantage of the hierarchical nature of contexts; and you would be right, as far as that goes. However, there is a problem; on loading the .mx file, you will find that your functions will not work. It turns out that for the context path to be modified when loading the .mx file, a dumped context must explicitly appear in the context path at the time of the dump. This means that your definitions are present, but only accessible by their full name, unless you fix the context path. The correct invocation of DumpSave explicitly names the contexts that are on the context path: DumpSave["app.mx", {"MyApp`Pack1`", "MyApp`Pack2`"}].

BeginPackage["MyApp`Pack1`"]

testfunc1[x_] := x^2

EndPackage[]

BeginPackage["MyApp`Pack2`"]

testfunc2[x_] := x^3

EndPackage[]

DumpSave["app.mx",  {"MyApp`"}]

Quit

<< app.mx

?testfunc1

?MyApp`Pack1`testfunc1

Quit

Interaction with DeclarePackage

Because you can always refer to a symbol by its full name, you could potentially have functionality in a package that is 'optional'; that is, it acts appropriately if the dependent package is loaded, but doesn't require the package to be loaded. For example, Graphics3D.m has some transformations of 2D primitives. The package code needs rules for the package-defined Arrow and Spline primitives; but we don't want to force Arrow.m and Spline.m to be loaded. The naive approach suggests that we could write something like the following:

myfunc[p_Graphics`Spline`Spline]:= (* do stuff to p *)

For many dependent packages, this approach will work quite well. myfunc will do the appropriate thing if the Spline.m package has been loaded; and if it hasn't loaded, then the rule won't get invoked.

However, with the standard packages, and any user packages that employ the DeclarePackage mechanism, there is a chance for some rather unintentional behavior. In particular, the standard packages make use of the Kernel/init.m stub loading mechanism, which allows you to say:

<< Graphics`

This preps the system to load desired graphics packages when their symbols are used, but does not actually load the package until a symbol from that package is used. The Graphics/Kernel/init.m package is filled with statements that look like:

DeclarePackage["Graphics`Spline`",
     {"RenderSpline", "Spline", "SplineDivision", "SplineDots",
      "SplinePoints"}]

If this has been invoked, then simply parsing any symbol from this list causes the Graphics`Spline` package to load. So, the first unintended side effect is that when the package containing myfunc loads, Spline.m also loads — which is exactly what we were trying to avoid.

Worse, since this occurs during parsing, which is to say, while your private package is loading — after the BeginPackage, but before the EndPackage — the context path will be restored. The DeclarePackage placed Graphics`Spline` on the context path before the BeginPackage, so that is OK; but if there is a dependent package, NumericalMath`Spline` in this case, it gets loaded without appearing on the context path, certainly not desired behavior.

The most effective approach seems to be prevent the name Graphics`Spline`Spline from parsing unless the function is actually evaluated. The first instinct is to use the technique described earlier: write Symbol["Graphics`Spline`Spline"]. This is fine for invocations of Spline embedded in the right-hand-side of myfunc; but for the pattern (the left-hand-side), this is a problem, since the pattern evaluates when the function definition is made (i.e., when the package is loaded, leaving us back at square one). One workaround is to modify the rule so that the symbol never appears:

myfunc[p:((_Symbol?(Context[#] === "Graphics`Spline`" &&
              SymbolName[#] === "Spline" &))[___])] := 
       (* do stuff as before *)

Contexts and Front End Programming

The implementation of a subset of the Mathematica language in the Front End can add some unusual twists to context handling.

Button Functions and Contexts

The first issue to be aware of is that when a button function is evaluated, it is sent to the kernel as a single expression to be parsed. This means that we have the same issue as we do with parsing a single expression at a command line: if you try to load a package via Needs and evaluate a function from that package, you need to have the full context name of the function in the button.

The next problem arises if you are sending your function from the kernel to the FE, but have the dependent package already loaded (on the context path) in the kernel. Then, the front end receives only the short name of the function, which may cause problems with later uses.

<< Graphics`Graphics`

ButtonBox["doit!",  ButtonEvaluator -> Automatic,  
        ButtonFunction  :>  (Needs["Graphics`Graphics`"]; 
            Graphics`Graphics`BarChart[{1, 2, 3, 2, 3}]),  
        Active -> True] // DisplayForm

The workaround is the same as described before; withhold parsing the name until evaluation time.

\(ButtonBox["doit!", \ ButtonEvaluator -> Automatic, \ 
        ButtonFunction\  :> \ \((Needs["Graphics`Graphics`"]; \(Symbol["Graphics`Graphics`BarChart"]\)[{1, 2, 3, 2, 3}])\), \ 
        Active -> True] // DisplayForm\)

Parsing Unknown Operators

With the advent of the typesetting front end, a large number of new operators was introduced. Many of these operators have no intrinsic meaning in the kernel, and have no behavior assigned to them. This leads to the question: how should these operators be parsed if a user inputs them? For example, a user inputs:

a⊕b

What should the kernel do in this case?

The design decision that was settled on was to effectively assign a short name for each of the new operators. Evaluating the operator expression above creates the equivalent symbol via its short name; this means that the symbol ends up in a context determined by the usual rules for evaluating a short name.

For the case of ⊕, the equivalent short name is CirclePlus. The usual rules for assigning a context to a short name during parsing are followed. Thus, if CirclePlus already exists in a context on the context path, or in the current context, the operator will correspond to that instantiation. This allows multiple user-defined packages to be loaded that define behavior for CirclePlus; the operator will be equivalent to the first one on the context path.

If CirclePlus isn't on the context path, then it is created in the current context, which in the usual default configuration, is Global`. This may be unexpected to users who expect all operators to correspond to System` functions, as they do with operators that have a specific meaning to the kernel — operators such as + always parse to their System` equivalent symbol, no matter the state of the current context or the context path. However, we gain useful flexibility in doing so, allowing user packages to define behaviors for these new operators without conflicting with each other.

Context[CirclePlus]