Advanced AuthorTools Topics

Lou D’Andria
Wolfram Research, Inc.
lou@wolfram.com
April 10, 2003

Writing is not necessarily something to be ashamed of, but do it in private and wash your hands afterwards. —Robert A. Heinlein

The AuthorTools add-on included in Mathematica 4.2 must handle notebooks of all sizes for a variety of reasons. This talk will discuss the AuthorTools utility functions underlying these features, and the design principles that guide them. The focus will be on using AuthorTools functions as a powerful approach to notebook processing and analysis. Enhancements to AuthorTools will also be discussed.

AuthorTools Design Goals

There is nothing in the programming field more despicable than an undocumented program. —Ed Yourdan

Type or Click

AuthorTools features are all implemented in Mathematica packages called by the AuthorTools palettes. These palettes were designed to expose many of the features in these packages, but not all, to keep them simple. Users of the packages can do everything that users of the palettes can, and more.

Handle Sets of Notebooks

Most book-sized projects are not contained in a single notebook. Rather, there is typically a notebook for each chapter, or perhaps even each section of each chapter, with other notebooks for pieces of the front matter and back matter. AuthorTools needed to find a way to manage projects like these that span multiple notebooks.

Handle Large Notebooks

While most of the necessary features could be implemented using existing methods of notebook processing — Get, NotebookGet, NotebookRead, … — these solutions generally don’t scale well. When handling notebooks with a large number of cells, or notebooks containing large PostScript graphics, they may work sporadically or not at all.

Unfortunately, we can make no assumptions about a typical notebook. Some users may want to use a single notebook to contain dozens of megabytes of graphical data, while others structure their notebooks to never contain more than a few hundred kilobytes. In order to appeal to both types of Mathematica users, AuthorTools had to devise the most robust and efficient way to process notebooks for its purposes.

Handle Files, Objects and Expressions

It’s often confusing to switch back and forth between file processing, expressions processing, and notebook object juggling. The majority of AuthorTools functions that handle notebooks at all should deal with all three of these, switching e.g. from a given notebook object to the related notebook file when necessary.

Build Help Browser Configuration Files

In order to be a useful tool suite for Mathematica authors, AuthorTools had to include features to make it easy to add user documentation to the Help Browser. That meant building BrowserCategory.m and BrowserIndex.nb configuration files.

Make Fast Things Fast

The Mathematica front end and kernel each have relative strengths and weaknesses. AuthorTools should avoid the weaknesses whenever possible. When a weakness is impossible to avoid — e.g., a function which is the only way to access a particular feature — AuthorTools should make every attempt isolate the feature, and not call it automatically without the user’s consent.

A prime example of this in Mathematica 4.2 is calculating page numbers. This calculation, which can only reliably be performed by the front end, must use the complete style hierarchy for the given notebook, as well as the print drivers, font metrics, and other related operating system information. As we’ll see later, page numbers are calculated only when explicitly requested, and then they’re cached.

Minimize Changes To User Notebooks

Some of these features require altering user notebooks a little (table of contents), a lot (bilateral cells), or somewhere in between (index). AuthorTools restricts itself to only those changes which the user specifically requests. Since the notebooks themselves must be altered, if only in a small way, we encourage users to work with a copy of their original notebooks.

Use Intermediate Formats

When building structured file results, AuthorTools should always create an intermediate expression form which can then be formatted. The benefit here is that other developers can access the intermediate, raw data form to write their own formatting functions for indexes, tables of contents, browser categories, etc.

Approaches to Notebook Processing

Writing is not a genteel profession. It’s quite nasty and tough and kind of dirty. —Rosemary Mahoney

There are several ways that you might consider processing your notebooks. Because notebooks are expressions, you can read the entire expression into the kernel using Get or NotebookGet. This method is very flexible, but it does not scale very well, and requires a lot of overhead regardless of the processing requirements.

Another approach is to write a script that will process your notebooks in the front end just as a user would with a mouse and keyboard. You might use SelectionMove, NotebookFind, and the like to select or extract the pieces of a notebook you’re interested in, and continue processing from there. This requires far less kernel memory than the previous approach, but solutions built up in this way need to be carefully considered to build robust, maintainable code. A complete knowledge of available front end packets and tokens is also required.

Because of its ability to handle very large notebooks, AuthorTools takes the latter approach whenever possible. At its crux are a very few pieces of front end technology.

CellTags

The CellTags mechanism allows you to store keywords associated with a cell, and find those cells with a given keyword very efficiently (NotebookFind, NotebookLocate, AuthorTools`Common`NotebokLookup).

TaggingRules

For storing even more information, the front end provides the TaggingRules option. This option can be applied at any scope — notebook, cell, etc. — and its value can be any Mathematica expression. By convention, AuthorTools sets this to a list of suboptions, TaggingRules -> {"opt1" -> data1, "opt2" -> data2, …}. TaggingRules is where, for example, AuthorTools stores indexing and pagination information for a notebook.

Information packets

Prior to Mathematica 4.2, the only way to get information about the current selection in a notebook was to read it in with NotebookRead. That would be quite wasteful if, for instance, you just want to know the style of the currently selected cell. However, a new packet (CellInformation) returns lots of information about the current selection, including how many cells are selected, whether they are evaluating, their style, etc.

There are similar packets that return general information about a notebook, or specific information relating counter values and other layout information to individual cells (NotebookInformation and NotebookGetLayoutInformationPacket respectively). These are used, for example, when AuthorTools must learn on which printed page a cell falls.

Notebook File Outline Cache

A notebook is a Mathematica expression, and every notebook file is just a file that contains an expression with head Notebook. When the front end writes a notebook file, it includes the expression with head Notebook, and some additional data within comments in the notebook file. These comments allow the front end fast access to portions of a notebook without having to read in the entire file. You can see these comments in a text editor.

FrontEnd Use

Most of the comments in a notebook file’s header and footer contain byte offset and newline information about various expressions in the notebook file, and are used internally by the front end to provide essentially random access the notebook file’s content. Think StreamPosition and SetStreamPosition in kernel lingo.

When opening a notebook, the front end checks for the presence of a special CacheID comment in the header. If it doesn’t find one, it parses the entire notebook file.

(*CacheID: 232*)
  
  (*NotebookFileLineBreakTest
  NotebookFileLineBreakTest*)
  (*NotebookOptionsPosition[    121983,       2355]*)
  (*NotebookOutlinePosition[    122811,       2384]*)
  (*  CellTagsIndexPosition[    122721,       2378]*)
  (*WindowFrame->Normal*)

If it does find a CacheID line, it then looks at the other header comments. The numbers in these comments indicate the byte offset from the beginning of the file to the location of these pieces of information, along with the number of newlines in that range. In this way, the front end can read in the notebook’s options without parsing the entire notebook, because it knows that the options start on line 2355, byte offset 121983. Similarly for the notebook outline and the cell tags index.

But what are the notebook outline and the cell tags index? These are both included among the comment footers that the front end writes. The notebook outline is a Notebook expression that includes an overview of the structure of the notebook, including almost everything about every cell except its content.

(*NotebookFileOutline
 Notebook[{
 
 Cell[CellGroupData[{
 Cell[1776, 53, 66, 2, 27, "Input",
   InitializationCell->True],
 Cell[1845, 57, 35, 1, 27, "Output"]
 }, Open  ]],
 
 Cell[CellGroupData[{
 Cell[1917, 63, 39, 1, 27, "Input"],
 Cell[1959, 66, 353, 9, 39, "Print",
   CellTags->"Info3252669347-9757967"]
 }, Open  ]],
 
 Cell[CellGroupData[{
 Cell[2349, 80, 96, 2, 27, "Input",
   InitializationCell->True],
 Cell[2448, 84, 22039, 540, 186, 4588, 320, "GraphicsData", "PostScript", \
 "Graphics"]
 }, Open  ]],
 
 Cell[CellGroupData[{
 Cell[24524, 629, 80, 1, 27, "Input"],
 Cell[24607, 632, 97220, 1715, 242, 20636, 765, "GraphicsData", "PostScript", \
 "Graphics"],
 Cell[121830, 2349, 137, 3, 27, "Output"]
 }, Open  ]]
 }
 ]
 *)

The numbers in these cells are byte offsets to the beginning of the cell and the beginning of the cells options.

This outline allows the front end to parse only those pieces of the notebook that are actually needed. If you never open a closed cell group in some notebook, then the front end can avoid reading those cells in from the file. By focusing on reading and displaying only those cells that a user sees, the front end can be more responsive.

Kernel Use

Notice that the notebook file outline cache described above is stored as a Mathematica expression in a comment. That means that the kernel can read the outline cache. In fact, through judicious use of SetStreamPosition, the kernel can be taught to read notebooks in the same way as the front end, reading only those pieces that the user happens to care about.

This is exactly what AuthorTools does.

Implementation Details

Do what you can, with what you have, where you are. —Theodore Roosevelt

Notebook Files as Streams

The AuthorTools functions for reading and using a notebook’s file outline cache are in the AuthorTools`Common` package.

Needs["AuthorTools`Common`"]

For out tests, we’ll use a seven megabyte file from the documentation that comes with Mathematica.

{FileByteCount[nbFile], nbFile}

NotebookCacheValidQ checks whether a notebook’s cache is intact. NotebookFileOptions extracts a notebook’s options directly from the notebook file.

If[NotebookCacheValidQ[nbFile], 
      FrontEndVersion /. NotebookFileOptions[nbFile], $Failed]

NotebookFileOutline extracts the notebook outline out of a notebook file.

{MaxMemoryUsed[], 
      Timing[fileOutline = NotebookFileOutline[nbFile];], 
      MaxMemoryUsed[]}

This has less overhead than reading the entire notebook expression, though you get less in return.

{MaxMemoryUsed[], Timing[Get[nbFile];], MaxMemoryUsed[]}

The structure looks familiar.

{Head[fileOutline],  
      fileOutline[[1, 2]]}

ExtractCells is the low-level function that translates cells from the file outline cache into cells with content from the actual notebook.

ExtractCells[nbFile, 
      fileOutline[[1, 2]]]

NotebookLookup is based on ExtractCells, and provides some heuristics for extracting information based on what sort of result the user wants. NotebookLookup also insulates the user from dealing with the file outline cache directly.

?NotebookLookup

The "CellOutline" token returns the distilled cell expressions directly from the file outline.

NotebookLookup[nbFile, "CellOutline"] // Length

You can give a pattern as the third argument to NotebookLookup to return only those cells whose outline cell matches the pattern.

NotebookLookup[nbFile, "CellOutline",  
      Cell[___, "OutputOnly", ___]]

With a different second argument, you can instruct NotebookLookup to return not just the outline for the found cells, but to extract the actual cell content.

NotebookLookup[nbFile, "CellExpression",  
      Cell[___, "OutputOnly", ___]]

For instance, when building an outline of a notebook, you might be interested in the content of cells of a particular style. This call to NotebookLookup returns a list of cell expressions, which are then displayed as a notebook.

NotebookLookup[nbFile, "CellExpression", 
          Cell[___, "Section" | "Input", ___]] // Notebook // 
      NotebookPut

Other supported second arguments are "CellPage", which returns the page number for each matching cell, and "CellIndex", which returns the index or offset number for each matching cell.

Other Caches

The file outline cache turns out to be incredibly useful for notebook processing. There are other caches that AuthorTools can use to make other processing just as efficient.

The notebook option TaggingRules can take any Mathematica expression as its value. The front end doesn’t use the value of this option in any way, but it will store it and let you manipulate it as you would any other option. By convention, the setting for the TaggingRules option is set to a list of rules, each of which has a string token on the left-hand side. It is in these suboptions that AuthorTools stores information about pagination and indexing.

There is also a function that lets you read these options directly from a notebook file, NotebookFileOptions, analogous to NotebookFileOutline.

TaggingRules  /.  NotebookFileOptions[nbFile]  //  Short

Note that there are no functions that go the other direction and write options directly to a notebook file. This is mainly because once an option changes, some of the cache information in the notebook file has to be rewritten. Thus, if you want to update the TaggingRules setting in a notebook, you’ll have to open the notebook and make changes with SetOptions as usual.

Utility Functions

In addition to those functions described above, other general purpose utility functions can be found in various AuthorTools packages. For more information, see the online help included with AuthorTools.

?CellInformation

?NotebookModifiedQ

?NotebookScan

?ProgressDialog

?SetOptionsDialog

?NotebookPageNumbers

?NotebookFolder

?NotebookFilePath

?NotebookCellTags

?ExportNotebook

Examples

For a list of all the ways technology has failed to improve the quality of life, please press three. —Alice Kahn

Wolfram Research

AuthorTools technology is used throughout Wolfram Research, for managing a variety of projects involving notebooks, including documentation bundled with Mathematica, custom versions, application packages, the system administrator guide, online information at documents.wolfram.com, the special functions poster, and even the custom display environment used by the Wolfram Education Group.

Guidebook Contents

This example shows how to extract all the heading cells from a set of notebooks, using Michael Trott’s Guidebook series as the source of our notebooks. Start with a list of files.

trottDir = 
        ToFileName[{NotebookFolder @ 
              EvaluationNotebook[], "trott"}];

trottFiles = FileNames["*.nb", trottDir];

Using Get and then extracting just those cells you want takes, and wastes, a lot of time and memory. Note that these are the *un*evaluated notebooks, so they are only a small fraction of the size they could be.

\(Timing[\(\(Cases[Get[#], 
              Cell[_, "ChapterNumber" | "Chapter" | "Section", ___], ∞] &\)\  /@ \ trottFiles;\)]\)

Using NotebookLookup is more efficient way to do the same thing.

\(Timing[\(cells = \(NotebookLookup[#, "CellExpression", 
                Cell[___, "ChapterNumber" | "Chapter" | "Section", ___]] &\)\  /@ \ trottFiles;\)]\)

We can build a simple table of contents.

NotebookPut @ Notebook[Flatten[cells]]

This table of contents uses different styles.

\(NotebookPut\ @\ 
      Notebook[Flatten[
            cells]\  //. \ {
Cell[a_, "ChapterNumber", 
                b___] :> 
              Cell[a, "Title", b], 
Cell[
                a_, "Chapter", b___] :> 
              Cell[a, "Subtitle", b], 
Cell[
                a_, "Section", b___] :> 
              Cell[a, "Text", 
                b], 
Cell[_, "Comment", ___] :> Sequence[]}]\)

Function Information

Instead of looking up functions in the Help Browser, you can find information directly in the notebooks that are displayed by the Help Browser.

\(\(refGuide = 
        ToFileName[{$TopDirectory, "Documentation", "English", "RefGuide"}, "RefGuide.nb"];\)\)

FileByteCount[refGuide]

Here’s a function that extracts cells from this notebook which correspond to a given keyword.

refGuideCells[str_] := 
      NotebookLookup[refGuide, "CellExpression", 
        Cell[___, 
          CellTags  ->  (str  |  {___, 
                  str, ___}), ___]]

This function displays the relevant cells in a new notebook.

functionInformation[str_] := 
      NotebookPut @ 
        Notebook[
refGuideCells @ str, 
          StyleDefinitions  ->  "HelpBrowser.nb", 
          WindowSize -> {500, 
              300}, 
WindowMargins  ->  {{Automatic, 
                5}, {Automatic, 
                5}}, 
Background  ->  
            RGBColor[1, 0.7, 1]]

Function names are among existing keywords.

functionInformation["ListConvolve"]

Loading Init Cells

Before version 3.0 of Mathematica, when the Get function was called on a notebook file, and it would find and evaluate all the initialization cells in that notebook without having to open the notebook in the front end. When the notebook file format changed to be a Mathematica expression starting in version 3, Get could no longer do this automatically. However, with the advent of NotebookLookup, this convenience is easy to re-implement.

Extract the inputs that are manifestly init cells.

notebookInitCells[nb_] := 
      NotebookLookup[nb, "CellExpression", 
        Cell[___, "Input", ___, 
          InitializationCell  ->  True, ___]]

Evaluate those cells by converting their inert typeset form to Mathematica expressions.

evaluateInitCells[nb_] := ToExpression[First @ #] &  /@  
        notebookInitCells[nb]

Leverage Intermediate Formats

Contents

This example uses the intermediate form used when building tables of contents to create the clickable outline that I’m using for this talk

nbObj = EvaluationNotebook[]

<< AuthorTools`MakeContents`

<< AuthorTools`MakeCategories`

c = 0;
tagCell[nb_] := 
      If[MemberQ[{"Title", "Section", "Subsection"}, 
          First["Style"  /.  CellInformation[nb]]], 
        AddCellTags[nb, "c:" <> ToString[++c]]]
NotebookScan[nbObj, tagCell];
NotebookSave[nbObj];

\(\(tocData = 
        AuthorTools`MakeContents`Private`RawContents[
          NotebookFilePath\ @\ nbObj, 
          SelectedCellStyles -> {"Title", "Section", "Subsection"}];\)\)

First[tocData]

\(outlineCell[{Cell[content_, style_, ___], 
          tag_String\  | \ {tag_String, ___}}] := 
Cell[
        TextData\ @\ \(List\ @\ \(ButtonBox\ @\ \(AuthorTools`MakeCategories`Private`cellContentsToString\ @\ 
                content\)\)\), 
"TOC" <> 
          style\ , 
PageWidth\  -> \ 
          Infinity, 
ButtonBoxOptions -> {
Active -> 
              True, 
ButtonEvaluator -> 
              None, 
ButtonFunction :> \((FrontEndExecute[{
NotebookLocate[{InputNotebook[], 
                          tag}], 
FrontEndToken[
                        InputNotebook[], "OpenCloseGroup"]}] &)\)}]\)

cellList = 
        Map[outlineCell,  {"CellExpression", "CellTags"} /. 
            List  @@@  tocData];

\(nb = 
      NotebookPut[
        Notebook[cellList, 
\ 
          WindowClickSelect\  -> \ 
            False, 
WindowFrame -> "Palette", 
StyleDefinitions\  -> \ "HelpBrowser.nb", 
ScrollingOptions -> {"VerticalScrollRange" -> Fit}, 
WindowMargins\  -> {{0, 
                Automatic}, {Automatic, 
                0}}, 
WindowSize -> {200, 
              606}, 
ClosingAutoSave -> 
            True
]]\)

talkOutlineFile = 
        ToFileName[{NotebookFolder @ 
              nbObj}, "TalkOutline.nb"];

NotebookSave[nb, talkOutlineFile]

NotebookClose[nb]

NotebookOpen[talkOutlineFile]

Index

The information that the AuhorTools`MakeIndex` package and palette insert into a user notebooks are contained almost complete in the TaggingRules for those notebooks.

cspFile = 
        ToFileName[{NotebookFolder @ 
              EvaluationNotebook[], "CSP"}, "12_misc.nb"];

You can handle the option setting directly.

Last["IndexEntries" /. (TaggingRules /. 
            NotebookFileOptions[cspFile])]

Or you can call the AuthorTools utility function that returns structured data, the internal precursor to creating a structured file.

AuthorTools`MakeIndex`Private`RawIndex[cspFile, 
        IncludeCellPage -> True] // Last

Browser Category

Help Browser configuration files contain special Mathematica expressions that are interpreted by the front end to display and index notebook content in the Help Browser. These files also require that the notebook files containing the content to be displayed are in sync with the configuration files.

AuthorTools can create these browser category files automatically, or you can get the raw data and do it yourself.

Off[MakeBrowserCategories::valid]

?MakeBrowserCategories

AuthorTools`MakeCategories`Private`RawBrowserCategories[cspFile] // 
      First

Either way, there's a function for writing out the resulting category files in a user-readable layout.

?WriteBrowserCategories

Forthcoming Features

Technology is a way of organizing the universe so that man doesn’t have to experience it. —Max Frisch

(* Internal use only *)

Conclusion

I think I did pretty well, considering I started out with nothing but a bunch of blank paper. —Steve Martin

AuthorTools implements a novel approach to notebook processing, reading information from notebooks in much the same way that the front end does. This flexible new approach doesn’t suffer the same limitations as traditional approaches, and so can be used to manipulate notebooks in ways that would otherwise be impractical or impossible.

This notebook can be downloaded from http://members.wri.com/lou/conf/.

Initialization

Load the package.

Needs["AuthorTools`"]

Our canonical sample file.

\(\(nbFile = 
        ToFileName[{$TopDirectory, "Documentation", "English", "GettingStarted"}, "Tour.nb"];\)\)