Various properties of ImportString[#,”TeX”]&

David P. Vasholz

Johns Hopkins University Applied Physics Laboratory

If the function ImportString[#,”TeX”]& worked perfectly, it would be powerful indeed; untold thousands of technical papers that appear in the world’s scientific journals and elsewhere also exist as strings of common typographical symbols written in the LATEX typesetting language.  If these strings could all be readily translated into Mathematica notebooks, one would be in a much stronger position to exploit the power of Mathematica to probe on a large scale the intellectual content of the paper themselves.  As of June 2011, ImportString[#,”TeX”]& falls short of this ideal, but it nonetheless appears to be an excellent starting point.  It is the purpose of this work to evaluate the present performance of ImportString[#,”TeX”]& by applying it to various examples, by pointing out areas of difficulty, and by suggesting possible avenues of improvement.

Getting started with ImportString[#,”TeX”]&

Abbreviation: run

run = ImportString[#,"TeX"]&;

Simplest possible good result

Whenever run operates upon a string and returns a result, it is always a notebook 
object.

run[""]

%//NotebookPut

Simplest possible not-so-good result

The following command can actually lock up the computer:

run["}"]

This problem has been reported to and reproduced by Technical Support ( 
TS[20973] )

Strings and/or lists of integers; tcc and fcc

tcc=ToCharacterCode; fcc=FromCharacterCode;

Range[65,90]

%//fcc

%//tcc

Equations often translate well

{92,98,101,103,105,110,123,101,113,117,97,116,105,111,110,125,92,109,97,116,104,98,98,123,71,125,95,119,40,120,44,121,44,122,41,61,45,92,102,114,97,99,123,49,125,123,52,32,92,112,105,32,94,50,125,32,92,102,114,97,99,123,92,112,97,114,116,105,97,108,32,125,123,92,112,97,114,116,105,97,108,32,122,95,48,125,92,108,101,102,116,40,92,115,117,109,32,95,123,110,61,49,125,94,123,92,105,110,102,116,121,32,125,32,92,105,110,116,95,123,45,92,105,110,102,116,121,32,125,94,123,92,105,110,102,116,121,32,125,32,101,94,123,105,10,121,32,92,98,101,116,97,32,125,32,92,105,110,116,95,123,45,92,105,110,102,116,121,32,125,94,123,92,105,110,102,116,121,32,125,32,92,102,114,97,99,123,101,94,123,105,32,120,32,92,97,108,112,104,97,32,125,32,92,97,108,112,104,97,32,94,50,32,92,112,115,105,32,95,110,92,108,101,102,116,40,92,98,101,116,97,32,94,50,43,92,97,108,112,104,97,32,94,50,32,92,109,117,32,44,122,92,114,105,103,104,116,41,32,92,112,115,105,32,95,110,92,108,101,102,116,40,92,98,101,116,97,32,94,50,43,92,97,108,112,104,97,32,94,50,10,92,109,117,32,44,122,95,48,92,114,105,103,104,116,41,125,123,92,98,101,116,97,32,94,50,32,92,103,97,109,109,97,32,43,92,97,108,112,104,97,32,94,50,32,92,103,97,109,109,97,32,32,92,109,117,32,45,92,97,108,112,104,97,32,94,50,32,92,108,97,109,98,100,97,32,95,110,92,108,101,102,116,40,92,98,101,116,97,32,94,50,43,92,97,108,112,104,97,32,94,50,32,92,109,117,32,92,114,105,103,104,116,41,125,32,92,44,32,100,92,97,108,112,104,97,32,32,92,44,32,100,92,98,101,116,97,32,92,114,105,103,104,116,41,92,101,110,100,123,101,113,117,97,116,105,111,110,125};

%//fcc

%//run

%//NotebookPut

But not all the time

DoubleBracketingBar[(Subscript[x, n])]==0 //TraditionalForm

s1=StringJoin["$",%//TeXForm//ToString,"$"]

s1//run

This problem reported to technical support ( [TS 15495] ).  Excellent 
workaround suggested by Paritosh Mokhasi:

paritosh[arg_String]:=StringReplace[arg,{"\\left\|"->"\\left|\\left|","\\right\|"->"\\right|\\right|"}]

s1//paritosh//run

%//NotebookPut

LATEX string examples; gould & najmi

gould

Is a sample of well-written LATEX  code by H. Gould

(See http://sip.clarku.edu/tutorials/TeX/introrev.html)

Show pdf file

Definition

gould="\\documentclass[prb,12pt]{revtex4}\n\n\\usepackage{amsmath}    % need for subequations\n\\usepackage{graphicx}   % need for figures\n\\usepackage{verbatim}   % useful for program listings\n\\usepackage{color}      % use if color is used in text\n\\usepackage{subfigure}  % use for side-by-side figures\n\\usepackage{hyperref}   % use for hypertext links, including those to external documents and URLs\n\n% don't need the following. simply use defaults\n\\setlength{\\baselineskip}{16.0pt}    % 16 pt usual spacing between lines\n\n\\begin{comment}\n\\pagestyle{empty} % use if page numbers not wanted\n\\end{comment}\n\n% above is the preamble\n\n\\begin{document}\n\n\\title{Introduction to \\LaTeX}\n\\author{Harvey Gould}\n\\date{December 5, 2006}\n\n\\maketitle\n\n\\section{Introduction}\n\\TeX\\ looks more difficult than it is. It is\nalmost as easy as $\\pi$. See how easy it is to make special\nsymbols such as $\\alpha$,\n$\\beta$, $\\gamma$,\n$\\delta$, $\\sin x$, $\\hbar$, $\\lambda$, $\\ldots$ We also can make\nsubscripts\n$A_{x}$, $A_{xy}$ and superscripts, $e^x$, $e^{x^2}$, and\n$e^{a^b}$. We will use \\LaTeX, which is based on \\TeX\\ and has\nmany higher-level commands (macros) for formatting, making\ntables, etc. More information can be found in Ref.~\\cite{latex}.\n\nWe just made a new paragraph. Extra lines and spaces make no\ndifference. Note that all formulas are enclosed by\n\\$ and occur in \\textit{math mode}.\n\nThe default font is Computer Modern. It includes \\textit{italics},\n\\textbf{boldface},\n\\textsl{slanted}, and \\texttt{monospaced} fonts.\n\n\\section{Equations}\nLet us see how easy it is to write equations.\n\\begin{equation}\n\\Delta =\\sum_{i=1}^N w_i (x_i - \\bar{x})^2 .\n\\end{equation}\nIt is a good idea to number equations, but we can have a\nequation without a number by writing\n\\begin{equation}\nP(x) = \\frac{x - a}{b - a} , \\nonumber\n\\end{equation}\nand\n\\begin{equation}\ng = \\frac{1}{2} \\sqrt{2\\pi} . \\nonumber\n\\end{equation}\n\nWe can give an equation a label so that we can refer to it later.\n\\begin{equation}\n\\label{eq:ising}\nE = -J \\sum_{i=1}^N s_i s_{i+1} ,\n\\end{equation}\nEquation~\\eqref{eq:ising} expresses the energy of a configuration\nof spins in the Ising model.\\footnote{It is necessary to process (typeset) a\nfile twice to get the counters correct.}\n\nWe can define our own macros to save typing. For example, suppose\nthat we introduce the macros:\n\\begin{verbatim}\n \\newcommand{\\lb}{{\\langle}}\n \\newcommand{\\rb}{{\\rangle}}\n\\end{verbatim}\n\\newcommand{\\lb}{{\\langle}}\n\\newcommand{\\rb}{{\\rangle}}\nThen we can write the average value of $x$ as\n\\begin{verbatim}\n\\begin{equation}\n\\lb x \\rb = 3\n\\end{equation}\n\\end{verbatim}\nThe result is\n\\begin{equation}\n\\lb x \\rb = 3 .\n\\end{equation}\n\nExamples of more complicated equations:\n\\begin{equation}\nI = \\! \\int_{-\\infty}^\\infty f(x)\\,dx \\label{eq:fine}.\n\\end{equation}\nWe can do some fine tuning by adding small amounts of horizontal\nspacing:\n\\begin{verbatim}\n \\, small space       \\! negative space\n\\end{verbatim}\nas is done in Eq.~\\eqref{eq:fine}.\n\nWe also can align several equations:\n\\begin{align}\na & = b \\\\\nc &= d ,\n\\end{align}\nor number them as subequations:\n\\begin{subequations}\n\\begin{align}\na & = b \\\\\nc &= d .\n\\end{align}\n\\end{subequations}\n\nWe can also have different cases:\n\\begin{equation}\n\\label{eq:mdiv}\nm(T) =\n\\begin{cases}\n0 & \\text{$T > T_c$} \\\\\n\\bigl(1 - [\\sinh 2 \\beta J]^{-4} \\bigr)^{\\! 1/8} & \\text{$T < T_c$}\n\\end{cases}\n\\end{equation}\nwrite matrices\n\\begin{align}\n\\textbf{T} &=\n\\begin{pmatrix}\nT_{++} \\hfill & T_{+-} \\\\\nT_{-+} & T_{--} \\hfill \n\\end{pmatrix} , \\nonumber \\\\\n& =\n\\begin{pmatrix}\ne^{\\beta (J + B)} \\hfill & e^{-\\beta J} \\hfill \\\\\ne^{-\\beta J} \\hfill & e^{\\beta (J - B)} \\hfill\n\\end{pmatrix}.\n\\end{align}\nand \n\\newcommand{\\rv}{\\textbf{r}}\n\\begin{equation}\n\\sum_i \\vec A \\cdot \\vec B = -P\\!\\int\\! \\rv \\cdot\n\\hat{\\mathbf{n}}\\, dA = P\\!\\int \\! {\\vec \\nabla} \\cdot \\rv\\, dV.\n\\end{equation}\n\n\\section{Tables}\nTables are a little more difficult. TeX\nautomatically calculates the width of the columns.\n\n\\begin{table}[h]\n\\begin{center}\n\\begin{tabular}{|l|l|r|l|}\n\\hline\nlattice & $d$ & $q$ & $T_{\\rm mf}/T_c$ \\\\\n\\hline\nsquare & 2 & 4 & 1.763 \\\\\n\\hline\ntriangular & 2 & 6 & 1.648 \\\\\n\\hline\ndiamond & 3 & 4 & 1.479 \\\\\n\\hline\nsimple cubic & 3 & 6 & 1.330 \\\\\n\\hline\nbcc & 3 & 8 & 1.260 \\\\\n\\hline\nfcc & 3 & 12 & 1.225 \\\\\n\\hline\n\\end{tabular}\n\\caption{\\label{tab:5/tc}Comparison of the mean-field predictions\nfor the critical temperature of the Ising model with exact results\nand the best known estimates for different spatial dimensions $d$\nand lattice symmetries.}\n\\end{center}\n\\end{table}\n\n\\section{Lists}\n\nSome example of formatted lists include the\nfollowing:\n\n\\begin{enumerate}\n\n\\item bread\n\n\\item cheese\n\n\\end{enumerate}\n\n\\begin{itemize}\n\n\\item Tom\n\n\\item Dick\n\n\\end{itemize}\n\n\\section{Figures}\n\nWe can make figures bigger or smaller by scaling them. Figure~\\ref{fig:lj}\nhas been scaled by 60\\%.\n\n\\begin{figure}[h]\n\\begin{center}\n\\includegraphics{figures/sine}\n\\caption{\\label{fig:typical}Show me a sine.}\n\\end{center}\n\\end{figure}\n\n\\begin{figure}[h]\n\\begin{center}\n\\scalebox{0.6}{\\includegraphics{figures/lj}}\n\\caption{\\label{fig:lj}Plot of the\nLennard-Jones potential\n$u(r)$. The potential is characterized by a length\n$\\sigma$ and an energy\n$\\epsilon$.}\n\\end{center}\n\\end{figure}\n\n\\section{Literal text}\nIt is desirable to print program code exactly as it is typed in a\nmonospaced font. Use \\verb \\begin{verbatim} and\n\\verb \\end{verbatim} as in the following example:\n\\begin{verbatim}\ndouble y0 = 10; // example of declaration and assignment statement\ndouble v0 = 0;  // initial velocity\ndouble t = 0;   // time\ndouble dt = 0.01; // time step\ndouble y = y0;\n\\end{verbatim}\nThe command \\verb \\verbatiminput{programs/Square.java}\\ allows\nyou to list the file \\texttt{Square.java} in the directory\nprograms.\n\n\\section{Special Symbols}\n\n\\subsection{Common Greek letters}\n\nThese commands may be used only in math mode. Only the most common\nletters are included here.\n\n$\\alpha, \n\\beta, \\gamma, \\Gamma,\n\\delta,\\Delta,\n\\epsilon, \\zeta, \\eta, \\theta, \\Theta, \\kappa,\n\\lambda, \\Lambda, \\mu, \\nu,\n\\xi, \\Xi,\n\\pi, \\Pi,\n\\rho,\n\\sigma, \n\\tau,\n\\phi, \\Phi,\n\\chi,\n\\psi, \\Psi,\n\\omega, \\Omega$\n\n\\subsection{Special symbols}\n\nThe derivative is defined as\n\\begin{equation}\n\\frac{dy}{dx} = \\lim_{\\Delta x \\to 0} \\frac{\\Delta y}\n{\\Delta x}\n\\end{equation}\n\\begin{equation}\nf(x) \\to y \\quad \\mbox{as} \\quad x \\to\nx_{0}\n\\end{equation}\n\\begin{equation}\nf(x) \\mathop {\\longrightarrow}\n\\limits_{x \\to x_0} y\n\\end{equation}\n\n\\noindent Order of magnitude:\n\\begin{equation}\n\\log_{10}f \\simeq n\n\\end{equation}\n\\begin{equation}\nf(x)\\sim 10^{n}\n\\end{equation}\nApproximate equality:\n\\begin{equation}\nf(x)\\simeq g(x)\n\\end{equation}\n\\LaTeX\\ is simple if we keep everything in proportion:\n\\begin{equation}\nf(x) \\propto x^3 .\n\\end{equation}\n\nFinally we can skip some space by using commands such as\n\\begin{verbatim}\n\\bigskip    \\medskip    \\smallskip    \\vspace{1pc}\n\\end{verbatim}\nThe space can be negative.\n\n\\section{Use of Color}\n\n{\\color{blue}{We can change colors for emphasis}},\n{\\color{green}{but}} {\\color{cyan}{who is going pay for the ink?}}\n\n\\section{\\label{morefig}Subfigures}\n\nAs soon as many students start becoming comfortable using \\LaTeX, they want\nto use some of its advanced features. So we now show how to place two\nfigures side by side.\n\n\\begin{figure}[h!]\n\\begin{center}\n\\subfigure[Real and imaginary.]{\n\\includegraphics[scale=0.5]{figures/reim}}\n\\subfigure[Amplitude and phase.]{\n\\includegraphics[scale=0.5]{figures/phase}}\n\\caption{\\label{fig:qm/complexfunctions} Two representations of complex\nwave functions.}\n\\end{center}\n\\end{figure}\n\nWe first have to include the necessary package,\n\\verb+\\usepackage{subfigure}+, which has to go in the preamble (before\n\\verb+\\begin{document}+). It sometimes can be difficult to place a figure in\nthe desired place.\n\nYour LaTeX document can be easily modified to make a poster or a screen\npresentation similar to (and better than) PowerPoint. Conversion to HTML is\nstraightforward. Comments on this tutorial are appreciated.\n\n\\begin{thebibliography}{5}\n\n\\bibitem{latex}Helmut Kopka and Patrick W. Daly, \\textsl{A Guide to\n\\LaTeX: Document Preparation for Beginners and Advanced Users},\nfourth edition, Addison-Wesley (2004).\n\n\\bibitem{website}Some useful links are\ngiven at \\url{<http://sip.clarku.edu/tutorials/TeX/>}.\n\n\\end{thebibliography}\n\n{\\small \\smallskip\\noindent Updated 5 December 2006.}\n\\end{document}";

najmi

Authored by Amir Najmi, a colleague at JHUAPL

Is the first chapter of a soon-to-be-published book on wavelets

Definition

najmi="\\documentclass[11pt,twoside]{book}\n\\usepackage{etex}\n\\reserveinserts{28}\n\\usepackage{latexsym}\n\\usepackage{amssymb}\n\\usepackage{amsmath}\n\\usepackage{amsthm}\n\\usepackage{mathrsfs}\n\\usepackage{graphicx}\n\\usepackage{makeidx}\n\\usepackage{fancyhdr}\n\\usepackage[section]{placeins}\n\\usepackage{array}\n\\usepackage{multirow}\n\\usepackage{rotating}\n\\usepackage{url}\n\\usepackage{setspace}\n%\\usepackage{times}\n%\\usepackage{mathptmx}\n\n\n\n\n\n\\makeindex\n\n\n%\\doublespacing\n\\begin{document}\n\n\n\\frontmatter\n\\pagenumbering{roman} \n\\title{Wavelets: \\\\\n   A Concise Course}\n\\author{Amir-Homayoon Najmi \\footnote{\\textit{Amir-Homayoon Najmi completed the Mathematical Tripos at Cambridge university in 1977, was a Fulbright scholar at the Relativity Center of University of Texas at Austin in 1978.  He obtained his D.Phil. at the Astrophysics Department of Oxford University in 1982 with a thesis on quantum field theory in curved space times.    He taught at the physics and mathematics departments of the University of Utah 1982-86 and was a research physicist at the Shell Oil Bellaire Geophysical Research Center 1986-1990.  Since 1990 he has been at Applied Physics Laboratory of the Johns Hopkins University and is currently in the Electromagnetic and Acoustics Group.  He teaches in the Physics and the Electrical Engineering programs of the Johns Hopkins University Whiting School Master's degree for professionals.}}, M.A. (Cantab.), D.Phil. (Oxon.)}\n\n\n\\maketitle\n\n\\normalsize\n\n\n\\include{book-dedication}\n\\include{book-preface}\n\\include{book-acknowledgement}\n\n\n\\thispagestyle{empty}\n\\newpage\n\\pagestyle{plain} \n\\tableofcontents\n\\listoftables\n\\listoffigures\n\\newpage\n\\mainmatter\n\\pagenumbering{arabic} \n\n\\theoremstyle{plain}       \\newtheorem{thm}{Theorem}\n\\theoremstyle{definition}  \\newtheorem{defn}{Definition}\n%\\theoremstyle{remark}      \\newtheorem{rem}{thm}{Remark}\n\n\n\n\n\n\n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n%\n%                                        CHAPTER 1\n%\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\n\n\n\\chapter{Analysis in Vector and Function Spaces}\n\\label{Chapter_AlgebraAnalysis}\n\n\n\\section{Introduction}\n\nIndependent variables of interest are either continuous or discrete.  If we denote by $t$ a continuous variable, $-\\infty <t<\\infty $, then  the range of $t$ defines the real line $\\mathbb{R}$.  If we denote a discrete variable by $n$, then the range  $n=0,\\pm 1,\\pm 2,\\cdots$ defines the set of integers $\\mathbb{Z}$.  Thus, we write $t \\in \\mathbb{R}$ and $n \\in \\mathbb{Z}$ introducing the symbol $ \\in $ to mean ``belongs to\" or ``is a member of\".  The negation of the latter is $\\notin$.  We define the range of ordered pairs $(x,y)$, $x,y \\in \\mathbb{R}$, to be the real plane and we denote it by ${\\mathbb{R}}^2$.  The set of all ``grid\" points $(m,n)$, $m,n \\in \\mathbb{Z}$, is denoted by ${\\mathbb{Z}}^2$.  There are times when we need to refer to the positive or negative ``half\" of the real line or the set of integers in which case we modify the notation in an obvious way.  For instance, ${\\mathbb{R}}^+$ denotes the positive real line and ${\\mathbb{Z}}^+$ denotes the positive integers.  The complex plane $\\mathbb{C}$ is defined as the range space of the complex variable $z=x+iy$, $x,y \\in \\mathbb{R}$ and $i\\equiv \\sqrt {-1}$.  \\footnote{Note that the symbol $\\equiv$ is used to mean ``defined\" or ``is equivalent to\" which are distinct from the equality sign used to mean ``is equal to\".}  Complex conjugation is denoted by $\\ast$, for example, the complex conjugate of $z=x+iy$ is $z^\\ast =x-i y$.  In addition, we use the symbol $\\forall$ to mean for all, $\\exists$ meaning there exists, $\\Leftrightarrow $ meaning if and only if, and set theoretic symbols $\\cup$ for union of sets and $\\cap$ for their intersection.\n\nWe think of all signals as real or complex valued functions of either a continuous time variable $-\\infty <t<\\infty $, e.g., $x(t)$, or a discrete time variable $n=0,\\pm 1,\\pm 2,\\cdots $, e.g., $x\\left[ n \\right]$.  Although many signals of interest are real quantities, some circumstances necessarily produce complex signals.  For instance, the Fourier transform (see section \\ref{Section_FT} of chapter \\ref{Chapter_AlgebraAnalysis}) of a real signal is a complex signal.  A second example is a signal processing operation known as complex demodulation whereby a real signal whose spectrum is limited to a small band of frequencies around a very high carrier frequency $\\omega_c$ is shifted down in frequency retaining the original shape of the spectrum but now centered around zero frequency (also known as DC).  The process of shifting down of the frequencies involves a multiplication of the original real signal with the complex exponential $\\exp \\left( {i\\omega _c t} \\right)$, thus producing a complex signal.  A third example of complex signals is provided by Magnetic Resonance Imaging (MRI) in which the circularly polarized changing magnetic field, the source of the MRI signal, results in two data streams with a $90^ \\circ$ phase difference that are combined together to form a single complex sequence of numbers.  Whether real or complex we think of the space of all signals as a space of functions with properties that we will discuss in this chapter.  Two spaces of particular interest are those in the following two definitions.   \n\\newtheorem{L2R}{Definition}\n\\begin{defn}\n\\label{FinitEnergyFunctions}\nThe space of continuous time signals $x(t)$ with finite energy, $\\int\\limits_{ - \\infty }^\\infty  {\\left| {x\\left( t \\right)} \\right|^2 } dt < \\infty $, is denoted by $L_2 \\left( \\mathbb{R} \\right)$, where $\\mathbb{R}$ is the real line.\n\\end{defn}\n\\newtheorem{L1R}{Definition}\n\\begin{defn}\nThe space of continuous time signals $x(t)$ that are absolutely integrable, i.e., functions $x(t)$ for which $\\int\\limits_{ - \\infty }^\\infty  {\\left| {x\\left( t \\right)} \\right|} dt < \\infty $, is denoted by $L_1 \\left( \\mathbb{R} \\right)$, where $\\mathbb{R}$ is the real line.\n\\end{defn}\nIn the next section we describe the meaning of the integrals in the above definitions.  It is important to note that in signal analysis there are two broad categories of functions of interest:  functions with finite energy (definition \\ref{FinitEnergyFunctions}), and functions such as thermal noise at a receiver that do not possess finite energy but have finite power (energy per unit time) defined below\n\\[\n\\mathop {{\\rm Lim}}  \\limits_{T \\to \\infty } \\frac{1}{{2T}}\\int\\limits_{ - T}^T {\\left| {x\\left( t \\right)} \\right|^2 dt} .\n\\]\nAlthough signals with finite energy are of interest, all functions with finite power (such as thermal noise) could be included if we assume that a window function of finite duration is applied: that is, the function is set to zero outside a finite time range.\n\n\n\n\n\n\\section{The Lebesgue Integral}\n\n\nAll integrals in this text are Lebesgue\\index{Lebesgue integral} integrals which are based on the concept of the Lebesgue measure \\cite{Burk71}.  The Lebesgue integral is a more general concept than that of the Riemann\\index{Riemann} integral which is based on the concept of ``the limit of a sum\".  A difficulty with the Riemann integral can be illustrated with the following example.  Consider the function $x(t)$ defined on the closed interval $[0,1]$ by the rule that it is equal to $1$ when $t$ is a rational number, and it is equal to $0$ when $t$ is irrational.  Proceeding with the concept of a Riemann integral we divide the interval into $N$ subintervals $\\left[ {t_n ,t_{n + 1} } \\right]$ of equal length $\\Delta t$, $t_0=0$ and $t_N=1$, and consider the sum\n\\[\n\\sum\\limits_{n = 0}^ {N} {x\\left( {\\bar t_n } \\right)\\Delta t} ~,\n\\]\nwhere $\\bar t_n$ is any point in the interval $\\left[ {t_n ,t_{n + 1} } \\right]$.  The limit of this sum as $N \\to \\infty $, if it exists, is the Riemann integral of our example function.  The result, of course, must be independent of the choice of evaluation point $\\bar t_n$.  It is clear, however, that no matter how large $N$ is the corresponding intervals contain at least one rational and one irrational number and the result of the sum can be made equal to $1$ if we always choose a rational number for all the points $\\bar t_n$, or zero if we choose irrationals!  Thus, the limit does not exist and the function is not integrable in the Riemann sense.  \n\nThe Lebesgue integral, on the other hand, is based on the concept of the Lebesgue measurable sets.  The simplest example of a measurable set is the closed interval $[T_1, T_2]$ in $\\mathbb{R}$ whose measure is the length of the interval $T_2-T_1$.  In fact, the Lebesgue measure of the semi open sets $(T_1, T_2]$ and $[T_1, T_2)$, and that of the open set $(T_1, T_2)$ are all given by the same length.  Thus, the Lebesgue measure of one or two points is zero, as is the measure of a countable\\index{countable} set of points, where countable means that the set can be put into a one to one correspondence with the integers \\footnote{Although we take the measure of a countable set to be zero, countability is not a necessary condition for a set to have zero measure.  The most well known example of an uncountable\\index{uncountable} set with zero measure is the Cantor\\index{Cantor} set that is created by repeatedly deleting the open middle thirds of a set of line segments. Starting with the closed interval $[0,1]$ we delete the open middle third $\\left( {{1 \\mathord{\\left/ {\\vphantom {1 3}} \\right. \\kern-\\nulldelimiterspace} 3},{2 \\mathord{\\left/ {\\vphantom {2 3}} \\right. \\kern-\\nulldelimiterspace} 3}} \\right)$ leaving the set $\\left[ {0,{1 \\mathord{\\left/ {\\vphantom {1 3}} \\right. \\kern-\\nulldelimiterspace} 3}} \\right] \\cup \\left[ {{2 \\mathord{\\left/ {\\vphantom {2 3}} \\right. \\kern-\\nulldelimiterspace} 3},1} \\right]$, and continue by removing the middle thirds of each of the remaining intervals.  The final removed set is then $\\mathop  \\cup \\limits_{j = 1}^\\infty  \\mathop  \\cup \\limits_{k = 0}^ {(3^{j-1}-1)}  3^{ - j} \\left( {1 + 3k,2 + 3k} \\right)$ which can be shown to have measure $1$.  The Cantor set is the uncountable closed set of remaining points and has measure $0$.}.  For instance, consider the set of all positive rational numbers ${\\mathbb{Q}}^+$ whose members are of the form ${n \\mathord{\\left/ {\\vphantom {n m}} \\right. \\kern-\\nulldelimiterspace} m}$, $n,m \\in {\\mathbb{Z}}^+$.  We can ``count'' them by writing them in the form of a matrix whose row is labeled by $n$ and whose column is labeled by $m$ and then mapping the $(1,1)$ element to $1$, elements $(2,1), (1,2)$ to $2,3$, elements $(3,1), (2,2), (1,3)$ to $4,5,6$, and so on.  Consequently the set ${\\mathbb{Q}}^+$ is countable and has measure zero.  \n\nNow consider the function $x(t)$ defined on the interval $[t_a, t_b]$ whose range is $[x_a, x_b]$.  In general, $x(t)$ is a many to one map between the two intervals in the sense that there may be many $t$ values that will give the same $x$ value.  The Riemann integral is defined by dividing the domain of the map $[t_a, t_b]$ into a number of equal length subintervals as was discussed in the last paragraph.  To define the Lebesgue integral we begin by dividing the range of the map into $N$ equal length subintervals $\n\\left[ {x_k ,x_{k + 1} } \\right]$, $k=0, \\cdots, N$, $x_{_0}=x_{_a}$, $x_{_N}=x_{_b}$.  Denoting the Lebesgue measure of the set $x^{-1} \\left({\\left[ {x_k ,x_{k + 1}} \\right]} \\right)$ (i.e., the set whose image under the function $x$ is the interval $\\left[ {x_k ,x_{k + 1} } \\right]$) by $M_k$, the Lebesgue integral of the function $x(t)$ on the interval $[t_a, t_b]$ is then defined by\n\\[\n\\mathop {Lim}\\limits_{N \\to \\infty } \\sum\\limits_{k = 0}^N {M_k x_k } ~, \n\\]\nprovided that the limit exists.  It is important to note, however, that if the Riemann integral of a given function exists, then it is equal to the Lebesgue integral and there is no distinction between the two in that case.  Later in this chapter when we define inner products and norms of functions, we will have to make no distinction between functions in $L_2(\\mathbb{R})$ that are the same everywhere except on a set of measure $0$.  For instance, the function defined earlier on $[0,1]$ that took the value $1$ on all the rationals and $0$ on all the irrationals, will be considered to be the same as the function $0$ defined on the same interval, and so its Lebesgue integral is $0$.  Thus, we are implicitly considering equivalence classes of functions \\footnote{Equivalence class structures arise from equivalence relations.  A binary relation $R$ on a set $\\left\\{ {x,y,z, \\ldots } \\right\\}$, written $x ~ R ~ y$ to denote the relation between $x$ and $y$, is said to be an equivalence relation if it is symmetric, i.e., :  $x~R~y \\Rightarrow y~R~x$, and it is reflexive, i.e., :  $x~R~x$, and finally it is transitive, i.e., :  $x~R~y, ~~ y~R~z  \\Rightarrow x~R~z$.  The equivalence class $[x]$ then contains all elements of the set that are related (equivalent) to $x$.} instead of individual functions: functions belonging to the same class only differ from each other on a set of measure $0$.  In other words, finite or countable number of points (as well as any other set of measure $0$) do not count!  \n\n\\newtheorem{AlmostEveryWhere}{Definition}\n\\begin{defn}\n\\label{AlmostEveryWhere}\nTwo functions $x(t)$ and $y(t)$ in $L_2(\\mathbb{R})$ are said to be the same almost everywhere if they differ from each other only on a set of measure zero, i.e., the measure of the set set $\\left\\{ {t \\in R: ~ x(t) \\ne y(t)} \\right\\}$ is zero.\n\\end{defn}\nHence, an equivalence class structure in $L_2(\\mathbb{R})$ can be defined where all functions that are the same almost everywhere belong to the same class.  The equivalence class structure of $L_2(\\mathbb{R})$ will allow us in later chapters to define inner products and norms unambiguously.     \n\n\n\n\\section{Discrete Time Signals}\n\n\nIn most modern applications a continuous time signal $x(t)$ is converted to a discrete time series denoted by $x\\left[ n \\right]$ representing the signal $x\\left( {n \\Delta t} \\right)$, where $\\Delta t$ is the sampling interval of the analogue to digital converter, and is related to the sampling frequency $f_s$ by $\\Delta t = {1 \\mathord{\\left/ {\\vphantom {1 {f_s }}} \\right. \\kern-\\nulldelimiterspace} {f_s }}$.  When $\\Delta t$ is measured in seconds the sampling frequency is measured in Hertz (Hz).  The space of finite energy  discrete time signals is defined as follows.   \n\\newtheorem{L2Z}{Definition}\n\\begin{defn}\nThe space of discrete time signals of finite energy $x\\left[ n \\right]$, $\\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {x\\left[ n \\right]} \\right|^2 }  < \\infty$, is denoted by $l_2 \\left( \\mathbb{Z} \\right)$, where $\\mathbb{Z}$ is the set of integers.  \n\\end{defn}\n\n\n\n\\section{Vector Spaces}\n\n\nThe most basic algebraic structure of interest is the vector space\\index{vector space} \\cite{Halm74}. In its most concise form it defines the space of all functions of interest to be such that if $s_1 \\left( t \\right)$ and $s_2 \\left( t \\right)$ are two signals in the space, then for all complex numbers $\\alpha $ and $\\beta $, the linear combination $\\alpha s_1 \\left( t \\right)+\\beta s_2 \\left( t \\right)$ also belongs to the space.  Thus,  the space is closed under the rules of addition of functions and multiplication of functions by complex numbers.  The set of real numbers $\\mathbb{R}$, complex numbers $\\mathbb{C}$, and rationals $\\mathbb{Q}$ are examples of a mathematical entity known as a field which is required in the definition of a vector space.  An abstract field is a set with notions of addition and multiplication and their corresponding inverses, namely, subtraction and division.\n\\newtheorem{VectorSpace}{Definition}\n\\begin{defn}\nA vector space $\\mathscr{V}$ defined over a field  $\\mathbb{F}$ is a collection of objects $\\mathbf{x}$ called vectors together with a binary operation called addition, and a scalar multiplication by the members of the field satisfying the following axioms.\n\\begin{itemize}\n\t\\item  To every pair, $\\mathbf{x}$ and $\\mathbf{y}$, of vectors corresponds a vector $\\mathbf{x+y}$ that is called their sum.  Addition of vectors is commutative and associative.\n\t\\item  There is a unique vector $\\mathbf{0}$ such that $\\mathbf{x+0=x}$ for all $\\mathbf{x} \\in \\mathscr{V}$.\n\t\\item  For every vector $\\mathbf{x} \\in \\mathscr{V}$ there is a unique vector $\\mathbf{-x}$ such that $\\mathbf{x+(-x)}=\\mathbf{0}$.\n\t\\item  For every vector $\\mathbf{x} \\in \\mathscr{V}$ and every scalar $\\alpha \\in \\mathbb{F}$ there corresponds a vector $\\alpha \\mathbf{x}$, the product of $\\alpha$ and $\\mathbf{x}$ and multiplication by scalars is associative.\n\t\\item  The product $1 \\mathbf{x}$ is equal to $\\mathbf{x}$ for every vector $\\mathbf{x} \\in \\mathscr{V}$. \n\t\\item  Multiplication by scalars is distributive with respect to vector addition, i.e., $\\alpha \\mathbf{(x+y)} = \\alpha \\mathbf{x} + \\alpha \\mathbf{y}$.\n\t\\item  Multiplication by vectors is distributive with respect to scalar addition, i.e., $(\\alpha+\\beta) \\mathbf{x} = \\alpha \\mathbf{x} + \\beta \\mathbf{x}$.\n\\end{itemize}\n\\end{defn}\nIf the field is the set of real numbers $\\mathbb{R}$ then $\\mathscr{V}$ is called a real vector space, and if the field is $\\mathbb{C}$ then $\\mathscr{V}$ is a complex vector space.  These are the only two fields of interest to us in this book.\n\n\n\\section{Linear Independence}\nA linear combination of vectors denoted by $\\sum\\limits_{j=1}^N {c_j \\mathbf{x}_j }$ is clearly a member of the vector space. \n\\newtheorem{LinearIndependence}{Definition}\n\\begin{defn}\nA set of vectors $\\left\\{ {\\mathbf{x}_j :\\;j=1,\\ldots ,N} \\right\\}$ is linearly independent provided \n\\begin{equation}\n\\label{Chapter1-1}\n\\sum\\limits_{j=1}^N {c_j \\mathbf{x}_j } ={0}\\Leftrightarrow c_j =0\\,~~ \\forall j ~.\n\\end{equation}\n\\end{defn}\n\\newtheorem{LinearDependence}{Theorem}\n\\begin{thm}\n\\label{LinearDependence}\nThe set of non zero vectors $\\left\\{ {\\mathbf{x}_j :\\;j=1,\\ldots ,N} \\right\\}$ is linearly dependent if and only if some $\\mathbf{x}_k$, $k=1,\\ldots ,N$, is a linear combination of the preceding vectors.\n\\end{thm}\n\n\\section{Basis And Basis Vectors}\n\n\\newtheorem{Basis}{Definition}\n\\begin{defn}\nA basis\\index{basis!finite} in a vector space consists of a set of linearly independent vectors, called basis vectors, such that every vector in the given space is a linear combination of the basis vectors.  A vector space is finite dimensional if it has a finite basis. \n\\end{defn}\nThe basis is not unique, although once chosen, the representation of any vector in terms of the basis (the ``components'') is unique.  Every linearly independent set can be extended to a basis by the following theorem.\n\\newtheorem{BasisExtend}{Theorem}\n\\begin{thm}\n\\label{BasisExtend}\nThe set of linearly independent vectors $\\left\\{ {\\mathbf{x}_j :\\;j=1,\\ldots ,M} \\right\\}$ in a finite dimensional vector space either forms a basis, or it can be extended to a set $\\left\\{ {\\mathbf{x}_j :\\;j=1,\\ldots ,N} \\right\\}$, where $N>M$ which is a basis.\n\\end{thm}\n\\newtheorem{Dimension}{Theorem}\n\\begin{thm}\n\\label{Dimension}\nThe number of elements in any basis of a finite dimensional vector space is the same as in any other basis.  This number $N$ is the dimension\\index{dimension} of the corresponding vector space.  In addition, every set of $N+1$ vectors in an $N$-dimensional vector space is linearly dependent. \n\\end{thm}\n\n\n\n\\newtheorem{SpanOfVectors}{Definition}\n\\begin{defn}\nGiven a set of vectors $\\left\\{ {\\mathbf{x}_j :\\;j=1,\\ldots ,N} \\right\\}$ which are not necessarily linearly independent, and a field $\\mathbb{F}$, the Span of this set is defined by \n\\begin{equation}\n\\label{Chapter1-2}\nSpan\\left\\{ {\\mathbf{x}_j :\\;j=1,\\ldots ,N} \n\\right\\}=\\left\\{ {\\sum\\limits_{j=1}^N {c_j \\mathbf{x}_j ,\\;\\forall c_j \\in \\mathbb{F}} } \\right\\} ~.\n\\end{equation}\n\\end{defn}\nNote that the Span of a set of vectors satisfies all the conditions required for a vector space.  So if the span of a set of vectors is not the entire space, then it is a proper subspace.  \n\n\\newtheorem{InfiniteDimensional VectorSpace}{Definition}\n\\begin{defn}\nAn infinite dimensional vector space is a vector space in which no linear span of the form \\eqref{Chapter1-2} is the entire space, for any integer $N$.\n\\end{defn}\n\\noindent In other words, for any given integer $N$, however large, and any set of vectors $\\mathbf{x}_j, ~ j=1, \\ldots, N$, there is at least one vector $\\mathbf{y}$ that is not in the linear span of those $N$ vectors.\n\nWe continue to denote a vector in a finite dimensional vector space by bold letters, e.g., ${\\mathbf{x}}$.  An infinite dimensional vector space in this text is a space of functions of a real variable, e.g. $x \\left( t \\right)$, which we often abbreviate to $x$, dropping the dependence on the real variable $t$.  Although many definitions and properties in this chapter use bold letters that refer to vectors in finite dimensional vector spaces, most are equally applicable to infinite dimensional space of functions.  The two notations are for the most part interchangeable unless otherwise stated.   \n\n\\newtheorem{InnerAndDirectSum}{Definition}\n\\begin{defn}\n\\label{InnerAndDirectSum}\nIf $\\mathbb{V}$ and $\\mathbb{W}$ are subspaces of the vector space $\\mathscr{V}$, the inner sum, denoted by  $\\mathbb{V+W}$, is a subspace defined by \n\\[\n\\mathbb{V+W} \\equiv \\left\\{ {\\mathbf{u}\\in \\mathscr{V}:\\;\\mathbf{u}=\\mathbf{v+w};\\;\\mathbf{v}\\in \\mathbb{V},\\mathbf{w}\\in \\mathbb{W}} \\right\\}, \n\\]\nwhile the direct sum of the two subspaces, denoted by $\\mathbb{V}\\oplus \\mathbb{W}$, is defined by \n\\[\n\\mathbb{V}\\oplus \\mathbb{W} \\equiv \\left\\{ {\\left( {\\mathbf{v},\\mathbf{w}} \\right): ~ \\mathbf{v}\\in \\mathbb{V},\\mathbf{w}\\in \\mathbb{W}} \\right\\}. \n\\]\n\\end{defn}\n\n\\noindent If $\\mathbb{V}$ and $\\mathbb{W}$ are disjoint, i.e. $\\mathbb{V}\\cap \\mathbb{W}=\\left\\{ {\\mathbf{0}} \\right\\}$, then $\\mathbb{V+W}$ and $\\mathbb{V}\\oplus \\mathbb{W}$ are isomorphic, i.e. they have identical algebraic structure.  \n\nIn order to do analysis we must be able to tell how close vectors are to each other, and so we introduce the concept of normed vector spaces in the next section \\cite{BaNa66}.\n\n\\section{Normed Vector Spaces}\n\n\\newtheorem{NormedSpace}{Definition}\n\\begin{defn}\nA Normed Vector Space is equipped with a non negative real valued function (the Norm), $\\left\\| {\\;\\cdot \\,} \\right\\|:\\mathscr{V}\\to {\\rm \\mathbb{R}}$, such that\n\\begin{subequations}\n\\label{Chapter1-3}\n\\begin{align}\n \\left\\| {\\mathbf{x}} \\right\\| & \\ge 0 ~ \\forall {x}\\in \\mathscr{V} ~, \\label{Chapter1-3a}\\\\ \n \\left\\| {\\mathbf{x}} \\right\\| & = 0\\Leftrightarrow \\mathbf{x=0} ~, \\label{Chapter1-3b}\\\\ \n \\left\\| {a \\mathbf{x}} \\right\\| & =\\left| a \\right|\\left\\| {\\mathbf{x}} \\right\\| ~, \\label{Chapter1-3c}\\\\ \n \\left\\| \\mathbf{x+y} \\right\\| & \\le \\left\\| {\\mathbf{x}} \\right\\|+\\left\\| {\\mathbf{y}} \n\\right\\|\\;\\quad \\forall \\mathbf{x},\\mathbf{y}\\in \\mathscr{V} \\label{Chapter1-3d} ~. \n \\end{align}\n\\end{subequations}\n\\end{defn}\n\\noindent Condition \\eqref{Chapter1-3d} is known as the triangle inequality\\index{triangle inequality}.  For instance, if $\\mathscr{V}$ is the infinite dimensional vector space of all square-integrable functions defined on the interval $\\left[ {a,b} \\right]\\in {\\rm \\mathbb{R}}$, the $L_2$ norm of a function ${x\\left( t \\right)}$ is defined by \n\\[\n\\left\\| {x\\left( t \\right)} \\right\\|_2 =\\left[ {\\int_a^b {\\left| {x\\left( t \\right)} \\right|^2dt} } \\right]^{{1 \\mathord{\\left/\n {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} ~. \n\\]\nThis can be generalized to an $L_p $ norm where $1\\le p<\\infty $ and $\\mathscr{V}$ is the vector space of all functions whose magnitude raised to the power $p$ is integrable\n\\[\n\\left\\| {x\\left( t \\right)} \\right\\|_p =\\left[ {\\int_a^b {\\left| {x\\left( t \\right)} \\right|^pdt} } \\right]^{{1 \\mathord{\\left/\n {\\vphantom {1 p}} \\right. \\kern-\\nulldelimiterspace} p}} ~.\n\\]\n\nThe norm enables us to define convergence in a normed vector space.  In the infinite dimensional space $L_{2}(\\mathbb{R})$, the space of functions with finite $L_2$ norm, three types of convergence can be defined.  \n\\newtheorem{ConvergenceInTheMean}{Definition}\n\\begin{defn}\n\\label{ConvergenceInTheMean}\nA sequence $x_{n}(t)$, $n = 0, 1, 2, \\ldots$ converges in the mean to the function $x(t)$ if $\\left\\| {x_n  - x} \\right\\| \\to 0$ for \n$n \\to \\infty$.  \n\\end{defn}\nUnless otherwise stated, this is the definition for convergence that is used throughout this book.  Here we will state the other two possible definitions for convergence.\n\\newtheorem{ConvergencePointwise}{Definition}\n\\begin{defn}\nA sequence $x_{n}(t)$, $n = 0, 1, 2, \\ldots$ converges pointwise to the function $x(t)$ if given the point $t$ and a small positive number $\\varepsilon$, an integer $N_{t}$ can be found for which $\\left| {x_n \\left( t \\right) - x\\left( t \\right)} \\right| < \\varepsilon $\nfor all $n \\ge N_t$.\n\\end{defn}\n\\newtheorem{ConvergenceUniform}{Definition}\n\\begin{defn}\nA sequence $x_{n}(t)$, $n = 0, 1, 2, \\ldots$ converges uniformly to the function $x(t)$ if for any small positive number $\\varepsilon$, an integer $N$ can be found that is independent of $t$ and for which $\\left| {x_n \\left( t \\right) - x\\left( t \\right)} \\right| < \\varepsilon $\nfor all $n \\ge N$.\n\\end{defn}\nGiven a sequence of functions $x_{n}(t)$ converging in the mean to the function $x(t)$, and using the triangle inequality we have\n\\[\n\\left\\| {x_n  - x_m } \\right\\| = \\left\\| {x_n  - x + x - x_m } \\right\\| \\le \\left\\| {x_n  - x} \\right\\| + \\left\\| {x_m  - x} \\right\\|~,\n\\]\nwhich in view of the convergence property, can be made as small as we please by taking the integers $n$ and $m$ sufficiently large.  This motivates the following definition.\n\\newtheorem{CauchySequence}{Definition}\n\\begin{defn}\n\\label{CauchySequence}\nA sequence $x_{n}(t)$, $n = 0, 1, 2, \\ldots $ is called a Cauchy sequence\\index{Cauchy sequence} if, for any $\\varepsilon > 0$ there is a positive integer $N$ (which could depend on $\\varepsilon$) such that $\\left\\| {x_n  - x_m } \\right\\| < \\varepsilon $ for $n,m \\ge N$.\n\\end{defn}\n\\noindent All convergent sequences are Cauchy sequences, but the converse is not true.\n\\newtheorem{ClosedSubspace}{Definition}\n\\begin{defn}\n\\label{ClosedSubspace}\nA subspace of a normed vector space is closed if all convergent sequences in that subspace converge to points within that subspace.\n\\end{defn}\n\n\n\n\\section{Inner Product}\nAn inner product\\index{inner product} can be used to project vectors in arbitrary directions, in addition to introducing the concept of orthogonality between vectors. \n\\newtheorem{InnerProduct}{Definition}\n\\begin{defn}\nThe inner product is a complex function $<\\cdot ,\\cdot >: \\mathscr{V}\\times \\mathscr{V}\\to {\\rm \\mathbb{C}}$ with the following properties \\footnote{The space $\\mathscr{V} \\times \\mathscr{V}$ denotes the set of all ordered pairs $\\left( {\\mathbf{x},\\mathbf{y}} \\right)$ where $\\mathbf{x},\\mathbf{y} \\in \\mathscr{V}$.} \n\\begin{subequations}\n\\label{Chapter1-4}\n\\begin{align}\n \\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle & = \\left\\langle {\\mathbf{y},\\mathbf{x}} \\right\\rangle ^\\ast \\label{Chapter1-4a} \\\\ \n \\left\\langle {\\mathbf{x}},a\\,{\\mathbf{y}} \\right\\rangle & = a \\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle \\label{Chapter1-4b} \\\\ \n \\left\\langle {\\mathbf{x+y},\\mathbf{z}} \\right\\rangle & = \\left\\langle {\\mathbf{x},\\mathbf{z}} \\right\\rangle + \\left\\langle {\\mathbf{y},\\mathbf{z}} \\right\\rangle \\label{Chapter1-4c}\\\\ \n \\left\\langle\\ {\\mathbf{x},\\mathbf{x}} \\right\\rangle \\;\\; & > 0,\\quad \\forall \\mathbf{x}\\ne \\mathbf{0} \\label{Chapter1-4d} \\\\ \n \\left\\langle {\\mathbf{x},\\mathbf{x}} \\right\\rangle & = 0\\Leftrightarrow \\mathbf{x}=\\mathbf{0} \\label{Chapter1-4e}\n\\end{align}\n\\end{subequations}\n\\end{defn}\nFor example, if $\\mathscr{V}={\\rm \\mathbb{R}}^n$ and $\\mathbf{x}=\\left[ {{\\begin{array}{*{20}c} {x_1 } \\hfill & {x_2 } \\hfill & \\cdots \\hfill & {x_n } \\hfill \\\\\\end{array} }} \\right]^T,\\,x_j \\in {\\rm \\mathbb{R}}$ and superscript $T$ denotes transposition, then \n\\begin{equation}\n\\label{Chapter1-5}\n\\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle = \\mathbf{x}\\cdot \\mathbf{y}=\\mathbf{x}^T \\mathbf{y}=\\sum\\limits_{j=1}^n {x_j y_j } \n\\end{equation}\nIf, on the other hand, $\\mathscr{V}={\\rm \\mathbb{C}}^n$ and $x_j \\in {\\rm \\mathbb{C}}$, then\n\\begin{equation}\n\\label{Chapter1-6}\n\\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle=\\mathbf{x}^\\ast \\cdot \\mathbf{y}=\\mathbf{x}^H \\mathbf{y}=\\sum\\limits_{j=1}^n {x_{_j }^\\ast y_j} \n\\end{equation}\nThe superscript $H$ denotes the Hermitian conjugate, $\\mathbf{x}^H\\equiv \\left[{{\\begin{array}{*{20}c}\n {x_1^\\ast } \\hfill & \\cdots \\hfill & {x_n^\\ast } \\hfill \\\\\\end{array} }} \\right]$. Note that the relation \n\\[\n\\left\\langle {a \\mathbf{x},\\mathbf{y}} \\right\\rangle =a^\\ast \\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle\n\\]\nis a consequence of the properties \\eqref{Chapter1-4a} and \\eqref{Chapter1-4b} of the inner product.  The usual example of an inner product on a function space is \n\\[\n\\left\\langle {x,y} \\right\\rangle  = \\int\\limits_{ - \\infty }^\\infty  {x^ *  \\left( t \\right)} y\\left( t \\right)dt\n\\]\nwhere  and $x$ and $y$ are square integrable complex valued functions of the continuous variable $t \\in \\mathbb{R}$.\n\nAn inner product space has a natural norm, called the induced norm\\index{norm}, associated with the inner product\n\\[\n\\left\\| {\\mathbf{x}} \\right\\|\\equiv \\sqrt {\\left\\langle {\\mathbf{x},\\mathbf{x}} \\right\\rangle} ~.\n\\]\nThere is a converse to the above relation in the sense that the knowledge of the norm in an inner product space is sufficient to recover the associated inner product, as shown in the following theorem known as the Polarization identity.\n\\newtheorem{PolarizationId}{Theorem}\n\\begin{thm}\n\\label{PolarizationId}\nLet $\\mathscr{V}$ be an inner product space and let $\\mathbf{x}$ and $\\mathbf{y} \\in \\mathscr{V}$.  Then, if $\\mathscr{V}$ is a real space\n\\[\n\\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle  \\equiv \\frac{1}{4}\\left\\| {\\mathbf{x} + \\mathbf{y}} \\right\\|^2  - \\frac{1}{4}\\left\\| {\\mathbf{x} - \\mathbf{y}} \\right\\|^2 ~,\n\\]\nand if $\\mathscr{V}$ is a complex space\n\\[\n\\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle  \\equiv \\frac{1}{4}\\left\\| {\\mathbf{x} + \\mathbf{y}} \\right\\|^2  - \\frac{1}{4}\\left\\| {\\mathbf{x} - \\mathbf{y}} \\right\\|^2  + \\frac{i}{4}\\left\\| {\\mathbf{x} + i\\mathbf{y}} \\right\\|^2  - \\frac{i}{4}\\left\\| {\\mathbf{x} - i\\mathbf{y}} \\right\\|^2 ~.\n\\]\n\\end{thm}\nThe induced norm satisfies the Cauchy-Schwartz\\index{Cauchy-Schwartz} inequality stated in the following theorem.  \n\\newtheorem{CauchySchwartz}{Theorem}\n\\begin{thm}\n\\label{CauchySchwartz}\nGiven an inner product space $\\mathscr{V}$ and two elements $\\mathbf{x}$ and $\\mathbf{y}$, then\n\\begin{equation}\n\\label{Chapter1-7}\n\\left| {\\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle} \\right|\\le \\left\\| {\\mathbf{x}} \n\\right\\|\\left\\| {\\mathbf{y}} \\right\\|\n\\end{equation}\nwith equality if, and only if, $\\mathbf{y}=a \\mathbf{x}$ for some (complex) constant $a$.\n\\end{thm}\nTwo vectors are orthogonal if their inner product is zero: $\\mathbf{x}\\bot \\mathbf{y}\\Leftrightarrow \\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle=0$. One can, using the Cauchy-Schwartz inequality, define an angle $\\theta$ between two vectors by \n\\begin{equation}\n\\label{Chapter1-8}\ncos\\left( \\theta \\right)\\equiv \\frac{\\left| {\\left\\langle {\\mathbf{x},\\mathbf{y}} \\right\\rangle} \\right|}{\\left\\| {\\mathbf{x}} \\right\\|\\left\\| {\\mathbf{y}} \n\\right\\|}, ~~ 0 \\le \\theta \\le \\frac{\\pi }{2} ~.\n\\end{equation}\n\nThe concept of orthogonality can be extended to subspaces of an inner product vector space. \n\\newtheorem{Vperp}{Definition}\n\\begin{defn}\n\\label{Vperp}\nGiven a subspace $\\mathbb{V}$ of an inner product space $\\mathscr{V}$, the space $\\mathbb{V} ^\\bot$ is defined to contain all vectors that are orthogonal to every vector in $\\mathbb{V}$.  That is, $<\\mathbf{v},\\mathbf{w}>=0$ whenever $\\mathbf{v}\\in \\mathbb{V}$ and $\\mathbf{w}\\in \\mathbb{V} ^ \\bot$.\n\\end{defn}\n\n\n\\newtheorem{VperpSubspace}{Theorem}\n\\begin{thm}\n\\label{VperpSubspace}\n$\\mathbb{V} ^\\bot$ is a subspace of $\\mathscr{V}$ and $\\mathbb{V} \\subset \\mathbb{V}^{ \\bot  \\bot }$ and $\\mathbb{V}^ \\bot   = \\mathbb{V}^{ \\bot  \\bot  \\bot }$.\n\\end{thm}\n\nIn transform theory a function $x(t)$ in $L_2(\\mathbb{R})$ is mapped to a set of transform coefficients by calculating the inner products of the given function with a set of basis function $\\xi_a(t)$.  Using theorem  \\ref{CauchySchwartz}, the magnitude squared of the inner product $\\left\\langle {\\xi _a ,x} \\right\\rangle$ satisfies the inequality  \n\\[\n\\left| {\\left\\langle {\\xi_a ,x} \\right\\rangle } \\right|^2  \\le {\\left\\| {\\xi _a } \\right\\|}^2  {\\left\\| x \\right\\|}^2\n\\]\nand so the quantity $\\left| {\\left\\langle {\\xi_a ,x} \\right\\rangle } \\right|^2$ can be used to find the matching value $a$ of the basis function that $x(t)$ may be known to be proportional to.  However, if the function of interest $x$ is not proportional to any of the basis functions, then the best match is found by minimizing the magnitude squared of the sum or the difference ${x \\pm \\xi_a }$.  Thus, the minima of the quantities\n\\begin{equation}\n\\label{Chapter1-9}\n\\left\\| {x \\pm \\xi_a } \\right\\|^2  = \\left\\| x \\right\\|^2  + \\left\\| {\\xi_a } \\right\\|^2  \\pm 2{\\mathop{\\rm Re}\\nolimits} \\left\\langle {\\xi _a ,x} \\right\\rangle ~,\n\\end{equation}\noccur at the maximum of $\\left( {{\\rm Re}\\left\\langle {\\xi_a ,x} \\right\\rangle } \\right)^2$.  \n\n\n\\section{Banach and Hilbert Spaces}\nAlthough in a normed space we have the means for doing analysis and study convergence, we are not guaranteed that converging sequences of vectors actually have a limit inside the space itself \\cite{BaNa66}. A Banach\\index{Banach} space is a normed space in which every convergent Cauchy sequence (see definition \\ref{CauchySequence}) of vectors converges to a vector inside the space.  We take this as our working definition of completeness\\index{completeness} \\footnote{A more general definition in a metric space relies on first defining the diameter of closed subspaces and then requiring that every decreasing sequence of non empty closed subsets of the corresponding space whose diameters tend to $0$ must have a non-empty intersection, i.e., a point that is common to all of them.}.\n\\newtheorem{BanachSpace}{Definition}\n\\begin{defn}\nA Banach space is a complete normed vector space. \n\\end{defn}\n\\newtheorem{HilbertSpace}{Definition}\n\\begin{defn}\nA Hilbert space $\\mathscr{H}$ is a complete inner product space, or equivalently, a Banach space whose norm is induced by an existing inner product, $\\left\\| x \\right\\| = \\sqrt {\\left\\langle {x,x} \\right\\rangle }$.\n\\end{defn}\nHilbert spaces commonly occur in mathematics and physics as function spaces.  Their study is, therefore, contained in a branch of mathematics known as functional analysis.  Hilbert spaces generalize the familiar Euclidean three dimensional space $\\mathbb{R}^3$: most important geometrical results in the latter have direct generalizations to Hilbert spaces.  For instance, orthogonal projections from the tip of a standard three dimensional vector on to an arbitrary plane has its analogue in Hilbert spaces in the form of orthogonal projections onto linear subspaces and is at the foundation of linear optimal filter theory.  Orthogonal projections are discussed in section \\ref{OrthogonalProjections}.   \n\nAn indispensable property of Hilbert spaces in almost all of physics and applied mathematics is that of separability: separable Hilbert spaces\\index{Hilbert space!separable} possess a countable orthonormal basis (see section \\ref{CompleteOrthonormalBases} for orthonormal bases).  The function space of interest to us is $L_2(\\mathbb{R})$ which is a separable Hilbert space.  One such basis can be found starting with the windowed (or short time) Fourier transform and its discretization in both time and frequency.  The resulting basis, although localized, have fixed time and frequency resolutions that are inversely proportional to each other.  The continuous wavelet transform is another approach to arrive at variable resolution functions that are localized in time and scale.  Both methods will be discussed in chapter \\ref{Chapter_TimeFrequencyScale}.\n\n\n\n\n\\section{Linear Operators, Operator Norm, the Adjoint Operator}\n\n\\newtheorem{LinearOperator}{Definition}\n\\begin{defn}\nGiven two vector spaces $\\mathscr{V}_1$ and $\\mathscr{V}_2$, a transformation (operator) $T$ between the two denoted by $T: \\mathscr{V}_1  \\to \\mathscr{V}_2$ is said to be linear if for every $\\mathbf{x},\\mathbf{y} \\in \\mathscr{V}_1$ and all $\\alpha, \\beta \\in \\mathbb{C}$, $T\\left( {\\alpha \\mathbf{x} + \\beta \\mathbf{y}} \\right) = \\alpha T\\left( \\mathbf{x} \\right) + \\beta T\\left( \\mathbf{y} \\right)$.  \n\\end{defn}\nFor instance, if the two vector spaces are $\\mathbb{C}^n$ and $\\mathbb{C}^m$, then $T$ is an $m \\times n$ matrix of complex numbers.  When the underlying spaces are both copies of $L_2 \\left( \\mathbb{R} \\right)$, a linear transformation takes the form of an integral operator, e.g., $X\\left( \\omega \\right) = \\int\\limits_{ - \\infty }^\\infty  {T\\left( {\\omega, t } \\right)} x\\left( t  \\right)dt$.  An example of such an operator is the Fourier transform (see equations \\eqref{Chapter1-56} and \\eqref{Chapter1-57}).  \n\nIn order to ensure that the image of a linear operator\\index{operator!linear} is in fact a member of $L_2(\\mathbb{R})$ we must extend the notion of norm to linear operators.  An operator norm\\index{operator!norm} must have all the required properties of a norm shown in equation \\eqref{Chapter1-3}.  The $p$-norm of a linear operator $T:\\mathscr{V}_1  \\to \\mathscr{V}_2 $ is defined by \\footnote{The definition of equation \\eqref{Chapter1-10} is equivalent to $\\left\\| T \\right\\|_p  = Max\\left\\{ {{{\\left\\| {Tx} \\right\\|_p } \\mathord{\\left/ {\\vphantom {{\\left\\| {Tx} \\right\\|_p } {\\left\\| x \\right\\|_p }}} \\right. \\kern-\\nulldelimiterspace} {\\left\\| x \\right\\|_p }};~x \\in \\mathscr{V}_1 ,\\left\\| x \\right\\|_p \\ne 0} \\right\\}$.}\n\\begin{equation}\n\\label{Chapter1-10}\n\\left\\| T \\right\\|_p  = Max \\left\\{ {\\left\\| {Tx} \\right\\|_p ; ~ x \\in \\mathscr{V}_1 {\\rm ~ and ~ }\\left\\| x \\right\\|_p  = 1} \\right\\}\n\\end{equation}\nwhere $\\left\\| x \\right\\|_p$ is the $L_p$ norm defined previously.  The $p$-norm of an operator satisfies the following inequalities \n\\begin{equation}\n\\label{Chapter1-11}\n\\left\\| {Tx} \\right\\|_p  \\le \\left\\| T \\right\\|_p \\left\\| x \\right\\|_p {\\rm  and }\\left\\| {T_1 T_2 } \\right\\|_p  \\le \\left\\| {T_1 } \\right\\|_p \\left\\| {T_2 } \\right\\|_p ~.\n\\end{equation}\nWhen the operator is defined $T:\\mathscr{V}  \\to \\mathscr{V}$ and it has an inverse we can show\n\\begin{equation}\n\\label{Chapter1-12}\n\\left\\| {T^{ - 1} } \\right\\|_p  =  \\frac{1}{{Min\\left\\{ {\\left\\| {Tx} \\right\\|_p ;\\left\\| x \\right\\|_p  = 1,x \\in \\mathscr{V}} \\right\\}}}\n\\end{equation}\n\\newtheorem{BoundedOp}{Definition}\n\\begin{defn}\nA bounded operator\\index{operator!bounded} is a linear operator with finite norm.  \n\\end{defn}\n\\newtheorem{BoundedOperator}{Theorem}\n\\begin{thm}\n\\label{BoundedOperator}\nA linear operator is bounded if, and only if, it is continuous.\n\\end{thm}\nThe following theorem describes a useful formula, known as the Neumann expansion\\index{Neumann expansion}, for bounded operators.\n\\newtheorem{NeumannExpansion}{Theorem}\n\\begin{thm}\n\\label{NeumannExpansion}\nFor a bounded operator $T$ with unity bound, i.e., $\\left\\| T \\right\\| < 1$, where the norm satisfies the inequalities \\eqref{Chapter1-11}, we have the following formula \\cite{MaWa64}.\n\\begin{equation}\n\\label{Chapter1-13}\n\\left( {1-T} \\right)^{-1} = \\sum\\limits_{k = 0}^\\infty  {T^k} ~.\n\\end{equation}\n\\end{thm}\n\nEvery bounded linear operator $T:\\mathscr{H}_1 \\to \\mathscr{H}_2 $ between two Hilbert spaces $\\mathscr{H}_1$ and $\\mathscr{H}_2$, has an associated adjoint\\index{operator!adjoint} $T^ + :\\mathscr{H}_2 \\to \\mathscr{H}_1 $.\n\\newtheorem{AdjointdOperator}{Definition}\n\\begin{defn}\n\\label{AdjointOperator}\nThe adjoint of a bounded linear operator $T:\\mathscr{H}_1 \\to \\mathscr{H}_2 $ is a linear operator $T^ + :\\mathscr{H}_2 \\to \\mathscr{H}_1 $ defined by the equation\n\\begin{equation}\n\\label{Chapter1-14}\n\\left\\langle {s_2,T s_1} \\right\\rangle  \\equiv \\left\\langle {T^ +  s_2, s_1} \\right\\rangle ~,\n\\end{equation}\nfor every $s_1 \\in \\mathscr{H}_1$ and $s_2 \\in \\mathscr{H}_2$.  When the two Hilbert spaces are the same, $\\mathscr{H}_1 = \\mathscr{H}_2 = \\mathscr{H}$, a linear bounded operator is self adjoint if $T = T^ +$.  \n\\end{defn}\nTo every linear operator $T:\\mathscr{H}_1 \\to \\mathscr{H}_2 $ there correspond two fundamental linear subspaces, namely, the range space\\index{operator!range space} $\\mathscr{R}_T $ and the null space\\index{operator!null space} $\\mathscr{N}_T$.  \n\\newtheorem{RangeNullSpace}{Definition}\n\\begin{defn}\nThe range space of a linear operator $T:\\mathscr{H}_1 \\to \\mathscr{H}_2 $ is the image of the linear map and it is a linear subspace of $\\mathscr{H}_2$: \n\\[\n\\mathscr{R}_T =T\\left( {\\mathscr{H}_1 } \\right)=\\left\\{ {\\mathbf{y}\\,\\in \\mathscr{H}_2 :\\,\\mathbf{y}=T\\mathbf{x},\\mbox{ for some }\\mathbf{x}\\in \\mathscr{H}_1 \\mbox{ }} \\right\\}. \n\\]\nThe null space is a linear subspace of $\\mathscr{H}_1 $ and contains all elements of $\\mathscr{H}_1 $ that map to the zero of $\\mathscr{H}_2 $: \n\\[\n\\mathscr{N}_T=\\left\\{ {\\mathbf{x}\\,\\in \\mathscr{H}_1 :\\,~ T\\mathbf{x}=\\mathbf{0}} \\right\\}~.\n\\]\n\\end{defn}\nThe range and null spaces of a linear operator and its adjoint are related through the following theorem.\n\\newtheorem{AandAPlusSpaces}{Theorem}\n\\begin{thm}\n\\label{AandAPlusSpaces}\nFor a bounded linear operator $T:\\mathscr{H}_1 \\to \\mathscr{H}_2 $ whose range space $\\mathscr{R}_{T}$ is closed, and whose adjoint range space $\\mathscr{R}_{T^+}$ is also closed \\footnote{Although all bounded linear operators between finite dimensional spaces have closed ranges, this is not necessarily the case in infinite dimensional spaces.  The theorem still holds, however, if we replace the corresponding range spaces with their closures.}, the following relations hold:\n\\begin{subequations}\n\\label{Chapter1-15}\n\\begin{align}\n{\\mathscr{R}_{T}}^\\bot & = \\mathscr{N}_{T^+}, {\\mathscr{N}_{T}}^\\bot  = \\mathscr{R}_{T^+}, \\label{Chapter1-15a}  \\\\\n{\\mathscr{R}_{T^+}}^\\bot & = \\mathscr{N}_{T}, {\\mathscr{N}_{T^+}}^\\bot  = \\mathscr{R}_{T} \\label{Chapter1-15b} ~,\n\\end{align}\nwhere the subspaces denoted by the symbol $\\bot$ are defined in definition \\ref{Vperp}.  In addition,\n\\[\n\\mathscr{H}_1  = \\mathscr{R}_{T^ +  }  \\oplus \\mathscr{N}_T ,~ \\mathscr{H}_2  = \\mathscr{R}_T  \\oplus \\mathscr{N}_{T^ +} ~,\n\\]\nwhere the direct sum\\index{direct sum} $\\oplus$ is defined in definition \\ref{InnerAndDirectSum}.\n\\end{subequations}\n\\end{thm}\n\nGiven a self adjoint (and bounded) positive definite operator $T$, and denoting its smallest and largest eigenvalues \\footnote{For a positive definite operator $T$, the matrix element $\\left\\langle {x,Tx} \\right\\rangle$ is real and positive for all $x \\in \\mathscr{V}$.  The eigenvalues and the eigenfunctions of the operator $T$ are defined by the equation $Tx = \\lambda x$, where $x$ is an eigenvector (eigenfunction) corresponding to the eigenvalue\\index{eigenvalue} $\\lambda$.} by $\\lambda _{\\min }$ and $\\lambda _{\\max }$, where $ 0< \\lambda _{\\min }  < \\lambda _{\\max } $, the spectral radius\\index{spectral radius} of $T$ is defined to be its largest eigenvalue $\\rho_T  \\equiv \\lambda _{\\max }$.  Following this, if $T$ is not a self adjoint operator we define the $l_2$ norm of $T$ by the following equation\n\\begin{equation}\n\\label{Chapter1-16}\n\\left\\| T \\right\\|_2  = \\sqrt {\\rho _{T^ + T} } ~ .\n\\end{equation}\nThis definition relies on the fact that the operator $T^+ T$ is self adjoint and positive definite for any operator $T$.  It is self adjoint because $\\left( {T^ +  T} \\right)^ +   = T^ +  T^{ +  + }  = T^ +  T$, and positive definite since $\\left\\langle {x,T^ + Tx} \\right\\rangle  = \\left\\langle {Tx,Tx} \\right\\rangle $.  The latter is real and positive by the postulates of an inner product shown in equation \\eqref{Chapter1-4}.  When $T$ is a self adjoint operator, i.e. $T^ + = T$, it follows that $\\left\\| {T} \\right\\|_2  = \\rho _T$.\n\\newtheorem{UnitaryOperator}{Definition}\n\\begin{defn}\n\\label{UnitaryOperator}\nA unitary\\index{operator!unitary} operator is one that satisfies the equation $T^ +  T = 1$, i.e., the operator has a left inverse which is the same as its adjoint.  \n\\end{defn}\n\n\n\n\n\n\\section{Reproducing Kernel Hilbert Space\\index{reproducing kernel!Hilbert space}}\n\nA special case of linear operators is a linear functional\\index{functional} as defined below \\cite{BaNa66}.\n\\newtheorem{LinearFunctional}{Definition}\n\\begin{defn}\n\\label{LinearFunctional}\nA linear functional on a Hilbert space $\\mathscr{H}$ is a linear map $f:\\mathscr{H} \\to \\mathbb{C}$.\n\\end{defn}\n\\noindent For instance, suppose that $\\phi  \\in \\mathscr{H}$ is fixed.  Then \n\\[\nf_\\phi  \\left[ x \\right] = \\left\\langle {\\phi ,x} \\right\\rangle  = \\int\\limits_{ - \\infty }^\\infty  {\\phi ^ *  \\left( t \\right)} x\\left( t \\right)dt, \\forall x \\in \\mathscr{H},  \n\\]\nis a linear functional.  An important example is the evaluation functional\\index{functional!evaluation} $\\mathscr{E}_t$.\n\\newtheorem{EvaluationFunctional}{Definition}\n\\begin{defn}\n\\label{EvaluationFunctional}\nThe evaluation functional $\\mathscr{E}_t$ is a linear map between a Hilbert space of functions $\\mathscr{H}$ and the complex numbers  $\\mathbb{C}$  \n\\begin{equation}\n\\label{Chapter1-17}\n\\mathscr{E}_t :\\mathscr{H} \\to \\mathbb{C}~, ~~ \\mathscr{E}_t [x] = x\\left( t \\right) ~,\n\\end{equation}\nwhere the functions are assumed to have been defined for every point $t \\in \\mathbb{R}$.\n\\end{defn}\n\\newtheorem{BoundedLinearFunctionals}{Definition}\n\\begin{defn}\nA linear functional $f$ on a Hilbert space is bounded\\index{functional!bounded} if there exists a constant $C$, such that $\\left| {f\\left[ x \\right]} \\right| \\le C\\left\\| x \\right\\|$, $\\forall x$ in the Hilbert space.  Note that $C$ is independent of $x$.\n\\end{defn}\n\\noindent An example is the Hilbert space $L_2 \\left( \\mathbb{R} \\right)$, and the linear functional \n\\[\nf_w \\left[ x \\right] \\equiv \\int\\limits_{ - \\infty }^\\infty  {w^ *  \\left( t \\right)x\\left( t \\right)dt} ~.\n\\]\nThen by the Cauchy-Schwartz inequality \\eqref{Chapter1-7} we have \n\\[\n\\left| {f_w \\left[ x \\right]} \\right| \\le \\left\\| w \\right\\|\\left\\| x \\right\\| ~,\n\\]\nand so we may take $C=\\left\\| w \\right\\|$.\n\\newtheorem{ContinuousLinearFunctional}{Definition}\n\\begin{defn}\nA linear functional $f$ is continuous\\index{functional!continuous} if, given that $x_n \\left( t \\right) \\to x\\left( t \\right)$ as $n \\to \\infty$, then $f\\left[ {x_n } \\right] \\to f\\left[ x \\right]$.\n\\end{defn}\n\\newtheorem{BoundedContinuousLinearFunctional}{Theorem}\n\\begin{thm}\n\\label{BoundedContinuousLinearFunctional}\nA linear functional $f$ defined on a Hilbert space is bounded if, and only if, it is continuous.\n\\end{thm}\nThis theorem, to be used below, can be proven as follows.  Given that $x_n  \\to x$ and if $f$ is bounded, we use linearity to write \n\\[\n\\left| {f\\left( {x_n } \\right) - f\\left( x \\right)} \\right| = \\left| {f\\left( {x_n  - x} \\right)} \\right| \\le C\\left\\| {x_n  - x} \\right\\| ~, \n\\]\nwhich proves the continuity of the functional.  Conversely, if $f$ is continuous but unbounded then there must exist a sequence $x_n$ such that  $f\\left[ {x_n } \\right] > n\\left\\| {x_n } \\right\\|$.  Now let $x'_n  = {{x_n } \\mathord{\\left/ {\\vphantom {{x_n } {\\left( {n\\left\\| {x_n } \\right\\|} \\right)}}} \\right.\n \\kern-\\nulldelimiterspace} {\\left( {n\\left\\| {x_n } \\right\\|} \\right)}}$.  Clearly $\\left\\| {x'_n } \\right\\| = n^{ - 1}$ and so $x'_n  \\to 0$.  Then $\\left| {f\\left[ {x'_n } \\right]} \\right| = {{f\\left[ {x_n } \\right]} \\mathord{\\left/ {\\vphantom {{f\\left[ {x_n } \\right]} {\\left( {n\\left\\| {x_n } \\right\\|} \\right) > 1}}} \\right.\n \\kern-\\nulldelimiterspace} {\\left( {n\\left\\| {x_n } \\right\\|} \\right) > 1}}$ which shows that $f\\left[ {x'_n } \\right]$ will not tend to zero, even though $x'_n  \\to 0$, thus contradicting the continuity assumption.  Consequently if $f$ is continuous it must be bounded.\n\nIn a Hilbert space, bounded linear functionals have a particularly simple representation, embodied in the Frechet-Riesz theorem.\n\\newtheorem{FrechetRiesz}{Theorem}\n\\begin{thm}\n\\label{FrechetRiesz}\n(Frechet-Riesz) If $f$ is a bounded linear functional on a Hilbert space $\\mathscr{H}$, then there exists a unique element $\\phi  \\in \\mathscr{H}$ such that $f\\left[ x \\right] = \\left\\langle {\\phi ,x} \\right\\rangle$ for all $x \\in \\mathscr{H}$.\n\\end{thm}\n\\newtheorem{BoundedEvaluationFunctional}{defn}\n\\begin{defn}\n\\label{BoundedEvaluationFunctional}\nThe evaluation functional $\\mathscr{E}_t$ of definition \\ref{EvaluationFunctional} is said to be bounded if given $x(t) \\in \\mathscr{H}$, defined for every value of the independent variable $t$, there exist constants $C_t$ (that depend on $t$ but not on $x$) for which the following inequality holds for all $x$, \n\\begin{equation}\n\\label{Chapter1-18}\n\\left| {x\\left( t \\right)} \\right| \\le C_t \\left\\| x \\right\\|~.\n\\end{equation}\n\\end{defn}\nIf the evaluation functional is bounded then theorem \\ref{FrechetRiesz} shows that $\\mathscr{E}_t [x]$ can be written as the inner product between a fixed element $e_{_t} \\in \\mathscr{H}$ and $x$, i.e., \n\\begin{equation}\n\\label{Chapter1-19}\nx(t) = \\left\\langle {e_{_t},x} \\right\\rangle = \\int\\limits_{ - \\infty }^\\infty {e_{_t}^ * \\left( t' \\right)x\\left( t'  \\right)dt'} ~.\n\\end{equation}\n\\newtheorem{ReproducingKernel}{Definition}\n\\begin{defn}\n\\label{ReproducingKernel}\nIn a Hilbert space with a bounded evaluation functional the reproducing kernel\\index{reproducing kernel} is defined by \\cite{Aron50}\n\\begin{equation}\n\\label{Chapter1-20}\nK\\left( {t,t'} \\right) \\equiv e_{_t} \\left(t' \\right) = \\left\\langle {e_{t'} ,e_t } \\right\\rangle\n\\end{equation}\n\\end{defn}\n\\noindent Equation \\eqref{Chapter1-19} can now be written in terms of the reproducing kernel\n\\begin{equation}\n\\label{Chapter1-21}\nx\\left( t \\right) = \\int\\limits_{ - \\infty }^\\infty {K^* \\left( {t,t'} \\right)} x\\left(t' \\right)dt' \\equiv \\left\\langle {K,x} \\right\\rangle _t ~, \n\\end{equation}\nwhere the subscript $t$ indicates that in the corresponding inner product the integration is over the variable $t'$ and that $t$ is held fixed.  This is the reproducing kernel property of any Hilbert space in which the evaluation functional is a bounded (equivalently, continuous) linear map.  Thus, we have the following theorem. \n\\newtheorem{Aronszajn}{Theorem}\n\\begin{thm}\n\\label{Aronszajn}\n(Aronszajn\\index{Aronszajn})  A necessary and sufficient condition that a Hilbert space possesses a reproducing kernel is that the evaluation functional be bounded.\n\\end{thm}\n\\newtheorem{KernelUniqueness}{Theorem}\n\\begin{thm}\n\\label{KernelUniqueness}\nIf a Hilbert space possesses a reproducing kernel then the kernel is unique.\n\\end{thm}\n\\noindent For if we have two reproducing kernels $K_1 \\left( {t,\\tau } \\right)$ and $K_2 \\left( {t,\\tau } \\right)$, then \n\\begin{align}\n\\label{Chapter1-22}\n\\left\\| {K_1 \\left( {t,\\tau } \\right) - K_2 \\left( {t,\\tau } \\right)} \\right\\|_t ^2 & = \\left\\langle {K_1 \\left( {t,\\tau } \\right) - K_2 \\left( {t,\\tau } \\right),K_1 \\left( {t,\\tau } \\right) - K_2 \\left( {t,\\tau } \\right)} \\right\\rangle _t  \\notag \\\\ \n  & = \\left\\langle {K_1 \\left( {t,\\tau } \\right) - K_2 \\left( {t,\\tau } \\right),K_1 \\left( {t,\\tau } \\right)} \\right\\rangle _t \\notag \\\\\n  & - \\left\\langle {K_1 \\left( {t,\\tau } \\right) - K_2 \\left( {t,\\tau } \\right),K_2 \\left( {t,\\tau } \\right)} \\right\\rangle _t \\notag  \\\\ \n  & = K_1 \\left( {t,t} \\right) - K_2 \\left( {t,t} \\right) - K_1 \\left( {t,t} \\right) + K_2 \\left( {t,t} \\right) \\notag \\\\\n  & = 0 ~.\n\\end{align}\n\\noindent The final result, of course, holds for all $t$ and so the reproducing kernel is unique.\n\n\n\n\\section{The Dirac Delta Distribution}\n\\label{DiracDelta}\n\n\nIn most Hilbert spaces the evaluation functional is unbounded.  For instance, functions in $L_2(\\mathbb{R})$ could be unbounded on sets of measure zero and so a reproducing kernel cannot be defined.  Examples of reproducing kernel Hilbert spaces include the space of band limited functions (a subspace of $L_2(\\mathbb{R})$ defined in section \\ref{BandLimitedSection}) as well as the range spaces of both the windowed Fourier transform and the continuous wavelet transform  (subspaces of $L_2({\\mathbb{R}}^2)$), as will be shown later.  \n\nThe Dirac delta\\index{Dirac!delta} distribution \\cite{GeSh64} \\footnote{The Dirac delta is popularly known as the Dirac delta function.  The usual definition ``zero everywhere except at one point\", however, clearly does not satisfy the basic requirements for a function.} $\\delta(t-t')$ is a singular form of a reproducing kernel existing in a space far larger than $L_2(\\mathbb{R})$:  the dual space of a space of ``sufficiently well behaved (good)\" functions.\n\\newtheorem{TestFunctions}{defn}\n\\begin{defn}\n\\label{TestFunctions}\nConsider the space of functions $\\mathscr{S}$ whose elements (in general, complex functions) are absolutely integrable on the real line $\\mathbb{R}$, and are infinitely many times differentiable at every $t \\in \\mathbb{R}$, and, together with any of their derivatives, decay to zero faster than any negative power of $t$ as $\\left| t \\right| \\to \\infty$ \\cite{Brem65}.  We will call this the space of test functions of rapid decay \\footnote{Technically, the space of test functions $\\mathscr{K}$, contained within $\\mathscr{S}$, is defined to include only functions with continuous derivatives of all order and with compact support.  Although, $\\mathscr{K}$ is used to define generalized functions, in practice $\\mathscr{S}$ is often used and so we take that as our function space for the definition of the Dirac delta and other generalized functions.}.\n\\end{defn}\n\\noindent An example of the sort of functions contained in $\\mathscr{S}$ is $\\exp (-t^2)$.  \n\\newtheorem{TestFunctionsDual}{defn}\n\\begin{defn}\n\\label{TestFunctionsDual}\nThe space of all linear functionals defined on $\\mathscr{S}$, whose range is either $\\mathbb{R}$ or $\\mathbb{C}$, is known as the dual space of $\\mathscr{S}$ and is denoted by $\\mathscr{S'}$.  \n\\end{defn}\n\\noindent Thus, if $\\phi_{_1} (t) \\in \\mathscr{S'}$, then $\\int\\limits_{ - \\infty }^\\infty  {\\phi_{_1} ^ *  \\left( t \\right)\\phi \\left( t \\right) dt}$ is a number (in general, complex).  The dual space $\\mathscr{S'}$ includes all functions in $\\mathscr{S}$ as well as all functions in $L_2(\\mathbb{R})$, and thus it includes regular functions, since\n\\[\n\\left| {\\int\\limits_{- \\infty }^\\infty {x^*(t) \\phi(t) dt}} \\right| < \\infty , ~ x \\in L_2(\\mathbb{R}) ~ {\\rm or} ~ x \\in \\mathscr{S}, ~ \\phi  \\in \\mathscr{S'}.\n\\]\nBut there is more:  $\\mathscr{S'}$ includes objects that are not ordinary functions.  For instance, consider the linear functional that maps every member of $\\mathscr{S}$ to the value of that member at a fixed point $t'$.  This linear map, a member of $\\mathscr{S'}$, is the Dirac delta generalized function (or, distribution).  Thus, the defining relation for the Dirac delta is\n\\begin{equation}\n\\label{Chapter1-23}\n\\int\\limits_{ - \\infty }^\\infty  {\\delta \\left({t-t'} \\right)\\phi \\left(t' \\right)dt'}  = \\phi \\left( {t} \\right) ~.\n\\end{equation}\nComparing the above with the reproducing kernel property \\eqref{Chapter1-21} suggests that the Dirac delta, that is only defined on functions in $\\mathscr{S}$, is a generalized reproducing kernel:  it is commonly used with members of $L_2(\\mathbb{R})$ which do not often even belong to $\\mathscr{S}$!  Although we can still use equation \\eqref{Chapter1-23} for functions in $L_2(\\mathbb{R})$ at points $t$ for which the function is defined, there is no meaning to the relation at points $t$ where the function is not defined (comprising a set of measure $0$).  Thus, the Dirac delta may be considered to be a singular (or generalized) reproducing kernel in $L_2(\\mathbb{R})$.\n\nAnother set of useful functions that are members of $\\mathscr{S'}$, but not $L_2(\\mathbb{R})$, are the complex exponentials $e_\\omega (t) \\equiv \\left( {2\\pi } \\right)^{ - {1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\exp (i \\omega t)$, $\\omega \\in \\mathbb{R}$; they are used to construct the Fourier transforms of members of $L_2(\\mathbb{R})$.  The variable $\\omega$ (also known as the frequency) is used as a continuous index to distinguish different exponential functions.  The inverse Fourier transform (the reconstruction formula) is based on the relation\n\\begin{equation}\n\\label{Chapter1-24}\n\\int\\limits_{ - \\infty }^\\infty  {e_\\omega ^ *  \\left( t \\right)e_\\omega  \\left( {t'} \\right)d\\omega }  = \\delta \\left( {t - t'} \\right) ~.\n\\end{equation}\nAs we shall see later, when defining the Fourier transform and its inverse, the above equation shows that any member of $L_2(\\mathbb{R})$ can be expanded in terms of $e_\\omega (t)$ \n\\begin{equation}\n\\label{Chapter1-25}\nx(t) = \\int\\limits_{-\\infty}^\\infty {e_\\omega  (t) X(\\omega)d\\omega }, ~~ x(t) \\in L_2(\\mathbb{R}),\n\\end{equation}\nwhere the expansion coefficients $X(\\omega)$ are\n\\begin{equation}\n\\label{Chapter1-26}\nX\\left( \\omega  \\right) = \\int\\limits_{ - \\infty }^\\infty  {{e_\\omega ^*(t)} x\\left( t \\right)dt} ~.\n\\end{equation}\nThus, the functions $e_\\omega (t)$ are a continuously labeled basis for members of $L_2(\\mathbb{R})$ within the space $\\mathscr{S'}$.  Equation \\eqref{Chapter1-24} is known as a (generalized ) resolution of identity\\index{resolution of identity} and often referred to as the completeness relation for the continuously indexed ``basis functions\" $e_\\omega (t)$.  However, there is no discretization of the frequency variable $\\omega$ of the form $\\omega_n = n\\Delta \\omega$, $n \\in \\mathbb{Z}$ for which the corresponding functions $e_{\\omega_{_n}} (t)$ satisfy the completeness relation.  We shall see later that in order to retain the completeness property using Fourier transforms we will have to use window functions of compact support.\n\nAs will be shown later in section \\ref{BandLimitedSection}, if we limit the frequency band of the function space $L_2(\\mathbb{R})$ to the interval $[-\\Omega, \\Omega]$, $\\Omega > 0$, the resulting Hilbert space (space of band limited functions\\index{band limited functions} that are magnitude squared integrable, a subspace of $L_2(\\mathbb{R})$) will admit a regular reproducing kernel that is the band limited form of the Dirac delta, namely, ${{\\sin \\left[ {\\Omega \\left( {t - t'} \\right)} \\right]} \\mathord{\\left/ {\\vphantom {{\\sin \\left[ {\\pi \\left( {t - t'} \\right)} \\right]} {\\pi \\left( {t - t'} \\right)}}} \\right. \\kern-\\nulldelimiterspace} {\\pi \\left( {t - t'} \\right)}}$, in the sense that\n\\[\n{{\\sin \\left[ {\\Omega \\left( {t - t'} \\right)} \\right]} \\mathord{\\left/ {\\vphantom {{\\sin \\left[ {\\Omega \\left( {t - t'} \\right)} \\right]} {\\left[ {\\pi \\left( {t - t'} \\right)} \\right]}}} \\right. \\kern-\\nulldelimiterspace} {\\left[ {\\pi \\left( {t - t'} \\right)} \\right]}} \\to \\delta ({t-t'}) , ~~ {\\rm as} ~ \\Omega   \\to \\infty ~.\n\\]\nThis reproducing kernel is a regular function (not a distribution) and is often used as an example of a Dirac delta convergent sequence.  We will return to the Dirac delta and reproducing kernels later in the context of frames and orthonormal bases.  \n\n\n\n\n\\section[Orthonormal Vectors]{Orthonormal Vectors}\n\nOrthogonal sets of vectors are particularly important in most applications and if, in addition, they are of unit norm then they are referred to as orthonormal\\index{orthonormal vectors} sets \\cite{Halm74}.\n\\newtheorem{Orthonormal}{Definition}\n\\begin{defn}\nA set of vectors $\\left\\{ {\\mathbf{x}_j } \\right\\}$ is said to be orthonormal provided $\\left\\langle {\\mathbf{x}_j ,\\mathbf{x}_k} \\right\\rangle = \\delta _{jk} $, where $\\delta _{jk} $ is the Kroenecker delta\\index{Kroenecker delta} \n\\begin{equation}\n\\label{Chapter1-27}\n\\delta _{jk} =\\left\\{ \n{\\begin{array}{rl}\n1 \\quad j=k \\\\ \n0 \\quad j\\ne k \\\\ \n\\end{array}} \\right.\n\\end{equation}\n\\end{defn}\n\nGiven any set of vectors $\\left\\{ {\\mathbf{x}_1 ,\\ldots ,\\mathbf{x}_n } \\right\\}$ the Gramm-Schmidt\\index{Gramm-Schmidt} method \\cite{MaWa64} is used to obtain an orthonormal set $\\left\\{ {\\mathbf{e}_1 ,\\ldots ,\\mathbf{e}_m } \\right\\}$ where $m\\le n$ and both sets of vectors span the same vector space.  The method is to choose $\\mathbf{e}_1  = {{\\mathbf{x}_1 } \\mathord{\\left/ {\\vphantom {{\\mathbf{x}_1 } {\\left\\| {\\mathbf{x}_1 } \\right\\|}}} \\right. \\kern-\\nulldelimiterspace} {\\left\\| {\\mathbf{x}_1 } \\right\\|}}$ and\n\\begin{equation}\n\\label{Chapter1-28}\n\\mathbf{e}'_k =\\mathbf{x}_k\n-\\sum\\limits_{l=1}^{k-1} {\\left\\langle {\\mathbf{x}_k ,~\\mathbf{e}_l} \\right\\rangle} \\mathbf{e}_l, ~ \\mathbf{e}_k  = \\left\\| {\\mathbf{e'}_k } \\right\\|^{-1} \\mathbf{e'}_k, ~ k \\ge 2. \n%  \\mathbf{e}_k =\\frac{\\mathbf{e}'_k }{\\left\\| {\\mathbf{e}'_k } \\right\\|}, ~ k \\ge 2.\n\\end{equation}\nFor example, if we use the space of square integrable real functions on $\\left[ {-1,1} \\right]$ with the inner product $\\left\\langle{f,g} \\right\\rangle=\\int\\limits_{-1}^1 {f\\left( t \\right)} g\\left( t \\right)dt$ , starting with the vectors $\\left\\{ {1,t,t^2,\\ldots ,t^n} \\right\\}$, the Gramm-Schmidt method will produce the Legendre polynomials up to degree n. If, on the other hand, we use the weighted inner product $\\left\\langle {f,g} \\right\\rangle  = \\int\\limits_{ - 1}^1 {f\\left( t \\right)} \\left( {1 - t^2 } \\right)^{ - {1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} g\\left( t \\right)dt$, the Gramm-Schmidt method will result in the Chebyshev polynomials up to degree n (see the exercises).\n\n\n\n\\section[Orthogonal Projections]{Orthogonal Projections}\n\\label{OrthogonalProjections}\n\n\n\nGiven a Hilbert space $\\mathscr{H}$ and two disjoint subspaces $\\mathbb{V}$ and $\\mathbb{W}$, $\\mathbb{V}\\cap \\mathbb{W}=\\left\\{ {\\mathbf{0}} \\right\\}$, a projection\\index{projection} onto $\\mathbb{V}$ is a mapping $P_v :\\mathscr{H}\\to \\mathbb{V}$ so that $P_v \\mathbf{x}=\\mathbf{v}$ where $\\mathbf{x}=\\mathbf{v+w},\\;\\mathbf{v}\\in \\mathbb{V}\\;\\mathbf{w}\\in \\mathbb{W}$. Alternatively, $P_w :\\mathscr{H}\\to \\mathbb{W}$ where $P_w \\mathbf{x}=\\mathbf{w}$.  Any projection operator $P$ must satisfy the equation $P^2=P$, which can be used to define a projection. The Cayley-Hamilton theorem\\footnote{The Cayley Hamilton theorem states that a square matrix $\\mathbf{A}$ whose characteristic polynomial is defined by $\\chi _\\mathbf{A} \\left( \\lambda  \\right) \\equiv \\left| {\\mathbf{A} - \\lambda \\mathbf{1}} \\right|$, satisfies the equation $\\chi _\\mathbf{A} \\left( \\mathbf{A} \\right) = 0$.} can be invoked to state that the eigenvalue equation $\\lambda ^2=\\lambda$ must be the characteristic equation for any projection operator; thus, the eigenvalues of a projection operator are 0 and 1. \n\nIf the projection is defined into the same space, i.e. $P:\\mathscr{H} \\to \\mathscr{H}$, then the range and the null spaces $\\mathscr{R}_P$ and $\\mathscr{N}_P$ are disjoint linear subspaces of the space $\\mathscr{H}$ (the only element common to both is the zero vector), and $\\mathscr{H}=\\mathscr{R}_p +\\mathscr{N}_p$, i.e. the range and the null spaces are algebraic complements and are not necessarily orthogonal.  \n\n\\newtheorem{OrthogonalProjection}{Definition}\n\\begin{defn}\n\\label{OrthogonalProjection}\nA projection $P:\\mathscr{H}\\to \\mathscr{H}$ is said to be orthogonal\\index{projection!orthogonal} if its range and null spaces are orthogonal to one another, i.e., $\\mathscr{R}_P  \\bot \\mathscr{N}_P$ \\cite{Luen69}. \n\\end{defn}\n\\newtheorem{OrthogonalProjectionTheorem1}{Theorem}\n\\begin{thm}\n\\label{OrthogonalProjectionTheorem1}\nA projection $P:\\mathscr{H}\\to \\mathscr{H}$ is orthogonal if, and only if, it is self adjoint.  \n\\end{thm}\n\\noindent If $P$ is self adjoint, then for every $x \\in \\mathscr{H}$ we write $x \\equiv Px + (1 - P)x$ is an identity.  Now clearly $Px \\in \\mathscr{R}_p$ and $\\left( {1 - P} \\right)x \\in \\mathscr{N}_p$, since $P\\left( {1 - P} \\right)x = \\left( {P - P^2 } \\right)x = 0$ ($P$ being a projection operator satisfies $P=P^2$).  Any member of $\\mathscr{R}_p$ is orthogonal to any member of $\\mathscr{N}_p$ since \n\\begin{equation}\n\\label{Chapter1-29}\n\\left\\langle {Px,\\left( {1 - P} \\right)y} \\right\\rangle  = \\left\\langle {x,P^ +  \\left( {1 - P} \\right)y} \\right\\rangle  = \\left\\langle {x,\\left( {P - P^2 } \\right)y} \\right\\rangle  = 0 ~.\n\\end{equation}\n\\newtheorem{OrthogonalProjectionTheorem2}{Theorem}\n\\begin{thm}\n\\label{OrthogonalProjectionTheorem2}\nA projection $P:\\mathscr{H}\\to \\mathscr{H}$ is orthogonal if $\\left\\| {Px} \\right\\| \\le \\left\\| x \\right\\|$ for every $x \\in \\mathscr{H}$.\n\\end{thm}\n\\noindent To show that $\\mathscr{R}_p$ is orthogonal to $\\mathscr{N}_p$, assume $x \\in \\mathscr{N}^{\\bot}_p$ (see definition \\ref{Vperp}).  Then $y = Px - x \\in \\mathscr{N}_p$ since $Py = P^2 x - Px = Px - Px = 0$.  So $Px = x + y$ where $\\left\\langle {x,y} \\right\\rangle  = 0$.  Thus,\n\\[\n\\left\\| x \\right\\|^2  \\ge \\left\\| {Px} \\right\\|^2  = \\left\\| x \\right\\|^2  + \\left\\| y \\right\\|^2  \\ge \\left\\| x \\right\\|^2 ~,\n\\]\nand therefore $y=0$, and $Px=x$, so that $x \\in \\mathscr{R}_p$.  This shows that $\\mathscr{N}^{\\bot}_p \\subset \\mathscr{R}_p$.  Conversely, if $x \\in \\mathscr{R}_p$, then $x=Px$ and we write $x=y+u$ with $y \\in \\mathscr{N}^{\\bot}_p$ and $u \\in \\mathscr{N}_p$.  Then, $x=Px=Py+Pu=Py=y$, since $y \\in \\mathscr{N}^{\\bot}_p$ and therefore $y \\in \\mathscr{R}_p$.  Hence, $x \\in \\mathscr{N}^{\\bot}_p$ so that $\\mathscr{R}_p \\subset \\mathscr{N}^{\\bot}_p$.\n\nOrthogonal projections are used to solve the problem of approximating a given vector in $\\mathscr{H}$ by a vector that lies entirely in a linear subspace of $\\mathscr{H}$.  For instance, consider the 3-dimensional Euclidean space $\\mathbb{R}^3$, a vector $\\mathbf{r}=\\left( {x,y,z} \\right)^T$ where $z\\ne 0$, and the xy-plane which is a linear subspace of $\\mathbb{R}^3$. The problem here is to find a vector $\\hat{\\mathbf{r}}$ in the xy-plane that is the best approximation to the original vector $\\mathbf{r}$ in the sense that $\\left\\| {\\mathbf{r}-\\hat {\\mathbf{r}}} \\right\\|$ attains its minimum value. The solution is provided by the orthogonal projection of the vector $\\mathbf{r}$ onto the xy-plane namely, $\\hat {\\mathbf{r}}=\\left( {x,y,0} \\right)^T$. The error in the approximation is given by the magnitude of the error vector $\\left( {0,0,z} \\right)^T$. The latter lies on the z-axis which is the subspace orthogonal to the approximating subspace, the xy-plane.  This example illustrates the orthogonal projection theorem \\cite{Luen69, Keen88}.\n\\newtheorem{OrthogonalProjectionTheorem3}{Theorem}\n\\begin{thm}\n\\label{OrthogonalProjectionTheorem3}\nGiven a Hilbert space $\\mathscr{H}$ and a closed linear subspace $\\mathbb{V}$ and a point $\\mathbf{x} \\in \\mathscr{H}$, there exists a unique point $\\mathbf{v} \\in \\mathbb{V}$ that is the orthogonal projection of $\\mathbf{x}$ onto $\\mathbb{V}$ and that minimizes the norm $\\left\\| \\mathbf{x - v} \\right\\|$.  In addition, we can write $\\mathscr{H} = \\mathbb{V} \\oplus \\mathbb{V}^ \\bot $, and $\\mathbb{V} = \\mathbb{V}^{ \\bot  \\bot } ~ $.\n\\end{thm}\n\n\\noindent Note that the result $\\mathbb{V} \\subset \\mathbb{V}^ {\\bot \\bot}$ of theorem \\ref{VperpSubspace} for an inner product space is now replaced by $\\mathbb{V} = \\mathbb{V}^ {\\bot \\bot}$ in a Hilbert space.  The difference is, of course, the completeness property of a Hilbert space.\n\n\n\n\\section[Multi Resolution Analysis Subspaces]{Multi Resolution Analysis Subspaces}\n\\label{MRASubspaces}\n\nAs described in the previous section, an orthogonal projection is used in the problem of approximating a function using the closest element of a given subspace.  An important concept in the theory and implementation of the discrete orthogonal wavelet transform is the concept of orthogonal projections into nested subspaces\\index{Nested subspaces} that satisfy a completeness property and multi resolution property \\cite{Mall89a, Mall99}. \n\\newtheorem{Nesting}{Definition}\n\\begin{defn}\nIn $L_2(\\mathbb{R})$, a set of closed subspaces $\\mathscr{V}_m \\subset L_2 (\\mathbb{R})$, $m \\in \\mathbb{Z}$, is said to satisfy the nesting property if \n\\begin{equation}\n\\label{Chapter1-30}\n\\mathscr{V}_m  \\subset \\mathscr{V}_{m - 1},~ m \\in \\mathbb{Z} ~.  \n\\end{equation}\n\\end{defn}\n\\newtheorem{SuccessiveApproximation}{Definition}\n\\begin{defn}\nA nested set of closed subspaces $\\mathscr{V}_m$ satisfying the completeness properties\n\\begin{equation}\n\\label{Chapter1-31}\n\\bigcup \\limits_{m \\in \\mathbb{Z}} \\mathscr{V}_m  = L_2 \\left( \\mathbb{R} \\right),~\\bigcap \\limits_{m \\in \\mathbb{\\mathbb{Z}}} \\mathscr{V}_m  = \\left\\{ 0 \\right\\} ~,\n\\end{equation}\nis said to have the successive approximation property\\index{Nested subspaces!successive approximation}.  For such a set of nested subspaces we also have\n\\[\nL_2 (\\mathbb{R}) = \\mathop {Lim}\\limits_{m \\to  - \\infty } \\mathscr{V}_m , ~~ \\left\\{ \\mathbf{0} \\right\\} = \\mathop {Lim}\\limits_{m \\to \\infty } \\mathscr{V}_m \n\\] \n\\end{defn}\n\nIf $L_2( \\mathbb{R} )$ admits a nested set of subspaces with the successive approximation property then any square integrable function can be approximated by an orthogonal projection onto any one of the subspaces $\\mathscr{V}_m$.  The process is iterative and is depicted in figure \\ref{fig:fig-Chapter1-1}.  Note that an approximation at stage $m$ produces two results:  one is the coarser approximation at stage $m+1$, and the other is the detail that is lost between the approximations at stage $m$ and $m+1$.  The former is an orthogonal projection into $\\mathscr{V}_{m+1}$, while the latter lies in \n$\\mathscr{V}_m^ \\bot$.\n\\begin{figure}[h!]\n\t\\begin{centering}\n\t\\includegraphics[width=3in]{./fig-Chapter1-1.pdf}\n\t\\caption{Approximation of a vector using nested subspaces and their orthogonal complements.}\n\t\\label{fig:fig-Chapter1-1}\n\t\\end{centering}\n\\end{figure}\n\n\n\\newtheorem{Multiresolution}{Definition}\n\\begin{defn}\n\\label{MRASubspacesDefn}\nA nested set of successive approximation subspaces $\\mathscr{V}_m$ is said to have the multi resolution property\\index{Nested subspaces!multi resolution} provided that \\cite{Daub92}\n\\begin{equation}\n\\label{Chapter1-32}\nx\\left( t \\right) \\in \\mathscr{V}_m  \\Leftrightarrow x\\left( {2t} \\right) \\in \\mathscr{V}_{m - 1} ~.\n\\end{equation}\nThe subspaces $\\mathscr{V}_m$ are then said to form a multi resolution analysis set of subspaces and are referred to as a MRA\\index{MRA}.\n\\end{defn}\n\nWe illustrate the concept of a MRA by producing a set of nested subspaces of $L_2(\\mathbb{R}$ satisfying the properties of completeness and multi resolution.  Consider the space $\\mathscr{V}_0$ consisting of all functions that are magnitude square integrable and that have constant values in all intervals of length $1$ of the form $[n, n+1]$, $n \\in \\mathbb{Z}$.  Similarly, consider the space $\\mathscr{V}_{-1}$, all functions that are magnitude square integrable and that have constant values in all intervals of length ${1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}$ of the form $\\left[ {{n \\mathord{\\left/ {\\vphantom {n 2}} \\right. \\kern-\\nulldelimiterspace} 2},{n \\mathord{\\left/ {\\vphantom {n 2}} \\right.\n \\kern-\\nulldelimiterspace} 2} + {1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\right]$, in addition to the space $\\mathscr{V}_{1}$, all functions that are magnitude square integrable and that have constant values in all intervals of length $2$ of the form $\\left[ {2n,2n + 2} \\right]$, $n \\in \\mathbb{Z}$.  Three typical functions from these three spaces are shown in figure \\ref{fig:fig-Chapter1-2} below.\n\\begin{figure}[h!]\n\t\\begin{centering}\n\t\\includegraphics[width=5in]{./fig-Chapter1-2.pdf}\n\t\\caption{Three typical functions that are constant on intervals of size $1/2$, $1$ and $2$.}\n\t\\label{fig:fig-Chapter1-2}\n\t\\end{centering}\n\\end{figure}\nEach one of these spaces is actually a subspace (the zero function is common to all of them, and a linear combination of two functions in any one space is again a function of the same type).  In addition, the spaces are nested, i.e., $\\mathscr{V}_{ - 1}  \\supset \\mathscr{V}_0  \\supset \\mathscr{V}_1$.  For instance, $\\mathscr{V}_1 \\subset \\mathscr{V}_0 $, since all functions that are constant on intervals $\\left[ {2n,2n+2} \\right]$, are also constant on intervals $\\left[ {n,n+1} \\right]$, but not vice versa.  The multi resolution property, easily verified, is interpreted as doubling the approximation resolution in going from $\\mathscr{V}_0 $ to $\\mathscr{V}_{-1}$ in the sense that if $x\\left( t \\right)\\in \\mathscr{V}_0 $ then $x\\left( {2t} \\right)\\in \\mathscr{V}_{-1} $.  Similarly the resolution is halved in going from $\\mathscr{V}_0 $ to $\\mathscr{V}_1 $ in the sense that if $x\\left( t \\right)\\in \\mathscr{V}_0 $ then $x\\left( {{t \\mathord{\\left/ {\\vphantom {t 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\right) \\in \\mathscr{V}_1 $.  Thus, we define the subspaces  $\\mathscr{V}_m$ of functions that have constant values in intervals of length $2^m$, of the form $\\left[ {2^m n,2^m \\left( {n + 1} \\right)} \\right]$, $m,n \\in \\mathbb{Z}$ and obtain a set of nested subspaces satisfying the completeness and the multi resolution properties, also known as the Haar MRA\\index{Haar!MRA}.  The Haar multi resolution approximation to a given function will be seen to consist of orthogonal projections onto the subspaces $\\mathscr{V}_m$ while the Haar wavelet transform coefficients will be the approximation errors, i.e., orthogonal transformations onto the orthogonal complement subspaces  $\\mathscr{V}_m ^\\bot$.\n\nThe key to the fast implementation of the discrete wavelet transform is the multi resolution property and the existence of complete and orthonormal bases in the subspaces $\\mathscr{V}_m$ and $\\mathscr{V}_m ^\\bot$.  We will discuss complete and orthonormal bases in the next section.\n\n\n\n\n\n\n\\section[Complete and Orthonormal Bases]{Complete and Orthonormal Bases in $L_2 \\left( \\mathbb{R} \\right)$}\n\\label{CompleteOrthonormalBases}\n\n\nThe concept of an orthonormal basis\\index{basis!infinite} in a finite dimensional vector space can be extended to an infinite dimensional Hilbert space $\\mathscr{H}$.  Later in this chapter we will study the more general concept of a frame but for now we define a complete and orthonormal basis\\index{basis!complete and orthonormal} with respect to which any element of a Hilbert space has a unique representation.  The concepts of decomposition of a function (signal) in terms of a basis set (analysis) and its inverse (reconstruction or synthesis) are what transform theory in general, and wavelet theory in particular, are based on.  For a complete and orthonormal basis denoted by $\\phi_n$, $n \\in \\mathbb{Z}$, the analysis and synthesis operations are summarized in one formula\n\\begin{equation}\n\\label{Chapter1-33}\nx = \\sum\\limits_n {\\left\\langle {\\phi _n, x} \\right\\rangle \\phi _n } ~,\n\\end{equation}\nwith the inner product representing the analysis operation and the entire sum representing the reconstruction or synthesis operation.  The right hand side is an infinite linear combination of the form $\\sum\\limits_n {c_n \\phi _n }$ and so we begin with conditions under which infinite sums of an orthonormal set converge \\cite{Keen88}. \n\n\\newtheorem{SeriesConvergence}{Theorem}\n\\begin{thm}\n\\label{SeriesConvergence}\nFor an orthonormal set $\\left\\{ {\\phi_n ,n \\in \\mathbb{Z}} \\right\\}$,  \n\\begin{equation}\n\\label{Chapter1-34}\n\\left\\langle {\\phi_n ,\\phi_m } \\right\\rangle  = \\delta _{nm}~,\n\\end{equation}\nthe infinite linear combination $\\sum\\limits_{n =  - \\infty }^\\infty  {c_n \\phi_n }$ converges if, and only if,\n\\[\n\\underline{c} = \\left[ { \\ldots ,c_n , \\ldots } \\right]^T  \\in l_2 \\left( \\mathbb{Z} \\right) ~~  \\Leftrightarrow ~~\\sum\\limits_{n= - \\infty }^\\infty {\\left| {c_n } \\right|^2} < \\infty ~. \n\\]\n\\end{thm}\nThe most widely used description of completeness of a basis $\\phi_n$, $n \\in \\mathbb{Z}$ is that for every $x \\in \\mathscr{H}$, there is a unique expansion \n\\begin{equation}\n\\label{Chapter1-35}\nx = \\sum\\limits_{n =  - \\infty }^\\infty  {c_n \\phi_n } ~, ~~ c_n \\in \\mathbb{C}~,\n\\end{equation}\n%with expansion coefficients $c_n$ given by \n%\\begin{equation}\n%\\label{Chapter1-35}\n%c_n =\\left\\langle {\\phi_n , x} \\right\\rangle ~,  \n%\\end{equation}\nwhere the equality expressed in equation \\eqref{Chapter1-35} is not a point-by-point equality, but equality in the sense of the convergence in the mean of definition \\ref{ConvergenceInTheMean}.  Thus, \n\\begin{equation}\n\\label{Chapter1-36}\n\\left\\| {x -\\sum\\limits_{n=-N}^{N} {c_n \\;\\phi _n  } } \\right\\| \\to 0,~~  {\\rm as} ~ N\\to \\infty ~.\n\\end{equation}\nThe following is a more formal definition of completeness.\n\\newtheorem{CompleteSet}{Definition}\n\\begin{defn}\nAn orthonormal set $\\left\\{ {\\phi_n} \\right\\}$ in a Hilbert space $\\mathscr{H}$ is said to be complete\\index{basis!complete} if it is contained in no other orthonormal set (i.e., it is a maximal orthonormal set). \n\\end{defn}\n\\noindent The following theorems show three ways to characterize complete and orthonormal sets of functions.\n\\newtheorem{CompleteSetTheorem1}{Theorem}\n\\begin{thm}\n\\label{CompleteSetTheorem1}\nAn orthonormal set $\\left\\{ {\\phi_n} \\right\\}$ in a Hilbert space $\\mathscr{H}$ is complete if, and only if, the only vector that is orthogonal to the set is the zero vector $\\mathbf{0}$.  \n\\end{thm}\n\\newtheorem{CompleteSetTheorem2}{Theorem}\n\\begin{thm}\n\\label{CompleteSetTheorem2}\nAn orthonormal set $\\left\\{ {\\phi_n} \\right\\}$ in a Hilbert space $\\mathscr{H}$ is complete if its closure is the entire space.  The closure of the set is defined as the set of all infinite sums $\\sum\\limits_{n =  - \\infty }^\\infty  {c_n } \\phi _n$ whose coefficients are in $l_2 \\left( \\mathbb{Z} \\right)$, i.e., $\\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {c_n } \\right|^2 }  < \\infty$. \n\\end{thm}\n\\newtheorem{CompleteSetTheoremParseval}{Theorem}\n\\begin{thm}\n\\label{CompleteSetTheoremParseval}\nAn orthonormal set $\\left\\{ {\\phi_n} \\right\\}$ in a Hilbert space $\\mathscr{H}$ is complete if Parseval's relation\\index{Parseval's relation} holds for every member of the space,\n\\begin{equation}\n\\label{Chapter1-37}\n\\left\\| x \\right\\|^2=\\sum\\limits_{n=- \\infty}^\\infty {\\left| {\\left\\langle {\\phi_n , x} \\right\\rangle } \\right|^2}, ~~ \\forall x \\in \\mathscr{H} ~.  \n\\end{equation}\n\\end{thm}\n\nThe following theorem summarizes the above results \\cite{Keen88}.\n\\newtheorem{CompleteSetTheorem3}{Theorem}\n\\begin{thm}\n\\label{CompleteSetTheorem3}\nFor an orthonormal set $\\left\\{ {\\phi_n ,n \\in \\mathbb{Z}} \\right\\}$ in a Hilbert space $\\mathscr{H}$ the following statements are equivalent:\n\\begin{itemize}\n  \\item {The set is complete.}\n  \\item {The only vector that is orthogonal to the set is the zero vector.}\n  \\item {For every $x$ in $\\mathscr{H}$ we have the expansion $x = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {\\phi _n ,x} \\right\\rangle \\phi _n }$.}\n  \\item {The closure of the set is the entire space $\\mathscr{H}$.}\n  \\item {For every $x$ in $\\mathscr{H}$, $\\left\\| x \\right\\|^2  = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {\\phi _n ,x} \\right\\rangle } \\right|^2 }$.}\n  \\item {For every $x$ and $y$ in $\\mathscr{H}$, $\\left\\langle {x,y} \\right\\rangle  = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {x,\\phi _n } \\right\\rangle \\left\\langle {\\phi _n ,y} \\right\\rangle }$.}\n\\end{itemize}\n\\end{thm}\n\nAn example of an orthogonal set that is not complete is the infinite set $\\left\\{ {\\sin \\left( {nt} \\right),\\;n\\in {\\rm Z},\\;t\\in \\left[ {0,2\\pi } \\right]} \\right\\}$. This set, although orthogonal with respect to the usual inner product \n\\begin{equation}\n\\label{Chapter1-38}\n\\int\\limits_0^{2\\pi } {\\sin \\left( {mt} \\right)} \\sin \\left( {nt} \\right)dt = \\pi \\delta_{mn} ~,\n\\end{equation}\nis not complete since any member of $\\left\\{ {\\cos \\left( {kt} \\right),\\;k\\in \\mathbb{Z},\\;t\\in \\left[ {0,2\\pi } \\right]} \\right\\}$ is orthogonal to it.\n\nWith respect to a complete and orthonormal basis, the expansion coefficients \n\\begin{equation}\n\\label{Chapter1-39}\nc_n  = \\left\\langle {\\phi _n ,x} \\right\\rangle  = \\int\\limits_{ - \\infty }^\\infty  {\\phi _n^ *  \\left( t \\right)} x\\left( t \\right)dt ~,\n\\end{equation}\nare the transform of a given function $x(t)$, and fully describe that function in the sense that the function $x(t)$ can be reconstructed from the transform coefficients at almost every point $t$, i.e., all points except for a set of measure zero \\footnote{The Gibbs phenomenon\\index{Gibbs phenomenon} is such an example.  Given a periodic function which is piecewise continuously differentiable with a countable number of discontinuities, then its Fourier series expansion converges to the value of the function everywhere except at the points of discontinuity.  If $N>>1$ denotes the number of terms in the Fourier series partial sum, then it will overshoot the discontinuity at one end while undershooting it at the other end by the same amount, nearly $18\\%$.}.  In other words, the transform relation is uniquely invertible in the form  \n\\begin{equation}\n\\label{Chapter1-40}\nx\\left( t \\right) = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {\\phi _n ,x} \\right\\rangle  } \\phi _n \\left( t \\right) ~.\n\\end{equation}\nAronszajn's theorem \\ref{Aronszajn} stated that a Hilbert space whose evaluation functional is bounded (and therefore continuous) admits a reproducing kernel\\index{reproducing kernel}.  If the Hilbert space has a complete orthonormal basis set $\\left\\{ {\\phi_n \\left( t \\right),n \\in \\mathbb{Z}} \\right\\}$ then the kernel has a simple representation described in the following theorem \\cite{Davi75}.\n\\newtheorem{BergmanKernel}{Theorem}\n\\begin{thm}\n\\label{BergmanKernel}\nFor a complete and orthonormal basis $\\left\\{ {\\phi_n \\left( t \\right),n \\in \\mathbb{Z}} \\right\\}$ in a Hilbert space, and for a fixed $t'$, the sum\n\\begin{equation}\n\\label{Chapter1-41}\nK\\left( {t ,t'} \\right) = \\sum\\limits_{n =  - \\infty }^\\infty  {\\phi_n^ *  \\left( {t} \\right)\\phi_n \\left( {t'} \\right)} \n\\end{equation}\nconverges uniformly and absolutely.  Furthermore, the function $K$ is the (unique) reproducing kernel for the Hilbert space.  \n\\end{thm}\nWe will see an example of this when we discuss the subspace of band limited functions within $L_2 \\left( \\mathbb{R} \\right)$ in section \\ref{BandLimitedSection}.  Completeness of an orthonormal system can now be stated in terms of the reproducing kernel \\cite{Davi75}.\n\\newtheorem{CompletenessAndKernel}{Theorem}\n\\begin{thm}\n\\label{CompletenessAndKernel}\nIn a Hilbert space with a reproducing kernel $K$, an orthonormal set $\\left\\{ {\\phi_n \\left( t \\right),n \\in \\mathbb{Z}} \\right\\}$ is complete if, and only if, \n\\begin{equation}\n\\label{Chapter1-42}\nK\\left( {t,t} \\right) = \\sum\\limits_{n =  - \\infty }^\\infty {\\left| {\\phi _n \\left(t \\right)} \\right|^2 } ~.\n\\end{equation}\n\\end{thm}\n\n\n\n\n\n\\section{The Dirac Notation}\n\n\nThe Dirac notation\\index{Dirac!notation} is a compact and useful way to describe completeness and orthonormality relations \\cite{Dira74}. Although the evaluation functional in $L_2 \\left( \\mathbb{R} \\right)$ is not bounded and so a proper reproducing Kernel (i.e, square integrable) does not exist in that space, a generalized function form of the reproducing Kernel can be written using the Dirac delta distribution defined in section \\ref{DiracDelta},\n\\begin{equation}\n\\label{Chapter1-43}\nx\\left( t \\right) = \\int\\limits_{ - \\infty }^\\infty  {\\delta \\left( {t - t'} \\right)x\\left( {t'} \\right)dt'}  ~.\n\\end{equation}\nThe function $x(t)$ is normally represented by $x$ (dropping the $t$ variable) since the value of the function at $t$ is the result of using the evaluation functional (see definition \\ref{EvaluationFunctional}).  Dirac introduced the notation $\\left| x \\right\\rangle$ and called it a ket vector in the corresponding Hilbert space.  A generalized evaluation functional along the lines of theorem \\ref{FrechetRiesz} is then defined so that \n\\begin{equation}\n\\label{Chapter1-44}\nx\\left( t \\right) = \\left\\langle {t,x} \\right\\rangle ~,\n\\end{equation}\nin analogy with equation \\eqref{Chapter1-19}.  Dirac chose the following notation for the above inner product\n\\begin{equation}\n\\label{Chapter1-45}\n\\left\\langle {t} \\mathrel{\\left | {\\vphantom {t x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle ~, \n\\end{equation}\nintroducing the bra vector $\\left\\langle t \\right|$.  The generalized reproducing kernel, in analogy to the definition \\eqref{Chapter1-20}, then becomes\n\\begin{equation}\n\\label{Chapter1-46}\n\\delta \\left( {t - t'} \\right) \\equiv \\left\\langle {t} \\mathrel{\\left | {\\vphantom {t {t'}}} \\right. \\kern-\\nulldelimiterspace} {{t'}} \\right\\rangle ~.\n\\end{equation}\nEquation \\eqref{Chapter1-43} then becomes\n\\begin{equation}\n\\label{Chapter1-47}\n\\left\\langle {t} \\mathrel{\\left | {\\vphantom {t x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle  = \\int\\limits_{ - \\infty }^\\infty  {dt'\\left\\langle {t}\n \\mathrel{\\left | {\\vphantom {t {t'}}} \\right. \\kern-\\nulldelimiterspace} {{t'}} \\right\\rangle \\left\\langle {{t'}} \\mathrel{\\left | {\\vphantom {{t'} x}}\n \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle } ~,\n\\end{equation}\nvalid for all ket vectors $\\left| x \\right\\rangle$.  Thus, we have the completeness, or resolution of identity\\index{resolution of identity}, relation \n\\begin{equation}\n\\label{Chapter1-48}\n\\int\\limits_{ - \\infty }^\\infty  {dt'\\left| {t'} \\right\\rangle \\left\\langle {t'} \\right|}  = \\mathbf{1} ~.\n\\end{equation}\nIf we further introduce the Einstein integration convention that all repeated continuous variables occurring in bras and kets are integrated over (we will later introduce the same convention for repeated discrete indices, i.e., the Einstein summation convention) we have\n\\begin{equation}\n\\label{Chapter1-49}\n\\left| t \\right\\rangle \\left\\langle t \\right| = \\mathbf{1} ~.\n\\end{equation}\nRepeated continuous variables, such as the variable $t$ in the above equation, are also known as dummy variables:  we can use $t'$ or any other variable in the above equation since an integration over the variable is implied.\n\nWe now make the following associations, defining the ket $\\left| {\\phi _n } \\right\\rangle$ and the bra $\\left\\langle {\\phi _n } \\right|$,\n\\begin{equation}\n\\label{Chapter1-50}\n\\phi _n \\left( t \\right) \\leftrightarrow \\left\\langle {t} \\mathrel{\\left | {\\vphantom {t {\\phi _n }}} \\right. \\kern-\\nulldelimiterspace}\n {{\\phi _n }} \\right\\rangle , ~~~ \\phi _n^ *  \\left( t \\right) \\leftrightarrow \\left\\langle {{\\phi _n }} \\mathrel{\\left | {\\vphantom {{\\phi _n } t}} \\right. \\kern-\\nulldelimiterspace}\n {t} \\right\\rangle  ~.\n\\end{equation}\nThe generalized reproducing kernel property of equation \\eqref{Chapter1-46} together with the completeness relation \\eqref{Chapter1-48}, then lead to\n\\begin{equation}\n\\label{Chapter1-51}\n\\left\\langle {\\phi _n ,x} \\right\\rangle  \\equiv \\int\\limits_{ - \\infty }^\\infty  {\\phi _n^ *  \\left( t \\right)x\\left( t \\right)dt}  = \\int\\limits_{ - \\infty }^\\infty  {\\left\\langle {{\\phi _n }} \\mathrel{\\left | {\\vphantom {{\\phi _n } t}} \\right. \\kern-\\nulldelimiterspace}\n {t} \\right\\rangle \\left\\langle {t} \\mathrel{\\left | {\\vphantom {t x}}\n \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle dt}  \\equiv \\left\\langle {{\\phi _n }}\n \\mathrel{\\left | {\\vphantom {{\\phi _n } x}} \\right. \\kern-\\nulldelimiterspace}\n {x} \\right\\rangle ~.\n\\end{equation}\nIn addition to the integration convention we introduce the Einstein summation convention:  An expression containing a repeated index is summed over that index.  For instance, using the Dirac notation the completeness relation \\eqref{Chapter1-40} becomes\n\\begin{equation}\n\\label{Chapter1-52}\n\\left| x \\right\\rangle  = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\phi _n } \\right\\rangle } \\left\\langle {{\\phi _n }}\n \\mathrel{\\left | {\\vphantom {{\\phi _n } x}}\n \\right. \\kern-\\nulldelimiterspace}\n {x} \\right\\rangle  ~,\n\\end{equation}\nwhich using the summation convention can be reduced to \n\\begin{equation}\n\\label{Chapter1-53}\n\\left| {\\phi _n } \\right\\rangle \\left\\langle {\\phi _n } \\right| \\equiv \\sum\\limits_{n = -\\infty }^\\infty {\\left| {\\phi_n } \\right\\rangle \\left\\langle {\\phi_n } \\right|}=\\mathbf{1} ~. \n\\end{equation}\n\\noindent Using the Dirac notation the orthonormality relation \\eqref{Chapter1-34} is now written as \\footnote{Using the resolution of identity \\eqref{Chapter1-55}, $\\left\\langle {{\\phi _m }} \\mathrel{\\left | {\\vphantom {{\\phi _m } {\\phi _n }}} \\right. \\kern-\\nulldelimiterspace} {{\\phi _n }} \\right\\rangle  \\equiv \\int\\limits_{ - \\infty }^\\infty  {\\left\\langle {{\\phi _m }} \\mathrel{\\left | {\\vphantom {{\\phi _m } t}} \\right. \\kern-\\nulldelimiterspace} {t} \\right\\rangle } \\left\\langle {t} \\mathrel{\\left | {\\vphantom {t {\\phi _n }}}\\normalfont \\right. \\kern-\\nulldelimiterspace} {{\\phi _n }} \\right\\rangle dt = \\int\\limits_{ - \\infty }^\\infty  {\\phi _m^ *  \\left( t \\right)} \\phi _n \\left( t \\right)dt$.}\n\\begin{equation}\n\\label{Chapter1-54}\n\\left\\langle {{\\phi _m }}\n \\mathrel{\\left | {\\vphantom {{\\phi _m } {\\phi _n }}}\n \\right. \\kern-\\nulldelimiterspace}\n {{\\phi _n }} \\right\\rangle  = \\delta _{mn} ~.\n\\end{equation}\n\\noindent A linear operator acting on a ket produces another ket, and together with its bra form we have the relations \n\\begin{equation}\n\\label{Chapter1-55}\nL\\left| \\phi  \\right\\rangle = \\left| \\psi  \\right\\rangle ~, ~~ \\left\\langle \\psi  \\right| = \\left\\langle \\phi  \\right|L^ +  ~.\n\\end{equation}\nFor instance, the functions $\\left( {2\\pi } \\right)^{ - {1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\exp \\left( {i\\omega t} \\right)$ in the Dirac notation are written as $\\left\\langle {t} \\mathrel{\\left | {\\vphantom {t \\omega }} \\right. \\kern-\\nulldelimiterspace} {\\omega } \\right\\rangle$ and their complex conjugates are   \n\\[\n\\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  t}} \\right. \\kern-\\nulldelimiterspace} {t} \\right\\rangle  = \\left( {2\\pi } \\right)^{ - {1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\exp \\left( { - i\\omega t} \\right) ~.\n\\]\nThus,\n\\[\n\\left\\langle {{t'}} \\mathrel{\\left | {\\vphantom {{t'} \\omega }} \\right. \\kern-\\nulldelimiterspace} {\\omega } \\right\\rangle \\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  t}} \\right. \\kern-\\nulldelimiterspace} {t} \\right\\rangle  = \\frac{1}{{2\\pi }}\\int\\limits_{ - \\infty }^\\infty  {e^{ i\\omega \\left( {t' - t} \\right)} d\\omega } = \\delta \\left( {t' - t} \\right) = \\left\\langle {{t'}} \\mathrel{\\left | {\\vphantom {{t'} t}} \\right. \\kern-\\nulldelimiterspace} {t} \\right\\rangle ~, \n\\]\nusing the completeness relation and the integration convention (integration over repeated variable $\\omega$ is assumed), again leading to the completeness relation (resolution of identity\\index{resolution of identity}) \n\\[\n\\left| \\omega  \\right\\rangle \\left\\langle \\omega  \\right| = \\mathbf{1} ~.  \n\\]\nThe functions $\\left( {2\\pi } \\right)^{ - {1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\exp \\left( {-i\\omega t} \\right)$ can be thought of as the Fourier basis functions\\index{Fourier!basis functions} in the  expansion of any $L_2(\\mathbb{R})$ function (excluding all sets of measure $0$ on which functions can be badly behaved), but they are not members of $L_2(\\mathbb{R})$ since they are clearly not square integrable.  One way to include these functions is by considering the dual space as discussed in section \\ref{DiracDelta}.  Although the existence of the Fourier transform will be shown in the next section, this formal point of view has the advantage of showing the deficiencies of the continuous Fourier transform and motivating the search for other continuous basis function (e.g., the windowed Fourier transform and the continuous wavelet transform) whose discretization will lead to frames and orthonormal bases.\n\n\n\n\\section{The Fourier Transform}\n\\label{Section_FT}\n\nThe continuous time Fourier transform\\index{Fourier!transform} can actually be defined for a larger class of functions than $L_2(\\mathbb{R})$, namely the class of absolutely integrable functions $L_1(\\mathbb{R})$, as described in the following theorem \\cite{Brem65, Korn88}.\n\\newtheorem{FTL1}{Theorem}\n\\begin{thm}\n\\label{FTL1}\nLet $x(t) \\in L_1(\\mathbb{R})$.  Then $X(\\omega)$ defined by\n\\begin{equation}\n\\label{Chapter1-56}\nX\\left( \\omega  \\right) = \\int\\limits_{ - \\infty }^\\infty  {x\\left( t \\right)e^{ - i\\omega t} dt} \n\\end{equation}\nis uniformly continuous and bounded for $\\omega \\in \\mathbb{R}$, and $X\\left( \\omega  \\right) \\to 0$ for $\\left| \\omega  \\right| \\to \\infty$.  In addition, if $X(\\omega) \\in L_1(\\mathbb{R})$, then\n\\begin{equation}\n\\label{Chapter1-57}\nx\\left( t \\right) = \\frac{1}{{2\\pi }}\\int\\limits_{ - \\infty }^\\infty  {X\\left( \\omega  \\right)e^{ + i\\omega t} d\\omega } ~,\n\\end{equation}\nwhere the equality is in the sense of equality in the mean (see definition \\ref{ConvergenceInTheMean}).  Note that $X(\\omega) \\in L_1(\\mathbb{R})$ is not implied by $x(t) \\in L_1(\\mathbb{R})$.\n\\end{thm}\n\n\\noindent A class of functions that are not absolutely integrable and still possess a Fourier transform also exists, e.g., the function ${{\\sin \\left( t \\right)} \\mathord{\\left/ {\\vphantom {{\\sin \\left( t \\right)} t}} \\right. \\kern-\\nulldelimiterspace} t}$.  These functions are covered by the following theorem.\n\\newtheorem{FTL1-2}{Theorem}\n\\begin{thm}\n\\label{FTL1-2}\nLet $x\\left( t \\right) = y\\left( t \\right)\\sin \\left( {\\omega t + \\phi } \\right)$.  If $y\\left( {t + k} \\right) < y\\left( t \\right)$, and if % \n\\[\n\\int\\limits_{\\left| t \\right| > \\varepsilon  > 0} {\\left| {{{x\\left( t \\right)} \\mathord{\\left/ {\\vphantom {{x\\left( t \\right)} t}} \\right.\n \\kern-\\nulldelimiterspace} t}} \\right|dt}  < \\infty ~,\n\\]\nthen $X(\\omega)$ exists.  In addition, the inverse transform exists and is given by equation \\eqref{Chapter1-57}.\n\\end{thm}\n\\noindent For functions in $L_2(\\mathbb{R})$ we have a stronger result.\n\\newtheorem{FTL2}{Theorem}\n\\begin{thm}\n\\label{FTL2}\n(Plancherel) Let $x(t) \\in L_2(\\mathbb{R})$.  Then $X(\\omega)$ defined by equation \\eqref{Chapter1-56} exists.  In addition,  $X(\\omega) \\in L_2(\\mathbb{R})$ and equation \\eqref{Chapter1-57} holds (in the sense of equality in the mean), i.e., we have \n\\[\nx_\\Omega  \\left( t \\right) = \\left( {2\\pi } \\right)^{ - 1} \\int\\limits_{ - \\Omega }^\\Omega  {X\\left( \\omega  \\right)} e^{i\\omega t} d\\omega ~, {\\rm then} ~~ \\mathop {Lim}\\limits_{\\Omega  \\to \\infty } \\left\\| {x_\\Omega   - x} \\right\\| = 0 ~.\n\\]\n\\end{thm}\n\\noindent The variable $\\omega$ denotes the continuous frequency.  If we use a factor of $\\left( {2\\pi } \\right)^{{{ - 1} \\mathord{\\left/ {\\vphantom {{ - 1} 2}} \\right. \\kern-\\nulldelimiterspace} 2}}$ on the right hand sides of equations \\eqref{Chapter1-56} and \\eqref{Chapter1-57}, we can write both of the equations in the Dirac notation\n\\[\nX\\left( \\omega  \\right) = \\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  t}} \\right. \\kern-\\nulldelimiterspace} {t} \\right\\rangle \\left\\langle {t}\n \\mathrel{\\left | {\\vphantom {t x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle  = \\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  x}}\n \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle , ~~ x\\left( t \\right) = \\left\\langle {t} \\mathrel{\\left | {\\vphantom {t \\omega }} \\right. \\kern-\\nulldelimiterspace}\n {\\omega } \\right\\rangle \\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle  = \\left\\langle {t}\n \\mathrel{\\left | {\\vphantom {t x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle ~,\n\\]\nwhich are equivalent to the Fourier reconstruction (inversion) formula\n\\[\n\\left| x \\right\\rangle  \\equiv \\left| \\omega  \\right\\rangle \\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  x}} \\right. \\kern-\\nulldelimiterspace}\n {x} \\right\\rangle ~.\n\\]\nThus, we adopt the formal view that the vectors $\\left| \\omega  \\right\\rangle$, $\\omega \\in \\mathbb{R}$ form a continuously labeled basis satisfying the completeness and orthonormality relations\n\\[\n\\left| \\omega  \\right\\rangle \\left\\langle \\omega  \\right| = \\mathbf{1}, ~~ \\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  {\\omega '}}}\n \\right. \\kern-\\nulldelimiterspace} {{\\omega '}} \\right\\rangle  = \\delta \\left( {\\omega  - \\omega '} \\right) ~.\n\\]\nThe orthonormality relation follows easily by inserting the resolution of identity\\index{resolution of identity} \\eqref{Chapter1-48}, or \\eqref{Chapter1-49}, in between the inner product, i.e., \n\\[\n\\left\\langle {\\omega }\n \\mathrel{\\left | {\\vphantom {\\omega  {\\omega '}}} \\right. \\kern-\\nulldelimiterspace} {{\\omega '}} \\right\\rangle  = \\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  t}}\n \\right. \\kern-\\nulldelimiterspace} {t} \\right\\rangle \\left\\langle {t} \\mathrel{\\left | {\\vphantom {t {\\omega '}}} \\right. \\kern-\\nulldelimiterspace} {{\\omega '}} \\right\\rangle  = \\left( {2\\pi } \\right)^{ - 1} \\int\\limits_{ - \\infty }^\\infty  {e^{ - it\\left( {\\omega  - \\omega '} \\right)} dt}  = \\delta \\left( {\\omega  - \\omega '} \\right)~.\n\\]\nWe also adopt the convention of using equations \\eqref{Chapter1-56} and \\eqref{Chapter1-57} as the definition of the forward and the inverse Fourier transform instead of their definition in terms of the normalized basis functions $\\left\\langle {\\omega } \\mathrel{\\left | {\\vphantom {\\omega  t}} \\right. \\kern-\\nulldelimiterspace} {t} \\right\\rangle$ and their complex conjugates, which include a factor of $\\left( {2\\pi } \\right)^{ - {1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}}$.  With this convention the equality of the energy in the time and the frequency domain known as the Parseval's relation\\index{Parseval's relation} \\footnote{If we include the factors of $\\left( {2\\pi } \\right)^{{{ - 1} \\mathord{\\left/ {\\vphantom {{ - 1} 2}} \\right. \\kern-\\nulldelimiterspace} 2}}$ on the right hand sides of equations \\eqref{Chapter1-56} and \\eqref{Chapter1-57}, then the $2 \\pi$ in the last integral of Parseval's relation \\eqref{Chapter1-58} must be omitted.}, i.e.,   \n\\begin{equation}\n\\label{Chapter1-58}\n\\left\\| x \\right\\|^2\\equiv \\int\\limits_{-\\infty }^\\infty {\\left| {x(t)} \\right|^2 dt}=\\frac{1}{2\\pi}\\int\\limits_{-\\infty }^\\infty {\\left| {X(\\omega)} \\right|^2 d\\omega } ~,\n\\end{equation}\nfollows from the following theorem.\n\\newtheorem{FTParseval}{Theorem}\n\\begin{thm}\n\\label{FTParseval}\nLet $x(t), y(t) \\in L_2(\\mathbb{R})$.  Denoting their Fourier transforms by $X(\\omega)$ and $Y(\\omega)$ we have\n\\begin{equation}\n\\label{Chapter1-59}\n\\int\\limits_{-\\infty }^\\infty {x^\\ast \\left( {t-k} \\right)y\\left( {t-l} \\right)} \\;dt=\\frac{1}{2\\pi }\\int\\limits_{-\\infty }^\\infty {X^\\ast \\left( \\omega \\right)e^{i\\omega k}\\;Y\\left( \\omega \\right)e^{-i\\omega l}} \\;d\\omega ~.\n\\end{equation}\n\\end{thm}\n\\noindent This can be proven using \n\\[\nx\\left( {t-k} \\right)=\\frac{1}{2\\pi }\\int\\limits_{-\\infty }^\\infty {X\\left( \\omega \\right)e^{i\\omega \\left( {t-k} \\right)}\\;d\\omega } ~.\n\\]\n\nThe rate of decay of the Fourier transform $X(\\omega)$ (as $\\omega  \\to \\infty$) is a measure of the smoothness of the function $x(t)$: the faster the decay, the smoother the function.  Similarly, the rate of decay of $x(t)$ is determined by the smoothness of $X(\\omega)$.  The following three theorems establish these results \\cite{Brem65, Korn88}.\n\\begin{thm}\n\\label{FTDecay1}\nGiven the function $x(t)\\in L_2(\\mathbb{R})$ assume that its Fourier transform $X(\\omega)$ is $N$ times continuously differentiable and all derivatives tend to $0$ as $\\left| \\omega  \\right| \\to \\infty$.  Then $t^N x\\left( t \\right) \\to 0$ as $t \\to \\infty$.\n\\end{thm}\n\\begin{thm}\n\\label{FTDecay2}\nGiven the function $x(t)\\in L_2(\\mathbb{R})$ assume that its Fourier transform $X(\\omega)$ decays sufficiently fast so that $\\omega ^N X\\left( \\omega  \\right)$ is absolutely integrable for some positive integer $N$.  Then $x(t)$ is $N$ times continuously differentiable and \n\\begin{equation}\n\\label{Chapter1-60}\n\\frac{{d^n }}{{dt^n }}x\\left( t \\right) = \\frac{1}{{2\\pi }}\\int\\limits_{ - \\infty }^\\infty  {\\left( {i\\omega } \\right)^n X\\left( \\omega  \\right)e^{i\\omega t} d\\omega }, ~~ 1 \\le n \\le N.\n\\end{equation}\n\\end{thm}\n\\begin{thm}\n\\label{FTDecay3}\nGiven the function $x(t)\\in L_2(\\mathbb{R})$ and its Fourier transform $X(\\omega)$ assume that $x(t)$ decays sufficiently fast so that $t^N x(t)$ is absolutely integrable for some positive integer $N$.  Then $X(\\omega)$ is $N$ times continuously differentiable and \n\\begin{equation}\n\\label{Chapter1-61}\n\\frac{{d^n }}{{d\\omega ^n }}X\\left( \\omega  \\right) = \\frac{1}{{2\\pi }}\\int\\limits_{ - \\infty }^\\infty  {\\left( { - it} \\right)^n x\\left( t \\right)e^{ - i\\omega t} dt}, ~~1 \\le n \\le N.\n\\end{equation}\n\\end{thm}\n\n\nAn important shortcoming of the continuously labeled vectors $\\left| \\omega  \\right\\rangle$, $\\omega \\in \\mathbb{R}$, is the absence of a discretization of the frequency variable that would produce a frame (see section \\ref{FramesInInfinite}) let alone an orthonormal basis.  The issue can be resolved only if we construct continuously labeled functions of $2$ variables by using time domain window functions with the vectors $\\left| \\omega  \\right\\rangle$;  a procedure known as the windowed Fourier transform.  If, on the other hand, we limit our Hilbert space to functions of compact support then the resulting Fourier transform, when discretized appropriately, leads to an orthonormal basis and the transform is known as the Fourier series expansion\\index{Fourier!series expansion}. \n\n\n\n\\section{The Fourier Series Expansion}\n\\label{FourierSeriesExpansion}\n\n\nThe space of square integrable functions with support in the interval $\\left[ {0,T_0} \\right]$ denoted by $L_2 \\left[ {0,T_0 } \\right]$ is an example of a Hilbert space with an orthonormal basis that results from a discretization of the exponential functions $\\exp \\left( {i\\omega t} \\right)$ at frequencies $\\omega _n  = n\\omega _0$ \\cite{Davi63}.  Thus, the set of functions\n\\begin{equation}\n\\label{Chapter1-62}\n\\phi_n(t) =  {T_0}^{ - {1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\exp \\left( {in\\omega _0 t} \\right), ~~ n \\in \\mathbb{Z},\n\\end{equation}\nis complete (see theorem \\ref{Riesz-Fischer} below) and orthonormal \\footnote{when proving orthonormality we use the result $\\mathop {Lim}\\limits_{\\varepsilon  \\to 0} \\left[ {{{\\left( {1 - e^{ - i\\omega _0 {T_0}\\varepsilon }} \\right)} \\mathord{\\left/ {\\vphantom {{\\left( {1 - e^{ - i\\omega _0 {T_0}\\varepsilon } } \\right)} {i\\omega _0 \\varepsilon }}} \\right. \\kern-\\nulldelimiterspace} {i\\omega _0 \\varepsilon }}} \\right] = T_0$} on the given interval provided that $\\omega _0 T_0 = 2\\pi$,\n\\begin{equation}\n\\label{Chapter1-63}\nT_0^{-1} \\int\\limits_0^{T_0} {e^{ - im\\omega _0 t} e^{in\\omega _0 t} dt}  = \\delta _{mn}, ~~\\omega _0 T_0 = 2\\pi ~.\n\\end{equation}\nUsing the Dirac notation and the summation convention we have\n\\begin{equation}\n\\label{Chapter1-64}\n\\left\\langle {t} \\mathrel{\\left | {\\vphantom {t n}} \\right. \\kern-\\nulldelimiterspace} {n} \\right\\rangle  \\equiv \\phi _n \\left( t \\right) ~, ~~ \\left\\langle {m} \\mathrel{\\left | {\\vphantom {m n}} \\right. \\kern-\\nulldelimiterspace} {n} \\right\\rangle=\\delta_{mn} , ~~ \\left| n \\right\\rangle \\left\\langle n \\right|=\\mathbf{1} ~,\n\\end{equation}\nwhere the requirement $\\omega _0 T_0 = 2\\pi$ is understood throughout this section.  A function $x(t)=\\left\\langle {t} \\mathrel{\\left | {\\vphantom {t x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle$ in this space has the Fourier series expansion\n\\begin{equation}\n\\label{Chapter1-65}\n\\left| x \\right\\rangle  = c_n \\left| n \\right\\rangle ,c_n  \\equiv \\left\\langle {n} \\mathrel{\\left | {\\vphantom {n x}} \\right. \\kern-\\nulldelimiterspace}\n {x} \\right\\rangle ~.\n\\end{equation}\nParseval's relation\\index{Parseval's relation} is now \n\\begin{equation}\n\\label{Chapter1-66}\n\\left\\| x \\right\\|^2  = \\int\\limits_0^{T_0} {\\left| {x\\left( t \\right)} \\right|^2 dt}  = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {c_n } \\right|^2 } ~.\n\\end{equation}\nThus, the transform coefficients $c_n$ have finite energy.  The space of all such sequences is the Hilbert space $l_2(\\mathbb{Z})$.  Starting with a sequence $\\left\\{ {c_n ,n \\in \\mathbb{Z}} \\right\\}$ in the Hilbert space $l_2 \\left( \\mathbb{Z} \\right)$ the Riesz-Fischer theorem\\index{Riesz-Fischer theorem} ensures the existence of a function $x(t)$ defined on the interval $\\left[ {0,T_0 } \\right]$ and whose Fourier series coefficients are the given sequence $c_n$.\n\\begin{thm}\n\\label{Riesz-Fischer}\n(Riesz-Fischer) Given a sequence $\\left\\{ {c_n ,n \\in \\mathbb{Z}} \\right\\}$ in the Hilbert space $l_2 \\left( \\mathbb{Z} \\right)$ and the functions $\\phi_n(t)$ defined in \\eqref{Chapter1-62} together with the constraint $\\omega _0 T_0 = 2\\pi$, there exists a (Lebesgue) measurable function $x(t)$ defined on the interval $\\left[ {0,T_0} \\right]$ such that\n\\[\n\\mathop {Lim}\\limits_{N \\to \\infty } \\int\\limits_0^{T_0 } {\\left| {x\\left( t \\right) -  \\sum\\limits_{n =  - N}^N {c_n \\phi _n \\left( t \\right)}} \\right|} ^2 dt = 0 ~.\n\\]\nIn addition, $c_n  = \\left\\langle {\\phi _n ,x} \\right\\rangle $ and Parseval's relation, equation \\eqref{Chapter1-66}, holds \\cite{Beal04}.\n\\end{thm}\nIt is common to use \n\\begin{equation}\n\\label{Chapter1-67}\nx\\left( t \\right) =  {T_0}^{-1} \\sum\\limits_{n =  - \\infty }^\\infty  {c_n } e^{ + in\\omega _0 t}, \n\\end{equation}\nas the Fourier reconstruction formula. The Fourier series coefficients are then given by\n\\begin{equation}\n\\label{Chapter1-68}\nc_n  = \\int\\limits_0^{T_0} {e^{ - in\\omega _0 t} x\\left( t \\right)dt} ,\n\\end{equation}\nomitting the $T_0^{-{1 \\mathord{\\left/ {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}}$ factor that would be present in $\\left\\langle {n} \\mathrel{\\left | {\\vphantom {n x}}\n \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle$, while Parseval's relation becomes\n\\begin{equation}\n\\label{Chapter1-69}\n\\left\\| x \\right\\|^2  = \\int\\limits_0^ {T_0} {\\left| {x\\left( t \\right)} \\right|^2 dt}  = {T_0}^{-1} \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {c_n } \\right|^2 } ~,\n\\end{equation}\nwhich is a special case of the more general equation\n\\begin{equation}\n\\label{Chapter1-70}\n\\int\\limits_0^{T_0} {x^ *  \\left( {t - \\tau _1 } \\right)y\\left( {t - \\tau _2 } \\right)dt}  = {T_0}^{-1} \\sum\\limits_{n =  - \\infty }^\\infty  {c_n^ *  } d_n e^{ + in\\omega _0 \\left( {\\tau _1  - \\tau _2 } \\right)} ,\n\\end{equation}\nwith $c_n$ and $d_n$ denoting the Fourier series coefficients of two function $x(t)$ and $y(t)$.\n\n\n\n\n\n\\section[The Discrete Time Fourier Transform]{The Discrete Time Fourier Transform}\n\nIf we consider the Hilbert space of discrete time sequences with finite energy $l_2(\\mathbb{Z}) $, i.e., $\\left\\{ {s\\left[ n \\right]:\\;\\sum\\limits_{j=-\\infty }^\\infty {\\left| {s\\left[ n \\right]} \\right|^2<\\infty } } \\right\\}$, then we have the discrete time Fourier transform coefficients \\cite{OpSc75}\n\\begin{equation}\n\\label{Chapter1-71}\nS\\left( \\omega \\right)=\\sum\\limits_{n=-\\infty }^\\infty {s\\left[ n \\right]} ~ e^{-in\\omega }\n\\end{equation}\nwhere \n\\begin{equation}\n\\label{Chapter1-72}\ns\\left[ n \\right]=\\frac{1}{2\\pi \n}\\int\\limits_{-\\pi }^\\pi {S\\left( \\omega \\right)\\,} e^{in\\omega }d\\omega \n\\end{equation}\nClearly the Fourier transform $S\\left( \\omega \\right)$ is periodic in the frequency variable $\\omega $ with a period of $2\\pi$. Consequently, the inversion formula is an integral over any interval of length $2\\pi $ which, without loss of generality, we take to be the interval $\\left[ {-\\pi ,\\pi } \\right]$ \\footnote{A discrete time sequence $s\\left[ n \\right]$ is usually the result of sampling of a continuous time function $s\\left( t \\right)$ at times $t = n \\Delta T = {n \\mathord{\\left/ {\\vphantom {n {f_s }}} \\right. \\kern-\\nulldelimiterspace} {f_s }}$.  The exponential function in the Fourier transform is then of the form $e^{ - i{{n\\omega } \\mathord{\\left/ {\\vphantom {{n\\omega } {f_s }}} \\right. \\kern-\\nulldelimiterspace} {f_s }}}$ where now $\\omega  \\in \\left[ { - \\pi f_s , + \\pi f_s ,} \\right]$.  It is common practice, without loss of generality, to take the sampling frequency in this case to be equal to $1$ Hz, and use a normalized frequency variable $\\omega$ in the range $\\left[ { - \\pi , + \\pi } \\right]$.}.  The periodicity in $\\omega $ now leads to the following relation (a special case of the Poisson summation formula \\footnote{The Poisson summation formula is $ 2\\pi \\sum\\limits_{n =  - \\infty }^\\infty  s \\left( {t + 2n\\pi } \\right) = \\sum\\limits_{k =  - \\infty }^\\infty  {S\\left( k \\right)} e^{ikt}$, where $S\\left( \\omega  \\right)$ is the Fourier transform of $s\\left( t \\right)$.})\n\\begin{equation}\n\\label{Chapter1-73}\n2\\pi \\sum\\limits_{n =  - \\infty }^\\infty  {e^{in\\left( {\\omega  - \\omega '} \\right)} }  = \\sum\\limits_{k =  - \\infty }^\\infty  {\\delta \\left( {\\omega  - \\omega ' + 2k\\pi } \\right)} \\end{equation}\nwhich can be used to derive Parseval's relation\\index{Parseval's relation} in the form \n\\begin{equation}\n\\label{Chapter1-74}\n\\sum\\limits_{n=-\\infty }^\\infty {\\left| {s\\left[ n \\right]} \\right|^2=} \\frac{1}{2\\pi }\\int\\limits_{-\\pi }^\\pi {\\left| {S\\left( \\omega \\right)} \\right|^2\\,} d\\omega\n\\end{equation}\nor in the more general form\n\\begin{equation}\n\\label{Chapter1-75}\n\\sum\\limits_{n=-\\infty }^\\infty {s^\\ast \\left[ {n-k} \\right]q\\left[ {n-l} \\right]=} \\frac{1}{2\\pi }\\int\\limits_{-\\pi }^\\pi {S^\\ast \\left( \\omega \n\\right)\\,} e^{-i\\omega k}Q\\left( \\omega \\right)e^{i\\omega l}d\\omega ~.\n\\end{equation}\n\n\n\n\n\\section[The Discrete Fourier Transform]{The Discrete Fourier Transform}\n\nThe space of discrete time sequences of finite energy that have a finite length $N$ is a finite dimensional vector space of dimension $N$.  An orthonormal basis for this space is provided by the functions \\cite{OpSc75} \n\\begin{equation}\n\\label{Chapter1-76}\np_{_{kn}}  = N^{{{ - 1} \\mathord{\\left/ {\\vphantom {{ - 1} 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\exp \\left( {{{2i\\pi kn} \\mathord{\\left/ {\\vphantom {{2i\\pi kn} N}} \\right. \\kern-\\nulldelimiterspace} N}} \\right), ~ ~ \\sum\\limits_{k = 0}^{N - 1} {p_{_{mk}}^ *  } p_{_{kn}}  = \\delta _{mn} ~.\n\\end{equation}\nFor $x[n], ~ n=0, \\ldots, N-1$ in this vector space we have the discrete and finite length Fourier series representation \n\\begin{equation}\n\\label{Chapter1-77}\nx\\left[ n \\right]=\\sum\\limits_{k=0}^{N-1} {X\\left[ k \\right]\\,p_{_n} \\left[ k \\right]} \\equiv \\frac{1}{\\sqrt N }\\sum\\limits_{k=0}^{N-1} {X\\left[ k \\right]\\,} e^{\\frac{2i\\pi kn}{N}},\n\\end{equation}\nwhere the coefficients are given by \n\\begin{equation}\n\\label{Chapter1-78}\nX\\left[ k \\right]=\\frac{1}{\\sqrt N }\\sum\\limits_{n=0}^{N-1} {x\\left[ n \\right]\\,} e^{-\\,\\frac{2i\\pi kn}{N}} ~.\n\\end{equation}\nParseval's relation is \n\\begin{equation}\n\\label{Chapter1-79}\n\\sum\\limits_{k=0}^{N-1} {\\left| {X\\left[ k \\right]} \\right|^2\\,} =\\sum\\limits_{n=0}^{N-1} {\\left| {x\\left[ n \\right]} \\right|^2\\,}\n\\end{equation}\n\nThe above equations define the Discrete Fourier Transform (DFT), that is often implemented by (and referred to as) the Fast Fourier Transform (FFT). A more conventional definition for the transform pair $x\\left[ n \\right] \\leftrightarrow X\\left[ k \\right]$ is given by \\cite{OpSc75}\n\\begin{equation}\n\\label{Chapter1-80}\nX\\left[ k \\right]=\\sum\\limits_{n=0}^{N-1} {x\\left[ n \\right]\\,} e^{-\\,\\frac{2i\\pi kn}{N}},\\;x\\left[ n \\right]=\\frac{1}{N}\\sum\\limits_{k=0}^{N-1} {X\\left[ k \\right]\\,} \ne^{\\frac{2i\\pi kn}{N}}\n\\end{equation}\nThe difference with the original definition is in the normalization of the transform pair. Parseval's relation is now \n\\begin{equation}\n\\label{Chapter1-81}\n\\sum\\limits_{k=0}^{N-1} {\\left| {X\\left[ k \\right]} \\right|^2\\,} =\\frac{1}{N}\\sum\\limits_{n=0}^{N-1} {\\left| {x\\left[ n \\right]} \\right|^2\\,} \n\\end{equation}\n\n\n\n\\section[Band Limited Functions]{Band limited functions and the sampling theorem}\n\\label{BandLimitedSection}\n\nA band limited\\index{band limited functions} function $x_\\Omega (t) \\in L_2(\\mathbb{R})$ is one whose Fourier transform $X_\\Omega (\\omega)$ has support in $\\left[{-\\Omega ,\\Omega } \\right]$, where $\\Omega$ is a positive real number.  The unitarity of the Fourier transform operation ensures that all band limited functions in $L_2(\\mathbb{R})$ form a closed subspace $L_2^\\Omega  \\left( \\mathbb{R} \\right)$.  The evaluation functional (see equations \\eqref{Chapter1-17} and \\eqref{Chapter1-19}) mapping $x_\\Omega   \\to x_\\Omega  \\left( t \\right)$ is bounded (and therefore continuous), for every function in $L_2^\\Omega  \\left( \\mathbb{R} \\right)$ and all $t$.  Aronszajn's theorem (theorem \\ref{Aronszajn}) then ensures that $L_2^\\Omega  \\left( \\mathbb{R} \\right)$ is a reproducing kernel Hilbert space\\index{reproducing kernel!Hilbert space}.  To find the kernel we use the inverse Fourier transform and the band limited property to write\n\\begin{equation}\n\\label{Chapter1-82}\nx_\\Omega  \\left( t \\right) = {(2 \\pi)}^{-1} \\int\\limits_{ - \\Omega }^\\Omega  {X_\\Omega  \\left( \\omega  \\right)} e^{ + i\\omega t} d\\omega ~.\n\\end{equation}\nWe substitute the Fourier transform \n\\begin{equation}\n\\label{Chapter1-83}\nX_\\Omega(\\omega) = \\int\\limits_{- \\infty }^\\infty  {x_\\Omega  (t')} e^{-i \\omega t' } dt' \n\\end{equation}\nand perform the integral over $\\omega$ to obtain \n\\begin{equation}\n\\label{Chapter1-84}\nx_\\Omega (t)=\\int\\limits_{-\\infty}^\\infty {\\frac{{\\sin \\left[ {\\Omega \\left({t-t'} \\right)} \\right]}}{{\\pi \\left( {t-t'} \\right)}}x_\\Omega (t') dt'} ~.\n\\end{equation}\nThis shows the reproducing kernel\\index{reproducing kernel} property of the associated Hilbert space $L_2^\\Omega  \\left( \\mathbb{R} \\right)$, with a real and symmetric kernel function  \\footnote{Note that the limit $\\Omega  \\to \\infty$ and using equation \\eqref{Chapter1-24} will result in the Dirac delta generalized reproducing kernel discussed in section \\ref{DiracDelta}.}\n\\begin{equation}\n\\label{Chapter1-85}\nK\\left( {t,t'} \\right) \\equiv {{\\sin \\left[ {\\Omega \\left( {t -t'} \\right)} \\right]} \\mathord{\\left/ {\\vphantom {{\\sin \\left[ {\\Omega \\left( {t - t' } \\right)} \\right]} {\\pi \\left( {t - t'} \\right)}}} \\right.\n \\kern-\\nulldelimiterspace} {\\pi \\left( {t - t'} \\right)}} ~.\n\\end{equation}\nAnother way to see this result is to note that the operator projecting a function into the space $L_2^\\Omega  \\left( \\mathbb{R} \\right)$\n\\begin{equation}\n\\label{Chapter1-86}\nP_\\Omega  \\left\\{ {x\\left( t \\right)} \\right\\} = {(2 \\pi)}^{-1} \\int\\limits_{ - \\Omega }^\\Omega  {X\\left( \\omega  \\right)e^{i\\omega t} d {\\omega}} \n\\end{equation}\nis an orthogonal projection.  This operator, in the frequency domain, has the form  \n\\begin{equation}\n\\label{Chapter1-87}\nP_\\Omega  \\left\\{ {X\\left( \\omega  \\right)} \\right\\} = X\\left( \\omega  \\right) \\times 1_{\\left[ { - \\Omega ,\\Omega } \\right]}  \\equiv X_\\Omega  \\left( \\omega  \\right)\n\\end{equation}\nwhere the function $1_{\\left[ { - \\Omega ,\\Omega } \\right]}$ is defined to be $1$ on the interval $\\left[ { - \\Omega ,\\Omega } \\right]$ and $0$ elsewhere, and its inverse Fourier transform is ${{\\sin \\left( {\\Omega t} \\right)} \\mathord{\\left/ {\\vphantom {{\\sin \\left( {\\Omega t} \\right)} {\\pi t}}} \\right. \\kern-\\nulldelimiterspace} {\\pi t}}$.  The inverse Fourier transform of $X_\\Omega  \\left( \\omega  \\right)$ is the convolution of $x_\\Omega (t)$ and ${{\\sin \\left( {\\Omega t} \\right)} \\mathord{\\left/ {\\vphantom {{\\sin \\left( {\\Omega t} \\right)} {\\pi t}}} \\right. \\kern-\\nulldelimiterspace} {\\pi t}}$, which is precisely equation \\eqref{Chapter1-84} (see equations \\eqref{Chapter2-1} and \\eqref{Chapter2-2} of chapter \\ref{Chapter_LTI} for theorems on convolution).  Thus, the reproducing kernel \\eqref{Chapter1-85} is a projection operator that maps any function in $L_2 \\left( \\mathbb{R} \\right)$ onto the subspace $L_2^\\Omega  \\left( \\mathbb{R} \\right)$. \n\nLet us define \n\\begin{equation}\n\\label{Chapter1-88}\np_\\tau \\left( t \\right) = {{\\sin \\left[ {\\Omega \\left( {t - \\tau} \\right)} \\right]} \\mathord{\\left/ {\\vphantom {{\\sin \\left[ {\\Omega \\left( {t - \\tau} \\right)} \\right]} {\\pi \\left( {t - \\tau} \\right)}}} \\right. \\kern-\\nulldelimiterspace} {\\pi \\left( {t - \\tau} \\right)}} = K \\left( {t,\\tau} \\right) ~.\n\\end{equation}\nWe note that $p_\\tau \\left( t \\right) = p_{\\tau - t } \\left( 0 \\right)$ and so the functions $p_{\\tau}(t)$ are essentially (except for an overall multiplicative constant) time shifted versions of a ``mother\" function ${\\mathbf{sinc}}\\left( {{{\\Omega t} \\mathord{\\left/ {\\vphantom {{\\Omega t} \\pi }} \\right. \\kern-\\nulldelimiterspace} \\pi }} \\right)$, where\n\\begin{equation}\n\\label{Chapter1-89}\n{\\mathbf{sinc}}\\left( t \\right) \\equiv {{\\sin \\left( {\\pi t} \\right)} \\mathord{\\left/ {\\vphantom {{\\sin \\left( {\\pi t} \\right)} {\\pi t}}} \\right.\n \\kern-\\nulldelimiterspace} {\\pi t}} ~.\n\\end{equation}\nThen we have \n\\begin{equation}\n\\label{Chapter1-90}\n\\int\\limits_{- \\infty }^\\infty {p_{\\tau} \\left( {t} \\right)p_{{\\tau}'} \\left( {t} \\right)dt} = p_{\\tau} \\left( {{\\tau}'} \\right) = p_{{\\tau}'} \\left( {\\tau} \\right) ~.\n\\end{equation}\nThe reproducing kernel equation shows that all elements of $L_2^\\Omega  \\left( \\mathbb{R} \\right)$ can be expanded in terms of the continuously parametrized functions $p_\\tau(t)$, $\\tau  \\in \\mathbb{R}$.  It is well known that a discretization of the shift parameter $\\tau$\n\\begin{equation}\n\\label{Chapter1-91}   \n\\tau_{_{n}}  = {{n\\pi } \\mathord{\\left/ {\\vphantom {{n\\pi } \\Omega }} \\right. \\kern-\\nulldelimiterspace} \\Omega } ~,\n\\end{equation}\nproduces an orthogonal basis \\cite{Keen88}.  The orthogonality follows from setting $\\tau = {{m\\pi} \\mathord{\\left/ {\\vphantom {{m\\pi } \\Omega }} \\right. \\kern-\\nulldelimiterspace} \\Omega }$ and ${\\tau}' = {{n\\pi} \\mathord{\\left/ {\\vphantom {{m\\pi } \\Omega }} \\right. \\kern-\\nulldelimiterspace} \\Omega }$ in equation \\eqref{Chapter1-90}.  Defining the linear bandwidth variable $B \\equiv {\\Omega  \\mathord{\\left/ {\\vphantom {\\Omega  {2\\pi }}} \\right. \\kern-\\nulldelimiterspace} {2\\pi }}$ we are led to the orthonormal basis functions\n\\begin{equation}\n\\label{Chapter1-92}\np_n \\left( t \\right)= \\sqrt {2B}\\;\\mbox{sinc}\\left[ {2B\\left( {t-\\frac{n}{2B}} \\right)} \\right] ~.\n\\end{equation}\nThe Fourier transform of $p_n(t)$ is identically zero outside the closed interval $\\left[ {-2\\pi B,+2\\pi B} \\right]$. Any band limited function $x_\\Omega\\left( t \\right)$ can be expressed in terms of this basis\n\\begin{equation}\n\\label{Chapter1-93}\nx_\\Omega \\left( t \\right)=\\sum\\limits_{n=-\\infty }^\\infty {c_n p_n \\left( t \\right)},\n\\end{equation}\nwith the expansion coefficients given by \n\\begin{equation}\n\\label{Chapter1-94}\nc_n  = \\left\\langle {p_n ,x_\\Omega  } \\right\\rangle  = x_\\Omega  \\left( {{n \\mathord{\\left/\n {\\vphantom {n {2B}}} \\right. \\kern-\\nulldelimiterspace} {2B}}} \\right) ~.\n\\end{equation}\nThus we arrive at the sampling theorem\\index{sampling theorem} \\footnote{Although the theorem is usually referred to as the Shannon sampling theorem, it was independently discovered by Nyquist, Whittaker, and Kotelnikov.} \\cite{Shan49, Shan98}.\n\\newtheorem{ShannonSamplingTheorem}{Theorem}\n\\begin{thm}\n\\label{SamplingTheorem}\nLet $x\\left( t \\right)$ be a continuous time signal whose Fourier transform $X\\left( \\omega \\right)$ is zero outside the interval $\\left[ {-2\\pi B,+2\\pi B} \\right]$, i.e., whose linear frequency content is limited to the band $\\left[ {-B,+B} \\right]\\ $.  If $\\Delta T$ is a positive constant satisfying the following inequalities \n\\begin{equation}\n\\label{Chapter1-95-0}\n0 < 2B ~ \\Delta T \\le 1~,\n\\end{equation}\nthen $x(t)$ can be uniquely reconstructed from its sampled values $x(n \\Delta T)$, $n \\in \\mathbb{Z}$ by the following interpolation formula \n\\begin{equation}\n\\label{Chapter1-95}\nx\\left( t \\right) = 2B ~ \\Delta T\\sum\\limits_{n =  - \\infty }^\\infty  {x\\left( {n\\Delta T} \\right)\\frac{{\\sin \\left[ {2\\pi B\\left( {t - n\\Delta T} \\right)} \\right]}}{{2\\pi B\\left( {t - n\\Delta T} \\right)}}} ~.\n\\end{equation}\nCondition \\eqref{Chapter1-95-0} is usually written in terms of the sampling frequency $f_s  \\equiv {1 \\mathord{\\left/ {\\vphantom {1 {\\Delta T}}} \\right. \\kern-\\nulldelimiterspace} {\\Delta T}}$ in the form\n\\[\nf_s  \\ge 2B ~,\n\\]\nand stated as the requirement that the signal sampling frequency should equal to or exceed twice the highest frequency of the signal.\n\\end{thm}\nThe sampling theorem \\ref{SamplingTheorem} applied to the $\\mathbf{sinc}$ function itself, at a sampling frequency equal to twice the highest frequency,  yields the following result\n\\begin{equation}\n\\label{Chapter1-96}\n\\frac{{\\sin \\left[ {2\\pi B\\left( {t - t'} \\right)} \\right]}}{{2\\pi B\\left( {t - t'} \\right)}} = \\sum\\limits_{n =  - \\infty }^\\infty  {\\frac{{\\sin \\left[ {2\\pi B\\left( {t - \\frac{n}{{2B}}} \\right)} \\right]}}{{2\\pi B\\left( {t - \\frac{n}{{2B}}} \\right)}}\\frac{{\\sin \\left[ {2\\pi B\\left( {t' - \\frac{n}{{2B}}} \\right)} \\right]}}{{2\\pi B\\left( {t' - \\frac{n}{{2B}}} \\right)}}},\n\\end{equation}\nwhich shows the reproducing kernel as an infinite sum over the products of the orthogonal basis functions.  This is an example of theorem \\ref{BergmanKernel} and equation \\eqref{Chapter1-41}.\n\n\n\\section[The Basis Operator]{The Basis Operator in $L_2 ( \\mathbb{R} )$}\n\nThe correspondence between a function and its transform coefficients with respect to an orthonormal basis expressed by equations \\eqref{Chapter1-39} and \\eqref{Chapter1-40} can be interpreted in terms of linear operators \\cite{DuSc52, DaGM86, Daub92}.  Consider the linear operator \n\\begin{equation}\n\\label{Chapter1-97}\nT_\\phi  :L_2 \\left( \\mathbb{R} \\right) \\to l_2 \\left( \\mathbb{Z} \\right) ~,\n\\end{equation}\nassociated with a basis (not necessarily orthonormal) $\\left\\{ {\\phi _n \\left( t \\right),n \\in \\mathbb{Z}} \\right\\}$, defined by  \n\\begin{equation}\n\\label{Chapter1-98}\nT_\\phi  \\left\\{ {x\\left( t \\right)} \\right\\} = \\left[ { \\ldots ,\\left\\langle {{\\phi _n }}\n \\mathrel{\\left | {\\vphantom {{\\phi _n } x}} \\right. \\kern-\\nulldelimiterspace}\n {x} \\right\\rangle , \\ldots } \\right]^T  ~.\n\\end{equation}\nIt is common to refer to this expansion as the analysis formula, or the forward transform, and to the linear operator as the analysis operator of the associated basis.  The corresponding adjoint operator, the synthesis operator, is a linear map \n\\begin{equation}\n\\label{Chapter1-99}\n{T^+}_\\phi  :l_2 \\left( \\mathbb{Z} \\right) \\to L_2 \\left( \\mathbb{R} \\right) ~,\n\\end{equation}\ndefined by the synthesis formula, or the inverse transform, \n\\begin{equation}\n\\label{Chapter1-100}\n{T^+}_\\phi ~ \\left\\{ {\\underline{c}} \\right\\} = \\sum\\limits_{n =  - \\infty }^\\infty  {c_n \\phi _n \\left( t \\right)} ,~~ \\underline{c} \\equiv \\left[ { \\ldots , c_n , \\ldots } \\right]^T ~.\n\\end{equation}\nThe basis operator\\index{operator!basis} is then defined by\n\\begin{equation}\n\\label{Chapter1-101}\n{T^+}_\\phi T_\\phi  :L_2 \\left( \\mathbb{R} \\right) \\to L_2 \\left( \\mathbb{R} \\right)\n\\end{equation}\nand can be expressed in the Dirac notation together with the summation convention, as\n\\begin{equation}\n\\label{Chapter1-102}\n{T^+}_\\phi T_\\phi = {\\left| {\\phi _n } \\right\\rangle } \\left\\langle {\\phi _n } \\right| ~, \n\\end{equation}\nor can be defined by its action on any function $x(t)$ (note the summation convention in the Dirac notation)\n\\begin{equation}\n\\label{Chapter1-103}\n{T^+}_\\phi T_\\phi ~ \\left\\{ {x\\left( t \\right)} \\right\\} = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {\\phi _n ,x} \\right\\rangle \\phi _n \\left( t \\right)} = \\left| {\\phi _n } \\right\\rangle \\left\\langle {{\\phi _n }} \\mathrel{\\left | {\\vphantom {{\\phi _n } x}}\n \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle ~.\n\\end{equation}\nThe basis operator ${T^+}_\\phi T_\\phi$ is clearly self adjoint and positive definite, and so we have\n\\begin{equation}\n\\label{Chapter1-104}\n\\left\\| {{T^+}_\\phi T_\\phi  } \\right\\|_2  = \\rho _{{T^+}_\\phi  T_\\phi}  = \\lambda _{\\max } \\end{equation}\nand\n\\begin{equation}\n\\label{Chapter1-105}\n\\left\\| {\\left( {{T^+}_\\phi T_\\phi  } \\right)^{ - 1} } \\right\\|_2 = \\lambda _{\\min }^{ - 1} ~ .\n\\end{equation}\nwhere $\\lambda_{max}$ and $\\lambda_{min}$ are the largest and smallest eigenvalues of the basis operator $T_\\phi ^ +  T_\\phi $.  The ratio ${{\\lambda _{{\\rm max}} } \\mathord{\\left/ {\\vphantom {{\\lambda _{{\\rm max}} } {\\lambda _{{\\rm min}} }}} \\right. \\kern-\\nulldelimiterspace} {\\lambda _{{\\rm min}} }}$ is the condition number\\index{condition number} of the same operator and its size is crucial in calculating the inverse operator in a numerically stable manner.   \n\nIf the basis functions are not orthonormal, as is the case in an over complete set in which the elements are not linearly independent, the numerical inversion of the basis operator ${T^+}_\\phi T_\\phi $ depends on how large its condition number is.  If, on the other hand, the analysis operator $T_\\phi $ is unitary then ${T^+}_\\phi  T_\\phi = \\mathbf{1}$, all the eigenvalues of the basis operator are equal to $1$ and the inverse is guaranteed to exist.  A unitary analysis operator indicates an orthonormal basis and for such an operator we have\n\\begin{equation}\n\\label{Chapter1-106}\n\\left\\| x \\right\\|^2  = \\left\\langle {x,x} \\right\\rangle  = \\left\\langle {x,{T^+}_\\phi T_\\phi  x} \\right\\rangle  = \\left\\langle {T_\\phi  x,T_\\phi  x} \\right\\rangle  = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {\\phi _n ,x} \\right\\rangle } \\right|^2 },\n\\end{equation}\nwhich is Parseval's relation\\index{Parseval's relation} \\eqref{Chapter1-37} \\footnote{In the Dirac notation $\\left\\| x \\right\\|^2  = \\left\\langle x \\right|T_\\phi ^ +  T_\\phi  \\left| x \\right\\rangle  = \\left\\langle {x} \\mathrel{\\left | {\\vphantom {x {\\phi _n }}} \\right. \\kern-\\nulldelimiterspace}\n {{\\phi _n }} \\right\\rangle \\left\\langle {{\\phi _n }} \\mathrel{\\left | {\\vphantom {{\\phi _n } x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle  \\equiv \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {{\\phi _n }} \\mathrel{\\left | {\\vphantom {{\\phi _n } x}}\n \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle } \\right|^2 }$.}.\n\n\n\\section[Biorthogonal Bases]{Biorthogonal Bases and Representations in $L_2 \\left( \\mathbb{R} \\right)$} \\label{SectionBiorthogonal}\n\n\nMore general than an orthogonal basis is the concept of biorthogonal bases\\index{basis!biorthogonal} that occur in the definition of a Riesz basis.  As we shall see later, biorthogonal bases allow more freedom in choosing wavelet functions of compact support.  The idea of biorthogonality is best explained for finite dimensional vector spaces.  Consider the xy-plane and the a pair of vectors $\\mathbf{v}_1  \\equiv \\left[ {0,1} \\right]^T$ and $\\mathbf{v}_2  \\equiv \\left[ {1 ,\\tan \\theta } \\right]^T$, and a second pair $\\mathbf{w}_1  \\equiv \\left[ {- \\tan \\theta ,1 } \\right]^T$ and $\\mathbf{w}_2  \\equiv \\left[ {1,0} \\right]^T$ for some angle $0 < \\theta  < {\\pi  \\mathord{\\left/ {\\vphantom {\\pi  2}} \\right. \\kern-\\nulldelimiterspace} 2}$.  Then $\\left\\langle {\\mathbf{w}_i , \\mathbf{v}_j } \\right\\rangle  \\equiv \\mathbf{w}_i^T \\mathbf{v}_j  = \\delta_{ij}$ and we call the two pairs biorthogonal.  The two pair of vectors are shown in figure \\ref{fig:fig-Chapter1-3} for $\\theta = 60^ \\circ$. \n\\begin{figure}[h!]\n\t\\begin{centering}\n\t\\includegraphics[width=2in]{./fig-Chapter1-3.pdf}\n\t\\caption{Two biorthogonal pairs of vectors.}\n\t\\label{fig:fig-Chapter1-3}\n\t\\end{centering}\n\\end{figure}\nNow any vector $\\mathbf{x} = \\left[ {x,y} \\right]^T$ can be expanded either in terms of the first pair, or in terms of the second pair.  The coefficients of expansion are then given by the inner product of the given vector with the alternate pair.  For instance, if $\\mathbf{x} = \\alpha_1 \\mathbf{v}_1  + \\alpha_2 \\mathbf{v}_2$, then $\\alpha_j = \\left\\langle {\\mathbf{w}_j , \\mathbf{x}} \\right\\rangle$.  Alternatively, if $\\mathbf{x} = \\beta_1 \\mathbf{w}_1  + \\beta_2 \\mathbf{w}_2$, then $\\beta_j = \\left\\langle {\\mathbf{v}_j , \\mathbf{x}} \\right\\rangle$ for $j = 1,2$.  Neither pair $\\left\\{ {\\mathbf{v}_1 , \\mathbf{v}_2 } \\right\\}$, nor $\\left\\{ {\\mathbf{w}_1 , \\mathbf{w}_2 } \\right\\}$, however, is an orthogonal set.\n\n\\newtheorem{RieszBais}{Definition}\n\\begin{defn}\n\\label{RieszBais}\nGiven a complete and orthonormal basis $\\left\\{ \\phi _n \\left( t \\right), ~ n \\in \\mathbb{Z} \\right\\}$, a new set of basis functions, a Riesz basis\\index{basis!Riesz}, is defined through a non unitary but invertible continuous linear operator $L$ (also known as a topological isomorphism) by the relations \\cite{Chri03}\n\\begin{equation}\n\\label{Chapter1-107}\n\\xi _n \\left( t \\right) = L\\phi _n \\left( t \\right) \\leftrightarrow \\left| {\\xi _n } \\right\\rangle  = L\\left| {\\phi _n } \\right\\rangle ~.\n\\end{equation}\n\\end{defn}\n\\noindent Since $L$ is not unitary, the new basis functions are not orthonormal \\footnote{If $L$ were unitary, i.e., $L^+ L =1$, then $\\left\\langle {\\phi _m ,\\phi _n } \\right\\rangle  = \\left\\langle {L^ +  L\\phi _m ,\\phi _n } \\right\\rangle  = \\left\\langle {\\xi _m ,\\xi _n } \\right\\rangle = \\delta_{mn}.$}.  We have \n\\begin{equation}\n\\label{Chapter1-108}\n\\left\\langle {\\phi _m ,\\phi _n } \\right\\rangle  = \\delta_{mn}  = \\left\\langle {L^{ - 1} L\\phi _m ,\\phi_n } \\right\\rangle  = \\left\\langle {L\\phi _m ,\\left( {L^{ - 1} } \\right)^ +  \\phi_n } \\right\\rangle \n\\end{equation}\nSince $L$ is a topological isomorphism, its inverse is also a continuous linear map and it can be used to define a second Riesz basis set $\\left\\{ \\chi_n \\left( t \\right), ~ n \\in \\mathbb{Z} \\right\\}$\n\\begin{equation}\n\\label{Chapter1-109}\n\\chi _n \\left( t \\right) \\equiv \\left( {L^{ - 1} } \\right)^ +  \\phi_n \\left( t \\right) \\leftrightarrow \\left| {\\chi _n } \\right\\rangle  = \\left( {L^{ - 1} } \\right)^ +  \\left| {\\phi _n } \\right\\rangle ~.\n\\end{equation}\nIt is easy to see that  \n\\begin{equation}\n\\label{Chapter1-110}\n\\left\\langle {{\\xi_m }} \\mathrel{\\left | {\\vphantom {{\\xi _m } {\\chi_n }}} \\right. \\kern-\\nulldelimiterspace} {{\\chi_n }} \\right\\rangle  = \\delta_{mn},\n\\end{equation}\nwhich shows the biorthogonal property of the two Riesz bases $\\left\\{ {\\xi _n \\left( t \\right)} \\right\\}$ and $\\left\\{ {\\chi _n \\left( t \\right)} \\right\\}$.  The corresponding analysis operators are denoted by $T_\\xi $ and $T_\\chi $, where\n\\begin{equation}\n\\label{Chapter1-111}\nT_\\xi  x\\left( t \\right) = \\left[ { \\ldots ,\\left\\langle {\\xi _n ,x} \\right\\rangle , \\ldots } \\right]^T ,~T_\\chi  x\\left( t \\right) = \\left[ { \\ldots ,\\left\\langle {\\chi _n ,x} \\right\\rangle , \\ldots } \\right]^T .\n\\end{equation}\nAny element of $L_2(\\mathbb{R})$ can now be represented in two equivalent ways in terms of the two biorthogonal sets.  Using the Dirac notation and the summation convention we have the following expansion in terms of the original orthonormal set\n\\[\n\\left| x \\right\\rangle  = \\left| {\\phi_n } \\right\\rangle \\left\\langle {{\\phi_n}} \\mathrel{\\left | {\\vphantom {{\\phi_n } x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle ~,\n\\]\nwhich upon acting on by the operator $L$ and using equation \\eqref{Chapter1-107} becomes\n\\[\nL\\left| x \\right\\rangle  = L\\left| {\\phi _n } \\right\\rangle \\left\\langle {{\\phi _n }} \\mathrel{\\left | {\\vphantom {{\\phi _n } x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle  = \\left| {\\xi _n } \\right\\rangle \\left\\langle {{\\phi _n }} \\mathrel{\\left | {\\vphantom {{\\phi _n } x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle ~.\n\\]\nNow using the adjoint of equation \\eqref{Chapter1-109} we write \n\\[\n\\left\\langle {\\phi_n } \\right| = \\left\\langle {\\chi_n } \\right|L ~,\n\\]\nand so for any $x(t)$ we have \n\\[\nL\\left| x \\right\\rangle  = \\left| {\\xi _n } \\right\\rangle \\left\\langle {\\chi _n } \\right|L\\left| x \\right\\rangle ~,\n\\]\nfrom which we arrive at the completeness relation\n\\begin{equation}\n\\label{Chapter1-112}\n{\\left| {\\xi _n } \\right\\rangle \\left\\langle {\\chi _n } \\right|}  = \\mathbf{1} ~,\n\\end{equation}\nwith the associated expansion formula\n\\begin{equation}\n\\label{Chapter1-113}\n\\left| x \\right\\rangle  = \\left\\langle {{\\xi_n }} \\mathrel{\\left | {\\vphantom {{\\xi_n } x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle \\left| {\\chi_n } \\right\\rangle .\n\\end{equation}\nOn the other hand, starting with the identity \n\\[\nL^+ \\left| x \\right\\rangle  = \\left| {\\phi_n } \\right\\rangle \\left\\langle {\\phi_n } \\right|L^+  \\left| x \\right\\rangle ~,\n\\]\nwe arrive at the completeness relation\n\\begin{equation}\n\\label{Chapter1-114}\n{\\left| {\\chi _n } \\right\\rangle \\left\\langle {\\xi _n } \\right|}  = \\mathbf{1} ~,\n\\end{equation}\nwith the associated expansion formula\n\\begin{equation}\n\\label{Chapter1-115}\n\\left| x \\right\\rangle  = \\left\\langle {{\\chi_n }} \\mathrel{\\left | {\\vphantom {{\\chi_n } x}} \\right. \\kern-\\nulldelimiterspace} {x} \\right\\rangle \\left| {\\xi_n } \\right\\rangle .\n\\end{equation}\n\nThus, biorthogonal Riesz bases are more general than orthogonal bases.  We will use biorthogonal bases in the construction of biorthogonal wavelets\\index{biorthogonal!wavelets} of compact support whose implementation does not require filter sequences of even length;  both even and odd lengths will be permitted in this context.  Even more general is the concept of a frame that will be introduced in the next two sections.  It is important to note, however, that both orthogonal and Riesz bases have the property that removing a single element from either set will destroy their completeness;  such bases are known as exact\\index{basis!exact}.  Riesz bases are also exact frames \\cite{Youn80}.  Before we introduce the concept of a frame in an infinite dimensional Hilbert space we will, in the next section, look at the problem of vector representation in a finite dimensional vector space, e.g., $\\mathbb{C}^n$. \n\n\n\n\\section{Frames\\index{frame} in A Finite Dimensional Vector Space}\n\nA vector $\\mathbf{x} \\in \\mathbb{C}^n$ is represented by the column $\\left[ {x_1 , \\ldots ,x_n } \\right]^T $ where $x_k  \\in \\mathbb{C}$, $k = 1, \\ldots ,n$.  The inner product is defined by\n\\[\n\\left\\langle {\\mathbf{z},\\mathbf{x}} \\right\\rangle  \\equiv \\mathbf{z}^ + \\mathbf{x} = \\sum\\limits_{k = 1}^n {\\mathbf{z}^*_{k}  \\mathbf{x}_{_k}} , ~~ \\mathbf{z}, \\mathbf{x} \\in \\mathbb{C}^n~.\n\\]\nConsider a linear mapping (operator) between $\\mathbb{C}^n$ and $\\mathbb{C}^m$, $n \\ge m$, represented by a complex $m \\times n$ matrix $\\mathbf{A}$, i.e., $\\mathbf{A} \\mathbf{x} = \\mathbf{y}$, $\\mathbf{x} \\in \\mathbb{C}^n$, $\\mathbf{y} \\in \\mathbb{C}^m$.  Denoting the columns of $\\mathbf{A}$ by $\\mathbf{c}_k$, $k = 1, \\ldots ,n$, it is easy to see that\n\\begin{equation}\n\\label{Chapter1-116}\n\\mathbf{A} \\mathbf{x} = \\sum\\limits_{k = 1}^n {x_{k} \\mathbf{c}_{k}}~, \n\\end{equation}\nthat is, the range space of the matrix $\\mathbf{A}$, denoted by $\\mathscr{R}(\\mathbf{A}) $ is the space spanned by the columns of the same matrix.  If we assume that $\\mathscr{R}(\\mathbf{A})$ is all of $\\mathbb{C}^m$, $m \\le n$, then there are $m$ independent columns of $\\mathbf{A}$.  The column rank\\index{column rank} of the matrix $\\mathbf{A}$ is the number of independent columns of the matrix, and defines the dimension of the range space $\\mathscr{R}(\\mathbf{A})$.  \n\nThe adjoint map (operator) denoted by $\\mathbf{A}^ +$ is defined between $\\mathbb{C}^m$ and $\\mathbb{C}^n$ and is the $n \\times m$ matrix $\\left( {\\mathbf{A}^ * }\\right)^T $ (the Hermitian conjugate of $\\mathbf{A}$).  The dimension of the range space $\\mathscr{R} (\\mathbf{A}^ +)$ is the column rank of the matrix $\\mathbf{A}^ +$ which is equal to the row rank\\index{row rank} of the matrix $\\mathbf{A}$ \\footnote{To be precise, the row rank of $\\mathbf{A}^*$, which is the same as the row rank of $\\mathbf{A}$.}, which is also equal to the column rank of the same matrix.  The rank $r_{\\mathbf{A}}$ of the matrix $\\mathbf{A}$ is the number of linearly independent columns (or rows) of that matrix.  The following theorem specifies the dimensions of the various range and null spaces \\cite{Stra88}.\n\\newtheorem{MatrixRank}{Theorem}\n\\begin{thm}\nIf a complex $m \\times n$ matrix $\\mathbf{A}$ has rank $r_{\\mathbf{A}}$ then,\n\\begin{itemize}\n\t\\item  $\\mathscr{R} (\\mathbf{A})$ and $\\mathscr{R} (\\mathbf{A}^+)$ have dimension $r_{\\mathbf{A}}$,\n\t\\item  $\\mathscr{N} (\\mathbf{A})$ has dimension $(n-r_{\\mathbf{A}})$ while $\\mathscr{N} (\\mathbf{A}^+)$ has dimension $(m-r_{\\mathbf{A}})$.\n\\end{itemize}\n\\end{thm}\n\nWhen $m=n=r_{\\mathbf{A}}$, the matrix $\\mathbf{A}$ is square and has full rank.  It is, therefore, non singular (has non zero determinant) and invertible.  Consequently, an equation of the form\n\\begin{equation}\n\\label{Chapter1-117}\n\\mathbf{A} \\mathbf{x} = \\mathbf{y}, ~ \\mathbf{x} \\in \\mathbb{C}^m,~ \\mathbf{y} \\in \\mathbb{C}^n, \n\\end{equation}\nhas the unique solution $\\mathbf{x} = \\mathbf{A}^{-1} \\mathbf{y}$ when $m=n=r_{\\mathbf{A}}$.  More generally, a solution exists only if $\\mathbf{y} \\in \\mathscr{R} (\\mathbf{A})$, i.e., if $\\mathbf{y}$ is a linear combination of the columns of $\\mathbf{A}$.  By theorem \\ref{AandAPlusSpaces} we have $\\mathscr{R} (\\mathbf{A}) = (\\mathscr{N} ({\\mathbf{A}}^+))^ \\bot$ and so if $\\mathbf{y} \\in \\mathscr{R} (\\mathbf{A})$ then $\\mathbf{y}$ must be orthogonal to $\\mathscr{N} ({\\mathbf{A}}^+)$.  In other words, the matrix equation \\eqref{Chapter1-117} has a solution if, and only if, $\\left\\langle {\\mathbf{z}, \\mathbf{y}} \\right\\rangle = 0$ for every $\\mathbf{z} \\in {\\mathbb{C}}^n$ that satisfies ${\\mathbf{A}}^+ \\mathbf{z} = \\mathbf{0}$.  \n\nIf a solution to \\eqref{Chapter1-117} exists then it is unique if, and only if, $\\mathscr{N}(\\mathbf{A}) = \\left\\{ \\mathbf{0} \\right\\}$.  Thus, when $m \\le n$ and $r_{\\mathbf{A}} < Min\\left( {m,n} \\right)$ the dimension of the null space $\\mathscr{N} (\\mathbf{A})$ is greater than zero and so if a solution to \\eqref{Chapter1-117} exists, it is not unique.  When multiple solutions exist, a unique solution whose norm is a minimum can be found using the following theorem \\cite{CaMe79, GoVL96}.\n\\newtheorem{Pseudoinverse}{Theorem}\n\\begin{thm}\nLet $\\mathbf{A}$ be a complex $m \\times n$ matrix, $m \\le n$.  Then the unique solution to the equation $\\mathbf{A} \\mathbf{x} = \\mathbf{y}$ whose norm $\\left\\| x \\right\\|$ is the smallest among all the solutions is $\\hat {\\mathbf{x}} = {\\mathbf{A}^+} \\mathbf{u}$ where $\\mathbf{u}$ is any solution to the equation $\\mathbf{A}\\mathbf{A}^+ \\mathbf{u}=\\mathbf{y}$.  In particular, when $r_{\\mathbf{A}}=m \\le n$, the minimum norm solution is\n\\begin{equation}\n\\label{Chapter1-118}\n\\hat {\\mathbf{x}} = {\\mathbf{A}^+}  \\left( {\\mathbf{A} {\\mathbf{A}^ +}} \\right)^{-1} \\mathbf{y} \\equiv {\\mathbf{A}_R ^\\dag} \\mathbf{y},\n\\end{equation}\nwhich also defines the right pseudo inverse\\index{pseudo inverse} $\\mathbf{A}_R ^\\dag$ satisfying\n\\begin{equation}\n\\label{Chapter1-119}\n\\mathbf{A} \\mathbf{A}_R^\\dag   = \\mathbf{1}.\n\\end{equation}\nFurthermore, if $\\mathbf{B}$ is a complex $n \\times m$ matrix, $n \\ge m$, whose rank is $m$, then the equation \n\\begin{equation}\n\\label{Chapter1-120}\n\\mathbf{B} \\mathbf{y} = \\mathbf{x}, ~ \\mathbf{y} \\in \\mathbb{C}^m,~ \\mathbf{x} \\in \\mathbb{C}^n, \n\\end{equation}\nhas no solution.  However, there is a unique vector $\\hat{\\mathbf{y}}$ that minimizes the norm $\\left\\| {\\mathbf{B} \\mathbf{y} - \\mathbf{x}} \\right\\|$ (for $l_2$ norm this is known as the least squares solution), namely, \n\\begin{equation}\n\\label{Chapter1-121}\n\\hat {\\mathbf{y}} = \\left( {\\mathbf{B}^ + \\mathbf{B}} \\right)^{-1} \\mathbf{B}^ + \\mathbf{x} \\equiv \\mathbf{B}_L^\\dag \\mathbf{x}\n\\end{equation}\nwhich also defines the left pseudo inverse $\\mathbf{B}_L ^\\dag$ satisfying\n\\begin{equation}\n\\label{Chapter1-122}\n\\mathbf{B}_L^\\dag \\mathbf{B}  = \\mathbf{1}.\n\\end{equation}\n\\end{thm}\n\n\\noindent Now consider the identity\n\\begin{equation}\n\\label{Chapter1-123}\n\\mathbf{y} = \\mathbf{A} \\mathbf{A}^+ \\left( {\\mathbf{A} \\mathbf{A}^+} \\right)^{-1} \\mathbf{y} = \\mathbf{A} \\mathbf{A}_R^\\dag \\mathbf{y}, ~~~ \\mathbf{y} \\in \\mathbb{C}^m ~,\n\\end{equation}\nor equivalently\n\\begin{equation}\n\\label{Chapter1-124}\ny_{_k}  = \\sum\\limits_{j = 1}^n {\\sum\\limits_{l = 1}^m {\\mathbf{A}_{kj} \\left[ {\\mathbf{A}_R^ \\dag } \\right]_{jl} y_{_l} } }, ~~k = 1, \\ldots ,m.\n\\end{equation}\nWe write\n\\[\n\\sum\\limits_{l = 1}^m {\\left[ {\\mathbf{A}_R^ \\dag} \\right]_{jl} y_{_l} }  = \\sum\\limits_{l = 1}^m {\\left[ {\\left( {\\mathbf{c}_j^ \\dag} \\right)^ *  } \\right]_{l} y_{_l} }  \\equiv \\left\\langle {\\mathbf{c}_j^ \\dag  ,\\mathbf{y}} \\right\\rangle,\n\\]\nwhere\n\\[\n\\left[ {\\left({\\mathbf{c}_j^ \\dag} \\right)} \\right]_l  = \\left[ {\\left( {\\mathbf{A}_R^ \\dag} \\right)^ *} \\right]_{jl}  = \\left[ {\\left( {\\mathbf{A}_R^ \\dag} \\right)^ +} \\right]_{lj}.\n\\]\nThus, $\\mathbf{c}_j^ \\dag$ is the $j$-th column of the $n \\times m$ matrix ${\\left( {\\mathbf{A}_R^ \\dag} \\right)^ +}$,\n\\[\n\\left( {\\mathbf{A}_R^ \\dag} \\right)^ +   \\equiv \\left[ {\\mathbf{c}_1^ \\dag , \\ldots , \\mathbf{c}_n^ \\dag} \\right],\n\\]\nand equation \\eqref{Chapter1-124} can be written as\n\\[\ny_{_k}  = \\sum\\limits_{j = 1}^n {\\mathbf{A}_{kj} \\left\\langle {\\mathbf{c}_j^ \\dag, \\mathbf{y}} \\right\\rangle }, ~ k = 1, \\ldots ,m,\n\\]\nwhich in view of the relation \\eqref{Chapter1-116} becomes\n\\begin{equation}\n\\label{Chapter1-125}\n\\mathbf{y} = \\sum\\limits_{j = 1}^n {\\left\\langle {\\mathbf{c}_j^ \\dag,y} \\right\\rangle \\mathbf{c}_j}, \n\\end{equation}\nwhere $\\mathbf{c}_j$ are the columns of the $m \\times n$ matrix $\\mathbf{A}$\n\\[\n\\mathbf{A} \\equiv \\left[ {\\mathbf{c}_1 , \\ldots , \\mathbf{c}_n} \\right].\n\\]\nEquation \\eqref{Chapter1-125} expresses a reconstruction formula for a vector $\\mathbf{y}$ in terms of a sum over the product of a set of vectors $\\mathbf{c}_k$ (columns of the matrix $\\mathbf{A}$) multiplied by a set of scalar transform coefficients, obtained in the analysis stage by taking inner products of $\\mathbf{y}$ with the columns of the Hermitian conjugate of the right pseudo inverse, and represented by ${\\left\\langle {\\mathbf{c}_k ^ \\dag ,\\mathbf{y}} \\right \\rangle }$.  \n\nA dual reconstruction formula to \\eqref{Chapter1-125} can be obtained starting with the identity\n\\begin{equation}\n\\label{Chapter1-126}\n\\mathbf{y} =  \\left( {\\mathbf{A} \\mathbf{A}^+} \\right)^{-1} (\\mathbf{A} \\mathbf{A}^+)\\mathbf{y} = ({\\mathbf{A}_R^\\dag})^+ {\\mathbf{A}}^+  \\mathbf{y}, ~~~ \\mathbf{y} \\in \\mathbb{C}^m ~,\n\\end{equation}\ninstead of \\eqref{Chapter1-123}.  Now we have\n\\begin{equation}\n\\label{Chapter1-127}\ny_{_k}  = \\sum\\limits_{j = 1}^n {\\sum\\limits_{l = 1}^m { \\left[({\\mathbf{A}_R^ \\dag)^+} \\right]_{kj}  (\\mathbf{A}^+)_{_{jl}}~ y_{_l} } }, ~~k = 1, \\ldots ,m, \n\\end{equation}  \nwhile this time\n\\[\n\\sum\\limits_{l = 1}^m {\\left[ {\\mathbf{A}^ +} \\right]_{jl} y_{_l}}  = \\left\\langle {\\mathbf{c}_j,\\mathbf{y}} \\right\\rangle,\n\\]\nwhere $\\mathbf{c}_j$ are the columns of the matrix $\\mathbf{A}$, and we find\n\\begin{equation}\n\\label{Chapter1-128}\n\\mathbf{y} = \\sum\\limits_{j = 1}^n {\\left\\langle {\\mathbf{c}_j,y} \\right\\rangle \\mathbf{c}_j ^ \\dag}, \n\\end{equation}\nwhere $\\mathbf{c}_j ^ \\dag$ are the columns of the $m \\times n$ matrix $(\\mathbf{A}_R^ \\dag)^ +$.\n\nThe reconstruction formula, however, is only useful when the right pseudo inverse matrix $\\mathbf{A}_R^\\dag$ can be computed in a stable fashion, which is possible only when the matrix $\\mathbf{A} \\mathbf{A}^ +$ has a reasonable condition number\\index{condition number}, that is, the magnitude of the ratio of the largest eigenvalue to the smallest eigenvalue is not very large \\footnote{Note that $\\mathbf{A} \\mathbf{A}^ +$ is Hermitian and positive semi definite and so has real and positive (or possibly zero) eigenvalues \\cite{GoVL96}.  The same holds for $\\mathbf{A}^+ \\mathbf{A}$.}.  But the eigenvalues of $\\mathbf{A} \\mathbf{A}^ +$ are the singular values\\index{singular values} of the matrix $\\mathbf{A}$ \\footnote{Any real or complex $m \\times n$ matrix $\\mathbf{A}$ can be factored as $\\mathbf{A} = \\mathbf{U} \\mathbf{\\Sigma} \\mathbf{V}^+$ where $\\mathbf{\\Sigma}$ is a $m \\times n$ real diagonal matrix of singular values $\\sigma_i \\ge 0$, $i=1, \\ldots ,k$ and $k = {\\rm Min}\\left( {m,n} \\right)$ \\cite{MoSt00}.  When $m \\ne n$, $\\mathbf{\\Sigma}$ is constructed from the singular values by placing them along the main diagonal and then placing zeros in other locations to preserve the $m \\times n$ size of the matrix, which in such cases will inevitably result in some zero columns (when $m < n$), or some zero rows (when $m > n$).  The singular values are the square roots of the intersection of the two sets of eigenvalues of the Hermitian and positive semi-definite matrices $\\mathbf{A} \\mathbf{A}^+$ and $\\mathbf{A}^+ \\mathbf{A}$.  Assuming, without loss of generality, that $m < n$ and defining the square $m \\times m$ diagonal matrix $\\mathbf{\\Lambda}_m  \\equiv {\\rm Diag}\\left[ {\\sigma _1^2 , \\ldots ,\\sigma _m^2 } \\right]$ and the square $n \\times n$ diagonal matrix $\\mathbf{\\Lambda}_n  \\equiv {\\rm Diag}\\left[ {\\sigma _1^2 , \\ldots ,\\sigma _n^2 } \\right]$ we have $\\mathbf{A} \\mathbf{A}^+ \\mathbf{U} = \\mathbf{U} \\mathbf{\\Lambda}$ and $\\mathbf{A}^+ \\mathbf{A} \\mathbf{V} = \\mathbf{V} \\mathbf{\\Lambda}$.  Thus, the eigenvectors of $\\mathbf{A} \\mathbf{A}^+$ are the columns of $\\mathbf{U}$, $\\mathbf{U} = \\left[ {\\mathbf{u}_1 , \\ldots ,\\mathbf{u}_n } \\right]$, while the eigenvectors of $\\mathbf{A}^+ \\mathbf{A}$ are the columns of $\\mathbf{V}$, $\\mathbf{V} = \\left[ {\\mathbf{v}_1 , \\ldots ,\\mathbf{v}_n } \\right]$.  In addition, $\\mathbf{U} \\mathbf{U}^+ = \\mathbf{1}$, $\\mathbf{V} \\mathbf{V}^+ = \\mathbf{1}$.  If the rank $r_{\\mathbf{A}} < m < n$ then $\\sigma _i  > 0$, $1 \\le i \\le r_{\\mathbf{A}}$, and $\\sigma _i=0$, $r_{\\mathbf{A}} < i \\le m$ and we have the representation $\\mathbf{A} = \\sum\\limits_{i = 1}^{r_{\\mathbf{A}}} {\\sigma_i \\mathbf{u}_i \\mathbf{v}_i^ +}$ which can now be used to compute any function of a matrix, namely $f(\\mathbf{A}) = \\sum\\limits_{i = 1}^{r_{\\mathbf{A}}} {f(\\sigma_i) \\mathbf{u}_i \\mathbf{v}_i^ +}$.}.  Denoting the extreme singular values by $\\sigma_{min}$ and $\\sigma_{max}$ we have the following inequalities,              \n\\begin{equation}\n\\label{Chapter1-129}\n\\sigma _{\\min } \\left\\| \\mathbf{y} \\right\\|^2  \\le \\left\\langle {\\mathbf{A}\\mathbf{A}^ + \\mathbf{y},\\mathbf{y}} \\right\\rangle \\le \\sigma _{\\max } \\left\\| \\mathbf{y} \\right\\|^2 , ~~ \\forall \\mathbf{y} \\in \\mathbb{C}^m ~.\n\\end{equation}\nUsing the reconstruction formula \\eqref{Chapter1-128} we have\n\\[\n\\left\\langle {\\mathbf{A} \\mathbf{A}^+ \\mathbf{y}, \\mathbf{y}} \\right\\rangle  = \\left\\langle {\\mathbf{y},\\mathbf{A} \\mathbf{A}^+ \\mathbf{y}} \\right\\rangle  = \\sum\\limits_{j = 1}^n {\\left\\langle {\\mathbf{c}_j ,y} \\right\\rangle \\left\\langle {y, \\mathbf{A} \\mathbf{A}^+ \\mathbf{c}_j^ \\dag} \\right\\rangle } ~.\n\\]\nUsing the definition of the right pseudo inverse we have \\cite{CaMe79}\n\\[\n\\left( {\\mathbf{A} \\mathbf{A}^+ \\mathbf{c}_j^ \\dag} \\right)_k  = \\left( {\\mathbf{A} \\mathbf{A}^+} \\right)_{ki} \\left[ {\\left( {\\mathbf{A}_R^ \\dag} \\right)^ +} \\right]_{ij}  = \\left( {\\mathbf{A} \\mathbf{A}^+} \\right)_{ki} \\left[ {\\left( {\\mathbf{A} \\mathbf{A}^+} \\right)^{-1} } \\right]_{il} \\mathbf{A}_{lj}  = \\mathbf{A}_{kj}~,\n\\]\nwhich is the $j$-th column of $\\mathbf{A}$ and so\n\\[\n\\left( {\\mathbf{A} \\mathbf{A}^+ \\mathbf{c}_j^ \\dag} \\right)  =  \\mathbf{c}_{j}~.\n\\]\nThe inequalities \\eqref{Chapter1-129} then become\n\\begin{equation}\n\\label{Chapter1-130}\n\\sigma _{\\min } \\left\\| \\mathbf{y} \\right\\|^2  \\le \\left\\langle {\\mathbf{A}\\mathbf{A}^ + \\mathbf{y},\\mathbf{y}} \\right\\rangle = \\sum\\limits_{j = 1}^n {\\left| {\\left\\langle {\\mathbf{c}_j , \\mathbf{y}} \\right\\rangle } \\right|^2 } \\le \\sigma _{\\max } \\left\\| \\mathbf{y} \\right\\|^2 , ~~ \\forall \\mathbf{y} \\in \\mathbb{C}^m ~.\n\\end{equation}\n\\noindent These are also known as frame inequalities\\index{frame!inequalities}.  Any set of $n$ vectors $\\left\\{ {\\mathbf{c}_k} \\right\\}$, satisfying the above inequalities is a frame\\index{frame} for $\\mathbb{C}^m$ and the quantities $\\sigma_{min}$ and $\\sigma_{max}$ are the associated frame bounds\\index{frame!bounds}.  When $\\sigma_{min} = \\sigma_{max}$, i.e., the condition number of $\\mathbf{A}$ is $1$, the frame is tight\\index{frame!tight} and it mimics an orthogonal basis even though the frame vectors might be linearly dependent.  If $\\sigma_{min} = \\sigma_{max}=1$ the frame becomes an orthogonal basis.  Given non tight frame vectors $\\mathbf{c}_k$, a tight frame can be constructed by choosing \n\\[\n\\mathbf{c'}_k  \\equiv \\left( {\\mathbf{A} \\mathbf{A}^+} \\right)^{{-1 \\mathord{\\left/ {\\vphantom {-1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\mathbf{c}_k ~,\n\\]\nwhere the square root operation is defined using the Singular Value Decomposition (SVD)\\index{singular values} of the associated matrix.  The matrix $\\mathbf{A} \\mathbf{A}^+$ is the finite dimensional analogue of the basis operator ${T^+}_{\\phi} T_\\phi$ defined in a Hilbert space, \\eqref{Chapter1-101}.  \n\n\n\\section[Frames in $L_2 \\left( \\mathbb{R} \\right)$]{Frames in $L_2 \\left( \\mathbb{R} \\right)$}\n\\label{FramesInInfinite}\n\nIn analogy to the previous section we generalize the concept of a Riesz basis\\index{basis!Riesz} in a Hilbert space by relaxing the exactness property of a Riesz basis (a Riesz basis is an exact frame\\index{frame!exact}) which allows for overcomplete (i.e., linearly dependent) sets that will lead to the concept of frames \\cite{Chri03}.  We begin with the definition of a basis and an unconditional basis\\index{basis!unconditional} in $L_2(\\mathbb{R})$.  \n\\newtheorem{UnconditionalBasis}{Definition}\n\\begin{defn}\n\\label{UnconditionalBasis}\nA set of functions $\\left\\{ {\\phi _n \\left( t \\right)} \\right\\}$, $n \\in \\mathbb{Z} $, whose closure is the entire space $L_2 \\left( \\mathbb{R} \\right)$ is a basis (or a Schauder basis\\index{basis!Schauder}) if every $x \\in  L_2(\\mathbb{R})$ has the representation  \n\\[\nx = \\sum\\limits_{n =  - \\infty }^\\infty  {c_n \\phi _n } ~,\n\\]\nwhere the coefficients $c_n$ are unique and depend on $x$.  Furthermore, if, for this basis, two positive constants $A$ and $B$ exist such that\n\\[\nA\\left\\| \\underline{c} \\right\\|^2  \\le \\left\\| {\\sum\\limits_{n =  - \\infty }^\\infty  {c_n \\phi _n } } \\right\\|^2  \\le B\\left\\| \\underline{c} \\right\\|^2 \n\\]\nfor every $\\underline{c} \\in l_2(\\mathbb{Z})$, then the basis is an unconditional basis.\n\\end{defn}\n\nThe concept of a frame\\index{frame} can be introduced by relaxing the uniqueness of the expansion coefficients in an unconditional basis.\n\\newtheorem{FrameDefinition}{Definition}\n\\begin{defn}\n\\label{FrameDefinition}\nA set of functions $\\left\\{ {\\phi _n \\left( t \\right)} , n \\in \\mathbb{Z} \\right\\}$ is a frame in $L_2 \\left( \\mathbb{R} \\right)$ if positive constants $A$ and $B$, the frame bounds\\index{frame!bounds}, exist such that \n\\begin{equation}\n\\label{Chapter1-131}\nA\\left\\| x \\right\\|^2  \\le \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {\\phi _n ,x} \\right\\rangle } \\right|^2 }  \\le B\\left\\| x \\right\\|^2 ~, \n\\end{equation}\nfor every $x \\in L_2(\\mathbb{R})$. If $A=B$ the frame is tight\\index{frame!tight}.\n\\end{defn}\n\\newtheorem{ExactFrameRiesz}{Theorem}\n\\begin{thm}\n\\label{ExactFrameRiesz}\nReferring to definition \\ref{FrameDefinition} and equation \\eqref{Chapter1-131}, if the frame is exact then it is a Riesz basis.  If $A=B=1$ then the frame is an orthonormal basis \\cite{Youn80, Daub92}.\n\\end{thm}\n\\noindent \n\\noindent Thus, an othonormal basis\\index{basis!orthonormal} is exact and tight with unity frame bounds.  Tight frames that are not orthonormal can be constructed by the union of two orthonormal bases \n\\[\n\\left\\{ {\\xi _n } \\right\\} = \\left\\{ {\\phi _n } \\right\\} \\cup \\left\\{ {\\psi _n } \\right\\}, ~ \\left\\langle {{\\xi _n }} \\mathrel{\\left | {\\vphantom {{\\xi _n } {\\xi _m }}}\n \\right. \\kern-\\nulldelimiterspace} {{\\xi _m }} \\right\\rangle  = \\left\\langle {{\\phi _n }} \\mathrel{\\left | {\\vphantom {{\\phi _n } {\\phi _m }}} \\right. \\kern-\\nulldelimiterspace}\n {{\\phi _m }} \\right\\rangle  = \\delta _{nm} \n\\]\nthat are not orthogonal to each other.  The resulting frame is not orthogonal and for any $x(t)$ we have \n\\[\n\\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {\\xi _n ,x} \\right\\rangle } \\right|^2  = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {\\phi _n ,x} \\right\\rangle } \\right|^2 }  + } \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {\\psi _n ,x} \\right\\rangle } \\right|^2 }  = 2\\left\\| x \\right\\|^2 ~,\n\\]\nwhich clearly shows the corresponding frame bounds to be equal to $2$.  If we choose the second basis to be an orthogonal but unnormalized set, e.g., $\\left\\| \\phi_n \\right\\|=2$, $\\forall ~ n$, then the resulting union is a frame that is neither tight, nor exact.\n\n\nIn analogy to equations \\eqref{Chapter1-97} and \\eqref{Chapter1-98} we define the frame analysis operator $T_\\phi$ and its adjoint ${T^+}_\\phi$, the frame synthesis operator.  Thus, we arrive at the definition of the frame operator\\index{frame!operator}.\n\\newtheorem{FrameOperatorDeinition}{Definition}\n\\begin{defn}\n\\label{FrameOperatorDeinition}\nGiven the frame functions $\\left\\{ {\\phi _n \\left( t \\right)} , n \\in \\mathbb{Z} \\right\\}$ we define the frame operator ${T^+}_\\phi T_\\phi$ by the equation\n\\begin{equation}\n\\label{Chapter1-132}\n{T^+}_\\phi T_\\phi  \\left\\{ {x\\left( t \\right)} \\right\\} = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {\\phi _n ,x} \\right\\rangle \\phi _n \\left( t \\right)}, ~ \\forall ~ x(t) \\in L_2(\\mathbb{R}).\n\\end{equation}\n\\end{defn}\n\\noindent Using the definition of the frame operator and its adjoint, the frame definition equation \\eqref{Chapter1-131} can be rewritten as\n\\begin{equation}\n\\label{Chapter1-133}\nA\\left\\| x \\right\\|^2  \\le \\left\\langle {x,{T^+}_\\phi T_\\phi  x} \\right\\rangle  \\equiv \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {\\phi _n ,x} \\right\\rangle } \\right|^2 }  \\le B\\left\\| x \\right\\|^2 ~.\n\\end{equation}\nThe analysis operator $T_\\phi$ is not unitary since the frame elements are not, in general, orthonormal.  The frame definition of equation \\eqref{Chapter1-133} implies that the frame operator ${T^+}_\\phi T_\\phi$ is bounded and positive definite, and hence invertible.  The operator norms of  ${T^+}_\\phi T_\\phi $ and its inverse are bounded by the frame bounds\\index{frame!bounds}.  Using equation \\eqref{Chapter1-10} we have \n\\begin{equation}\n\\label{Chapter1-134}\n\\left\\| {{T^+}_\\phi T_\\phi  } \\right\\| \\le B, ~~ \\left\\| {\\left( {{T^+}_\\phi T_\\phi  } \\right)^{ - 1} } \\right\\| \\le A^{ - 1} ~.\n\\end{equation}\nThe ratio ${B \\mathord{\\left/ {\\vphantom {B A}} \\right. \\kern-\\nulldelimiterspace} A}$ is known as the frame redundancy ratio\\index{frame!redundancy ratio} and can be used as an approximation to the condition number\\index{condition number} of the operator ${T^+}_\\phi T_\\phi $.  A finite frame ratio that is not too large and not too small is required for stable synthesis or recovery, i.e. reconstruction of a signal from its transform coefficients, as will be seen below in the frame representation equation.  An orthonormal basis $\\hat \\phi _n \\left( t \\right)$ can be constructed for a frame with a reasonable frame ratio, using the square root of the inverse of the frame operator ${T^+}_\\phi T_\\phi $, \n\\begin{equation}\n\\label{Chapter1-135}\n\\hat \\phi _n \\left( t \\right) \\equiv \\left( {{T^+}_\\phi T_\\phi  } \\right)^{ - {1 \\mathord{\\left/\n {\\vphantom {1 2}} \\right. \\kern-\\nulldelimiterspace} 2}} \\phi _n \\left( t \\right) ~.\n\\end{equation}\n\\newtheorem{DualFrameDefinition}{Definition}\n\\begin{defn}\n\\label{DualFrameDefinition}\nGiven a frame $\\left\\{ \\phi_n, ~ n \\in \\mathbb{Z} \\right\\}$ with an invertible frame operator we define the dual frame\\index{frame!dual} elements by\n\\begin{equation}\n\\label{Chapter1-136}\n\\xi _n \\left( t \\right) = \\left( {{T^+}_\\phi T_\\phi  } \\right)^{-1} \\phi _n \\left( t \\right), ~ n \\in \\mathbb{Z}~.\n\\end{equation}\n\\end{defn}\n\\noindent The inverse of this relation shows the linear dependence of the original frame elements (vectors), i.e.,\n\\begin{equation}\n\\label{Chapter1-137}\n\\phi _n \\left( t \\right) = {T^+}_\\phi ~ T_\\phi ~ {\\xi _n (t)}  = \\sum\\limits_{m =  - \\infty }^\\infty  {\\left\\langle {\\phi _m ,\\xi _n } \\right\\rangle \\phi _m \\left( t \\right)} ~.\n\\end{equation}\n\nThe dual frame is a new frame with frame bounds\\index{frame!bounds} $B^{ - 1}$ and $A^{ - 1}$, i.e., it satisfies the defining frame equation \\eqref{Chapter1-131} with the new bounds \\cite{Daub92},\n\\begin{equation}\n\\label{Chapter1-138}\nB^{ - 1} \\left\\| x \\right\\|^2  \\le \\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {\\left\\langle {\\xi _n ,x} \\right\\rangle } \\right|^2 }  \\le A^{ - 1} \\left\\| x \\right\\|^2 ~.\n\\end{equation}\nThe frame representation for functions in $L_2 \\left( \\mathbb{R} \\right)$ is now expressed in the following dual frame reconstruction\\index{frame!dual reconstruction} equations \n\\begin{equation}\n\\label{Chapter1-139}\nx\\left( t \\right) = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {\\phi _n ,x} \\right\\rangle \\xi _n \\left( t \\right)}  = \\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {\\xi _n ,x} \\right\\rangle \\phi _n \\left( t \\right)} ~.\n\\end{equation}\nThe last equation recast in terms of frame operators is simply\n\\begin{equation}\n\\label{Chapter1-140}\nx\\left( t \\right) = {T^+}_\\xi T_\\phi  \\left\\{ {x\\left( t \\right)} \\right\\} = {T^+}_\\phi T_\\xi  \\left\\{ {x\\left( t \\right)} \\right\\} ~.\n\\end{equation}\nThis equation is valid for all $x\\left( t \\right) \\in L_2 \\left( \\mathbb{R} \\right)$, and therefore\n\\begin{equation}\n\\label{Chapter1-141}\n{T^+}_\\phi T_\\xi  = {T^+}_\\xi T_\\phi = \\mathbf{1}\n\\end{equation}\nThe dual reconstruction equations \\eqref{Chapter1-139} are the solutions to two dual minimization problems as stated in the following theorem \\cite{Daub92}.  \n\\newtheorem{DualMinimumNormSolutions}{Theorem}\n\\begin{thm}\n\\label{DualMinimumNormSolutions}\nThe coefficients $c_n  \\in l_2 (\\mathbb{Z})$ in the expansion \n\\[\nx\\left( t \\right) = \\sum\\limits_{n =  - \\infty }^\\infty  {c_n \\phi _n \\left( t \\right)}\n\\]\nthat minimize the square of the norm of the difference between the left and the right hand sides, are not unique.  A minimum norm solution, i.e., a solution for which the quantity $\\sum\\limits_{n =  - \\infty }^\\infty  {\\left| {c_n } \\right|^2 }$ is a minimum among the set of all possible solutions, is given by $c_n  = \\left\\langle {\\xi _n ,x} \\right\\rangle$. Similarly, the coefficients $c'_n  \\in l_2 (\\mathbb{Z})$ that minimize the square of the norm of the difference between the two sides of the equation\n\\[\nx\\left( t \\right) = \\sum\\limits_{n =  - \\infty }^\\infty  {c'_n \\xi_n \\left( t \\right)} \n\\]\nand have minimum norm is given by $c'_n  = \\left\\langle {\\phi _n ,x} \\right\\rangle$.\n\\end{thm}\nThe reconstruction of a function from its coefficients via equations \\eqref{Chapter1-139} must be unique, i.e., the equality $\\left\\langle {\\phi _n ,x_1 } \\right\\rangle  = \\left\\langle {\\phi _n ,x_2 } \\right\\rangle$ must imply that the two underlying functions are the same, $x_1(t) = x_2(t)$ where the equality is understood to mean $\\left\\| {x_1  - x_2 } \\right\\| = 0$.  Equivalently, we must have\n\\begin{equation}\n\\label{Chapter1-142}\n\\left\\langle {\\phi _n ,e} \\right\\rangle  = 0 \\Leftrightarrow e\\left( t \\right) = 0 ~.\n\\end{equation}\nThe frame inequalities \\eqref{Chapter1-131} and \\eqref{Chapter1-138} ensure that the above uniqueness condition is satisfied \\cite{Daub92}.  \n\\newtheorem{DualFrameReconstruction}{Theorem}\n\\begin{thm}\n\\label{DualFrameReconstruction}\nA function $x(t)$ in $L_2(\\mathbb{R})$ can be uniquely reconstructed from the coefficients $\\left\\langle {\\phi _n ,x} \\right\\rangle$ or $\\left\\langle {\\xi _n ,x} \\right\\rangle$ if, and only if, the functions $\\phi_n(t)$ and $\\xi_n(t)$ form dual frames.\n\\end{thm}\n\n\nThe following theorem shows the close connection between a frame and a reproducing kernel\\index{reproducing kernel} in a reproducing kernel Hilbert space\\index{reproducing kernel!Hilbert space}   \\cite{Daub92, BeAg04}.\n\\newtheorem{DualFrameKernel}{Theorem}\n\\begin{thm}\n\\label{DualFrameKernel}\nA Hilbert space with a frame and frame bounds that allow the construction of dual frames $\\left\\{ {\\phi _n ,\\xi _n } \\right\\}$, admits a unique reproducing kernel that has the following representation\n\\begin{equation}\n\\label{Chapter1-143}\nK\\left( {t,t'} \\right) = \\sum\\limits_{n =-\\infty }^\\infty {\\xi_n^* \\left(t \\right)\\phi_n \\left(t' \\right)} ~,\n\\end{equation}\nand that is self adjoint, i.e., $K\\left( {t,t'} \\right) = K^ *  \\left( {t',t} \\right)$.\n\\end{thm}\n\\noindent Equation \\eqref{Chapter1-41} is a special case of this result applied to an orthonormal basis.\n\n\n\\section[Dual Frames]{Dual Frame Construction Algorithm}\n\nThe dual frame\\index{frame!dual} reconstruction formulae expressed in equation \\eqref{Chapter1-139} can only be used when the dual frame elements are known.  If only the frame elements are available then an iterative scheme can be used to approximate the dual frame elements.  The approximation is useful only when the frame ratio ${B \\mathord{\\left/ {\\vphantom {B A}} \\right. \\kern-\\nulldelimiterspace} A}$ is not too much larger than $1$.\n\nUsing the definition of the dual frame in equation \\eqref{Chapter1-136} the reconstruction formula \\eqref{Chapter1-139} can be written as\n\\[\nx\\left( t \\right) = \\sum\\limits_{n=-\\infty }^\\infty {\\left\\langle {\\phi_n ,x} \\right\\rangle} \\left( {{T^+}_\\phi T_\\phi} \\right)^{-1} \\phi_n \\left( t \\right)\n\\]\nwhich is equivalent to \n\\begin{equation}\n\\label{Chapter1-144}\n\\left( {1 - \\frac{2}{{A + B}}{T^+}_\\phi  T_\\phi  } \\right) x\\left( t \\right) = x\\left( t \\right) - \\frac{2}{{A + B}}\\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {\\phi _n ,x} \\right\\rangle \\phi _n \\left( t \\right)} ~.\n\\end{equation}\nIt can be shown that \\cite{Daub92}          \n\\begin{equation}\n\\label{Chapter1-145}\n\\left\\| {1 - \\frac{2}{{A + B}}{T^+}_\\phi T_\\phi} \\right\\| \\le \\frac{{B - A}}{{B + A}} < 1 ~,\n\\end{equation}\nand so according to theorem \\ref{NeumannExpansion} and equation \\eqref{Chapter1-13} the operator \n\\[\n\\left({1-\\left({1-\\frac{2}{{A + B}}{T^+}_\\phi T_\\phi} \\right)} \\right)^{-1} = \\frac{{A + B}}{2}\\left({{T^+}_\\phi T_\\phi} \\right)^{-1} \n\\]\nhas a convergent Neumann expansion\\index{Neumann expansion}.  Thus,\n\\begin{equation}\n\\label{Chapter1-146}\n\\left( {{T^+}_\\phi T_\\phi} \\right)^{-1}  = \\frac{2}{{A + B}}\\sum\\limits_{k = 0}^\\infty  {\\left( {1 - \\frac{2}{{A + B}}{T^+}_\\phi T_\\phi} \\right)} ^k ~.\n\\end{equation}\nAn approximation to the dual frame vectors is obtained by using the first $N$ terms of the Neumann series in equation \\eqref{Chapter1-136}.  Thus, \n\\begin{equation}\n\\label{Chapter1-147}\n\\xi _{_{nN}} \\left( t \\right) \\equiv \\frac{2}{{A + B}}\\sum\\limits_{k = 0}^{N-1} {\\left( {1 - \\frac{2}{{A + B}}{T^+}_\\phi  T_\\phi  } \\right)} ^k \\phi _n \\left( t \\right) ~.\n\\end{equation}\nThe approximated dual frame when used in equation \\eqref{Chapter1-140} gives the approximate reconstructed signal.  The norm of the error in the reconstruction can be shown to decay exponentially,\n\\begin{equation}\n\\label{Chapter1-148}\n\\left\\| {x - \\sum\\limits_{n =  - \\infty }^\\infty  {\\left\\langle {\\phi _n ,x} \\right\\rangle \\xi _{_{nN}} } } \\right\\| \\le \\left( {\\frac{{B - A}}{{B + A}}} \\right)^{N} \\left\\| x \\right\\| ~.\n\\end{equation}\n\nFrames provide a powerful method of signal representation because of their redundancy.  Among important advantages of a redundant basis are better approximations of the continuous wavelet transform coefficients,  robustness to quantization effects and fewer restrictions in the choice of analyzing basis functions which makes it easier to design functions matching specific signal characteristics.  In addition, a signal can be successfully reconstructed even when some coefficients are missing.  However, frames can be difficult to use for signal reconstruction since the reconstruction formula requires the computation of the dual frame elements, which as has been noted above, is a cumbersome and approximate process.  Orthonormal basis functions (tight frames with bounds $A=B=1$), on the other hand, are far easier to deal with.  For wavelets of compact support we will study both orthonormal and biorthogonal basis functions, since as we shall see later, the transform coefficients can be calculated using finite impulse response (FIR) filters, and without the need to compute any of the basis functions.\n\nLocalization of signals both in time and frequency, or in time and scale are major motivations for studying the windowed Fourier transform and the continuous wavelet transform:  both are linear maps between $L_2 \\left( \\mathbb{R} \\right)$ and $L_2 \\left( {\\mathbb{R}}^2 \\right)$ and both transforms use two continuous parameters: time shift and frequency for the windowed Fourier transform, and time shift and scale for the continuous wavelet transform.  The property that is common to both is that the transform coefficients for a function $x(t)$ are obtained by taking the inner product between $x(t)$ and a family of analyzing functions that are constructed from the same mother window, in the case of the windowed Fourier transform, and the same mother wavelet, in the case of the continuous wavelet transform.  \n\nThe process of building frames and orthonormal bases in either case is based on discretization of the continuous variables on which the analyzing functions depend.  This results in analyzing functions that carry two integer indices: one for time translation and one for frequency, in the case of the windowed Fourier transform, and one for time translation and one for scale, in the continuous wavelet transform case.  In either case we may represent the basis functions by $\\xi _{mn} \\left( t \\right)$ and the corresponding inner products by $c_{mn}$ with the following analysis and synthesis (reconstruction) equations  \n\\begin{equation}\n\\label{Chapter1-149}\n~c_{mn}  = \\left\\langle {\\xi _{mn} ,x} \\right\\rangle ,~x\\left( t \\right) = \\sum\\limits_{n =  - \\infty }^\\infty  {c_{mn} } \\xi _{mn} \\left( t \\right),\n\\end{equation}\nand, in the Dirac notation and summation over repeated indices, the completeness relation\n\\begin{equation}\n\\label{Chapter1-150}\n\\left| {\\xi _{mn} } \\right\\rangle \\left\\langle {\\xi _{mn} } \\right| = \\mathbf{1} ~.\n\\end{equation}\nIf the associated frames are actually orthonormal bases then we will have the additional orthonormality relations\n\\begin{equation}\n\\label{Chapter1-151}\n\\left\\langle {{\\xi _{mn} }}\n \\mathrel{\\left | {\\vphantom {{\\xi _{mn} } {\\xi _{lk} }}}\n \\right. \\kern-\\nulldelimiterspace}\n {{\\xi _{lk} }} \\right\\rangle  = \\delta _{ml} \\delta _{nk}  ~.\n\\end{equation}\n\n\n%\\clearpage\n\\section{Exercises}\n\n\n\n\\paragraph{1}\n\nConsider the set of functions $\\left\\{ {1,t,t^2 , \\ldots ,t^{N - 1} } \\right\\}$ defined on the closed interval $\\left[ { - 1,1} \\right]$.  The members of this set are $N$ linearly independent functions.  Using the inner product $\\left\\langle {f,g} \\right\\rangle  \\equiv \\int\\limits_{ - 1}^1 {f\\left( t \\right)} g\\left( t \\right)dt$ and the Gram-Schmidt procedure find a set of $N$ orthonormal polynomials (the latter are known as the Legendre polynomials).\n\n\n\\paragraph{2}\n\nRepeat the last problem using the same set of functions, but this time use the following inner product\n$\\left\\langle {f,g} \\right\\rangle  \\equiv \\int\\limits_{ - 1}^1 {f\\left( t \\right)} \\left( {1 - t^2 } \\right)^{ - \\frac{1}{2}} g\\left( t \\right)dt$.  The resulting orthonormal functions are known as the Chebyshev polynomials $T_0 \\left( t \\right) \\equiv \\pi ^{ - \\frac{1}{2}}$, and $T_n \\left( t \\right) \\equiv \\left( {\\frac{\\pi }{2}} \\right)^{ - \\frac{1}{2}} \\cos \\left[ {n\\cos ^{ - 1} \\left( t \\right)} \\right]$, $n = 1,2, \\ldots$.\n\n\\paragraph{3}\n\nProve equation \\eqref{Chapter1-9}.\n\n\n\n\\paragraph{4}\n\nProve the completeness relation \\eqref{Chapter1-114}.\n\n\n\n\\paragraph{5}\n\nConsider the function defined to be $e^{ - \\left| n \\right|}$ on intervals $[n, n+1]$, $n \\in \\mathbb{Z}$.  Show that this function is square integrable and is in the subspace $\\mathscr{V}_0$.\n\n\n\\paragraph{6}\n\nConsider a continuous time function $f\\left( t \\right) = \\sum\\limits_{n =  - \\infty }^\\infty  {h\\left[ n \\right]} \\phi \\left( {t - n} \\right)$.  Calculate the Fourier transform of the function expressing your result in terms of the Fourier transforms $H\\left( \\omega  \\right)$ and $\\Phi \\left( \\omega  \\right)$.\n\n\n\\paragraph{7}\n\nProve the result \n\\[\n\\int\\limits_{ - \\infty }^\\infty  {\\frac{{\\sin \\left( {a\\pi t} \\right)}}{{a\\pi t}}e^{ - i\\omega t} dt = \\left\\{ \\begin{array}{l}\n {1 \\mathord{\\left/ {\\vphantom {1 a}} \\right. \\kern-\\nulldelimiterspace} a}, - a\\pi  < \\omega  < a\\pi  \\\\ \n 0, ~~~~~~~ {\\rm       otherwise} ~, \\\\ \n \\end{array} \\right.} \n\\]\nby calculating the inverse Fourier transform of the right hand side.  Use this result to find the Fourier transform of $p_k\\left(t \\right)$ as defined in equations \\eqref{Chapter1-92} and \\eqref{Chapter1-89} \\cite{MoSt00}.\n\n\\paragraph{8}\n\nGiven a signal $s\\left( t \\right)$ whose band is limited to $\\left[ { - B, + B} \\right]$, we use the completeness of the functions $p_k \\left( t \\right)$ (see the last problem and equations \\eqref{Chapter1-92} and \\eqref{Chapter1-89}) to write $s\\left( t \\right) = \\sum\\limits_{k =  - \\infty }^\\infty  {c_k p_k \\left( t \\right)}$.  Using the orthonormality of the functions $\\left\\{ {p_k \\left( t \\right)} \\right\\}$ we find the coefficients $c_k  = \\int\\limits_{ - \\infty }^\\infty  {s\\left( t \\right)p_k \\left( t \\right)} dt$.  Use theorem \\ref{FTParseval} and the Fourier transform result of the previous problem to calculate this coefficient.  This is a proof of the sampling theorem\\index{sampling theorem} (theorem \\ref{SamplingTheorem}).\n\n\\paragraph{9}\nUsing the reproducing kernel\\index{reproducing kernel} function of equation \\eqref{Chapter1-85}, show that the kernel is a projection operator from $L_2 \\left( \\mathbb{R} \\right)$ into $L_2^\\Omega  \\left( \\mathbb{R} \\right)$.\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n%\n%                                     END OF CHAPTER 1\n%\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\n\n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n%\n%                                        CHAPTER 2\n%\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\n";

Some properties

Head/@{gould,najmi}

StringLength/@{gould,najmi}

gould//run

najmi//run

In general, the larger and more elaborate a LATEX input string is, the more likely run will fail

How can run be modified so that it always produces the “correct” notebook?

Basic strategy

Create a table of string patterns, each of which corresponds to a LATEX construct

Partition the input string into substrings, each of which matches a particular pattern table entry

run will perform flawlessly on some pattern types; for others it must be modified

Take all the Notebook objects produced by run (or its modification) and extract the relevant cells

Take the extracted cells and form the appropriate Notebook output

Pattern table generation; ptab

Curly bracket fidelity; curlyQ

Curly brackets play a fundamental role in LATEX syntax

Numerical codes :

"{}"//tcc

We demand that all extracted substrings exhibit “curly bracket fidelity” i.e. give True when operated upon by curlyQ:

curlyQ[var_String]:=Module[{q1,q2,q3},
q1=Select[var//tcc,Or[#==123,#==125]&]/.{123->1,125->-1};
q2=(q1/.List->Plus)==0;
q3=And@@((#≥0)&/@Drop[FoldList[Plus,0,q1],1]);
And[q2,q3]]

Grid[{#,#//curlyQ}&/@{"{ab}","}a{b}","{{a{{bc}d}}","{}{a{{bc}d}}"},Frame->All,Alignment->Right]

curlyQ/@{gould,najmi}

Pattern examples

StringExpression["\\begin{equation}",Shortest[s__],"\\end{equation}"]

StringExpression["\\section",Condition[Shortest[s__],curlyQ[s]],"\\end{section}"]

Construction details; ptab

The following generates a variety of important sectioning patterns.

s123[arg_String]:="\\"~~arg~~"{"~~(Shortest[s__]/;curlyQ[s])~~"}"

s91[arg_String]:="\\"~~arg~~"["~~Shortest[__]~~"]{"~~(Shortest[s__]/;curlyQ[s])~~"}"

pattgen12391[arg_List]:=Module[{q1,q2},q1=ToString/@arg;q2={s123[#],s91[#]}&/@q1//Flatten;Grid[{q2//Length//Range,q2}//Transpose,Frame->All,Alignment->Left]]

{chapter,section,subsection,subsubsection,footnote,index,label,ref,eqref};

q1=pattgen12391[%];

StringCases[#,"\\begin{"~~Shortest[s__]~~"}"->s]&/@Flatten[{najmi,gould}]//Flatten//Union;

%/."document"->{}//Flatten;

("\\begin{"~~#~~"}"~~ Shortest[s__]~~"\\end{"~~#~~"}")&/@%;

Grid[Transpose[{%//Length//Range,%}],Frame->All,Alignment->Left];

q1;

{q1[[1]]//Transpose//Last,%%[[1]]//Transpose//Last}//Flatten;

q1=Grid[Transpose[{%//Length//Range,%}],Frame->All,Alignment->Left];

"\\["~~Shortest[s__] ~~"\\]";

{q1[[1]]//Transpose//Last,%}//Flatten

{title,author,date };

s123/@ToString/@%

{q1[[1]]//Transpose//Last,%}//Flatten

ptab=Grid[Transpose[{%//Length//Range,%}],Frame->All,Alignment->Left]

Partitioning the input string

Analogous to DNA sequencing

Main function: partition

proc[arg_List]:=Module[{q1,q2},
q1=Transpose[{Drop[arg,-1],Drop[arg,1]}];
q2=q1/.{a_[b_Integer,c_],u_[v_Integer,w_]}->{a[b,c],u[v,w],c<v};
q1=Cases[q2/.{a_,b_,False}->Rule[b,{}],_Rule,{0,∞}];
arg/.q1//Flatten]

partition[arg_String,ptab_Grid]:=Module[{q1,q2},
(*Label & locate all patterns wrt ptab*) 
q1={#,StringPosition[arg,ptab[[1,#,2]]]}&/@Range[ptab[[1]]//Length];
(*Use triple integer representations*)
q2=q1/.{a_Integer,b_List}:> a/@b;
q1=q2/.a_[{b_,c_}]->a[b,c]//Flatten;
(*Order by initial string position*)
q2=Sort[q1/.a_[b_Integer,c_]->b[c,a]]/.b_Integer[c_,a_]->a[b,c];(*Delete nonconforming elements*)
q1=FixedPoint[proc,q2];
(*Insert junk DNA segments in form 0[m,n]*)
q2=Transpose[{Drop[q1,-1],Drop[q1,1]}]/.{a_[b_,c_],u_[v_,w_]}->{a[b,c],0[c+1,v-1],u[v,w]};
(*Eliminate duplicates & flatten*)
q1={(q2/.{a_,b_,c_Integer[u__]}->{a,b}),Last[Last[q2]]}//Flatten;
(*Take care of endpoints*)
q2=q1/.{a_[b_,c_],u__}->{0[1,b-1],a[b,c],u} ;
q1=q2/.{u__,a_[b_,c_]}->{u,a[b,c],0[c+1,arg//StringLength]};
q2=q1/.(0[a_,b_]/;a>b)->{}//Flatten;
(*Delete nonconforming elements*)
FixedPoint[proc,q2]]

partition[gould,ptab]

%//Length

partition[najmi,ptab]

(%/.a_[_Integer,_]->a//Tally)/.{a_Integer,b_}->{b,a}//Sort//Reverse;

Grid[%/.{a_,b_Integer}:>{a,b,If[b== 0,{},ptab[[1,b,2]]]},Frame->All,Alignment->Left]

Mapping “genes” (outputs of partition) to cells

A function must be written for each line of ptab

Mappings of "sectioning" patterns; chapter, section, subsection

cases[a_,b_]:=Cases[a,b,{0,∞}]

rules[chapter][arg_String,ptab_Grid]:=Module[{r0,r1,r2,r3,r4},
r2=r4=partition[arg,ptab];
r0=r1=cases[r2,#]&/@{1[__],2[__]}//Flatten;
r2=r1/.a_Integer[b_,c_]:>StringTake[arg,{b,c}];
(*Remove "clutter"*)
r3=Rule[#,""]&/@ptab[[1,{11,12,13,14},2]];
(*Previous step obviously requires expected entries in lines 11-14 of ptab*)
r1=StringReplace[#,r3]&/@r2;
r2=StringCases[#,___~~"{"~~(Shortest[s__]/;curlyQ[s])~~"}"~~EndOfString->s]&/@r1//Flatten;
r1=(#[[1]]&/@(run/@r2)//Flatten)/."Text"->"Section";
{r4,Transpose[{r0,r1}]/.{a_,b_Cell}->Rule[a,b]}]

rules[section][arg_String,ptab_Grid]:=Module[{r0,r1,r2,r3,r4},
r2=r4=partition[arg,ptab];
r0=r1=cases[r2,#]&/@{3[__],4[__]}//Flatten;
r2=r1/.a_Integer[b_,c_]:>StringTake[arg,{b,c}];
(*Remove clutter*)
r3=Rule[#,""]&/@ptab[[1,{11,12,13,14},2]];
r1=StringReplace[#,r3]&/@r2;
r2=StringCases[#,___~~"{"~~(Shortest[s__]/;curlyQ[s])~~"}"~~EndOfString->s]&/@r1//Flatten;
r1=(#[[1]]&/@(run/@r2)//Flatten)/."Text"->"Subsection";
{r4,Transpose[{r0,r1}]/.{a_,b_Cell}->Rule[a,b]}]

rules[subsection][arg_String,ptab_Grid]:=Module[{r0,r1,r2,r3,r4},
r2=r4=partition[arg,ptab];
r0=r1=cases[r2,#]&/@{5[__],6[__]}//Flatten;
r2=r1/.a_Integer[b_,c_]:>StringTake[arg,{b,c}];
(*Remove clutter*)
r3=Rule[#,""]&/@ptab[[1,{11,12,13,14},2]];
r1=StringReplace[#,r3]&/@r2;
r2=StringCases[#,___~~"{"~~(Shortest[s__]/;curlyQ[s])~~"}"~~EndOfString->s]&/@r1//Flatten;
r1=(#[[1]]&/@(run/@r2)//Flatten)/."Text"->"Subsubsection";
{r4,Transpose[{r0,r1}]/.{a_,b_Cell}->Rule[a,b]}]

rules[#][gould,ptab][[2]]&/@{chapter,section,subsection}//Flatten//TableForm

Mappings of "equation display" patterns; equation, slashbrack

rules[equation][arg_String,ptab_Grid]:=Module[{r0,r1,r2,r3,r4},
r2=r4=partition[arg,ptab];
(*Be sure line 27 of ptab is appropriate*)
r0=r1=cases[r2,27[__]] ;
(*Do not forget to define paritosh[ ] *)
r2=r1/.a_Integer[b_,c_]:>paritosh[StringTake[arg,{b,c}]];
r1=run/@r2;
r2=r1/.$Failed->Cell["$Failed","DisplayFormulaNumbered"] ;
r1=r2/.Notebook[a_,___]->a;
r2=r1/."NumberedEquation"->"DisplayFormulaNumbered";
r1=r2/.Cell[a__]->Cell[a, TextAlignment->-0.5];
{r4,Transpose[{r0,r1}]/.{a_Integer[b_,c_],u_}->Rule[a[b,c],u]}]

rules[slashbrack][arg_String,ptab_Grid]:=Module[{r0,r1,r2,r3,r4},
r2=r4=partition[arg,ptab];
(*Be sure line 39 of ptab is appropriate*)
r0=r1=cases[r2,39[__]] ;
(*Do not forget to define paritosh[ ] *)
r2=r1/.a_Integer[b_,c_]:>paritosh[StringTake[arg,{b,c}]];
r1=run/@r2;
r2=r1/.$Failed->Cell["$Failed","DisplayFormulaNumbered"] ;
r1=r2/.Notebook[a_,___]->a;
r2=r1/."NumberedEquation"->"DisplayFormulaNumbered";
r1=r2/.Cell[a__]->Cell[a, TextAlignment->-0.5];
{r4,Transpose[{r0,r1}]/.{a_Integer[b_,c_],u_}->Rule[a[b,c],u]}]

rules[#][najmi,ptab][[2]]&/@{equation,slashbrack}//Flatten

cases[%,_Cell]//Flatten//Notebook//NotebookPut

Forming cells into notebooks

Definition of cells2notebook

cells2notebook[arg_String,ptab_Grid,keywords_List]:=Module[{q1,q2},
q1=rules[#][arg,ptab]&/@keywords//Transpose ;
q2=q1/.{a_,b_}:>{a[[1]],b//Flatten};
q1=(q2/.{a_,b_}:>(a/.b));
q2=Rule[#,Cell[ToString[#],"Text"]]&/@Select[q1,IntegerQ[Head[#]]&];
q1/.q2//Flatten//Notebook]

Some results

Application to najmi

cells2notebook[najmi,ptab,{chapter,section,subsection,equation,slashbrack}]//Timing

%//Last//NotebookPut

Application to gould

cells2notebook[gould,ptab,{chapter,section,subsection,equation,slashbrack}]//Timing

%//Last//NotebookPut

Comments & conclusions

A general pattern-matching approach for translating LATEX code into Mathematica notebooks has been presented

Small improvements in run would be very helpful

This notebook is structured with a view to enable others to reproduce and extend the results presented here