A Practical Guide to the Propagation and Analysis
of Experimental Error

Anthony J. Muscat
Department of Chemical and Environmental Engineering
University of Arizona
Tucson, AZ 85721

INTRODUCTION

Quantifying uncertainty in a set of measurements or data is an important part of the process of experimentation. Uncertainty arises even in well designed and executed experiments due to variations in the apparatus and conditions of the experiment from run to run. Statistical inference is a tool for drawing conclusions from data that contain fluctuations and are consequently not perfect. Statistics also provides information that can be used to determine the cost of improving the certainty of the conclusions drawn from a data set.

This hyperlinked Mathematica 4 document presents a practical guide to error analysis or more properly statistical inference. The intent is to present the equations needed to compute common statistical parameters with enough of the underlying theory to skillfully apply statistical inference to data obtained in a typical university laboratory course. The examples given to illustrate use of the equations are taken from an undergraduate semiconductor processing laboratory to obtain the etching rate of silicon dioxide films in aqueous fluoride-bearing solutions. Section 1 examines confidence intervals and describes how to compute them to assess the uncertainty in the thickness of a silicon dioxide film before etching. Section 2 builds on the confidence interval presentation to test the significance of statements made about the film thickness data. Section 3 shows how to combine uncertainty when subtracting film thickness measurements to compute the amount etched and when dividing the amount etched by the time to compute an overall etching rate. Section 4 discusses fitting a linear model to data and shows how to quantify the uncertainty in the slope and intercept of the best fit line to obtain the etching rate and confidence limits from a set of etching measurements as a function of time.

Basic probability laws and random sampling concepts underlying the error analysis equations used are touched on very briefly in this document and in the accompanying notes at the end of the text. The interested reader is referred to the annotated references for resources that inspired this document and that provide a broader and more complete understanding of statistical inference. The best resources the author found for learning about statistical inference and how to apply it are "Introduction to the Practice of Statistics" written by Moore and McCabe and the interactive software program ActivStats written by Velleman. Velleman also is the creator of a software program called DataDesk that is excellent for multivariate data analysis.

Some comments on using features in Mathematica 4 are appropriate. If this is a live notebook running Mathematica 4, evaluate it by selecting Kernel:Evaluation:Evaluate Notebook from the top drop down menu. Mathematica works by evaluating commands. The commands are shown on the input lines below and are denoted by In followed by a line number in square brackets. The output resulting from that input is shown on the subsequent line and is denoted by Out followed by a line number in square brackets. The hyperlinks or blue underlined text are linked to material in other parts of this document or a URL if a web address. To return from a hyperlink jump in this document use the Go Back command in the Find menu of Mathematica or click on the 3 triangle, if the toolbar is showing. There are several places in the notebook called "Try Its" in which you the reader are encouraged to change parameters and explore what happens. To evaluate commands after editing them, hold down the Shift key while simultaneously pressing the Enter key. This action sends the command to the Mathematica kernel, which is the computational engine of Mathematica. To learn more about Mathematica and the statistics capabilities see books by Blachman and Williams, Ruskeepää, and Wolfram. 

Plans to update this work include adding a section on design of experiments and creating more examples and "Try Its". The most current version can be found at the following web address:
http://www.che.arizona.edu/Directory/Faculty/Muscat/index.html.
Please send comments on this module to the author at muscat@erc . arizona . edu.

version: August 27, 2000

CONTENTS

INTRODUCTION

SYMBOLS AND ABBREVIATIONS

1	FUNDAMENTALS OF CONFIDENCE INTERVALS
		TRY IT: Histogram and Mean of a Set of 14 Experiments
		TRY IT: Comparison of Standard Normal and t Distributions
		Example 1: Mean/Confidence Interval for (Subscript[SiO, 2]) Film Thickness Measurements
		TRY IT: Number of Measurements to Achieve a Specific Uncertainty

2	HYPOTHESIS TESTING
		Example 2: Significance of a Mean (Subscript[SiO, 2]) Film Thickness

3	PROPAGATION OF RANDOM ERROR
	2.1 Uncertainty in a Value Computed by Addition and Subtraction
		Example 3: Mean and Confidence Interval for (Subscript[SiO, 2]) Film Thickness
		Etched in an Aqueous HF/methanol Solution
	2.2 Uncertainty in a Value Computed by Multiplication and Division
		Example 4: Mean and Confidence Interval for Nominal (Subscript[SiO, 2])
		Etching Rate—Combining Different Types of Uncertainty

4	LEAST-SQUARES FIT TO LINEAR DATA
		Example 5: Calculation of the Slope (Rate) and Confidence Interval for
		(Subscript[SiO, 2]) Etching in an HF/water Solution

NOTES

REFERENCES

Symbols and Abbreviations

Symbols
b	intercept of line
m	slope of line
N	number of measurements in a data set or sample
NM	number of measurements for computations and simulations
 p	upper α/2 critical value
P	probability
s	sample standard deviation 
(Subscript[s, c])	most probable uncertainty
(Subscript[S, b])	standard deviation of intercept of line
(Subscript[S, m])	standard deviation of slope of line
(Subscript[S, y])	standard deviation of response (y)
t	mean difference standardized with sample standard deviation
t*	upper α/2 critical value for the t(N-1) distribution
x	value of a measurement
\(TraditionalForm\`x\&_\)	sample mean
w	width of the confidence interval or one-half the accuracy
z	mean difference standardized with population standard deviation

α	1 - decimal equivalent of confidence level 
α/2	critical value for distribution
μ	true or population mean
σ	population standard deviation
(Subscript[σ, m])	standard deviation of the mean and equal to σ/(Sqrt[N])

Subscript Abbreviations
i	measurement number
m	mean
est	estimate

<< Statistics`NormalDistribution`; 
<< Graphics`Graphics`;
<< Graphics`MultipleListPlot`;
<< Statistics`ConfidenceIntervals`;<< Statistics`HypothesisTests`;<< Statistics`LinearRegression`;

SECTION 1
FUNDAMENTALS OF CONFIDENCE INTERVALS

One objective of doing an experiment and collecting data is to draw a conclusion. A conclusion may be a statement (for example, one treatment works better than another) or a value of a parameter (for example, an estimate of the rate of a chemical reaction). Chances are though that if an experiment is repeated several times, the result will not be the same each time but will instead vary. Variations can often be reduced by thoughtful experimentation but not completely eliminated. One approach to improve the precision and arrive at a trustworthy result is to repeat the experiment several times under the same conditions and take an average. If a very large number of repetitions were made, then the experimental average should be close to the true result. This approach requires time and resources to make the large number of measurements needed to be close to the true result, which may be very costly and it turns out unnecessary in practice if principles of statistical inference are used.

Consider an experimental study in which 14 measurements are made.  The 14 values from one particular sample are given the name SampleDist1 and are shown below in curly brackets. The 14 measurements form a sample, since they are a subset of all possible measurements that could have been obtained from the experiment. The number 14 is not special. It was chosen to be representative of the number of repetitions that might be done in a typical laboratory experiment. The results of a simulation of 14 experimental measurements and a histogram of the results are shown in Figure 1. A histogram displays the frequency or the number of measurements in a given interval as a function of the measured value. The average or mean value in this set of measurements is defined by

				\(x\&_ = ∑\+\(i = 1\)\%N x\_i/N\)						(1)

where (Subscript[x, i]) is the value of measurement i and N is the total number of measurements, in this case 14. A calculation of the mean using equation (1) and the Mathematica command for the 14 member sample set are shown in the output line following the histogram. The mean changes each time a set of values is generated in the SampleDist1 set as a result of using the RandomArray command, which randomly chooses elements from the normal distribution.

SampleDist1 = 
      SetPrecision[RandomArray[NormalDistribution[60, 10], 14], 3]

Block[{$DisplayFunction = Identity}, 
plot1 = 
          ListPlot[SampleDist1,   Frame -> True, 
            PlotRange -> {{1, 14}, {0, 100}}, AxesOrigin -> {0, 0}, 
            FrameLabel -> {"Experiment  Number", "Measured  \
Value", " ", " "}, DefaultFont :> {"Arial", 10}, 
            PlotStyle -> {RGBColor[1, 0, 0],  PointSize[0.04]}, 
            AspectRatio -> 1.2]; 
histo1 = 
          Histogram[SampleDist1, Frame -> True, 
            FrameLabel -> {"Value", "Frequency", " ", " \
"}, DefaultFont :> {"Arial", 10}, 
            AspectRatio -> 1.2]];
Show[GraphicsArray[{plot1, histo1}], 
        GraphicsSpacing -> 0.1];

Figure 1: Simulation and histogram of 14 experimental measurements

average = (∑_(i = 1)^14) 
            SampleDist1[[i]]/14

Mean[
      SampleDist1]

TRY IT: Histogram and Mean of a Set of 14 Experiments
Place the cursor anywhere in the following command and evaluate the command by pressing Shift-Return to generate a new set of 14 values, a histogram, and mean. There is no parameter that needs to be changed here, simply reevaluate the command to generate a new data set. This command simulates the results that would be obtained from 14 repetitions of one experiment. Watch how the histogram and mean change each time the command is evaluated and a new sample is generated. Some questions to think about as new results are obtained by repeatedly evaluating this command are as follows. Is there a greater change in the histogram or the mean from run to run? How large is the interval containing all of the mean values from each sample? Does this interval change with the sample set? Does the histogram have a particular shape? What does the shape of the histogram appear to indicate?

SampleDist1 = 
      SetPrecision[RandomArray[NormalDistribution[60, 10], 14], 
        3]
Block[{$DisplayFunction = Identity}, 
plot1 = 
          ListPlot[SampleDist1,   Frame -> True, 
            PlotRange -> {{1, 14}, {0, 100}}, AxesOrigin -> {0, 0}, 
            FrameLabel -> {"Experiment  Number", "Measured  \
Value", " ", " "}, DefaultFont :> {"Arial", 10}, 
            PlotStyle -> {RGBColor[1, 0, 0],  PointSize[0.04]}, 
            AspectRatio -> 1.2]; 
histo1 = 
          Histogram[SampleDist1, Frame -> True, 
            FrameLabel -> {"Value", "Frequency", " ", " \
"}, DefaultFont :> {"Arial", 10}, 
            AspectRatio -> 1.2]];
Show[GraphicsArray[{plot1, histo1}], GraphicsSpacing -> 0.1]; 
    Mean[SampleDist1]

The mean for the sample set containing 14 values is usually close to a value of 60, but typically varies between 50 and 70 (it was set up that way with the RandomArray command) . Values as low as 40 and as high as 80 are possible, however. These variations in the value of the mean in a real experiment are the result of experimental error. Variations are caused by a finite precision in the instruments used to make the measurements and slight changes in the conditions at the time a given experiment is performed among other factors. One way to characterize the variation is to compute the standard deviation of the measurements. The standard deviation is a measure of the width of the histogram of values around the average or mean value and is defined by

			s=(Sqrt[\(∑\+\(i = 1\)\%N\((x\_i - x\&_)\)\^2/\((N - 1)\)\)]).					(2)

The standard deviation is the square root of the sum of the squares of the deviations of the measurements from the mean averaged by N-1. The factor N-1 is used rather than N to compute the average because only N-1 differences can vary freely. The last difference \(TraditionalForm\`x\_i - x\&_\) can be found once all of the others are known since the sum of all of the differences \(TraditionalForm\`x\_i - x\&_\) always equals zero. The number N-1 is called the degrees of freedom. The square of the standard deviation is called the variance (s^2).

			\(s\^2 = ∑\+\(i = 1\)\%N\((x\_i - x\&_)\)\^2/\((N - 
                    1)\)\)					(3)

The variance and the moment of inertia I from physics defined by

			I = (∑_(i = 1)^N) (Subscript[m, i]) (Subscript[r, i]^2)							(4)

where (Subscript[m, i]) is the mass of particle i and (Subscript[r, i]) is the distance of particle i from an origin are similar in that both quantities depend on a squared term which quantifies how far away something is from a point of reference. In the case of the moment of inertia, the squared term indicates how far away a mass is from an axis. So for two objects of the same mass, the one having more of the mass farther from a fixed axis has a larger moment of inertia about that axis. In the case of the variance, the squared term indicates how far away the data is from the average. So for two data sets with the same average, the one that contains points farther from the average has a larger variance from the average, which is just a point of reference. Likewise the data set with the larger variance has the larger standard deviation which is the square root of the variance. 

A calculation of the standard deviation for the last sample set generated above follows.

average = (∑_(i = 1)^14) SampleDist1[[i]]/
            14;
stddev = 
      Sqrt[(∑_(i = 1)^14)(((SampleDist1[[i]] - average))^2)/(14 - 1)] // N

StandardDeviation[SampleDist1] // N

Another sample set containing 14 measurements using the same experimental apparatus will usually have a different mean and standard deviation as shown below using the RandomArray command to model the experiment.

SampleDist2 = 
        RandomArray[NormalDistribution[60, 10], 14];

Mean[SampleDist2]

StandardDeviation[SampleDist2]

We could repeat this process, in principle, generating an infinite number of samples each containing 14 measurements, each with its own mean and standard deviation. The infinite sample set would contain all possible sets of 14 values that could be generated using the same experimental apparatus. A plot of the complete set of discrete histograms each containing 14 values added together then normalized yields the continuous curve shown in Figure 2. The vertical axis is the fraction of the whole rather than absolute number of values as in Figure 1. The curve or distribution is continuous because a very large number of experimental measurements are included, in fact, an infinite number, and this curve is called the normal distribution.

Distributions of experimental measurements or the shape of the histogram usually are completely determined by one or two parameters. The distribution in Figure 2 is symmetric about the center or mean, has only one peak, and is bell-shaped. This curve is called a normal distribution and the position and shape of the curve are completely determined by the mean μ and standard deviation σ. The mean is the average of all the experimental measurements and, in this case, it is the real or true mean since there are an infinite number of measurements included. An infinite number of measurements were not made to obtain the continuous distribution rather an equation was used that describes what would happen if an infinite number of measurements could be made. The true mean is also called the population mean. The sample mean often referred to simply as the mean is the average of all the experimental measurements from a particular sample of the entire population. Sample means were computed above for two samples of 14 measurements. In practice, the true mean is impossible to compute since an infinite number of measurements can not be made. The mean of the continuous distribution of every possible 14 measurement data set in Figure 2 is 60. How close are the means of particular 14 measurement data sets computed above to the value of 60 for the infinite population?

The standard deviation quantifies the spread of the sample values about the mean so provides a measure of the variation of the measurements due to experimental error. The standard deviation is the square root of the moment of the values with respect to the average value used as a reference. The standard deviation of the distribution in Figure 2 is 10. Note that the two sample sets of 14 values each (SampleDist 1 and SampleDist2 above) have means and standard deviations close to the true values. This observation suggests that an estimate of the true or population mean can be obtained using a finite (small) number of measurements.

Plot[PDF[NormalDistribution[60, 10], x], {x, 20, 100}, 
        PlotRange -> {{20, 100}, {0, 0.041}}, Frame -> True, 
        FrameLabel -> {"Value", "Fraction", " ", " "}, 
        DefaultFont :> {"Arial", 10}, 
        PlotStyle -> {RGBColor[1, 0, 0],  
            Thickness[0.01]} ];

Figure 2: Normal distribution

Mean[NormalDistribution[60, 10]]

StandardDeviation[NormalDistribution[60, 10]]

The mean of all finite sets of samples fluctuate around the true mean. The value of the true mean can never be known in practice, however, since an infinite amount of data is needed. The principles of statistical inference, however, can be used to estimate a true mean using a finite experimental data set obtained in the laboratory. Every experimental measurement made can be thought of as one value drawn from the infinite data set containing all possible values. If experimental measurements have an equal and independent chance of being selected (drawn) from the infinite data set then the mathematical properties of the infinite data set can be used to infer approximate values of parameters, such as an estimate of the mean. An estimate of the true mean can be established without knowing the exact value of the true mean. The important thing here is that the experimental measurements be independent. Practically this means that the measurements in the laboratory be made under the same conditions each time and that a serious effort is made to minimize extraneous influences (such as those from the experimenter) on the outcome of the measurements.

If a set of independent measurements can be made for a particular system, then the Central Limit Theorem states that a plot of the means of each sample data set also forms a normal or Gaussian distribution. In other words, the set of all means calculated using 14 data points, using the example above, is distributed normally about the true sample mean μ. This assertion is valid, however, only when the infinite data set is itself Gaussian. Although this can never be proven or tested because collecting an infinite amount of data is not practical, experience shows that with large data sets a Gaussian distribution is very closely approached. Some science museums have demonstrations of this observation using ping-pong balls dropped onto a series of horizontal pegs fixed on a board in an enclosed, oblong shell. A distribution of balls collected at the bottom of the pegboard is obtained that is very close to normal after 50-100 balls are dropped. The Gaussian distribution is based on random fluctuations which are very well approximated by most physical experiments in the laboratory. The balls in the pegboard experiment have an equal chance of falling one way or the other when they hit a peg in their downward path to the collection cylinders. More balls collect in the middle because there are more paths or a higher probability for the balls to reach the middle than the ends. Putting all of these concepts together, the mean value of independently selected data is distributed normally about the true mean, which is a characteristic quantity.

Before defining the relationship between the true mean of all possible values μ and the sample mean \(TraditionalForm\`x\&_\) obtained from experimental measurements, two important properties of the distribution of sample means are needed. Using the 14 member sample as an example, the distribution of sample means is the set of averages computed from all possible 14 member samples of the infinite set. The first important property of this distribution is that just like the infinite data set which is centered about the true mean, the distribution of sample means is also centered about the true mean. The second property is that the distribution of sample means is narrower than the original distribution. What this means practically is that the distribution of sample means cluster more closely around the true mean than do individual measurements. The precision of a sample containing several measurements (the sample means) is better then the precision of any single measurement. How much better? The standard deviation of the means (Subscript[σ, m]) can be shown to be equal to

				(Subscript[σ, m]) = σ/(Sqrt[N])					(5)

where N is the number of measurements and σ is the true standard deviation. The sample means are consequently distributed more tightly about the true mean than individual measurements by a factor of (Sqrt[N]).

The best estimate of the true standard deviation σ can be shown to be equal to

				\(TraditionalForm\`σ ≃ σ\_est = \@\(\(∑\((x\_i - x\&_)\)\^2\)\/\(N - 1\)\)\)					(6)

which is the same as the sample standard deviation s in equation (2). The important point about equation (6) is that an estimate of the true standard deviation can be calculated directly using the values (Subscript[x, i]) from a set of N measurements as was done above for the standard deviation of a sample of 14 measurements. Combining equations (5) and (6) shows that the standard deviation of the means is given approximately by

				\(TraditionalForm\`σ\_m ≃ \@\(\(∑\((x\_i - x\&_)\)\^2\)\/\(N(N - 1)\)\)\)	.					(7)

The standard deviation of the means can consequently be calculated from experimental measurements. Combining equations (2) and (7) yields

					(Subscript[σ, m]) ≃ (s/(Sqrt[N]))   					(8)

which shows that the sample standard deviation s can be used to estimate the standard deviation of the means. When data is used to estimate a standard deviation as in equation (8), the standard deviation is called the standard error.

Confidence intervals for a parameter are directly proportional to standard deviations, which indicates the importance of understanding the types of standard deviations and how each is related to another. A confidence interval is a range defined between a lower and an upper bound, which quantifies the percentage of time that this range contains the parameter of interest. A range computed from data must be related to the spread in the data so it is not surprising that confidence intervals are related to standard deviations, but unlike standard deviations, confidence intervals provide a measure or probability of how certain the range contains the parameter. Absolute certainty requires an infinite number of measurements, but if 95% or 99% certainty is good enough then a more modest number of measurements is all that is required. Understanding statistical inference provides some control over the trade-off between certainty (confidence) and cost (number of measurements).

A confidence interval depends on the parameter distribution chosen for the calculations. The normal distribution shown in Figure 2 is one possible selection and is completely determined by the population mean μ and population standard deviation σ. The Multinormal Distribution package in Mathematica contains normal, Student t, Wishart, Hotelling (T^2), and quadratic form distributions. The normal distribution is often tabulated or available as a function of one parameter called z. The z parameter expresses the number of standard deviations that a sample mean is away from the true mean. The parameter z is defined by

					z≡((\(x\&_ - μ\))/(σ/(Sqrt[N])))					(9)

and is called a standardized sample mean since it is the ratio of the difference between the sample and populations means to the standard deviation of the mean. Standardization allows the tables in books to be applied to many different problems. The z parameter in equation (9) is not practical for computing confidence intervals from small data sets, however, since the population standard deviation σ must be known, which requires an infinite or at the very least a large amount of data. The sample standard deviation s does not require a large number of data points and can consequently be computed from any data set. Substituting the sample standard deviation s for the population standard deviation in equation (9) produces a new parameter called the t statistic defined by

					t≡((\(x\&_ - μ\))/(s/(Sqrt[N])))					        (10)

This substitution is equivalent to using the standard error given by equation (8) for the standard deviation of the sample mean. Associated with the t statistic is a distribution called the "Student's  t" or just t distribution. The t distribution has a larger spread than the standard normal distribution since the sample standard deviation s depends on the number of measurements made. As the number of measurements gets large, s approaches σ,  t approaches z, and the t distribution approaches the standard normal distribution, as expected.

NM = 
        5;
Block[{$DisplayFunction = Identity}, 
plot2 = 
          Plot[PDF[NormalDistribution[0, 1], x], {x, -5, 5}, 
            Frame -> True, AspectRatio -> 2, 
            PlotRange -> {{-5, 5}, {0, 0.42}}, 
            FrameLabel -> {"Value", "Fraction", " ", " \
"}, DefaultFont :> {"Arial", 10}, 
            PlotStyle -> {RGBColor[1, 0, 0],  
                Thickness[0.004]} ]; 
plot3 = 
          Plot[PDF[StudentTDistribution[3 - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[0, 0, 1],  
                Thickness[0.004]}]; 
plot4 = 
          Plot[PDF[StudentTDistribution[10 - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[1, 1, 0],  
                Thickness[0.004]}]; 
plot5 = 
          Plot[PDF[StudentTDistribution[NM - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[0, 0, 0],  Thickness[0.01], 
                Dashing[{ .03}]}]];

TRY IT: Comparison of standard normal and t distributions
How many measurements (NM) need to be made in order for a t distribution to closely match the standard normal distribution?
To find out, change the numerical value of NM, which is initially set at 5, in the command that follows the plotting button below (evaluate the command by typing shift-enter) until the dashed black line (t distribution for NM measurements) matches the red line (standard normal distribution) in Figure 3. The blue and yellow lines are for 3 and 10 measurements, respectively. The average of the standard normal distribution is fixed at a value of 0 with a standard deviation of 1.

ButtonBox[\(StyleBox[
            RowBox[{"Generate", \ " ", \ " ", \ "plots", \ " ", \ "of", \ " ", \ "the", "
", "Standard", \ " ", \ "Normal", \ " ", \ "and", \ "
", \ "Student t", " ", "Distributions"}], \ FontFamily\  -> \ "Helvetica"]\), ButtonEvaluator  ->  "Automatic",ButtonFunction  :>  (NotebookPut[
                  Notebook[{Cell[
                        GraphicsData["PostScript",  DisplayString[#2]], 
                        ShowCellBracket  ->  False, 
                        ImageSize -> 288]},  
                    WindowSize  ->  {325, 
                        570},   WindowElements  ->  {}, 
                    WindowTitle -> "Figure 3"]] & ), ButtonData :>
Show[plot2,plot3,plot4,plot5,DisplayFunction  ->  Identity],Active  ->  True]  // DisplayForm

NM = 5   ;
Block[{$DisplayFunction = 
            Identity}, 
plot2 = 
          Plot[PDF[NormalDistribution[0, 1], x], {x, -5, 5}, 
            Frame -> True, AspectRatio -> 2, 
            PlotRange -> {{-5, 5}, {0, 0.42}}, 
            FrameLabel -> {"Value", "Fraction", " ", " \
"}, DefaultFont :> {"Arial", 10}, 
            PlotStyle -> {RGBColor[1, 0, 0],  
                Thickness[0.004]} ]; 
plot3 = 
          Plot[PDF[StudentTDistribution[3 - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[0, 0, 1],  
                Thickness[0.004]}]; 
plot4 = 
          Plot[PDF[StudentTDistribution[10 - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[1, 1, 0],  
                Thickness[0.004]}]; 
plot5 = 
          Plot[PDF[StudentTDistribution[NM - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[0, 0, 0],  Thickness[0.01], 
                Dashing[{ .03}]}]];

NM = 5;
Block[{$DisplayFunction = Identity}, 
plot2 = 
          Plot[PDF[NormalDistribution[0, 1], x], {x, -5, 5}, 
            Frame -> True, AspectRatio -> 2, 
            PlotRange -> {{-5, 5}, {0, 0.42}}, 
            FrameLabel -> {"Value", "Fraction", " ", " \
"}, DefaultFont :> {"Arial", 10}, 
            PlotStyle -> {RGBColor[1, 0, 0],  
                Thickness[0.004]} ]; 
plot3 = 
          Plot[PDF[StudentTDistribution[3 - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[0, 0, 1],  
                Thickness[0.004]}]; 
plot4 = 
          Plot[PDF[StudentTDistribution[10 - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[1, 1, 0],  
                Thickness[0.004]}]; 
plot5 = 
          Plot[PDF[StudentTDistribution[NM - 1], x], {x, -5, 5} , 
            PlotStyle -> {RGBColor[0, 0, 0],  Thickness[0.01], 
                Dashing[{ .03}]}]];
Show[plot2, plot3, plot4, plot5];

Figure 3: Comparison of standard normal and t distributions

For the relatively small number of measurements usually made in practice, the value of s depends on the number of measurements N as does the t statistic. In Figure 3, the blue and yellow distributions are for N = 3 and 10, respectively, and the red curve is the standard normal distribution. Since the number of independent \(TraditionalForm\`x\_i - x\&_\) differences is equal to N-1, there are N-1 degrees of freedom and the t distribution is designated t(N-1). The blue and yellow curves in Figure 3 are the t(2) and t(9) distributions, respectively.

The fluctuation of the sample mean around the true mean can be quantified by computing a confidence interval. The t distribution is used to calculate a confidence interval for a sample mean using data. The confidence interval for a sample mean quantifies the percentage of the time that the range contains the true mean. All confidence intervals contain two pieces of information: the interval which gives the range of values and the confidence level which gives the probability that the range contains the parameter of interest. The confidence level is usually given in terms of both a P-value and α, which are related as follows.

					P-value = 1 - α				        (11)

The P-value or percentage if multiplied by a factor of 100 expresses the fraction of the t distribution contained in the confidence interval. The parameter α is called the significance level. Higher P-values indicate that more of the t distribution is within the confidence interval, which is another way of saying that the confidence interval is more likely to contain the true mean. The value α gives the fraction of the t distribution outside of the confidence interval.

A level 1 - α confidence interval for the true mean μ is defined by

				\(x\&_\)  ±  \(t\^*\) (s/(Sqrt[N]))					       (12)
 
where t* is the upper α/2 critical value for the t(N-1) distribution. Only the area contained by the upper α/2 critical value is used since the t distribution is symmetric. The quantity  \(t\^*\) (s/(Sqrt[N])) is called the margin of error.

For example, consider the following sample distribution of 14 measurements.

SampleDist3 = 
      SetPrecision[RandomArray[NormalDistribution[60, 10], 14], 
        3]
plot6 = 
        ListPlot[SampleDist3,   Frame -> True, 
          PlotRange -> {{1, 14}, {0, 100}}, AxesOrigin -> {0, 0}, 
          FrameLabel -> {"Experiment  Number", "Measured  \
Value", " ", " "}, DefaultFont :> {"Arial", 10}, 
          PlotStyle -> {RGBColor[1, 0, 0],  PointSize[0.04]}, 
          AspectRatio -> 1.2];

The sample mean and standard deviation can be computed as follows.

Mean[SampleDist3]
StandardDeviation[SampleDist3]

The 95% confidence interval computed using these results requires a value for t*. Values for t* can be found in books on statistics. The values are tabulated as a function of the number of degrees of freedom and the upper α/2 critical value, which is sometimes called  p (not to be confused with an upper case P meaning probability). A selection of values is shown in Table 1 below. The t* values can also be found directly using Mathematica as follows.

conLevel = 95;
NM = 14  ;
degreesFreedom = 
        NM - 1  ;
tStatStar = 
      Quantile[StudentTDistribution[degreesFreedom], 
          1 - ((1 - (conLevel/100))/2)] // N

Combining these results yields the 95% confidence interval using the sample measurements. The lower bound is computed as follows.

μlower = 
      Mean[SampleDist3] - 
        tStatStar*StandardDeviation[SampleDist3]/(Sqrt[NM])

The upper bound is found similarly.

μupper = 
      Mean[SampleDist3] + 
        tStatStar*StandardDeviation[SampleDist3]/(Sqrt[NM])

The confidence interval

{μlower, μupper}

consequently has a 95% chance of containing the true mean. Mathematica can compute the confidence interval directly using the command StudentTCI[\(TraditionalForm\`x\&_\),(Subscript[σ, m]),N - 1] using computed values for the sample mean \(TraditionalForm\`x\&_\), standard error of sample mean (Subscript[σ, m]), and degrees of freedom N - 1 (N is the size of the sample).

StudentTCI[Mean[SampleDist3], StandardDeviation[SampleDist3]/(Sqrt[NM]), 
      degreesFreedom]

Table 1 contains values for t* as a function of the degrees of freedom and the probability values. Note that 10 to 20 degrees of freedom are commonly encountered in experimental measurements and the t* values for 68%, 95%, and 99% confidence levels are approximately 1, 2, and 3, respectively, in this range, which is an easy rule of thumb to remember to obtain an estimate of the confidence limits from the sample standard deviation.

table1 = 
        Table[{i, 
            Quantile[StudentTDistribution[i], 
                1 - ((1 - (68/100))/2)] // N, 
            Quantile[StudentTDistribution[i], 
                1 - ((1 - (95/100))/2)] // N, 
            Quantile[StudentTDistribution[i], 
                1 - ((1 - (99/100))/2)] // N}, {i, 
            20}];
table2 = 
        Table[{i, 
            Quantile[StudentTDistribution[i], 
                1 - ((1 - (68/100))/2)] // N, 
            Quantile[StudentTDistribution[i], 
                1 - ((1 - (95/100))/2)] // N, 
            Quantile[StudentTDistribution[i], 
                1 - ((1 - (99/100))/2)] // N}, {i, 30, 100, 
            10}];

{{" ", "  Confidence", "Levels", "     "}, {"deg \
freedom", "   68%", "95%", "   99%"}} // 
      TableForm
PaddedForm[TableForm[table1], {7, 3}, 
      NumberPadding -> {" ", " "}]
PaddedForm[TableForm[table2], {7, 3}, 
      NumberPadding -> {" ", " "}]

Table 1: t* Values

A confidence interval consequently provides a range both below and above the sample mean that contains the true mean a given percentage of the time. The percentage depends on the confidence level chosen. Note that a confidence interval is not an interval about the sample mean in which the next experimental measurement can be expected. It is much more powerful than that, since it provides a range of values that contain the true mean with a quantified level of confidence based on a finite and small number of experimental measurements. The width of the interval decreases in proportion to the number of experimental data points available as shown by the decrease in t* as a function of the degrees of freedom in Table 1. Ironically the true mean can never be known with absolute certainty, however by giving up on this certainty a range of values can be found that contains the true mean with very high confidence.

A 95% confidence limit means that the computed interval contains the true mean 95% of the time. This idea can be illustrated by generating 100 random samples of the 14 measurement set used previously, computing confidence intervals for each member of the set, and counting how many intervals contain the true mean of 60.

sample100 = 
          Table[RandomArray[NormalDistribution[60, 10], 
              14], {100}];  
confidenceIntervals = 
        Map[MeanCI[#] &, sample100];
Length[
        Select[confidenceIntervals,  #[[1]]  <  
              60  <  #[[2]]  &]]

The latter value gives the number of samples that produced a 95% confidence interval containing the true mean of 60. This number is equal or close to 95 and varies each time the command is run since 100 new random samples are generated each time. A plot of all 100 confidence intervals in the order generated follows (see Ruskeepää).

ciPlot[confidenceIntervals_,mu_,n_,opts___] := Show[Graphics[{Line[{{1,mu}, {n,mu}}],
  Table[Line[{{i,confidenceIntervals[[i,1]]}, {i,confidenceIntervals[[i,2]]}}], {i,n}]}, 
  Axes->{False,True}, AspectRatio->0.3, PlotRange->All, opts]];

ciPlot[confidenceIntervals,  60,  100, 
          DefaultFont :> {"Arial", 10}];

Note that the confidence intervals are different sizes and while most contain the true mean of 60 some do not. The confidence intervals (bars) sorted in ascending order of the magnitude of the mean (shown by points) follows. Does the number of confidence intervals (or CIs) that do not contain the true mean agree with the number computed above of CIs that do contain the true mean?

me = Transpose[{Range[100], Sort[Map[Mean,sample100]]}];

ciPlot[Sort[confidenceIntervals, Mean[#1]<Mean[#2]&], 60, 100, 
  Epilog->{PointSize[0.009], Map[Point,me]},DefaultFont:>{"Arial",10}];

Example 1: Mean and Confidence Interval of Silicon Dioxide Film Thickness

Three measurements of the silicon dioxide film thickness on a silicon wafer were made using an ellipsometer. An ellipsometer uses interference between light rays reflected off of each interface in a layered thin film structure to measure the thickness and refractive index of each layer in the film. The values measured were 4500, 4530, and 4580 Å. A human hair is approximately 50 µm (50 x (10^(-6)) m) thick and 1 Å = (10^(-4)) µm so the (Subscript[SiO, 2]) films were approximately 0.5 µm thick or one order of magnitude (10x) thinner than a human hair. What range of values contains the true mean with 95% and 99% confidence?

The mean of the three measurements is 4536.7 Å, which is computed as follows.

NM =. ; s =. ;

dataex1 = {(Subscript[x, 1]) -> 4500, (Subscript[x, 2]) -> 4530, 
          (Subscript[x, 3]) -> 4580}  ;

\(x\&_ = \((∑\+\(i = 1\)\%NM x\_i)\)/NM\)

NM = 3;
\(mex1 = Evaluate[x\&_ /. dataex1] // N\)

Mathematica can also be used to make the computation directly.

Mean[{(Subscript[x, 1]), (Subscript[x, 2]), (Subscript[x, 3])} /. dataex1] // N

\(x\&_ =. ; NM =. ; s =. ;\)

The standard deviation is 40.4 Å, which is computed as follows.

\(s = 
      Sqrt[∑\+\(i = 1\)\%NM\((x\&_ - x\_i)\)^2/\((NM - 1)\)]\)

\(NM = 3; \ x\&_ = \((∑\+\(i = 1\)\%NM x\_i)\)/NM; 
    Evaluate[x\&_ /. dataex1] // N;\)

sdevex1 = Evaluate[s /. dataex1] // N

StandardDeviation[{(Subscript[x, 1]), (Subscript[x, 2]), (Subscript[x, 3])} /. dataex1] // N

Note that the standard deviation is approximately equal to 40 which is the average of the deviation around the median or middle value as shown below.

medianex1 = 4530;

averagedeviation1 = (Abs[4500 - medianex1] + 
            Abs[4580 - medianex1])/(NM - 1)

\(x\&_ =. ; NM =. ;\)

There are 2 degrees of freedom in the sample data set containing NM = 3 points, since N - 1 = 2. The (Subscript[t, N - 1]) values for the 95% and 99% confidence levels are 4.303 and 9.925, respectively, from Table 1 or directly from Mathematica.

tCI95 = 
      Quantile[StudentTDistribution[3 - 1], 
          1 - ((1 - (95/100))/2)] // N
tCI99 = 
      Quantile[StudentTDistribution[3 - 1], 
          1 - ((1 - (99/100))/2)] // N

The equation for the margin of error or one half of the confidence interval is

\(ConfIntex1 = 
      tCI*Sqrt[∑\+\(i = 1\)\%NM\((x\&_ - 
                    x\_i)\)^2/\((NM*\((NM - 1)\))\)]\)

\(NM = 3; \ x\&_ = \((∑\+\(i = 1\)\%NM x\_i)\)/NM; 
    Evaluate[x\&_ /. dataex1] // N;\)

The margin of error for the 95% confidence level is 100.4 Å, which is computed as follows.

marginError95 = 
      Evaluate[ConfIntex1 /. dataex1] /. 
        tCI -> tCI95
clim95ex1 = marginError95;

The lower and upper confidence limits are computed as follows.

lowerCI95 = mex1 - marginError95

upperCI95 = mex1 + marginError95

Using Mathematica,

MeanCI[{(Subscript[x, 1]), (Subscript[x, 2]), (Subscript[x, 3])} /. dataex1, 
      ConfidenceLevel -> 0.95]

or

StudentTCI[mex1, sdevex1/Sqrt[NM], NM - 1, 
      ConfidenceLevel -> 0.95]

The confidence interval from 4436 to 4637 Å contains the true mean with 95% certainty.

The margin of error for the 99% confidence level is 231.6 Å, which is computed as follows.

marginError99 = 
      Evaluate[ConfIntex1 /. dataex1] /. 
        tCI -> tCI99
clim99ex1 = marginError99;

The lower and upper confidence limits are computed as follows.

lowerCI99 = mex1 - marginError99
upperCI99 = mex1 + marginError99

MeanCI[{(Subscript[x, 1]), (Subscript[x, 2]), (Subscript[x, 3])} /. dataex1, 
      ConfidenceLevel -> 0.99]

StudentTCI[mex1, sdevex1/Sqrt[NM], NM - 1, 
      ConfidenceLevel -> 0.99]

A 99% confidence interval is from 4305 to 4768 Å.

\(x\&_ =. ; NP =. ; tCI =. ;\)

Another way of expressing the confidence interval estimate is as follows.

		μ   =   4537   ±   100  Å  (with 95% confidence)
or
		μ   =   4537   ±   232  Å  (with 99% confidence).

Increased certainty in finding the true mean in a given range of values comes at the expense of increasing the size of the confidence intervals or error bars. Observe that these confidence intervals are much larger than the standard deviation of the data, by a factor of 2-6 times, in this case. The standard deviation, in fact, provides the limits that have a 68% chance of containing the true mean. How can the confidence intervals be decreased in size in order to obtain a better estimate of the true mean?

TRY IT: One of the uses of statistical inference is to compute the number of experimental measurements needed to achieve a particular accuracy with a specified level of confidence. Suppose that an accuracy of ±50 Å is desired with 95% confidence. How many measurements are required to achieve this level of accuracy and confidence?

Try to compute the answer yourself or click the triangle on the left of the next cell for the equation to use. The answer is developed in three parts.

Clicking the triangle at left opens or closes the text in this cell group.
Equation Development for Sample Size Calculation.

Note the accuracy desired is equal to the margin of error or one-half the width of the confidence interval, which can be given the symbol w. Using equation (12) for the margin of error indicates that

		w = 2t*(s/((Sqrt[N])))	⟹	N=(((((\(2 \( t\^*\) s\))/w)))^2)			     (13)

The sample standard deviation is not known before the measurements are made. It must either be estimated based on past experiments or guessed.

Numerical Computation for Sample Size

Using the sample standard deviation from the three measurements made (past experiments), the number of measurements to improve the accuracy to 50 Å is obtained by changing the numerical value of NM in the first line below and evaluating the cell (Shift-Enter) until the value of w on the output cell equals 100 (2 x 50 Å).

NM = 3;
s = 
        StandardDeviation[{(Subscript[x, 1]), (Subscript[x, 2]), (Subscript[x, 3])} /. dataex1] // 
          N;
tCI95 = 
        Quantile[StudentTDistribution[NM - 1], 
            1 - ((1 - (95/100))/2)] // 
          N;
w = 2*tCI95*(s/Sqrt[NM])

Answer and Analysis

Since 5 measurements yield an interval of 129 Å which is slightly wider than desired and 6 measurements yield an interval of 85 Å which is smaller than desired, 6 measurements must be made to reach the desired accuracy of ±50Å or an interval of 100 Å.

It is instructive to plot the magnitude of the interval as a function of the number of measurements made to determine if there are any break points in the benefit (accuracy) versus cost (number of measurements) curve.

s = 
        StandardDeviation[{(Subscript[x, 1]), (Subscript[x, 2]), (Subscript[x, 3])} /. dataex1] // 
          N;
tCI95func[z_] = 
        Quantile[StudentTDistribution[z], 
            1 - ((1 - (95/100))/2)] // 
          N;
w = 
table3 = 
          Table[{i + 1, 2*tCI95func[i]*(s/Sqrt[i + 1])}, {i, 
              48}];  
ListPlot[table3, DefaultFont -> {"Arial", 10}, 
        PlotRange -> {{0, 50}, {0, 210}}, PlotJoined -> False, 
        Frame -> True, 
        FrameLabel -> {"Number of Measurements", "Accuracy   (\
Å)"}, 
        PlotStyle -> {RGBColor[1, 0, 0],  
            PointSize[0.02]}];

The accuracy drops off quickly with the number of measurements made and reaches a break point at approximately 10 measurements. Making more than 10 measurements consequently does not improve the accuracy all that much relative to making less than this number of measurements. Note that this number is a strong function of the standard deviation used which was estimated based on a small number of measurements (three in this case). The standard deviation typically improves with the number of measurements made which will alter the position of the break point. Practically, however, the sample standard deviation can not be determined until the measurements have been made so an estimate is necessary.

This example indicates that only a relatively small number of measurements (10 in the case considered) are needed to obtain an accurate estimate of the true mean with a high degree of confidence.

SECTION 2
HYPOTHESIS TESTING

A confidence interval contains the population or true value of a parameter a specified percentage of the time. Confidence intervals can consequently be used to accept or reject a statement or hypothesis about the parameter. This procedure is called a significance test. The significance of a hypothesis is based on the value of a test statistic that gives the probability that the statement being tested is true. The statement being tested is called the null hypothesis, which typically asserts that the experiment has had no effect or has a particular value or range of values outside of the desired value. Assuming an undesired result (the null hypothesis) is standard in hypothesis testing. The alternative hypothesis is the statement that is accepted if the probability of the null hypothesis is small enough. In other words, if an effect (null hypothesis) is not supported by the results of the experiments it can be rejected with a certain level of confidence.

A significance test requires a null hypothesis (Subscript[H, 0]) and one or more alternative hypotheses (Subscript[H, a]). A significance level α is chosen that specifies how much evidence against (Subscript[H, 0]) is needed to reject it. The calculation of a test statistic using (Subscript[H, 0]) such as the t statistic provides a measure of how well the experimental data support (Subscript[H, 0]). The P-value gives the probability that (Subscript[H, 0]) is true. If the P-value is less than or equal to α, then the null hypothesis can be rejected or is statistically significant at level α. There are many more details to significance testing that can be found in the references.

Example 2: Significance of a Mean (Subscript[SiO, 2]) Film Thickness

In example 1, the 99% confidence interval for the mean (Subscript[SiO, 2]) film thickness on a silicon wafer was from 4305 to 4769 Å (μ   =   4537   ±   232  Å). If a supplier states that the film thickness is 4800 Å, then this claim could be rejected at the 1% significance level. This is called a two-sided significance test since the hypothesized mean (Subscript[SiO, 2]) thickness falls outside of the level 1% confidence interval for μ. On the other hand if the supplier states that the film thickness is 4700 Å, then this claim could not be rejected at the 1% significance level, since this value is within the 99% confidence interval.

Is the mean film thickness value obtained from measurements significantly different than 4700 Å? The hypotheses are as follows

			(Subscript[H, 0]) :  μ  =  4700 Å
			(Subscript[H, a]) :  μ  ≠  
          4700 Å
			
Compute the t statistic using results from example 1.

muNull = 4700;
NM = 3;
tstat = (mex1 - muNull)/(sdevex1/Sqrt[NM]) // N

Compute the P-value using the Mathematica command StudentTPValue using a two-sided test.

StudentTPValue[tstat, NM - 1, TwoSided -> True]

The difference between the sample mean film thickness and the supplier value is significant at the 2% level. The supplier value of 4700 Å lies just outside the 98% confidence interval, which is from 4374 to 4699 Å.

StudentTCI[mex1, sdevex1/Sqrt[NM], NM - 1, 
      ConfidenceLevel -> 0.98]

Is the supplier value significantly different than the sample mean value? The answer to this question depends on the needs of the user who processes the wafers on which the films are deposited. On one hand, there is only a 2% probability that the mean film thickness value is 4700 Å rather than 4537 Å. If the wafers were put into a manufacturing process that required the additional material in order to produce an acceptable product, then the difference is significant. The user should reject the wafer. On the other hand, if the manufacturing process were designed to tolerate wafers with starting (Subscript[SiO, 2]) film thicknesses between 4500 and 4800 Å, then the difference would not be significant. The user would accept the wafer in this case.

Suppose that the user must have wafers containing less than 4700 Å of (Subscript[SiO, 2]) 99% of the time in order to have an acceptable product yield. Should the user put the wafer having a mean film thickness of 4537 Å into production?

The hypotheses are as follows.

			(Subscript[H, 0]) :  μ  =  4700 Å
			(Subscript[H, a]) :  μ  ≺  
          4700 Å
			
This is a one-sided significance test. The P-value is computed as follows.

StudentTPValue[tstat, NM - 1, TwoSided -> False]

There is less than a 1% chance that the wafer with a sample mean film thickness of 4537 Å has a 4700 Å (Subscript[SiO, 2]) layer so the user should choose to put this wafer into production. Mathematica can provide this result if the significance level is specified as follows.

data = {4500, 4530, 
          4580}  ;
MeanTest[data, 4700, SignificanceLevel -> 0.01] // 
      TableForm

SECTION 3
PROPAGATION OF RANDOM ERROR

The uncertainty in a value computed from other values whose confidence intervals or uncertainties are known is called propagation of error. There are several possible approaches to finding the uncertainty in computed values. An approach will be used here to find the most probable uncertainty for the operations of addition, subtraction, multiplication, and division. The reader is referred to the text by Baird listed in the References for a more complete discussion.

Addition and Subtraction

The most probable uncertainty or (Subscript[s, c]) in a value computed by the sum or difference of values whose standard deviations are known is the square root of the sum of the squares of all the standard deviations. If (Subscript[s, i]) is the standard deviation of value i, then the standard deviation of the computed value (Subscript[s, c]) is

				(Subscript[s, c])   =   (Sqrt[(∑_(i = 1)^n) (Subscript[s, i]^2)]).					(14)

where n is the number of individual standard deviations contributing. This equation assumes that all values from i = 1.  ..  n are distributed normally. If this is not the case, then another procedure most be used to combine different types of uncertainty (see Baird, pp. 53-54).

A confidence interval is computed by first finding the confidence intervals at a chosen (confidence) level for each parameter in the sum or difference, then taking the square root of the sum of squares of these component confidence intervals.

Example 3: Mean and Confidence Interval of Etched (Subscript[SiO, 2]) Film

The silicon dioxide layer whose mean thickness and confidence interval were calculated in example 1 was etched in a 1:1:100 solution of HOH:HF:methanol at 41°C for 3600 seconds (60 min). After etching and quenching the reaction, an ellipsometer was used to measure the thickness of silicon dioxide remaining on the surface. The following three values were obtained on three different places on the wafer: 4218, 4270, and 4284 Å. The mean of the three measurements is 4257 Å and the standard deviation is 35 Å, which are computed as follows.

dataex3 = {4218, 4270, 4284};

mex3 = Mean[dataex3] // N

The difference between the initial and final mean layer thicknesses gives the mean etched thickness which in this case is 279 Å.

MeanInitial = mex1; MeanFinal = mex3;

AmountEtched = MeanInitial - MeanFinal

sdevex3 = StandardDeviation[dataex3] // N

The 95% confidence interval for the etched data set is from 4171 to 4344 Å and is computed as follows.

ci95ex3 = MeanCI[dataex3, ConfidenceLevel -> 0.95]

The 95% margin of error or confidence limit is one-half of the difference.

clim95ex3 = (1/2)*(ci95ex3[[2]] - 
            ci95ex3[[1]])

The 99% confidence interval for the second data set is from 4058 to 4457 Å and is computed as follows.

ci99ex3 = MeanCI[dataex3, ConfidenceLevel -> 0.99]

The 99% margin of error or confidence limit is one-half of the difference.

clim99ex3 = (1/2)*(ci99ex3[[2]] - 
            ci99ex3[[1]])

The 95% margin of error for the difference between the initial and final layer thicknesses is computed by taking the square root of the sum of the 95% confidence limits of the initial and final silicon dioxide thicknesses squared. The 95% margin of error for the difference is 132 Å.

climEtch95 = (Sqrt[(clim95ex1^2) + (clim95ex3^2)])

The 99% margin of error is 306 Å.

climEtch99 = (Sqrt[(clim99ex1^2) + (clim99ex3^2)])

The 95% and 99% margins of error that cover the true difference in the silicon dioxide thickness due to etching are as follows

		μ   =   279   ±   132  Å  (with 95% confidence)
or
		μ   =   279   ±   306  Å  (with 99% confidence).

Note that the 99% confidence interval would decrease dramatically if more experimental measurements were available.

Multiplication and Division

The relative error of a product or quotient is determined by the relative errors of values forming the computed result. The method for finding the standard deviation of the computed value requires three steps:
	(1) Calculate the relative standard deviation for each value.
	(2) Calculate the overall relative standard deviation of the computed value.
	(3) Calculate the overall absolute standard deviation of the computed value.
The relative standard deviation is defined as the ratio of the standard deviation to the mean.

Example 4: Mean and Confidence Interval of (Subscript[SiO, 2]) Etching Rate—Combination of Different Types of Uncertainty

The amount etched divided by the etching time is one measure of the etching rate if the amount etched is linear with time and the intercept is zero. Assuming that the time of 3600 seconds is accurate to ±10 seconds, the mean etching rate and standard deviation can be computed as follows.

The mean etching rate is 0.078 Å/s.

time = 3600;
etchRate = AmountEtched/time // N

(1) Compute relative standard deviations for each value. For the amount etched, the standard deviation of the difference in film thickness is divided by the amount etched.

sRelEtch = (Sqrt[(sdevex1^2) + (sdevex3^2)])/AmountEtched // N

The time is a quantity that does not follow a normal distribution but has been assumed to have fixed limits. The uncertainty in the time consequently can not be combined with the uncertainty in the amount etched which has been treated as a normal distribution, unless some manipulation is done (See Baird p. 53-54 for details).

We need to compute limits on the time that are analogous to one standard deviation or 68% limits in the amount etched. Since the absolute range of possibility for the time is assumed to be ±10 seconds, the margin of error or uncertainty limits are 0.68 of this result or (0.68)(10).

sRelTime = (0.68)*10/time // N

(2) Calculate the overall relative standard deviation of the result from the root mean square of the individual values.

sRelCombined = (Sqrt[(sRelEtch^2) + (sRelTime^2)])

(3) Calculate the overall absolute standard deviation of the result.

sCom67 = (etchRate)*sRelCombined // N

The mean etching rate is consequently 0.078  ±  0.015 Å/s to 68% certainty.

The margins of error for 95% certainty are not simply a t-factor multiplied by these limits, since different quantities are being combined. Instead the 95% margin of error must be calculated and combined separately as follows.

95% Limits:

sRelEtch95 = climEtch95/AmountEtched // N

sRelTime95 = (0.95)*10/time // N

sRelCom95 = (Sqrt[(sRelEtch95^2) + (sRelTime95^2)])

sCom95 = etchRate*sRelCom95

etchRate - sCom95
etchRate + sCom95

The interval from 0.041 to 0.114 Å/s contains the true etching rate with 95% certainty or

			μ  =  0.078  ±  0.037  Å/s with 95% confidence.

99% Limits:

sRelEtch99 = climEtch99/AmountEtched // N

sRelTime99 = (0.99)*10/time // N

sRelCom99 = (Sqrt[(sRelEtch99^2) + (sRelTime99^2)])

sCom99 = etchRate*sRelCom99

etchRate - sCom99
etchRate + sCom99

The interval from -0.007 to 0.162 Å/s contains the true etching rate with 99% certainty or

			μ  =  0.078  ±  0.085  Å/s with 99% confidence.

SECTION 4
LEAST-SQUARES FIT TO LINEAR DATA

Suppose that a set of data is available where N values of a variable y have been measured as a function of another variable x. If the x variable is known exactly or much more precisely than the y variable, then the uncertainty in x can be neglected compared to the uncertainty in y. For example, values of the amount of (Subscript[SiO, 2 ])film etched may be known as a function of time, where the time is known much more precisely than the amount of film that has been etched away. If the equation of the best line has the form

					y = mx + b .						(15)
					
then the values of the slope and intercept that minimize the sum of the squared errors are

		\(TraditionalForm\`m = \((N \(∑\((\(x\_i\) 
                      y\_i)\)\)\  - \ ∑\(x\_i\) \(∑y\_i\))\)/\((N \(∑x\_i\%2\)\  - \ \((∑x\_i)\)\^2)\)\)				(16)
and
		\(TraditionalForm\`b\  = \ \((∑\(x\_i\%2\) \(∑y\_i\) - ∑\(x\_i\) \(∑\((\(x\_i\) 
                        y\_i)\)\))\)/\((N \(∑x\_i\%2\)\  - \ \((∑x\_i)\)\^2)\)\) .			(17)
		
The standard deviations of the slope and intercept are calculated using the distribution of y values about the best fit line by

			\(TraditionalForm\`S\_y\  = \ \@\(∑\(\(\((\(\((y\_i - \((mx\_i + b)\))\)\+\ \))\)\^2\)\+\ \)/\((N - 2)\)\)\)			(18)
and
			\(TraditionalForm\`S\_m\  = \ 
        S\_y\ *\ \ \@\(\(N/\((N \(∑x\_i\%2\)\  - \ \((∑x\_i)\)\^2)\)\)\+\ \)\)			(19)
and
			\(TraditionalForm\`S\_b\  = \ 
        S\_y\ *\ \ \@\(\(∑x\_i\%2/\((N \(∑x\_i\%2\)\  - \ \((∑x\_i)\)\^2)\)\)\+\ \)\) .			(20)

These standard deviations are 68% confidence intervals. Higher level intervals can be obtained using the t-distribution technique described in the discussion on confidence intervals with the degrees of freedom equal to N - 2. Note that the degrees of freedom is equal to N - 2 for a least-squares fit instead of the familiar N - 1 (Moore and McCabe, p. 653).

Example 5: Calculation of the Rate and Confidence Interval of (Subscript[SiO, 2]) Etching

Compute the etching rate of a (Subscript[SiO, 2]) film on a silicon wafer surface from the slope of a plot of the amount etched at various times. The following data are available for the etching of nominal 12,000 Å (Subscript[SiO, 2]) films on silicon substrates in 1:100 (0.55 M) HF:water solutions at 52°C. The data is given as {time, amount etched, ±95% margins of error} triplets.

\({{"t(sec)", "Amount Etched(Å)", "±95% Limits"}, {120, 384, 112}, {300, 1010, 145}, {600, 1713, 
          152}, {1200, 3945, 175}, {1800, 5770, 230}} // TableForm\)

Check that the data are indeed linear before fitting to a line. The plot in Figure 4 appears to indicate that the data are linear with time. The data points are plotted with error bars constructed from the 95% confidence limits.

exppoints = {{120, 384, 112}, {300, 1010, 145}, {600, 1713, 
            152}, {1200, 3945, 175}, {1800, 5770, 230}};

errorPlot = 
        ErrorListPlot[exppoints, 
          DefaultFont :> {"Arial", 10}, Frame -> True, 
          FrameLabel -> {"time (sec)", "Amount  Etched  (\
Å)", " ", " "}, 
          AspectRatio -> 1.0];

Figure 4: (Subscript[SiO, 2]) Etched Amounts in 0.55 M HF/water Solutions at 55°C

The 95% confidence limits on each of the points is not used to compute the error in the slope using the least-squares method. Instead the scatter in the data is used to quantify the uncertainty in the slope and intercept.

Using the equations given above for m,  (Subscript[S, m]),  b,  (Subscript[S, b]), we find that the best fit line parameters and standard deviations as follows.

NP = 5 ;

timeex5 = {(Subscript[x, 1]) -> 120, (Subscript[x, 2]) -> 300, (Subscript[x, 3]) -> 600, 
        (Subscript[x, 4]) -> 1200, (Subscript[x, 5]) -> 1800}

amtEtchedex5 = {(Subscript[y, 1]) -> 384, (Subscript[y, 2]) -> 1010, (Subscript[y, 3]) -> 1713, 
        (Subscript[y, 4]) -> 3945, (Subscript[y, 5]) -> 5770}

slope = (NP*(∑_(i = 1)^NP)((Subscript[x, i])*
                      (Subscript[y, i])) - (∑_(i = 1)^NP) (Subscript[x, i])*(∑_(i = 1)^NP) (Subscript[y, i]))/(NP*(∑_(i = 1)^NP) (Subscript[x, i]^2) - ((∑_(i = 1)^NP) (Subscript[x, i]))^2);

m = 
      Evaluate[Evaluate[slope /. timeex5] /. amtEtchedex5] // N

intercept = ((∑_(i = 1)^NP) (Subscript[x, i]^2)*(∑_(i = 1)^NP) (Subscript[y, i]) - (∑_(i = 1)^NP) (Subscript[x, i])*(∑_(i = 1)^NP)((Subscript[x, i])*
                        (Subscript[y, i])))/(NP*(∑_(i = 1)^NP) (Subscript[x, i]^2) - ((∑_(i = 1)^NP) (Subscript[x, i]))^2);

b = 
      Evaluate[Evaluate[intercept /. timeex5] /. amtEtchedex5] // 
        N

sdAmtEtchex5 = (Sqrt[((∑_(i = 1)^NP)((Subscript[y, i]) - (m*(Subscript[x, i]) + b))^2)/(NP - 2)]);

(Subscript[S, y]) = 
      Evaluate[Evaluate[sdAmtEtchex5 /. timeex5] /. amtEtchedex5]

sdSlope = 
        (Subscript[S, y])*(Sqrt[NP/(NP*(∑_(i = 1)^NP) (Subscript[x, i]^2) - ((∑_(i = 1)^NP) (Subscript[x, i]))^2)]);

(Subscript[S, m]) = 
      Evaluate[Evaluate[sdSlope /. timeex5] /. amtEtchedex5]

sdIntercept = 
        (Subscript[S, y])*(Sqrt[(∑_(i = 1)^NP) (Subscript[x, i]^2)/(NP*(∑_(i = 1)^NP) (Subscript[x, i]^2) - ((∑_(i = 1)^NP) (Subscript[x, i]))^2)]);

(Subscript[S, b]) = 
      Evaluate[Evaluate[sdIntercept /. timeex5] /. amtEtchedex5]

The value of the slope is consequently 3.24 Å/s with a standard deviation of 0.0967 Å/s. The value of the intercept is -37.2 Å with a standard deviation of 98.1 Å. The standard deviation of the y-vales is 133.7 Å. Since there are 5 data points, the number of degrees of freedom is N - 2 = 3 and using Table 1 the t-factors are 3.182 for the 95% limit and 5.841 for the 99% limit.

ConInt = tCI*(Subscript[S, m])

ConInt /. tCI -> 3.182

ConInt /. tCI -> 5.841

The margins of error for the etching rate are

		r  =   3.24   ±   0.31  Å/s  (with 95% confidence)
and
		r  =   3.24   ±   0.56  Å/s  (with 99% confidence).

The equation for the best fit line is

fitFn = m x + b

Mathematica has several built-in commands for both linear and nonlinear regression including statistical information.

dataex5 = {{120, 384}, {300, 1010}, {600, 1713}, {1200, 3945}, {1800, 
          5770}}

Fit[dataex5, {1, x}, x]

datafit = 
      Regress[dataex5, {1, x}, x, 
        RegressionReport -> {BestFit, RSquared, EstimatedVariance, 
            ParameterTable, ParameterCITable, ANOVATable}, 
        ConfidenceLevel -> 0.95]

Show[errorPlot,  
        Plot[fitFn, {x, 120, 1800}, DisplayFunction -> Identity, 
          PlotPoints -> 20, 
          PlotStyle -> {{RGBColor[1, 0, 0], Thickness[0.005]}, {}}], 
        DisplayFunction -> $DisplayFunction, 
        AspectRatio -> 1.0];

Figure 5: (Subscript[SiO, 2])Etched Thickness in 0.55 M HF/Water Solutions at 55°C and Linear Least Squares Fit of the Data (solid line).

The linear-least-squares fit gives a good representation of the five data points (Figure 5). Note that the best fit line goes through most but not all of the ranges defined by the 95% error bars of the individual data points.

NOTES

1. The curve in figure 2 provides the fraction of measurements at a given value and is the result of plotting the probability density function \(TraditionalForm\`{1/\((\(\@\(2  π\)\) σ)\)\ } 
        exp[\(-\((x - μ)\)\^2\)/\((2  σ\^2)\)]\), where μ is the mean and σ is the standard deviation. This probability density function is derived from the normal distribution. The normal distribution is also referred to as the Gaussian distribution after the mathematician C. F. Gauss (1777–1855) who developed many of its properties. The reason why the normal or Gaussian distribution successfully models many physical situations is that it is based on the premise that the variation in a value is due to a large number of small fluctuations that occur randomly in both positive and negative directions. Experimental measurements of most physical systems are subject to the same random perturbations, which explains the success of the normal or Gaussian distribution in describing experimental data.

2. A random sample means that each experimental measurement has an equal and independent chance of being selected (measured). The chance of making a particular selection (measurement) is not affected by another selection (measurement).

3. The Central Limit Theorem states that the sample mean distribution approaches the normal distribution as the sample size increases.

4. Precision and Accuracy. Precision is how reproducible a measured value is from experiment to experiment. Accuracy is how well a measured value corresponds to its true value. A measured value can be precise without being accurate, but all accurately measured values are precise.

5. Two Types of Experimental Error: Systematic and Random. Systematic errors are those that result from an instrument that is not calibrated properly or a measurement technique that causes the measurements to be consistently high or consistently low. Every effort should be made to avoid systematic errors: instruments should be calibrated regularly and the experimenter must strive to avoid bias in taking measurements. Random or independent errors are due to positive and negative fluctuations that occur due to the finite precision of most instruments and due to small changes in process parameters, such as temperature and pressure, in a real experiment.

ANNOTATED REFERENCES

Baird, D. C. Experimentation: An Introduction to Measurement Theory and Experiment Design. Underwood Cliffs, New Jersey: Prentice-Hall, Inc., 3rd ed., 1995. This text contains a good, readable discussion of statistics applied to scientific experimenting and contains sections on experimental design, experiment evaluation, and report writing as well.

Blachman, Nancy and Colin P. Williams. Mathematica: A Practical Approach. Upper Saddle River, New Jersey: Prentice Hall, 2nd ed., 1999. Excellent beginners introduction and intermediate resource to Mathematica.

Kinney, John J. Probability: An Introduction with Statistical Applications. New York: John Wiley & Sons, 1997.

Moore, David S. and George P. McCabe. Introduction to the Practice of Statistics.New York: W.H. Freeman, 1989. This is the first edition of a truly remarkable book about statistics. The basics are all covered in a very readable and interesting format complete with examples. Hopefully later editions are just as good, since the first edition is out of print.

Moore, David S. The Active Practice of Statistics. New York: W.H.Freeman, 1997. See the reference to ActivStats below.

Rabinovich, Semyon. Measurement Errors: Theory and Practice. New York: American Institute of Physics, 1995.

Ruskeepää, Heikki. Mathematica Navigator. New York: Academic Press, 1999. This book and the accompanying compact disk are an excellent reference for learning to apply the functionality of Mathematica. This work is more advanced than the book by Blachman and Williams, but is easily accessible. Highly recommended. Chapter 25 is devoted to the statistics capabilities of Mathematica and contains many useful commands. One can not learn statistical inference with this book and compact disk combination, but learning to apply the concepts and techniques of statistical inference using Mathematica can be accomplished.

Tufte, Edward R. The Visual Display of Quantitative Information. Cheshire, Connecticut: Graphics Press, 1983. This is a classic work on preparing and interpreting all sorts of graphs. If data and the meaning derived from it intrigues you, then this book will provide insight.

Velleman, Paul F. Data Desk 6.0 Statistics Guide. Ithaca, New York: Data Description, Inc., 1997. Data Desk 6 is a piece of software designed for the analysis of multivariable data. It is well documented, intuitive, and very efficient at what is was designed to do. It does not generate publishable quality graphs, but it excels at giving the user the freedom to discover relationships among variables. The statistics functions are extensive. The web site is www.datadesk.com.

Velleman, Paul F. ActivStats. Ithaca, New York: Data Description, Inc., 1997. ActivStats is a piece of software for learning and reviewing statistics. It contains both descriptive text and examples, many of which are interactive, in a very easy to use and pleasing interface. ActivStats was designed specifically to go with the book by D. Moore, The Active Practice of Statistics, 1997, W.H.Freeman, New York. A deeper understanding of confidence intervals and hypothesis testing can be obtained by playing with the interactive tools in ActivStats. Consult another reference to become familiar with the material in order to get the most out of it. Perhaps The Active Practice of Statistics by Moore or Introduction to the Practice of Statistics, which is a work that Moore coauthored.

Wolfram, Stephen. The Mathematica Book. New York: Cambridge University Press and Champaign, IL: Wolfram Media, Inc., 1999. An encyclopedic resource about Mathematica by its creator. The full text of the book is contained in the help browser in Mathematica 4. Consult it for complete details on commands.