Smirnov Test of Significance of Mathematica in 
High School Word Problem Achievement
Quiz 3

Off[General::spell1, General::spell]

{ylist,zlist} = 
   Transpose[
     ReadList[
       OpenRead["d:\\Susann\\quiz3c.prn"],
       {Real, Real}]];

Exam scores of students who were taught word problem solving with pencil and paper are read into the variable "ylist". (Missing data, coded with "-1", are stripped out.)

ylist = Sort[Select[ylist,#>=0.&]]

ylist = 
{24., 28., 30., 32., 32., 40., 42.00000000000001, 

  42.00000000000001, 44.00000000000001, 

  44.00000000000001, 44.00000000000001, 

  44.00000000000001, 44.00000000000001, 

  44.00000000000001, 46.00000000000001, 

  46.00000000000001, 48.00000000000001, 

  48.00000000000001, 48.00000000000001, 50., 

  52.00000000000001, 56.00000000000001, 

  56.00000000000001, 58.00000000000001, 

  58.00000000000001, 58.00000000000001, 

  58.00000000000001, 58.00000000000001, 60., 

  62.00000000000001, 62.00000000000001, 

  62.00000000000001, 62.00000000000001, 

  62.00000000000001, 62.00000000000001, 

  64.00000000000001, 66.00000000000001, 

  66.00000000000001, 68.00000000000001, 

  68.00000000000001, 68.00000000000001, 

  68.00000000000001, 68.00000000000001, 70., 70., 70., 

  70., 70., 70., 70., 72.00000000000001, 

  72.00000000000001, 72.00000000000001, 

  74.00000000000001, 76.00000000000001, 

  76.00000000000001, 78.00000000000001, 80., 80., 80., 

  82., 82., 86., 86., 86., 88., 88., 90., 92., 92., 92., 

  92., 94., 96., 96., 96., 96., 96., 96., 98., 100., 

  100., 100., 100., 100., 100., 100., 100., 100.}

Exam scores of students who were taught word problem solving with Mathematica are read into the variable "zlist". (Missing data, coded with "-1", are stripped out.)

zlist = Sort[Select[zlist,#>=0.&]]

zlist=
{22., 32., 44.00000000000001, 46.00000000000001, 

  48.00000000000001, 50., 52.00000000000001, 

  52.00000000000001, 52.00000000000001, 

  54.00000000000001, 56.00000000000001, 60., 

  64.00000000000001, 64.00000000000001, 

  64.00000000000001, 64.00000000000001, 

  66.00000000000001, 68.00000000000001, 

  72.00000000000001, 72.00000000000001, 

  74.00000000000001, 74.00000000000001, 

  78.00000000000001, 78.00000000000001, 

  78.00000000000001, 80., 84., 84., 84., 86., 86., 86., 

  88., 88., 88., 88., 88., 90., 90., 90., 90., 92., 92., 

  92., 92., 92., 92., 92., 92., 92., 94., 94., 96., 96., 

  96., 96., 96., 96., 96., 96., 96., 96., 96., 98., 98., 

  100., 100., 100., 100., 100., 100., 100., 100., 100., 

  100., 100., 100., 100., 100., 100.}

{ny,nz} = {Length[ylist],Length[zlist]}

Define empirical (cumulative) distribution functions for the y and z data sets:

yint[x_] := Length[Select[ylist,#<=x&]]/ny

zint[x_] := Length[Select[zlist,#<=x&]]/nz

xlist = Union[ylist,zlist];

Off[General::spell, General::spell1]

The Smirnov two-sample, one-sided test statistic, for testing the hypothesis that the students who used pencil and paper scored lower, in general, than the students who used Mathematica, is T1+. It is computed as the maximum by which the "y" empirical distribution exceeds the "z" empirical distribution. (Note that if the y cumulative distribution is higher, then it is to the left as well; this happens if the "y" scores are lower than the "z" scores.)

T1plus = N[Max[yint/@xlist - zint/@xlist]]

For these large samples (greater than 20 is "large"), the test statistic is scaled by its standard deviation for comparison with a normal distribution to find the critical level.

T1plus / Sqrt[(1/ny + 1/nz)] // N

Needs["Statistics`NormalDistribution`"]

normal[x_] := CDF[NormalDistribution[0,1],x]

p = 1. - normal[T1plus / Sqrt[(1/ny + 1/nz)]]

This critical level of significance, p = 0.7%, indicates strongly that the use of Mathematica is statistically significant in improving these high school student's test scores in algebra word problems.

Plot[{yint[x],zint[x]},{x,Min[xlist]-1,Max[xlist]+1},
     PlotStyle->{{},{Dashing[{.01,.01}]}},
     AxesLabel->{
     FontForm["Exam Score (%)",{"Times-Italic",10}],
     FontForm["Empirical CDF", {"Times-Italic",10}]
     }];

Comparison of Empirical Distributions:

Quiz 3 Scores With vs. Without Computer

Solid line	=	Without Computer
Dashed line	= 	With Computer
Smirnov Test Statistic:	T1+  = .372
Critical level: 			p = .0079

Reject the null hypothesis
Accept the alternate hypothesis:
	Students who did use the computer scored significantly higher
		 than students who did not use the computer.

The above plot of the empirical distributions, solid = paper and pencil, dashed = Mathematica, shows the extent of the improvement attributable to the use of Mathematica. For example, the median score improved from 70 to 90. It is clear that these differences are important as well as statistically significant.

Needs["Statistics`DescriptiveStatistics`"]

Median[ylist]

Median[zlist]