NAME¶

SYNOPSIS¶

DESCRIPTION¶

•

Descriptive statistical parameters (mean, minimum, maximum, standard deviation)

•

Estimates of the distribution in the form of histograms and quantiles

•

Basic testing of hypotheses

•

Probability and cumulative density functions

GENERAL PROCEDURES¶

::math::statistics::mean data

Determine the mean value of the given list of data.

::math::statistics::min data

Determine the minimum value of the given list of data.

::math::statistics::max data

Determine the maximum value of the given list of data.

::math::statistics::number data

Determine the number of non-missing data in the given list

::math::statistics::stdev data

Determine the sample standard deviation of the data in the given list

::math::statistics::var data

Determine the sample variance of the data in the given list

::math::statistics::pstdev data

Determine the population standard deviation of the data in the given list

::math::statistics::pvar data

Determine the population variance of the data in the given list

::math::statistics::median data

Determine the median of the data in the given list (Note that this requires sorting the data, which may be a costly operation)

::math::statistics::basic-stats data

Determine a list of all the descriptive parameters: mean, minimum, maximum, number of data, sample standard deviation, sample variance, population standard deviation and population variance.

(This routine is called whenever either or all of the basic statistical parameters are required. Hence all calculations are done and the relevant values are returned.)

::math::statistics::histogram limits values ?weights?

Determine histogram information for the given list of data. Returns a list consisting of the number of values that fall into each interval. (The first interval consists of all values lower than the first limit, the last interval consists of all values greater than the last limit. There is one more interval than there are limits.)

Optionally, you can use weights to influence the histogram.

::math::statistics::corr data1 data2

Determine the correlation coefficient between two sets of data.

::math::statistics::interval-mean-stdev data confidence

Return the interval containing the mean value and one containing the standard deviation with a certain level of confidence (assuming a normal distribution)

::math::statistics::t-test-mean data est_mean est_stdev confidence

Test whether the mean value of a sample is in accordance with the estimated normal distribution with a certain level of confidence. Returns 1 if the test succeeds or 0 if the mean is unlikely to fit the given distribution.

::math::statistics::test-normal data confidence

Test whether the given data follow a normal distribution with a certain level of confidence. Returns 1 if the data are normally distributed within the level of confidence, returns 0 if not. The underlying test is the Lilliefors test.

::math::statistics::lillieforsFit data

Returns the goodness of fit to a normal distribution according to Lilliefors. The higher the number, the more likely the data are indeed normally distributed. The test requires at least five data points.

::math::statistics::quantiles data confidence

Return the quantiles for a given set of data

::math::statistics::quantiles limits counts confidence

Return the quantiles based on histogram information (alternative to the call with two arguments)

::math::statistics::autocorr data

Return the autocorrelation function as a list of values (assuming equidistance between samples, about 1/2 of the number of raw data)

The correlation is determined in such a way that the first value is always 1 and all others are equal to or smaller than 1. The number of values involved will diminish as the "time" (the index in the list of returned values) increases

::math::statistics::crosscorr data1 data2

Return the cross-correlation function as a list of values (assuming equidistance between samples, about 1/2 of the number of raw data)

The correlation is determined in such a way that the values can never exceed 1 in magnitude. The number of values involved will diminish as the "time" (the index in the list of returned values) increases.

::math::statistics::mean-histogram-limits mean stdev number

Determine reasonable limits based on mean and standard deviation for a histogram Convenience function - the result is suitable for the histogram function.

::math::statistics::minmax-histogram-limits min max number

Determine reasonable limits based on a minimum and maximum for a histogram

Convenience function - the result is suitable for the histogram function.

::math::statistics::linear-model xdata ydata intercept

Determine the coefficients for a linear regression between two series of data (the model: Y = A + B*X). Returns a list of parameters describing the fit

::math::statistics::linear-residuals xdata ydata intercept

Determine the difference between actual data and predicted from the linear model.

Returns a list of the differences between the actual data and the predicted values.

::math::statistics::test-2x2 n11 n21 n12 n22

Determine if two set of samples, each from a binomial distribution, differ significantly or not (implying a different parameter).

Returns the "chi-square" value, which can be used to the determine the significance.

::math::statistics::print-2x2 n11 n21 n12 n22

Determine if two set of samples, each from a binomial distribution, differ significantly or not (implying a different parameter).

Returns a short report, useful in an interactive session.

::math::statistics::control-xbar data ?nsamples?

Determine the control limits for an xbar chart. The number of data in each subsample defaults to 4. At least 20 subsamples are required.

Returns the mean, the lower limit, the upper limit and the number of data per subsample.

::math::statistics::control-Rchart data ?nsamples?

Determine the control limits for an R chart. The number of data in each subsample (nsamples) defaults to 4. At least 20 subsamples are required.

Returns the mean range, the lower limit, the upper limit and the number of data per subsample.

::math::statistics::test-xbar control data

Determine if the data exceed the control limits for the xbar chart.

Returns a list of subsamples (their indices) that indeed violate the limits.

::math::statistics::test-Rchart control data

Determine if the data exceed the control limits for the R chart.

Returns a list of subsamples (their indices) that indeed violate the limits.

::math::statistics::test-Kruskal-Wallis confidence args

Check if the population medians of two or more groups are equal with a given confidence level, using the Kruskal-Wallis test.

::math::statistics::analyse-Kruskal-Wallis args

Compute the statistical parameters for the Kruskal-Wallis test. Returns the Kruskal-Wallis statistic and the probability that that value would occur assuming the medians of the populations are equal.

::math::statistics::group-rank args

Rank the groups of data with respect to the complete set. Returns a list consisting of the group ID, the value and the rank (possibly a rational number, in case of ties) for each data item.

::math::statistics::test-Wilcoxon sample_a sample_b

Compute the Wilcoxon test statistic to determine if two samples have the same median or not. (The statistic can be regarded as standard normal, if the sample sizes are both larger than 10. Returns the value of this statistic.

::math::statistics::spearman-rank sample_a sample_b

Return the Spearman rank correlation as an alternative to the ordinary (Pearson's) correlation coefficient. The two samples should have the same number of data.

::math::statistics::spearman-rank-extended sample_a sample_b

Return the Spearman rank correlation as an alternative to the ordinary (Pearson's) correlation coefficient as well as additional data. The two samples should have the same number of data. The procedure returns the correlation coefficient, the number of data pairs used and the z-score, an approximately standard normal statistic, indicating the significance of the correlation.

::math::statistics::kernel-density data opt -option value ...

] Return the density function based on kernel density estimation. The procedure is controlled by a small set of options, each of which is given a reasonable default.

The return value consists of three lists: the centres of the bins, the associated probability density and a list of computational parameters (begin and end of the interval, mean and standard deviation and the used bandwidth). The computational parameters can be used for further analysis.

MULTIVARIATE LINEAR REGRESSION¶

::math::statistics::tstat dof ?alpha?

Returns the value of the t-distribution t* satisfying

    P(t*)  =  1 - alpha/2
    P(-t*) =  alpha/2

for the number of degrees of freedom dof.

Given a sample of normally-distributed data x, with an estimate xbar for the mean and sbar for the standard deviation, the alpha confidence interval for the estimate of the mean can be calculated as

      ( xbar - t* sbar , xbar + t* sbar)

The return values from this procedure can be compared to an estimated t-statistic to determine whether the estimated value of a parameter is significantly different from zero at the given confidence level.

::math::statistics::mv-wls wt1 weights_and_values

Carries out a weighted least squares linear regression for the data points provided, with weights assigned to each point.

The linear model is of the form

    y = b0 + b1 * x1 + b2 * x2 ... + bN * xN + error

and each point satisfies

    yi = b0 + b1 * xi1 + b2 * xi2 + ... + bN * xiN + Residual_i

Arguments:

::math::statistics::mv-ols values

Carries out an ordinary least squares linear regression for the data points provided.

This procedure simply calls ::mvlinreg::wls with the weights set to 1.0, and returns the same information.

# Store the value of the unicode value for the "+/-" character
set pm "\u00B1"

# Provide some data
set data {{  -.67  14.18  60.03 -7.5  }
          { 36.97  15.52  34.24 14.61 }
          {-29.57  21.85  83.36 -7.   }
          {-16.9   11.79  51.67 -6.56 }
          { 14.09  16.24  36.97 -12.84}
          { 31.52  20.93  45.99 -25.4 }
          { 24.05  20.69  50.27  17.27}
          { 22.23  16.91  45.07  -4.3 }
          { 40.79  20.49  38.92  -.73 }
          {-10.35  17.24  58.77  18.78}}

# Call the ols routine
set results [::math::statistics::mv-ols $data]

# Pretty-print the results
puts "R-squared: [lindex $results 0]"
puts "Adj R-squared: [lindex $results 1]"
puts "Coefficients $pm s.e. -- \[95% confidence interval\]:"
foreach val [lindex $results 2] se [lindex $results 3] bounds [lindex $results 4] {
    set lb [lindex $bounds 0]
    set ub [lindex $bounds 1]
    puts "   $val $pm $se -- \[$lb to $ub\]"
}

STATISTICAL DISTRIBUTIONS¶

•

The normal or Gaussian distribution

•

The uniform distribution - equal probability for all data within a given interval

•

The exponential distribution - useful as a model for certain extreme-value distributions.

•

The gamma distribution - based on the incomplete Gamma integral

•

The chi-square distribution

•

The student's T distribution

•

The Poisson distribution

•

PM - binomial,F.

•

The probability density (pdf-*)

•

The cumulative density (cdf-*)

•

Quantiles for the given distribution (quantiles-*)

•

Histograms for the given distribution (histogram-*)

•

List of random values with the given distribution (random-*)

::math::statistics::pdf-normal mean stdev value

Return the probability of a given value for a normal distribution with given mean and standard deviation.

::math::statistics::pdf-exponential mean value

Return the probability of a given value for an exponential distribution with given mean.

::math::statistics::pdf-uniform xmin xmax value

Return the probability of a given value for a uniform distribution with given extremes.

::math::statistics::pdf-gamma alpha beta value

Return the probability of a given value for a Gamma distribution with given shape and rate parameters

::math::statistics::pdf-poisson mu k

Return the probability of a given number of occurrences in the same interval (k) for a Poisson distribution with given mean (mu)

::math::statistics::pdf-chisquare df value

Return the probability of a given value for a chi square distribution with given degrees of freedom

::math::statistics::pdf-student-t df value

Return the probability of a given value for a Student's t distribution with given degrees of freedom

::math::statistics::pdf-beta a b value

Return the probability of a given value for a Beta distribution with given shape parameters

::math::statistics::cdf-normal mean stdev value

Return the cumulative probability of a given value for a normal distribution with given mean and standard deviation, that is the probability for values up to the given one.

::math::statistics::cdf-exponential mean value

Return the cumulative probability of a given value for an exponential distribution with given mean.

::math::statistics::cdf-uniform xmin xmax value

Return the cumulative probability of a given value for a uniform distribution with given extremes.

::math::statistics::cdf-students-t degrees value

Return the cumulative probability of a given value for a Student's t distribution with given number of degrees.

::math::statistics::cdf-gamma alpha beta value

Return the cumulative probability of a given value for a Gamma distribution with given shape and rate parameters

::math::statistics::cdf-poisson mu k

Return the cumulative probability of a given number of occurrences in the same interval (k) for a Poisson distribution with given mean (mu)

::math::statistics::cdf-beta a b value

Return the cumulative probability of a given value for a Beta distribution with given shape parameters

::math::statistics::random-normal mean stdev number

Return a list of "number" random values satisfying a normal distribution with given mean and standard deviation.

::math::statistics::random-exponential mean number

Return a list of "number" random values satisfying an exponential distribution with given mean.

::math::statistics::random-uniform xmin xmax number

Return a list of "number" random values satisfying a uniform distribution with given extremes.

::math::statistics::random-gamma alpha beta number

Return a list of "number" random values satisfying a Gamma distribution with given shape and rate parameters

::math::statistics::random-chisquare df number

Return a list of "number" random values satisfying a chi square distribution with given degrees of freedom

::math::statistics::random-student-t df number

Return a list of "number" random values satisfying a Student's t distribution with given degrees of freedom

::math::statistics::random-beta a b number

Return a list of "number" random values satisfying a Beta distribution with given shape parameters

::math::statistics::histogram-uniform xmin xmax limits number

Return the expected histogram for a uniform distribution.

::math::statistics::incompleteGamma x p ?tol?

Evaluate the incomplete Gamma integral

                    1       / x               p-1
      P(p,x) =  --------   |   dt exp(-t) * t
                Gamma(p)  / 0

::math::statistics::incompleteBeta a b x ?tol?

Evaluate the incomplete Beta integral

DATA MANIPULATION¶

::math::statistics::filter varname data expression

Return a list consisting of the data for which the logical expression is true (this command works analogously to the command foreach).

::math::statistics::map varname data expression

Return a list consisting of the data that are transformed via the expression.

::math::statistics::samplescount varname list expression

Return a list consisting of the counts of all data in the sublists of the "list" argument for which the expression is true.

::math::statistics::subdivide

Routine PM - not implemented yet

PLOT PROCEDURES¶

::math::statistics::plot-scale canvas xmin xmax ymin ymax

Set the scale for a plot in the given canvas. All plot routines expect this function to be called first. There is no automatic scaling provided.

::math::statistics::plot-xydata canvas xdata ydata tag

Create a simple XY plot in the given canvas - the data are shown as a collection of dots. The tag can be used to manipulate the appearance.

::math::statistics::plot-xyline canvas xdata ydata tag

Create a simple XY plot in the given canvas - the data are shown as a line through the data points. The tag can be used to manipulate the appearance.

::math::statistics::plot-tdata canvas tdata tag

Create a simple XY plot in the given canvas - the data are shown as a collection of dots. The horizontal coordinate is equal to the index. The tag can be used to manipulate the appearance. This type of presentation is suitable for autocorrelation functions for instance or for inspecting the time-dependent behaviour.

::math::statistics::plot-tline canvas tdata tag

Create a simple XY plot in the given canvas - the data are shown as a line. See plot-tdata for an explanation.

::math::statistics::plot-histogram canvas counts limits tag

Create a simple histogram in the given canvas

THINGS TO DO¶

•

F-test-stdev

•

interval-mean-stdev

•

histogram-normal

•

histogram-exponential

•

test-histogram

•

test-corr

•

quantiles-*

•

fourier-coeffs

•

fourier-residuals

•

onepar-function-fit

•

onepar-function-residuals

•

plot-linear-model

•

subdivide

EXAMPLES¶

# Simple example:
# - Generate data (as a cheap way of getting some)
# - Perform statistical analysis to describe the data
#
package require math::statistics

#
# Two auxiliary procs
#
proc pause {time} {
   set wait 0
   after [expr {$time*1000}] {set ::wait 1}
   vwait wait
}

proc print-histogram {counts limits} {
   foreach count $counts limit $limits {
      if { $limit != {} } {
         puts [format "<%12.4g\t%d" $limit $count]
         set prev_limit $limit
      } else {
         puts [format ">%12.4g\t%d" $prev_limit $count]
      }
   }
}

#
# Our source of arbitrary data
#
proc generateData { data1 data2 } {
   upvar 1 $data1 _data1
   upvar 1 $data2 _data2

   set d1 0.0
   set d2 0.0
   for { set i 0 } { $i < 100 } { incr i } {
      set d1 [expr {10.0-2.0*cos(2.0*3.1415926*$i/24.0)+3.5*rand()}]
      set d2 [expr {0.7*$d2+0.3*$d1+0.7*rand()}]
      lappend _data1 $d1
      lappend _data2 $d2
   }
   return {}
}

#
# The analysis session
#
package require Tk
console show
canvas .plot1
canvas .plot2
pack   .plot1 .plot2 -fill both -side top

generateData data1 data2

puts "Basic statistics:"
set b1 [::math::statistics::basic-stats $data1]
set b2 [::math::statistics::basic-stats $data2]
foreach label {mean min max number stdev var} v1 $b1 v2 $b2 {
   puts "$label\t$v1\t$v2"
}
puts "Plot the data as function of \"time\" and against each other"
::math::statistics::plot-scale .plot1  0 100  0 20
::math::statistics::plot-scale .plot2  0 20   0 20
::math::statistics::plot-tline .plot1 $data1
::math::statistics::plot-tline .plot1 $data2
::math::statistics::plot-xydata .plot2 $data1 $data2

puts "Correlation coefficient:"
puts [::math::statistics::corr $data1 $data2]

pause 2
puts "Plot histograms"
.plot2 delete all
::math::statistics::plot-scale .plot2  0 20 0 100
set limits         [::math::statistics::minmax-histogram-limits 7 16]
set histogram_data [::math::statistics::histogram $limits $data1]
::math::statistics::plot-histogram .plot2 $histogram_data $limits

puts "First series:"
print-histogram $histogram_data $limits

pause 2
set limits         [::math::statistics::minmax-histogram-limits 0 15 10]
set histogram_data [::math::statistics::histogram $limits $data2]
::math::statistics::plot-histogram .plot2 $histogram_data $limits d2
.plot2 itemconfigure d2 -fill red

puts "Second series:"
print-histogram $histogram_data $limits

puts "Autocorrelation function:"
set  autoc [::math::statistics::autocorr $data1]
puts [::math::statistics::map $autoc {[format "%.2f" $x]}]
puts "Cross-correlation function:"
set  crossc [::math::statistics::crosscorr $data1 $data2]
puts [::math::statistics::map $crossc {[format "%.2f" $x]}]

::math::statistics::plot-scale .plot1  0 100 -1  4
::math::statistics::plot-tline .plot1  $autoc "autoc"
::math::statistics::plot-tline .plot1  $crossc "crossc"
.plot1 itemconfigure autoc  -fill green
.plot1 itemconfigure crossc -fill yellow

puts "Quantiles: 0.1, 0.2, 0.5, 0.8, 0.9"
puts "First:  [::math::statistics::quantiles $data1 {0.1 0.2 0.5 0.8 0.9}]"
puts "Second: [::math::statistics::quantiles $data2 {0.1 0.2 0.5 0.8 0.9}]"

•

There is a strong correlation between two time series, as displayed by the raw data and especially by the correlation functions.

•

Both time series show a significant periodic component

•

The histograms are not very useful in identifying the nature of the time series - they do not show the periodic nature.

BUGS, IDEAS, FEEDBACK¶

KEYWORDS¶

CATEGORY¶