Descriptive Statistics and Preprocessing

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Written against MATLAB R2026b. Examples labelled "Octave 11.3.0 output" were run in GNU Octave 11.3.0 using only code that Octave also accepts (mean, median, std, bounds, movmean, normalize, rescale, corrcoef, cov, and base-language code). Octave 11.3.0 has no smoothdata, isoutlier, filloutliers, rmoutliers, histcounts or discretize, so examples that use them are checked line by line against the R2026b documentation and labelled "not run here". Octave output layout differs from MATLAB’s.

Descriptive statistics summarise a data set with a few numbers: where it is centred, how spread out it is, how two variables move together. Preprocessing prepares data before analysis: it smooths noise, rescales variables to a common range and deals with outliers. Base MATLAB covers all of this without a toolbox for the common cases. Missing values are on Missing Data, the heatflow log is read on Reading and Writing Tabular Data and plotting results is on 2-D Plotting.

Basic statistics

The summary functions share one calling pattern: they work down the columns of a matrix (the first dimension whose size is not 1) unless you give a dimension.

Function Returns

mean, median, mode

Centre of the data: average, middle value, most frequent value.

std, var

Spread. Both divide by n-1 by default (sample estimate); a weight of 1 divides by n.

min, max

Smallest and largest values; a second output gives the index.

bounds

Smallest and largest value in one call: [lo,hi] = bounds(A).

range

Largest minus smallest.

The dimension and the whole-array forms are the arguments to remember: mean(A,2) averages each row, and mean(A,"all") averages every element, replacing the older mean(A(:)). The NaN flag is the third kind: "omitmissing" skips missing values and "includemissing" keeps them. As the Missing Data page explains, the older spelling "omitnan" still works. By default mean, median, std and var include missing values, so one NaN makes the result NaN, whereas min and max skip it.

Three readings of a small plate-temperature matrix, with one lost reading, were run in Octave 11.3.0 (the flag spelling "omitnan" is used because that is what Octave accepts; the MATLAB spelling "omitmissing" is equivalent):

A = [20.1 20.8 21.6; 22.4 NaN 24.1; 25.0 25.6 26.3];
disp(mean(A))                      % column means
disp(mean(A,2))                    % row means
disp(mean(A,"omitnan"))            % skip the NaN
disp(mean(A(:),"omitnan"))         % all elements (MATLAB: mean(A,"all","omitmissing"))
disp(median(A,"omitnan"))
disp(std(A,0,1,"omitnan"))
disp(var([1 2 3 4]))               % divides by n-1
disp(var([1 2 3 4],1))             % divides by n
[lo,hi] = bounds(A(:));
disp([lo hi])
disp(range([3 9 4]))
disp(mode([1 2 2 3 3]))            % ties: the smallest value wins
disp(max(A))                       % NaN is skipped by default
disp(max(A,[],"all"))
[m,i] = max([4 9 2]);
disp([m i])

Octave 11.3.0 output:

  22.5000      NaN  24.0000
  20.8333
      NaN
  25.6333
   22.500   23.200   24.000
23.238
   22.400   23.200   24.100
   2.4515   3.3941   2.3516
1.6667
1.2500
   20.100   26.300
6
2
   25.000   25.600   26.300
26.300
   9   2

The first line holds the column means (the middle one is NaN because of the lost reading), the next three lines the row means (the middle row is NaN), and the following line the column means with the gap skipped (23.2 is the mean of the two readings that exist). The max lines show that max skips NaN by default.

Source: MATLAB documentation: Descriptive Statistics and Insights, mean

Cumulative statistics and differences

cumsum returns running totals and diff returns differences between neighbours, one element shorter than its input. cummax, cummin and cumprod follow the same pattern. A running total of energy and the temperature change between consecutive samples are typical uses.

disp(cumsum([1 2 3 4]))
disp(diff([20 22 25 25 24]))

Octave 11.3.0 output:

    1    3    6   10
   2   3   0  -1

Dividing diff(T) by diff(t) gives a finite-difference rate of change, which is how a cooling rate is estimated from a log; see Integration, ODEs and Optimization for the integral counterpart.

Source: MATLAB documentation: Descriptive Statistics and Insights

Moving statistics

movmean, movmedian, movmax, movmin, movstd, movvar, movsum, movmad and movprod compute a statistic over a sliding window; movfun applies any function you give. According to the movmean reference page, a window length k that is odd is centred on the current element; an even k covers the current and previous elements; [kb kf] takes kb elements back and kf forward; and by default the window is truncated at the ends, so only the elements that exist are averaged. The "Endpoints" name-value argument changes that (for example "discard"), and SamplePoints gives uneven time stamps.

A moving median ignores a single spike that a moving mean smears out. One bad reading (35.0) in a plate sensor series:

t = [20.1 20.4 35.0 20.9 21.1 21.3 21.2 21.8];
disp(movmean(t,3))
disp(movmean(t,3,"Endpoints","discard"))
disp(movmedian(t,3))
disp(movmean(t,[2 0]))             % trailing window: this and the 2 previous

Octave 11.3.0 output:

   20.250   25.167   25.433   25.667   21.100   21.200   21.433   21.500
   25.167   25.433   25.667   21.100   21.200   21.433
   20.250   20.400   20.900   21.100   21.100   21.200   21.300   21.500
   20.100   20.250   25.167   25.433   25.667   21.100   21.200   21.433

The moving mean drags the spike across three outputs; the moving median removes it. The trailing form is the one that uses only past data, which matters for real-time filtering.

Source: MATLAB documentation: movmean

Smoothing with smoothdata

B = smoothdata(A) smooths with a moving average whose window size is chosen from the data. smoothdata(A,method) selects another method, and [B,winsize] = smoothdata(___) also returns the window it used. According to the reference page, smoothdata(A,"sgolay") uses a Savitzky-Golay filter and smoothdata(A,"movmedian",5) a five-element moving median; the robust "rlowess" method suits data with outliers, and the documentation’s smoothing example shows that a moving mean follows a noisy sinusoid but misses its valleys while the Savitzky-Golay method follows them better. NaN values are ignored by default ("includenan" keeps them), and SamplePoints smooths relative to real time stamps, because otherwise samples are treated as equally spaced integers.

t = [20.1 20.4 35.0 20.9 21.1 21.3 21.2 21.8];
[s,w] = smoothdata(t,"sgolay");     % Savitzky-Golay, window chosen automatically
sm    = smoothdata(t,"movmedian",3);
sr    = smoothdata(t,"rlowess",5);  % robust to the spike

(Not run here: Octave 11.3.0 has no smoothdata.) Use smoothdata when you want a smooth curve for display or modelling, and movmean/movmedian when you need exact control of the window. Smoothing is not free: it lowers peaks and shifts them, so never smooth the data you later test hypotheses on without saying so. Filtering in the frequency sense is on Fourier Analysis.

Source: MATLAB documentation: smoothdata, Data Smoothing and Outlier Detection

Normalising and rescaling

N = normalize(A) returns the z-score of each column: centred on 0 with standard deviation 1. normalize(A,dim) selects the dimension. normalize(A,method) offers other methods: the reference page lists "norm" (vector norm, with Inf for the infinity norm), "range", and the combination "center" with "scale", for example normalize(A,"center","median","scale","mad"). [N,C,S] = normalize(___) returns the centre and scale used, so you can apply the same transformation to new data. rescale(A,l,u) maps the data linearly onto [l,u] (default [0,1]).

t = [20.1 20.4 35.0 20.9 21.1 21.3 21.2 21.8];
z = normalize(t);
fprintf('%.4f ',z); fprintf('\n');
fprintf('%.4f %.4f\n', abs(mean(z)), std(z));
disp(normalize([2 4 6],"range"))
disp(rescale([2 4 6]))
disp(rescale([2 4 6],-1,1))

Octave 11.3.0 output:

-0.5263 -0.4661 2.4609 -0.3659 -0.3258 -0.2857 -0.3057 -0.1854
0.0000 1.0000
        0   0.5000   1.0000
        0   0.5000   1.0000
  -1   0   1

The z-score of the spike is 2.46, the only value far from zero. Normalising before comparing variables with different units (temperature in degrees, flux in watts) keeps one from dominating a distance or a fit. Compute the centre and scale on the training data only and reuse C and S on the rest.

Source: MATLAB documentation: normalize

Detecting and handling outliers

TF = isoutlier(A) returns a logical array. According to the reference page, by default an outlier is a value more than three scaled median absolute deviations (MAD) from the median; isoutlier(A,"mean") uses three standard deviations from the mean, "percentiles" takes a two-element threshold such as [10 90], and a moving method such as isoutlier(A,"movmedian",5) finds local outliers. [TF,L,U,C] = isoutlier(___) also returns the lower and upper thresholds and the centre. The documentation’s example finds the outliers 100 and 300 in [57 59 60 100 59 58 57 58 300 61 62 60 62 58 57].

Two functions act on the detection: filloutliers(A,fillmethod) replaces outliers (for example with "previous", "next", "linear", "spline" or "center"), and rmoutliers(A) removes them. The smoothing documentation example fills an injected outlier with the neighbour to its right using filloutliers(Anoise,"next") and removes it with rmoutliers(Anoise).

t = [20.1 20.4 35.0 20.9 21.1 21.3 21.2 21.8];
TF     = isoutlier(t);                  % scaled MAD rule
tFill  = filloutliers(t,"linear");      % replace by interpolation
[tKeep,TFr] = rmoutliers(t);            % drop them

(Not run here.) The default rule is easy to write in base code, which also shows what it does. The following was run in Octave 11.3.0:

t = [20.1 20.4 35.0 20.9 21.1 21.3 21.2 21.8];
med  = median(t);
madv = 1.4826*median(abs(t-med));        % scaled MAD
isout = abs(t-med) > 3*madv;
disp(isout)
clean = t; clean(isout) = NaN;
disp(clean)
x = 1:numel(t);
filled = clean;
filled(isout) = interp1(x(~isout), clean(~isout), x(isout));
disp(filled)

Octave 11.3.0 output:

  0  0  1  0  0  0  0  0
  20.1000  20.4000      NaN  20.9000  21.1000  21.3000  21.2000  21.8000
   20.100   20.400   20.650   20.900   21.100   21.300   21.200   21.800

The constant 1.4826 scales the MAD so it is comparable to a standard deviation for normal data. Whether to fill, remove or keep an outlier is a decision about the physics. A thermocouple glitch is an error; a real hot spot is the answer. Report how many points were changed.

Source: MATLAB documentation: isoutlier, Data Smoothing and Outlier Detection

Histogram counts and binning

[N,edges] = histcounts(x) counts how many values fall in each bin, choosing the bins automatically; histcounts(x,edges) uses edges you give, and discretize(x,edges) returns the bin index of every element instead of counts. These are the numeric counterparts of the histogram plot (R2014b) that replaced the older hist.

t = [20.1 20.4 35.0 20.9 21.1 21.3 21.2 21.8];
edges = [20 21 22 36];
N   = histcounts(t,edges);      % 3 4 1
bin = discretize(t,edges);      % bin number of each reading

(Not run here: Octave 11.3.0 has neither function.) The count in a base-language run, using histc, gave 3 4 1:

   3   4   1

Bins are half-open, [20,21), [21,22), except the last, which also includes its right edge.

Source: MATLAB documentation: Descriptive Statistics and Insights

Covariance and correlation

cov(a,b) returns the 2-by-2 covariance matrix of two vectors (the output shape of two-vector calls differs in old releases; check the version history of cov), and corrcoef(a,b) returns the matrix of correlation coefficients, with 1 on the diagonal. For a matrix, both treat columns as variables. A normalisation flag of 1 divides by n rather than n-1.

a = [1 2 3 4 5]; b = [2.1 3.9 6.2 7.8 10.1];
disp(corrcoef(a,b))
disp(cov(a,b))
disp(cov(a,b,1))

Octave 11.3.0 output:

   1.0000   0.9987
   0.9987   1.0000
   2.5000   4.9750
   4.9750   9.9270
   2.0000   3.9800
   3.9800   7.9416

A correlation near 1 says the two series move linearly together; it does not say one causes the other, and a single outlier can dominate it. The Descriptive Statistics and Insights topic page introduces linear correlation.

Source: MATLAB documentation: Descriptive Statistics and Insights

Base MATLAB and the Statistics and Machine Learning Toolbox

Everything above is base MATLAB. Probability distributions and fitting (fitdist, normpdf, normcdf), hypothesis tests (ttest, anova1), regression models (fitlm), clustering and classification, and summary helpers such as skewness, kurtosis and iqr need the Statistics and Machine Learning Toolbox; check the function page for its product. Base MATLAB does include polyfit/polyval regression (see Elementary Math, Polynomials and Interpolation) and the backslash least-squares solve (see Linear Algebra). Random data for simulations is on Random Numbers and Reproducibility. The product list is on Toolboxes and Add-On Products.

Pitfalls

  • Statistics of data with gaps. mean, std and friends return NaN unless you pass "omitmissing", while max and min silently skip them.

  • Wrong dimension. Matrices are summarised down columns. Samples stored in rows need dim 2.

  • Sample versus population. std and var divide by n-1 by default.

  • Smoothing before outlier removal. A spike smoothed by a moving mean spreads over its neighbours and is then harder to detect. Remove or flag outliers first, or use a median-based method.

  • Edge effects. Windows are truncated at the ends by default, so the first and last values use fewer points.

  • Uneven sampling. Without SamplePoints, smoothing assumes equally spaced samples.

  • Leakage. Compute centre and scale values on training data, then apply them to the rest.

Where older books differ

Books for MATLAB 5 to 7 compute statistics with mean, median, std and var on columns, loop for moving averages or call filter, and normalise by hand with bsxfun or repmat; implicit expansion (R2016b) removes that need ((A-mean(A))./std(A), or simply normalize). The hist function they use for counting is replaced by histogram (R2014b) and histcounts. Everything else on this page is newer than those books: the moving functions movmean and its relatives (R2016a), smoothdata, isoutlier and bounds (R2017a), rescale (R2017b), normalize (R2018a), the "all" dimension option (R2018b) and the "omitmissing" and "includemissing" flags (R2023a, the newer names of "omitnan" and "includenan"). The nanmean, nanstd and similar functions of the Statistics Toolbox are superseded by the NaN flags.

See Also

References