Missing Data

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Written against MATLAB R2026b. The sessions in the "Octave 11.3.0 output" blocks were run in GNU Octave 11.3.0, using only code that Octave also accepts (isnan, logical indexing, interp1). Octave 11.3.0 has no missing, ismissing, standardizeMissing, fillmissing or rmmissing (they belong to a statistics package that was not installed), so every example that uses those functions is checked line by line against the R2026b documentation and labelled "not run here".

A missing value marks a data point that is absent, unrecorded or unusable. The documentation gives two reasons to care. A missing value can be meaningful (a sensor was switched off), but more often it is an unreliable or unusable point, and a computation that does not account for it returns a misleading answer. MATLAB handles the problem in three layers: each data type has its own standard missing value; a family of functions finds, standardises, fills and removes missing values; and many statistics functions take a flag that ignores them. Tabular data is on Tables, dates on Dates and Times and categories on Categorical Arrays.

The standard missing value of each type

There is no single missing value; the representation depends on the type.

Data type Standard missing value Notes

double, single

NaN

Created by 0/0, Inf-Inf, reading text that is not a number, or assignment.

duration, calendarDuration

NaN

Displays as NaN.

datetime

NaT (not-a-time)

Displays as NaT.

string

<missing>

A missing string is not the same as an empty string "".

categorical

<undefined>

A categorical element that belongs to no category.

cell array of character vectors

{''}

ismissing treats an empty cell as missing.

Integer types, logical, char

None

They cannot hold a missing value. int8(NaN) is 0 (Octave 11.3.0 run below; verify the MATLAB result on the integer-conversion documentation, not run here).

The missing value (R2017a) is a type-independent way to write any of these. Assigning missing to an array or table element converts it to the standard missing value of that type. The documentation example builds one vector of each type this way:

xDouble      = [missing 1 2 3 4]                                  % 1x5 double, first is NaN
xDatetime    = [missing datetime(2014,1:4,1)]                     % first is NaT
xString      = [missing "a" "b" "c" "d"]                          % first is <missing>
xCategorical = [missing categorical({'cat1' 'cat2' 'cat3' 'cat4'})]   % first is <undefined>

(Not run here.) The documentation shows the displayed values NaN, NaT, <missing> and <undefined> respectively. missing itself is not a numeric type: concatenated with a double it becomes NaN, with a string it becomes <missing>. To set elements of a table or timetable, assign missing to the element or the variable slice, as the missing reference page does for a timetable row.

Source: MATLAB documentation: Missing Data in MATLAB, missing

How NaN behaves

NaN is defined by IEEE arithmetic, and it spreads. These are the rules that cause most trouble. The results below were produced by Octave 11.3.0; the comparison, propagation, max and sort rules are the documented MATLAB behaviour too, and for unique see its reference page (not run here).

x = [1 3 NaN 4 NaN NaN 5];
disp(isnan(x))
disp(NaN == NaN)
disp(isequal(NaN,NaN))
disp(isequaln(NaN,NaN))
disp(mean(x))
disp(sum(x))
disp(max(x))
disp(sort([3 NaN 1 2]))
disp(sort([3 NaN 1 2],'descend'))
disp(unique([NaN NaN 1]))

Octave 11.3.0 output:

  0  0  1  0  1  1  0
0
0
1
NaN
NaN
5
     1     2     3   NaN
   NaN     3     2     1
     1   NaN   NaN

  • Comparison is always false. NaN == NaN is 0, NaN > 1 is 0, and NaN ~= NaN is 1. Never test with x == NaN; use isnan(x) or ismissing(x). (isequal also treats NaNs as different; isequaln treats them as equal.)

  • Arithmetic propagates. mean and sum of a vector that contains one NaN return NaN. The documentation points out that a plot silently skips the NaN points, so the picture can look fine while the average is NaN.

  • Extrema skip it. max and min ignore NaN by default (the documented default flag for them is "omitmissing"), so max(x) above is 5, while mean and sum include missing values by default. The two behaviours in one program are a classic source of confusion.

  • Sorting. sort puts NaN last for ascending order and first for descending order, as the run shows. The 'MissingPlacement' option ('first', 'last', 'auto') controls it.

  • unique keeps each NaN. The run shows two NaN values after unique, because each is "different" from the other. See Sets and Sorting.

  • Integers. int8(NaN) is 0 in the Octave run, so converting data that contains NaN to an integer type silently turns the gaps into zeros. Convert only after filling or removing.

Finding missing values

TF = ismissing(A) returns a logical array of the size of A with 1 for each standard missing value of its type. For a table it checks each variable according to its own type, so one call finds missing values in numeric, string, categorical, datetime and cell variables together. isnan is meant for numeric data.

TF = ismissing(A,indicator) replaces the default definition: the values in indicator are the only ones treated as missing, and the standard missing values are ignored unless you list them. For a numeric array, ismissing(A,[0 -99]) treats 0 and -99 as missing and does not treat NaN as missing. For a table, give a cell array with one indicator per type (the cleaning page uses ismissing(messyTable,{string(missing),NaN,-99})). TF = ismissing(___,OutputFormat=format) selects the form of the result for tables. anymissing(A) (R2022a) returns one logical value, true if any element of A is missing, which reads better than any(ismissing(A),"all") in an if.

A = [3 NaN 5 6 7 NaN NaN 9];
TF = ismissing(A)         % 0 1 0 0 0 1 1 0, the documentation result
if anymissing(A)
    disp("There are gaps")
end

(Not run here.)

Source: MATLAB documentation: ismissing, anymissing

Standardising odd placeholders

Real files use placeholders that are not standard missing values: -99, -999, "N/A", ".", Inf. B = standardizeMissing(A,indicator) replaces every value that matches indicator with the standard missing value for the data type, and it takes the same kind of indicator as ismissing, with DataVariables= to restrict it for tables.

A = [0 1 5 -99 8 3 4 -99 16];
B = standardizeMissing(A,-99)      % 0 1 5 NaN 8 3 4 NaN 16
T = standardizeMissing(T,{Inf,"N/A"})   % in every variable of a table, by type

(Not run here; the first result is the documentation’s own.) When you read the file with readtable, the TreatAsMissing name-value argument does the same job at import time, for example TreatAsMissing=[".","NA"]; any placeholder that survives import as a valid number, such as -99, still needs standardizeMissing. In base MATLAB without the function, the equivalent for numbers is a logical assignment, shown in the Octave session below.

Source: MATLAB documentation: standardizeMissing

Filling with fillmissing

F = fillmissing(A,method) replaces missing entries. By default it works along the first dimension whose size is not 1 (columns of a matrix); a dim argument selects another. [F,TF] = fillmissing(___) also returns which entries were filled. These are the choices.

Method Fills with

"constant",v

The constant v; a vector gives one constant per column, and a cell array one per table variable.

"previous", "next", "nearest"

The previous, next or nearest nonmissing value.

"linear"

Linear interpolation of the neighbouring nonmissing values.

"spline", "pchip", "makima"

Cubic interpolation: piecewise cubic spline, shape-preserving piecewise cubic, and modified Akima.

"mean", "median", "mode" (R2026a)

The mean, median or mode of the nonmissing values along the operating dimension.

"movmean" or "movmedian", window

A moving mean or median over a window length; an integer window is centred, [b f] takes b points back and f forward, and a duration is used for datetime sample points.

"knn", "knn",k

Values from the nearest rows (Euclidean distance), or the mean of the k nearest, for matrices and tables.

fillfun,gapwindow

A function handle @(xs,ts,tq) ... that you write; it receives the sample data, their locations and the missing locations.

Name-value arguments refine the fill:

  • SamplePoints gives the x-locations when the data are not equally spaced (a vector, or a table variable name); it must be sorted and unique. This is how timestamps drive the interpolation.

  • EndValues controls leading and trailing gaps; the default "extrap" applies the chosen method, and "previous", "next", "nearest", "none", or a scalar or vector can replace it.

  • MaxGap leaves gaps wider than the limit unfilled. A numeric value, or a duration for datetime sample points, is accepted.

  • DataVariables restricts a table to some variables; ReplaceValues=false appends the filled variables instead of replacing the originals.

  • MissingLocations (R2024a) is a logical array or table that says which entries count as missing, instead of the standard definition.

fillmissing has existed since R2016b (the reference page’s version history); the mean, median and mode methods were added in R2026a.

A = [1 3 NaN 4 NaN NaN 5];
fillmissing(A,"previous")      % 1 3 3 4 4 4 5   (documentation result)
fillmissing(A,"constant",0)    % 1 3 0 4 0 0 5

(Not run here.)

Source: MATLAB documentation: fillmissing

Removing with rmmissing

R = rmmissing(A) removes missing entries from a vector, and any row that contains one from a matrix or table. rmmissing(A,dim) selects rows or columns, MinNumMissing=n removes only the rows with at least n missing values, DataVariables limits the check to named table variables, and [R,TF] = rmmissing(___) returns the logical vector of removed rows or columns.

A = [1 3 NaN 6 NaN];
R = rmmissing(A)               % 1 3 6, the documentation result

(Not run here.) Removing rows of a matrix is what the logical-indexing form does in plain code: A(~any(isnan(A),2),:) keeps the complete rows. The choice between filling and removing is a decision about the data, not a coding one. Removing rows from a time series breaks equal spacing; filling invents values. Record which readings were filled (the TF output).

Source: MATLAB documentation: rmmissing

Ignoring missing values in statistics

Many summary functions can skip missing values without a separate cleaning step. A flag is passed after the data (and the dimension):

xDouble = [NaN 1 2 3 4];
sum(xDouble)                 % NaN
sum(xDouble,"omitnan")       % 10, the documentation result
mean(xDouble,"omitmissing")  % 2.5

  • "omitmissing" ignores the missing values of any supported type; "includemissing" keeps them. These generic flags are the current form. The type-specific forms "omitnan", "omitnat" and "omitundefined" (and the include counterparts) are still accepted; "omitnan" is what older code and books use. The version history of mean lists the generic flags under R2023a; check the page of each function you use.

  • For mean, sum, std and similar functions the default is to include missing values, so the answer is NaN. For max and min the default is to omit them. State the flag explicitly so the choice is visible.

In base MATLAB the same result is obtained by selecting the valid data: mean(x(~isnan(x))) (this form is also in the Octave run below). Statistics on tables are in Tables and the descriptive-statistics functions in Descriptive Statistics and Preprocessing.

Cleaning the sensor log

The heatflow scenario (The heatflow scenario) records thermocouple readings in sensors.csv. The logger writes -999 when a reading is lost and NA when a field is blank. The first listing, in plain base-language code, was run in Octave 11.3.0. It uses a single channel, readings every 5 minutes, and -999 as the placeholder:

t   = 0:5:35;
raw = [20.1 20.8 -999 22.4 -999 -999 25.0 25.6];
temp = raw;
temp(temp == -999) = NaN;           % standardise the placeholder
disp(temp)
missingNow = isnan(temp)
fprintf('%d of %d readings missing\n', nnz(missingNow), numel(temp));
good = ~missingNow;
filled = temp;
filled(missingNow) = interp1(t(good), temp(good), t(missingNow));   % linear fill
disp(filled)
fprintf('mean with gaps: %g\n', mean(temp));
fprintf('mean of valid readings: %.4f\n', mean(temp(good)));
disp(class(int8(NaN)))
disp(int8(NaN))

Octave 11.3.0 output:

  20.1000  20.8000      NaN  22.4000      NaN      NaN  25.0000  25.6000
missingNow =

  0  0  1  0  1  1  0  0

3 of 8 readings missing
   20.100   20.800   21.600   22.400   23.267   24.133   25.000   25.600
mean with gaps: NaN
mean of valid readings: 22.7800
int8
0

MATLAB shows the numbers in its own layout (the disp lines differ in spacing). The run illustrates the whole pipeline: standardise, count, fill by interpolation over the time axis, and compare the mean that includes gaps (NaN) with the mean of the valid data. Note that interp1 returns the interpolated values only inside the range of good points; a missing reading at the end of the log is not filled by interp1 in this form, whereas fillmissing with the default EndValues extrapolates.

The same job on the real log, as a table, uses the missing-data functions (MATLAB R2026b; not run here; the column names Time, SensorID and Temperature are those assumed for sensors.csv):

T = readtable("data/sensors.csv",TextType="string",TreatAsMissing=["NA" "."]);
T = standardizeMissing(T,-999,DataVariables="Temperature");

summary(T)                                    % NumMissing per variable
fprintf("%d missing readings\n",nnz(ismissing(T.Temperature)));

% Fill each sensor separately, using the timestamps as the x-axis,
% and leave gaps wider than 15 minutes as they are.
for id = unique(T.SensorID)'
    rows = T.SensorID == id;
    T.Temperature(rows) = fillmissing(T.Temperature(rows),"linear", ...
        SamplePoints=T.Time(rows),MaxGap=minutes(15));
end

T = rmmissing(T,DataVariables="Temperature");   % drop what could not be filled
avg = mean(T.Temperature,"omitmissing")

The loop fills each sensor on its own, because interpolating across rows from different sensors would mix them. The last line is shown with the flag for completeness: after rmmissing nothing is missing, so the flag has no effect. When you keep the unfilled rows, the flag is what keeps the mean from becoming NaN. Reading CSV files is on Reading and Writing Tabular Data.

Source: MATLAB documentation: Clean Messy and Missing Data in Tables

Pitfalls

  • x == NaN is never true. Use isnan or ismissing.

  • isnan on a string or datetime. isnan is for numeric data; for string, categorical and datetime data use ismissing, isundefined (categorical) or isnat (datetime).

  • The empty string is not missing. "" is a valid text value and ismissing("") is false; <missing> is the missing string. A file that uses blank fields needs TreatAsMissing or standardizeMissing(T,"").

  • mean, sum, std return NaN; max, min skip it. Pass the flag explicitly.

  • Filling across the wrong axis. A matrix is filled column by column by default. For samples stored in rows, pass dim as 2. For timestamps, pass SamplePoints; without it the points are assumed equally spaced.

  • Converting to integers. int8(NaN) is 0 (shown above). Fill or remove first.

  • Silent plots. plot skips NaN, so a gap looks like a break in the line rather than a warning. Count the missing values before plotting.

  • Filling the test set. Filling with the mean of the whole data set leaks information when you later split the data for a model; compute the fill value from the training part only.

Where older books differ

Books written for MATLAB 5 to 7 treat missing data as NaN only, test it with isnan, and remove or replace it with logical indexing (x(isnan(x)) = 0), as in the base-language code above. The missing value (R2017a), ismissing for all types, standardizeMissing, fillmissing (R2016b), rmmissing, anymissing (R2022a), NaT for dates, <missing> for strings and the "omitmissing" flag are all newer than those books. The older 'omitnan' form still works. The functions of the Statistics and Machine Learning Toolbox named nanmean, nansum and similar, which older code uses, are superseded by the "omitmissing" flag in base MATLAB.

See Also

References