Large Data, Datastores and Tall Arrays

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Written against MATLAB R2026b. GNU Octave 11.3.0 has no datastore, no tall arrays and no matfile, so the datastore, tall array and matfile examples on this page are not run here: they were checked line by line against the R2026b documentation and show no captured output. One example, a hand-written block-by-block reader built from fopen and textscan, does run in Octave and shows its captured output, because it shows by hand what a datastore automates.

Most MATLAB code assumes the whole data set fits in memory as one array or table. When it does not (one file larger than memory, or thousands of small files), there is no single tool. The documentation overview names three layers: begin with a datastore that reads small portions at a time; put a tall array on top of it to use familiar functions such as mean on data with any number of rows; for harder problems, write a MapReduce algorithm. A fourth, smaller tool, matfile, reads and writes parts of variables in a MAT-file.

Source: MATLAB documentation: Large Files and Big Data

The heatflow scenario at scale

The running heatflow scenario logs thermocouple readings in sensors.csv with the columns Time, SensorID and Temperature (see Tables and Reading and Writing Tabular Data). One day of that file reads comfortably with readtable. Imagine instead a year of logs, one file per day, sensors_2026-01-01.csv and so on, together larger than the machine’s memory. The examples below use that folder of files, and the file names are placeholders.

Terminology

A datastore is an object that refers to a file or a collection of files (or other data) and reads them in pieces. It holds the location and the reading settings, not the data.
A read returns the next piece. Subsequent calls continue from where the previous one stopped; reset goes back to the start.
ReadSize is how much one read returns. For a tabular text datastore the default is 20000 rows.
A tall array is an array with an unknown, possibly huge, number of rows, backed by a datastore. A tall table is the table form.
Deferred evaluation (the documentation also says lazy evaluation) means operations on a tall array are queued rather than executed.
gather runs the queued operations, takes the minimum number of passes through the data, and returns an ordinary in-memory result.
A pass is one full trip through the underlying data.

Datastores

A datastore is useful when each file might be too large for memory, or when the files in a collection have arbitrary names. All the files in one datastore must have the same structure and formatting: the same data type in the same order, separated by the same delimiter. Which datastore type you get depends on the data.

Type of file or data Datastore type

Type of file or data	Datastore type
Text files with column-oriented data, including CSV	`TabularTextDatastore`
Image files that `imread` supports	`ImageDatastore`
Spreadsheet files in a supported Excel format	`SpreadsheetDatastore`
Key-value data that are inputs to or outputs of `mapreduce`	`KeyValueDatastore`
Parquet files with column-oriented data	`ParquetDatastore`
Custom file formats, with a function you provide to read them	`FileDatastore`
Checkpoints of tall arrays	`TallDatastore`

Text files with column-oriented data, including CSV

TabularTextDatastore

Image files that imread supports

ImageDatastore

Spreadsheet files in a supported Excel format

SpreadsheetDatastore

Key-value data that are inputs to or outputs of mapreduce

KeyValueDatastore

Parquet files with column-oriented data

ParquetDatastore

Custom file formats, with a function you provide to read them

FileDatastore

Checkpoints of tall arrays

TallDatastore

datastore(location) picks the type from the files and returns the matching object; the type-specific functions (tabularTextDatastore, imageDatastore, fileDatastore and so on) create it directly and expose more settings. The location can be one file, a wildcard such as *.csv (all files with that extension), or a whole folder, and remote locations such as Amazon S3 or Azure Blob Storage are accepted (see the Work with Remote Data topic in the documentation).

ds = datastore("sensors_*.csv", "TreatAsMissing", "NA", "MissingValue", NaN);
ds.SelectedVariableNames = ["SensorID", "Temperature"];
preview(ds)

Not run here (no datastore in Octave). TreatAsMissing says which text means a missing value in numeric columns (the documentation example uses the text NA); SelectedVariableNames limits the table that preview, read and readall return to the variables of interest, which also saves reading time and memory. preview returns the first rows (eight in the documentation example) without reading the data into memory. The TabularTextDatastore display lists its many text-format properties (Delimiter, NumHeaderLines, TextscanFormats and others); the same options you would give detectImportOptions and readtable are mostly reached here as properties.

Source: MATLAB documentation: datastore

Reading in pieces: read, hasdata and reset

ds = datastore("sensors_*.csv", "TreatAsMissing", "NA");
ds.SelectedVariableNames = "Temperature";
ds.ReadSize = 15000;          % rows per read
reset(ds)                     % back to the start before (re)reading
hottest = -Inf;
while hasdata(ds)
    T = read(ds);             % next block as a table
    hottest = max(hottest, max(T.Temperature));
end
hottest

Not run here. read(ds) returns the next block; hasdata(ds) is true while unread data remains; reset(ds) returns the datastore to its initial state so the same data can be read again. Calling read inside a while hasdata loop, doing an intermediate calculation on each block and combining the intermediate results at the end, is the documented pattern, and the example above follows it with a running maximum (a quantity that combines correctly block by block; a plain mean of block means would be wrong if blocks differ in size). ds.ReadSize = "file" makes each read return a whole file, which is appropriate when each individual file fits in memory. [data, info] = read(ds) also returns information about the extracted data, including metadata. If everything you selected fits in memory, T = readall(ds) returns it in one go.

Source: MATLAB documentation: Getting Started with Datastore

Source: MATLAB documentation: read

What a datastore automates

The same idea, written by hand with low-level file functions (see Low-Level File I/O and the File System), runs in Octave. This program writes a 1000-row log of sensor and temperature, then reads it 300 rows at a time with textscan, keeping only a running maximum and two counters in memory.

folder = fullfile(pwd, "large_tmp");
if ~exist(folder, "dir"), mkdir(folder); end
f = fullfile(folder, "log.csv");
fid = fopen(f, "w");
fprintf(fid, "sensor,temperature\n");
for k = 1:1000
  fprintf(fid, "S%d,%.2f\n", mod(k,4)+1, 20 + mod(k*7,31)/2);
end
fclose(fid);

fid = fopen(f, "r");
fgetl(fid);                       % skip the header line
blockSize = 300;                  % rows per block, like ReadSize
runningMax = -Inf; nBlocks = 0; nRows = 0;
while ~feof(fid)
  C = textscan(fid, "%s %f", blockSize, "Delimiter", ",");
  if isempty(C{2}), break; end
  runningMax = max(runningMax, max(C{2}));
  nRows = nRows + numel(C{2});
  nBlocks = nBlocks + 1;
end
fclose(fid);
fprintf("rows=%d blocks=%d max=%.2f\n", nRows, nBlocks, runningMax);

Octave 11.3.0 output (the folder name is a scratch folder; the numbers come from the generated data):

rows=1000 blocks=4 max=35.00

The file is never held in memory: 4 blocks of 300, 300, 300 and 100 rows pass through the loop. A datastore does the same bookkeeping (the file position, the block size, moving to the next file, the variable types) and returns tables instead of cell arrays. The fopen/textscan version is also the way to read a format no datastore covers, though a FileDatastore with a custom read function is the supported route in MATLAB.

Transforming, combining and partitioning

The documentation groups further datastore functions under these headings: subset, partition or shuffle a datastore; combine or transform datastores; integrate with MapReduce and Tall Arrays; and a class hierarchy for developing a custom datastore. This page names them as pointers only; the Datastore topic page lists the functions of each group, and a custom datastore (the matlab.io.Datastore class family) belongs with classdef basics once the reading logic outgrows FileDatastore.

Source: MATLAB documentation: Datastore

Tall arrays

A tall array is an in-memory-looking array that is really backed by a datastore. Tall arrays can have any number of rows, and many core operators and functions work on them as on in-memory arrays, so A + B or mean(A) look the same; MATLAB handles the chunking in the background. t = tall(ds) creates one: the underlying type matches what read and readall of the datastore return, so a datastore that returns tables gives a tall table (or tall timetable), and one that returns numeric arrays gives a tall numeric array. The number of rows is shown as M because it is not yet known.

ds = datastore("sensors_*.csv", "TreatAsMissing", "NA");
tt = tall(ds)                       % M-by-3 tall table; nothing computed yet
valid = tt.Temperature(~isnan(tt.Temperature));
lo  = min(valid);                   % queued, not run
hi  = max(valid);                   % queued, not run
avg = mean(valid);                  % queued, not run
[lo, hi, avg] = gather(lo, hi, avg) % one evaluation, as few passes as possible

Not run here. Only the last line does work. The first lines build a description of the computation; gather then combines the queued operations where possible and takes the minimum number of passes through the data (the documentation example shows several statistics computed in "Pass 1 of 1"). Because the number of passes dominates run time on big data, request results only when necessary, and ask for several at once in one gather call, as above, so they share passes. gather is the bridge back to in-memory arrays: you cannot control an if or a while with a tall logical, but you can after it is gathered, and since gather returns the whole result, make sure that result fits in memory. A tall result that is itself large should not be gathered at all (see below).

For data with a time per row, tall timetables are supported: table2timetable on a tall table with one tall datetime variable makes a tall timetable (see Timetables). tall(A) also converts an in-memory array into a tall array, which the documentation recommends for quick prototyping and debugging, and from R2019b it lets MATLAB avoid whole-array temporary copies and work in smaller blocks.

Source: MATLAB documentation: Tall Arrays for Out-of-Memory Data

Source: MATLAB documentation: tall

Source: MATLAB documentation: gather

Saving results and checkpoints

save of a tall array saves its state but copies no data; the small MAT-file works only while the original data files remain in place. To write the data itself, write(location, TA) evaluates the tall array and writes the results to disk without bringing them into memory (unlike gather), and ds = datastore(location); TA = tall(ds); later recreates the tall array from those files. This is a checkpoint, a good way to keep the result of expensive preprocessing in a form that is more efficient to load.

cleaned = tt(~isnan(tt.Temperature), :);
write("cleaned_sensors", cleaned);       % evaluates and writes to a folder
ds2 = datastore("cleaned_sensors");      % later: reopen the checkpoint
tt2 = tall(ds2);

Not run here. The folder name is a placeholder; the write and datastore calls follow the documented checkpoint recipe, and the row-filtering line relies on tall tables accepting a tall logical row index, which the Index and View Tall Array Elements topic describes.

Which functions work

Not every function accepts a tall array. The documentation keeps a filtered function list (Function List, Tall Arrays); check it before building a pipeline, because an unsupported call errors out only when you reach it. Tall arrays are also supported by several toolboxes, for parallel and cluster execution and for machine learning; that is outside Base MATLAB and is not covered here.

Partial MAT-file access with matfile

matfile is the right tool when the data is one large variable in a MAT-file rather than a collection of files. It reads and writes parts of variables without loading the whole file, and needs a Version 7.3 file for partial access; the example, the creation rules and the caveats are on Workspace Variables and MAT-Files. It is not a datastore: it indexes into the arrays of one file, while a datastore iterates over blocks of many files.

Source: MATLAB documentation: matfile

Memory considerations

Before reaching for these tools, check whether the problem is memory use rather than data size. The documentation lists "Resolve Out of Memory Errors" and "Strategies for Efficient Use of Memory" among the topics under Large Files and Big Data, and the memory-allocation topic describes how MATLAB allocates arrays. Preallocation, the right numeric class, avoiding copies and sparse storage are covered where performance is discussed; this page does not repeat them. Memory-mapped files (memmapfile) are another route to large binary files and are only named here.

Source: MATLAB documentation: Memory Allocation

Common errors and pitfalls

Mistake What happens and what to do

Mistake	What happens and what to do
Gathering a huge tall result	`gather` returns an in-memory array, so MATLAB can run out of memory. Reduce first (filter, aggregate, take a few rows), or `write` the result to disk instead.
Calling `gather` after every line	Each call may make its own pass through the data. Queue the calculations and gather several results in one call.
Using a tall logical in `if` or `while`	Control flow needs an in-memory logical: `gather` it first.
Reading with `read` after the data is exhausted	`hasdata` is false; `read` past the end raises an error (not run here; wording varies by release). Loop with `while hasdata(ds)` and call `reset(ds)` to start again.
Forgetting `reset` before a second loop	The second loop sees no data because the datastore is already at its end.
Mixing files with different columns or delimiters in one datastore	All files in a datastore must share one structure; separate datastores, or a custom `FileDatastore` reader, are needed otherwise.
Averaging block means with unequal blocks	A mean of per-block means is wrong when block sizes differ; keep a running sum and count, or use a tall `mean`.
Moving the source files after saving a tall array	A saved tall array stores only its state; the original data files must still be at the same location.
Expecting a tall array to be faster on a small file	Tall arrays are for data too large for memory; for data that fits, an ordinary table is simpler and faster.

Gathering a huge tall result

gather returns an in-memory array, so MATLAB can run out of memory. Reduce first (filter, aggregate, take a few rows), or write the result to disk instead.

Calling gather after every line

Each call may make its own pass through the data. Queue the calculations and gather several results in one call.

Using a tall logical in if or while

Control flow needs an in-memory logical: gather it first.

Reading with read after the data is exhausted

hasdata is false; read past the end raises an error (not run here; wording varies by release). Loop with while hasdata(ds) and call reset(ds) to start again.

Forgetting reset before a second loop

The second loop sees no data because the datastore is already at its end.

Mixing files with different columns or delimiters in one datastore

All files in a datastore must share one structure; separate datastores, or a custom FileDatastore reader, are needed otherwise.

Averaging block means with unequal blocks

A mean of per-block means is wrong when block sizes differ; keep a running sum and count, or use a tall mean.

Moving the source files after saving a tall array

A saved tall array stores only its state; the original data files must still be at the same location.

Expecting a tall array to be faster on a small file

Tall arrays are for data too large for memory; for data that fits, an ordinary table is simpler and faster.

Where older books differ

Books for MATLAB R2013 and earlier know only fopen, textscan and MAT-file loading for big files; datastore and mapreduce arrived in R2014b. The block-by-block pattern above is the manual ancestor of datastore.
tall arrays and gather arrived in R2016b, after most books, and tall(A) for in-memory arrays is R2019b. Datastore and tall array details such as the properties of TabularTextDatastore have changed between releases; for example, the documentation page now displays VariableNamingRule where older pages show PreserveVariableNames.
mapreduce is still documented, with datastores as its input and tall arrays presented as the more intuitive alternative for the same kind of problem; new code starts with a datastore and tall arrays, and mapreduce is mentioned here as the route for algorithms that do not fit the tall array functions.
Names passed as 'Name' character vectors in older code are written as double-quoted strings in current code; the page examples use the comma form that the datastore documentation itself shows.