Serialization: RDS, RData and JSON

R Data Import/Export observes that the easiest data to import is a simple text file, and that when data arrive in a proprietary binary format the simplest course is often to export them as text from the program that made them (R Data Import/Export: Imports). Serialization is the other direction: R talking to R, and sometimes R talking to other programs. The formats fall into three families.

Source: R Internals: Serialization Formats

The rest of the page takes the families in turn. The rule of thumb for what to use is short: saveRDS() for one R object that only R will read, JSON or CSV for anything that leaves R, and save() and load() only when several named objects must travel together and you control the file.

saveRDS() writes a single R object to a file and readRDS() returns it. The help page says that this differs from save() and load(), which save and restore one or more named objects into an environment, and that the single object can be restored under a different name: because readRDS() is a function that returns a value, the caller chooses the name, and nothing in the calling environment is overwritten. The help page’s own note on save() goes further and says that for saving single R objects, saveRDS() is mostly preferable to save(), notably because of the functional nature of readRDS() as opposed to load(). R itself uses .rds files for package metadata and for the help.search() databases, and .rds is the usual extension.

The signature is saveRDS(object, file = "", ascii = FALSE, version = NULL, compress = TRUE, refhook = NULL) and readRDS(file, refhook = NULL). The file argument is a path or a connection; ascii = TRUE writes text instead of bytes; version selects the format version (see the versions section below); compress takes TRUE (gzip), FALSE or one of "gzip", "bzip2", "xz" and "zstd"; and refhook lets a function handle environments and other reference objects (see the section on serialize()). saveRDS() returns NULL invisibly.

Source: R documentation: readRDS and saveRDS

Four things in that output are worth reading twice. First, identical(shelf_books, books) is TRUE: the round trip is exact, including the integer type of year, the NA values, the row names and every string. A CSV round trip cannot promise that (see the fidelity section below). Second, the book object came back with its class, so autoprinting called print.book() again; the method itself is not in the file, it is found by the normal S3 dispatch in the session that reads it (see S3 Classes and Generics). Third, storing several objects with saveRDS() means putting them in a list, which is a good habit anyway because the list names document the content. Fourth, the two error messages are plain: a missing file is a warning from the connection followed by an error, and reading a save() file with readRDS() fails with unknown input format, because the two functions use different headers.

The help page also documents infoRDS(), which reads the header of a file written by saveRDS() or serialize() and returns the format version, the version of R that wrote it, the minimum version that can read it, the data representation and the native encoding of the writer. It cannot tell whether a file is a serialization at all, nor whether it is valid. The versions section shows it at work.

save() writes the objects whose names you give and load() recreates them with the same names. The names to save can be given as symbols or strings in the dots, or as a character vector in list =; they are looked up in envir, which defaults to the calling environment. The arguments ascii, version, compress and compression_level mean what they mean in saveRDS(), and eval.promises and precheck are specific to save(): by default promises are forced before saving, and the existence of all the objects is checked before the file is opened, so a typo does not leave a half-written file.

Source: R documentation: load

load() returns the character vector of the names it created, invisibly, which is the only reliable way to find out what a file contained without looking inside. The warning in the help page for load() is the reason for the third and fourth blocks of that script: load(file) replaces all existing objects with the same names in the current environment, typically the workspace, and so can overwrite important data; the help page calls it considerably safer to use envir = to load into a different environment, or to attach(file), which loads it into a new entry in the search path. On this site the first of the two is the recommendation. attach() is mentioned here because old code and the help page use it; it changes the search path for every function that runs afterwards, which is the problem described in Scoping, Environments and Closures, and it only warns about masked objects instead of preventing them. The pattern in the script, e <- new.env(); load(file, envir = e) followed by e$name or get("name", envir = e), turns load() into the function-like call that readRDS() already is. The local() version in the script keeps the names out of the calling environment altogether.

load(verbose = TRUE) prints each name just before it is restored; the help page says it is mainly intended for debugging, and that a value greater than one also prints names for attributes and deeper parts. When a file’s objects hold references to a namespace that cannot be loaded (for example a function or formula created inside a package that is not installed on the reader’s machine), load() replaces the reference with the global environment and warns; the warning names the first such object, but there may be more. This behaviour was added in R 3.0.1.

The two functions are equally capable; the difference is in how they are used.

Source: R Internals: Serialization Formats

The help page for readRDS() has a warning that is easy to miss: files produced by saveRDS() (or by serialize() to a file connection) are not suitable as an interchange format between machines, for example to download from a website, while the files produced by save() carry a header that identifies the file type and so are better protected against erroneous use. In practice this advice is about robustness, not about portability of the bytes: the binary format is the same on every platform. And the conventional extension is only a convention: R does not look at it. .RData is also the name of the workspace file that R loads at startup, which is a good reason not to use that name for a data file you save by hand.

dput() writes an ASCII text representation of an object, as R code, to the console, a file or a connection. The object name is not written (unlike dump()); if the object is a function its environment is stripped, so scoping information can be lost. dget() evaluates the text and returns the object. The help page states plainly that this is not a good way to transfer objects between R sessions, that dump() is better, and that save() and saveRDS() are designed for transporting R data, work with objects that dput() does not handle, and are much faster. Its real use is to show someone an object: a data frame in a question on a forum or an issue tracker, in a form that can be pasted straight back into R.

Source: R documentation: dput and dget

The output shows how the classes are built from their parts: a data frame is a structure() of a list with row.names and class attributes (the compact row.names = c(NA, 3L) means "three automatic row names"), a factor is an integer vector with levels and class, and a Date is the number of days since 1970-01-01 with the class "Date", which is exactly the representation described in Dates and Times. That is also why dput() is the best way to report a bug involving a data frame: dput(head(books, 3)) is everything someone else needs to reproduce it, with no file to attach.

The control argument is a character vector of deparsing options. The default is c("keepNA", "keepInteger", "niceNames", "showAttributes"); control = "all" is the most thorough deparsing, control = NULL the simplest, dropping even the attributes (the book object came back as a plain list of six numbers and strings), and "useSource" shows the saved source of a function instead of deparsing it. For numbers, "digits17" writes 17 significant digits, which makes the round trip exact for doubles on most platforms, and "hexNumeric" writes the exact binary value as a hexadecimal fraction; R 4.0.0 added "digits17" to control = "all" and a new control = "exact" that guarantees numeric exactness through "hexNumeric". The default writes 15 digits, so a number like one third does not survive: the script shows identical() returning FALSE for the default and TRUE for "digits17". deparse() returns the same text as a character vector (and deparse1() as one string) and takes the same control; it breaks long output into elements of about 60 characters by default, so a data frame needs more than one.

Functions need a word of their own. dput() removes the environment, and with Rscript or options(keep.source = FALSE) the original formatting and comments are not kept either, so the function prints in R’s own layout. The last lines of the script show the damage: the closure add_two carries the value n = 2 in its environment, the deparsed copy does not, and calling it fails with object 'n' not found. This is the reason to use saveRDS() for functions, as the section on closures shows.

serialize() is the lower-level interface to the same machinery that saveRDS() and save() use: serialize(object, connection, ascii, xdr = TRUE, version = NULL, refhook = NULL). With connection = NULL it returns a raw vector, which is how an object is put in a database column, sent through a socket, hashed or compared; unserialize() accepts either a connection or a raw vector. Objects serialized to a connection can be read by readRDS() and the other way round.

Source: R documentation: serialize and unserialize

The R Internals chapter on serialization formats describes what those bytes are. A stream starts with a format marker: X for XDR, A for ASCII and B for native word order binary, each followed by a newline; then three integers giving the format version, the version of R that wrote the stream and the minimum version of R needed to read it (and, in version 3, the name of the writer’s native encoding). After that the object is written recursively. In the XDR format integers and doubles are written big-endian, so a file written on one platform reads on any other; in the ASCII format integers are written with %d, doubles with %.16g and bytes as two hexadecimal digits. Environments that are not the global environment, a package environment or a namespace are reference objects: they are written once, in a hash table, and every later reference is written as an index, which is how sharing inside one object survives a round trip.

Choosing between the variants is a matter of what the reader is. xdr = FALSE writes the machine’s own byte order; the help page notes that, because almost all systems in current use are little-endian, it avoids byte-shuffling at both ends when data go from one little-endian machine to another or between processes on the same machine, and can speed serialization and unserialization up by a factor of up to three. It is for process-to-process transfer, not for files that others read. ascii = TRUE or NA writes text; ascii = NA (from R 3.2.0) writes doubles as hexadecimal fractions so that no value is lost in a decimal conversion, and for the same reason the save() help page calls ASCII saves mainly of historical interest: they are almost always slower to read and write, compress worse and, when written in decimal, may not restore doubles exactly. For a text-mode connection the default of ascii is TRUE, and ascii = FALSE there is an error.

The warning in the help page for serialize() is worth repeating: the interface has been stable since R 2.4.0, but the format may change in future versions of R, so serialization should not be used for long-term storage of R objects. The dependable archive for a table is a text format such as CSV; the dependable archive for an R object is an RDS file together with the note of which R version wrote it, and infoRDS() records that for you.

saveRDS() and save() compress by default. compress = TRUE means gzip; the character strings "gzip", "bzip2", "xz" and "zstd" choose the algorithm, and compress = FALSE writes the plain stream. compression_level (in save()) defaults to 6 for gzip and to 9 for bzip2 and xz. The compression is handled by the connection that R opens for a file name, which is why compress is ignored when file is a connection: the connection decides. R Internals states that compression is applied to the whole file stream, including the headers, so the files can be uncompressed or re-compressed by external programs, and that load() and readRDS() read gzip, bzip2 and xz when reading from a file and gzip when reading from a connection. The save() help page adds that it is possible to compress later, with gzip, bzip2 or xz, a file saved with compress = FALSE, and that a saved file can be uncompressed and re-compressed under a different scheme (tools::resaveRdaFiles() does it from within R).

"zstd" is newer than the others: it was first supported in R 4.5.0, when R gained a zstdfile connection type, and it needs optional operating-system support, so the help pages advise against it as an interchange format. The same pages give a measurement of a 46 MB object, saved as 35 MB without compression in 2 seconds, 22 MB with gzip in 8 seconds, 19 MB with bzip2 in 13 seconds and 9.4 MB with xz in 40 seconds, and say that the figures are indicative and that xz compressed unusually well there. The script below measures four objects of our own: the two shelf data frames, a six-thousand-row copy of loans made by repeating its rows 100 times (a highly repetitive table), and a table of 5,000 random readings (data that compress poorly).

Source: R documentation: save and save.image

The table tells three stories. For the tiny books table the algorithms are within a few bytes of each other, and compression turns the 1,733 bytes of the plain stream into about 770; at this size the choice is irrelevant. For the repetitive big_loans table the picture is the textbook one: gzip makes 477,905 bytes into 15,432, bzip2 into 6,529, xz into 3,284 and zstd into 5,461. For random doubles (readings) nothing helps much, because the digits are noise: the best result is a halving. The ASCII table confirms the help page: the plain ASCII stream is larger than the binary one, and after gzip it is the same size for a small table and noticeably worse for numbers (52,890 bytes against 44,641 for readings). The magic-number block shows how the reader tells the algorithms apart: gzip starts with 1f 8b, bzip2 with BZh (42 5a 68), xz with fd 37 7a 58 5a 00, zstd with 28 b5 2f fd, and the uncompressed stream starts directly with the X marker. Note that readings also shows version 3 at work: the id column is an ALTREP sequence, which version 3 stores compactly (with version = 2 the gzip file is 57,330 bytes instead of 44,641).

For speed on large objects the help page suggests, instead of a bigger algorithm, a parallel compressor reached through a pipe() connection (pigz, pbzip2, xz -T), because a connection passed as file is exactly what makes the external tool usable; it warns that parallel compression is effective only for large objects.

An R object is more than its values. Closures, formulas and environments carry references to other environments, and serialization follows those references. The rules are the ones in R Internals: package, namespace and global environments are written by name, as pseudo-types, and every other environment is written out in full with its enclosure, its frame and its attributes.

Source: R Internals: Serialization Formats

The consequences are all visible in that output.

Objects that hold a handle owned by something outside R do not survive. A database connection from DBI, an xml2 document, a socket or a httpuv server is an external pointer to memory or to a descriptor owned by C code in the writing process, which cannot exist in the reading process; the pointer is saved and the thing it pointed at is not. The documented way to handle it is refhook, shown above, or, better, to save the data and the recipe for reconnecting (Databases with DBI).

The serialization format has been compatible for a long time, and the changes are few. Version 2 was the format from R 1.4.0 (December 2001) to R 3.5.3 (March 2019), expanded in back-compatible ways. Version 3 was introduced in R 3.5.0 and became the default in R 3.6.0. The save() help page spells out the consequence: a version number that did not change does not mean that saved files are necessarily backwards compatible. A saved image can be loaded into an earlier version of R that supports its version unless use is made of later additions (for version 2, raw vectors, external pointers and some S4 objects; long vectors, introduced in R 3.0.0, can be loaded only on 64-bit platforms).

Source: R documentation: readRDS and saveRDS

Read the first two blocks together: a default file reports version 3, the R version that wrote it, min_reader_version of "3.5.0", the xdr format and the native encoding (shown as "UTF-8" on the machine that produced this output), while a file written with version = 2 reports min_reader_version "2.3.0" and has no encoding field. The header block shows the magic numbers of save() files.

What happens when an old R meets a version-3 file is a hard stop with a message of a fixed shape. R’s unserializer reads the version number, the writer’s version and the minimum reader version from the header; a version that the running R does not know is rejected with the text cannot read workspace version V written by R W; need R M or newer. R before 3.5.0 has no branch for version 3, so an R 3.4 session that reads a version-3 file written by R 4.6.1 ends in that same error, with the wording cannot read workspace version 3 written by R 4.6.1; need R 3.5.0 or newer (the check is in src/main/serialize.c, source of the R-4-6 branch). An old R cannot be run for this page, so the script reproduces the message with the one thing R 4.6.1 can reject in the same way: a header whose version field is patched to 4, which an R that only knows versions 2 and 3 cannot read either. The remedy for sharing with an old reader is saveRDS(x, file, version = 2) or save(..., version = 2), at the cost of the ALTREP compactness; R_DEFAULT_SAVE_VERSION and R_DEFAULT_SERIALIZE_VERSION set to 2 change the default for a whole session, but R reads them once, when the first serialization takes place, so they must be set before R starts (the script starts a child Rscript process to show it). R NEWS for 3.6.0 notes that, for maximal backward compatibility, the vignette.rds and partial.rdb files that R CMD build generates stay in version 2, and that resave produces version 2 unless the original is already version 3.

Reading a serialized object is not like reading a CSV file. A CSV reader turns text into numbers and strings. A deserializer rebuilds R objects, and some R objects are code: functions, language objects, promises (delayed computations) and active bindings (functions that run each time a name is read). All of them can be saved, and all of them come back as they were. Nothing in readRDS() or load() limits what a file may contain.

Source: R documentation: makeActiveBinding and the other binding functions

The script is deliberately harmless, and it shows the two places where the code in a file runs. Reading the file ran nothing; the code ran the moment the name balance was read, and the promise ran the moment later was evaluated. A file that is read by a script and then used by it, which is what a file is for, gives the code its moment. Functions are the same: reading a function is harmless, calling it is not (the second and third blocks). None of this is a bug in R; it is what faithful serialization means. R itself has had defects in the deserializer, and the National Vulnerability Database records one: CVE-2024-27322 describes deserialization of untrusted data in R versions from 1.4.0 up to, and not including, 4.4.0, which let a crafted RDS file or R package run arbitrary code when it was interacted with (NVD record for CVE-2024-27322). Running a current R closes that particular hole and not the class of problem: the general rule stays that deserialising runs R internals on bytes chosen by whoever made the file.

The practical rules follow.

dget() deserves the same suspicion: it evaluates the file as R code, so a .R file from outside is a program.

JSON (JavaScript Object Notation, RFC 8259) is the usual format at the edge of a program: it is text, every language reads it, and it is what web APIs speak. It has six kinds of value: objects, arrays, strings, numbers, true/false and null. R has more: integers and doubles, factors, dates, NA, matrices, data frames, classes, functions and environments. A JSON library for R must therefore map a rich set of types onto a poor one and back. jsonlite (labelled package documentation) implements a class-based mapping that its author describes in a journal article (The jsonlite Package: A Practical and Consistent Mapping Between JSON Data and R Objects) and in the quickstart vignette (Getting started with JSON and jsonlite): its main strength is a bidirectional mapping between JSON and the most important R types, so that data frames round trip without manual restructuring. It has no pkgdown site; its help pages are on the CRAN reference pages, linked below. toJSON() and fromJSON() are documented as drop-in replacements for the functions of the same names in the older rjson and RJSONIO packages, with a somewhat more consistent mapping.

YAML is a text format for nested data with an emphasis on being easy for people to read and write, which is why it is used for configuration. R Data Import/Export names the yaml package as the R interface (R Data Import/Export: XML). yaml 2.3.12 (labelled package documentation) provides as.yaml() and write_yaml() to write, yaml.load() and read_yaml() to read.

Source: yaml package documentation: as.yaml

Two properties of YAML are worth knowing. The package reads yes and no as logicals (YAML 1.1 booleans), and as.yaml() therefore quotes the strings "yes" and "no" when it writes them; a country code such as no in a hand-written file turns into FALSE. And a tag !expr asks the parser to evaluate R code: yaml.load() leaves the text alone and warns that evaluating requires an explicit eval.expr = TRUE, which is the same rule as the one above for RData, applied by default. Do not pass eval.expr = TRUE to a file you did not write. The reference pages are yaml.load, read_yaml and write_yaml.

XML is the other structured text format, and R Data Import/Export notes that the XML package provides general facilities and that xml2 is another interface to the libxml2 C library; xml2 1.6.0 was installed for this page but not used, and read_xml() is where to start (xml2 package documentation: read_xml).

Parquet and Feather are columnar binary formats read by Python, Spark, DuckDB and many other systems; in R they come with the arrow package (arrow documentation for R), with write_parquet() and read_parquet() (arrow package documentation: read_parquet). arrow was not installed for this page, so nothing was run and the fragment below is only the shape of the calls. For a large table that other programs also read, a columnar file is the better interchange format than CSV or JSON (it keeps column types and is usually much smaller), and the only one here that does not require parsing text.

Source: arrow package documentation: read_parquet

R Data Import/Export lists HDF5 and netCDF packages (h5, rhdf5, RNetCDF, ncdf4) under binary data formats, for array-oriented scientific data (R Data Import/Export: Binary data formats), and spreadsheets and the formats of other statistical systems are in Reading and Writing Text Data.

Source: R documentation: save and save.image