Memory and Copy Semantics

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Checked against R 4.6.1 (2026-06-24) on 2026-10-20; R 4.6.2 is scheduled for 2026-10-29. Everything on this page is base R (the base, utils and methods packages that ship with it), every example runs, and the lintr 3.4.0 default linters were used only to check the scripts. The community packages pryr and lobstr, which older tutorials use to look at memory, and callr are named in prose and not run (pryr and lobstr are not installed in the environment used for this page). R keeps every object it is using in memory, and it makes the object-passing rules of a value-based language cheap by copying late: an assignment shares, and the copy is made at the moment one of the sharers is modified. This page explains what a value looks like in memory (the node, its header and its size), how the rule "copy when modified and shared" is decided (reference counting, introduced in R 4.0.0, watched with tracemem()), which operations modify in place and which copy, how to measure memory (object.size(), Rprofmem(), Rprof()), what the garbage collector and gc() report and do, why a process may not shrink after objects are removed, the alternative representations of the ALTREP framework (compact sequences and wrappers), and the long vectors that raised the limit of two billion elements.

Every example is a script that was run with Rscript on R 4.6.1 with warnings printed as they happen (options(warn = 1)), and the lines that start with #> are what it printed. The scripts pass the default lintr linters. Two kinds of output are edited, and the page says so where it happens. The addresses that tracemem() prints are replaced by 0x... (a real address changes on every run). The numbers in the gc() table and in the gcinfo() messages are replaced by <n>, because they depend on the machine, on the packages loaded and on the history of the session. No timing is shown: copies are counted with tracemem(), and big allocations are listed in bytes with Rprofmem(), so the results do not depend on the speed of the machine. The sizes in bytes are those of a 64-bit build of R (an object header of 48 bytes, a double of 8 bytes), and they are the same on every 64-bit platform. The names books, loans, shelf_today, loan_days and the book() constructor come from the running library-shelf scenario described on the section index: a script that needs them starts with source("scenario.R").

This page goes deeper than Assignment, Names and Values, which introduces copy-on-modify with a figure and the first tracemem() examples; read that page first if "two names, one object" is new. The speed side of these topics, timing and the profilers that sample time, belongs to Performance, Profiling and Benchmarking, and the view of a SEXP from C, with PROTECT() and allocation, belongs to Interfacing C, C++ and Fortran. Preallocation and the cost of growing a vector are covered in Vectorisation and Iteration, and the details of long vectors as a numeric type in Atomic Vectors and Types; this page connects them to what R does with memory.

How R stores an object

A node with a header

When R evaluates x <- c(1, 2, 3) it allocates a node in its heap, fills it with the three doubles and binds the name x in an environment to a pointer to that node. The pointer is what C code calls a SEXP, and the structure it points to is a SEXPREC (or, for vectors, a VECTOR_SEXPREC); R Internals calls both kinds of structure nodes (R Internals: SEXPs). Every node starts with a 64-bit header, sxpinfo, followed by three pointers: to the attributes of the object and to the previous and next node of the lists that the garbage collector keeps. A node that is not a vector (a pairlist cell, a closure, an environment, a promise, a symbol) then carries three more pointers, which is why the manual quotes 56 bytes for a node on a 64-bit platform. A vector node carries instead its length and its "true length" (the room allocated, which can be more than the length), and then the data, counted in 8-byte cells. The header says what kind of node it is, and the bits that matter for this page are these:

Header field What it holds What R code can see of it

type (5 bits)

The SEXPTYPE: NILSXP, LGLSXP, INTSXP, REALSXP, STRSXP, VECSXP, CLOSXP, ENVSXP, LANGSXP, …​

typeof()

obj (1 bit)

The node has a class attribute

is.object()

alt (1 bit)

The node is an ALTREP object, that is, its data are not an ordinary array

nothing directly; see the ALTREP sections below

gp (16 bits)

General-purpose flags: the S4 bit, the bits that lock a binding or an environment, the bit that marks an active binding, the encoding of a string

isS4(), lockBinding(), bindingIsLocked()

mark, gcgen, gccls

The garbage collector’s mark, the generation of the node and its allocation class (small vector classes, large vectors)

gc() and gcinfo() report their effects

debug, trace

The function is under debug(); the object is under trace() or tracemem()

isdebugged(), trace(), tracemem()

named (16 bits)

The field that R Internals lists under this name; since R 4.0.0 it is a reference count, the number of references to the node

the effect: whether a modification copies

Source: R Internals: Rest of header

R Internals still shows the field under its old name and says in a note that the NAMED mechanism has been replaced by reference counting; the section on the contents of a node lists, for each type, what the node points at: a closure holds its formals, its body and its environment, an environment holds a frame (a tagged pairlist or a hash table), its enclosure and the table, a promise holds a value, an expression and an environment (R Internals: The data, with the list of types in R Internals: SEXPTYPEs). The next script shows the type of the values that the language talks about every day. Atomic vectors, lists and expression vectors are vector nodes; symbols, calls, closures, environments and promises are the other kind; a formula is an ordinary call (language) whose class attribute is the bit that is.object() reads; an S4 object has type S4, which R 4.4.0 renamed OBJSXP internally and which typeof() still reports as "S4".

# Every R value is a node with a type. typeof() reports the type of the node
# that a name points at, and is.object() reads one bit of its header.
source("scenario.R")

setClass("Point", slots = c(x = "numeric"))
objects <- list(
  NULL, TRUE, 1L, 1, 1i, "a", as.raw(1L), list(1), quote(x), quote(f(x)),
  expression(x), pairlist(a = 1), function(x) x, sum, `if`, new.env(),
  y ~ x, books, new("Point", x = 1)
)
labels <- c(
  "NULL", "TRUE", "1L", "1", "1i", "\"a\"", "as.raw(1L)", "list(1)",
  "quote(x)", "quote(f(x))", "expression(x)", "pairlist(a = 1)",
  "function(x) x", "sum", "`if`", "new.env()", "y ~ x", "books",
  "new(\"Point\", x = 1)"
)
data.frame(
  value = labels,
  type = vapply(objects, typeof, character(1L)),
  has_class_attribute = vapply(objects, is.object, logical(1L))
)
#>                  value        type has_class_attribute
#> 1                 NULL        NULL               FALSE
#> 2                 TRUE     logical               FALSE
#> 3                   1L     integer               FALSE
#> 4                    1      double               FALSE
#> 5                   1i     complex               FALSE
#> 6                  "a"   character               FALSE
#> 7           as.raw(1L)         raw               FALSE
#> 8              list(1)        list               FALSE
#> 9             quote(x)      symbol               FALSE
#> 10         quote(f(x))    language               FALSE
#> 11       expression(x)  expression               FALSE
#> 12     pairlist(a = 1)    pairlist               FALSE
#> 13       function(x) x     closure               FALSE
#> 14                 sum     builtin               FALSE
#> 15                `if`     special               FALSE
#> 16           new.env() environment               FALSE
#> 17               y ~ x    language                TRUE
#> 18               books        list                TRUE
#> 19 new("Point", x = 1)          S4                TRUE

# The node types that exist, with the number of nodes of each type that the
# running session holds (the counts are not shown here, they change all the
# time).
names(memory.profile())
#>  [1] "NULL"        "symbol"      "pairlist"    "closure"     "environment"
#>  [6] "promise"     "language"    "special"     "builtin"     "char"
#> [11] "logical"     "integer"     "double"      "complex"     "character"
#> [16] "..."         "any"         "list"        "expression"  "bytecode"
#> [21] "externalptr" "weakref"     "raw"         "S4"

Source: R documentation: typeof

  • The data frame books has type list and the class bit set: a data frame is a list of columns with attributes, and the object system sits on top of the node, not inside it.

  • sum is a builtin and if is a special: both are nodes that point into a table of primitives in the interpreter and hold no code of their own. A closure (function(x) x) holds R code.

  • memory.profile() counts the nodes of each type that currently exist in the session, named by the types. The help page says that the types and their uses are listed in the C header Rinternals.h. The names above include the ones that R code rarely meets (char for the string nodes that character vectors point at, bytecode for compiled code, weakref and externalptr).

R can print the header of any object with the call .Internal(inspect(x)). It exists, and it is the quickest way to see a reference count, an (compact) sequence or a wrapper object, but it is an internal call and not part of the supported interface: its output has changed between releases (the NAM(2) that older tutorials show is REF(2) in R 4.6.1) and it prints addresses, so no example on this page depends on it. The community package lobstr offers obj_addr(), ref() and obj_size() for the same purposes, and the chapter "Names and values" of the community book Advanced R (Advanced R: Names and values, community material) builds its explanation on them; this page uses base R only.

How much a vector takes: header, data and size classes

A vector node is a header of 48 bytes on a 64-bit build followed by the data, rounded up to a multiple of 8 bytes. Small vectors are allocated from pages in classes of fixed sizes, and large vectors are allocated one by one with malloc. R Internals describes the allocation classes: non-vector nodes are class 0, "small" vector nodes are classes 1 to 5 for data of up to 8, 16, 32, 64 and 128 bytes, the larger ones are malloc-ed individually, and the small ones come from pages of about 2000 bytes (R Internals: Allocation classes). The gc() report counts the two kinds of storage separately: Ncells are the nodes (56 bytes each on a 64-bit build) and Vcells are the 8-byte cells that hold vector data. The Writing R Extensions manual puts it the same way: Ncells are used for everything but vector contents, "including all the administrative overhead for vectors such as type and length information" (Writing R Extensions: Profiling R code for memory use).

# object.size() models how much memory a vector node takes: a fixed header,
# then the data in 8-byte cells. Small vectors are rounded up to a size class.
sizes <- vapply(0:20, \(n) as.numeric(object.size(numeric(n))), numeric(1L))
data.frame(length = 0:20, bytes = sizes, data_bytes = sizes - sizes[[1L]])
#>    length bytes data_bytes
#> 1       0    48          0
#> 2       1    56          8
#> 3       2    64         16
#> 4       3    80         32
#> 5       4    80         32
#> 6       5    96         48
#> 7       6    96         48
#> 8       7   112         64
#> 9       8   112         64
#> 10      9   176        128
#> 11     10   176        128
#> 12     11   176        128
#> 13     12   176        128
#> 14     13   176        128
#> 15     14   176        128
#> 16     15   176        128
#> 17     16   176        128
#> 18     17   184        136
#> 19     18   192        144
#> 20     19   200        152
#> 21     20   208        160

# Integers and logicals take 4 bytes per element, doubles 8, and a character
# vector holds one 8-byte pointer per element to a shared string node.
size_of <- function(x) as.numeric(object.size(x))
c(
  logical = size_of(logical(100L)),
  integer = size_of(integer(100L)),
  double = size_of(numeric(100L)),
  complex = size_of(complex(100L)),
  raw = size_of(raw(100L)),
  character = size_of(character(100L)),
  list = size_of(vector("list", 100L))
)
#>   logical   integer    double   complex       raw character      list
#>       448       448       848      1648       176       904       848

# Attributes are separate nodes, so they add to the size.
x <- numeric(100L)
size_of(x)
#> [1] 848
names(x) <- paste0("v", seq_along(x))
size_of(x)
#> [1] 7408

Source: R documentation: object.size

  • object.size(numeric(0)) is 48 bytes: the header with nothing after it. One double adds 8 bytes, two add 16, and three or four both cost 32, because the data are rounded up to the next size class. The steps in the table (8, 16, 32, 48, 64 and 128 bytes of data, then 8 bytes per element beyond 16 doubles) are the size classes. object.size() models one step, 48 bytes, that the list in R Internals does not show, which is one more reason to read its result as an estimate.

  • A logical or integer takes 4 bytes per element, a double 8, a complex 16 and a raw 1. A character vector holds one 8-byte pointer per element to a string node, so character(100) is 48 + 800 bytes plus one shared empty-string node.

  • A list of 100 empty slots also holds 100 pointers. What the slots point at is counted separately.

  • Attributes are nodes of their own and add to the size: naming the 100 doubles added 6560 bytes, almost all of it the 100 name strings.

Copy-on-modify and reference counting

What happens at a modification

An assignment never copies. y <- x binds a second name to the node of x, and passing an argument to a function binds the parameter to the caller’s node without copying (a promise, covered in Lazy Evaluation and Promises). The copy is made later, when a modification is about to happen. The language definition describes a modification such as x[[2]] <- value as the call x <- "[[<-"(*tmp*, 2, value = value), with the old value of x held under the temporary name *tmp* (The R Language Definition: Subset assignment). The built-in replacement function receives the node and asks one question: is anything else able to see this node? If the answer is yes it duplicates the node, changes the duplicate and returns it, and the assignment binds x to the result; if the answer is no it changes the node it was given and returns it, and nothing is allocated. The figure in Assignment, Names and Values shows the first case. The question is answered by the reference count in the header.

Before R 4.0.0 the answer came from the NAMED field, which took the values 0, 1 and 2 (R Internals: 0 means that no other SEXP shares data with the object, and 2 or more that it should be duplicated before a change, needed or not). The field was raised when a second name appeared and was not lowered again when a name went away, so an object that had once had two names stayed marked as shared. R 4.0.0 replaced it with reference counting. The NEWS entry, under the heading Reference Counting, says that the count is used instead of NAMED for deciding when objects can be safely mutated in base C code, that this reduces the need for copying in some cases, and that almost no impact is expected on packages that follow the supported coding practices (R NEWS: Changes in R 4.0.0). The count goes up when a binding, a list element or an attribute starts to refer to the node and goes down when a binding is removed or rebound, so the question has an exact answer at the moment of a modification, with the limits described below. The Writing R Extensions section on named objects and copying still carries the old explanation of NAMED with a note that it has been replaced (Writing R Extensions: Named objects and copying). From C, the rule that follows is unchanged: .Call and .External do not copy their arguments, so code that modifies one must duplicate it first.

Reading tracemem output

tracemem(x) sets the trace bit in the header of the node that x is bound to and returns a string that identifies the node (its address, in angle brackets). From then on the interpreter prints a message whenever it duplicates that node, and the copy is traced too. The message has the form tracemem[<old address> -> <new address>]: <calls>, where the calls are the R functions running when the copy was made, innermost first (it is the stack that the interpreter saw, not a source location). The help page says that the messages go to standard output, that type coercion and the copying of arguments of .C and .Fortran are reported too, and that retracemem() marks a new object as a copy of a traced one. It also says that the function is primitive and a compile-time option, because tracing makes R run a little more slowly even when nothing is traced; the CRAN macOS and Windows builds have it, and capabilities("profmem") tells whether a given build does. R Installation and Administration lists the configure switch, --enable-memory-profiling, which compiles support for Rprofmem() and tracemem() and is off by default (R Installation and Administration: Configuration options).

In the transcripts the identifier that tracemem() prints and the addresses in the messages are shown as 0x.... Each message is one copy, which is how the scripts below count copies: they capture what tracemem() prints with capture.output() and count the lines that start with tracemem[.

# tracemem() prints one line per copy of the traced object. The names after the
# colon are the calls that were running when the copy was made.
bump <- function(v) {
  v[[1L]] <- v[[1L]] + 1
  v
}
x <- c(10, 20, 30)
tracemem(x)
#> [1] "<0x...>"

cat("read only: no copy\n")
#> read only: no copy
sum(x)
#> [1] 60

cat("a function that modifies its argument: one copy, made inside it\n")
#> a function that modifies its argument: one copy, made inside it
y <- bump(x)
#> tracemem[0x... -> 0x...]: bump

cat("a loop in lapply(): the closure modifies x once per call\n")
#> a loop in lapply(): the closure modifies x once per call
invisible(lapply(1:2, \(i) {
  x[[i]] <- 0
  sum(x)
}))
#> tracemem[0x... -> 0x...]: FUN lapply
#> tracemem[0x... -> 0x...]: FUN lapply

cat("new objects made by arithmetic: only the unary minus is reported\n")
#> new objects made by arithmetic: only the unary minus is reported
doubled <- x * 2
summed <- x + x
negated <- -x
#> tracemem[0x... -> 0x...]:

cat("x[] duplicates the whole vector and is reported\n")
#> x[] duplicates the whole vector and is reported
everything <- x[]
#> tracemem[0x... -> 0x...]:

cat("top level: x has one name again, so no copy\n")
#> top level: x has one name again, so no copy
x[[3L]] <- 99
untracemem(x)

cat("coercion to another type is reported too\n")
#> coercion to another type is reported too
counts <- c(1, 2, 3)
invisible(tracemem(counts))
storage.mode(counts) <- "integer"
#> tracemem[0x... -> 0x...]:
untracemem(counts)

Source: R documentation: tracemem

  • The read-only call sum(x) printed nothing: reading never copies.

  • y <- bump(x) printed one message and the stack ends in bump. Inside the function the parameter v is bound to the same node as the caller’s x, so the first assignment to v[[1L]] has to copy; the copy belongs to the function and x is unchanged. This is how a function can never change its argument behind the caller’s back, and the price is one copy, paid by the function that modifies.

  • In lapply() the anonymous function runs once per element, and each call modifies x in its own frame: two calls, two copies, and the stack names FUN and lapply. The global x is not changed (the function assigns to a local copy).

  • The unary minus is reported, binary arithmetic is not (x * 2 and x + x allocate a new vector without a message), and x[] is reported because it duplicates the whole vector. The Writing R Extensions section on tracing copies says that a message is also printed when a new object of the same size is created in arithmetic operations; in this build only the unary operator did. Read the list as "what duplicate does", not as "every allocation": for allocations, use Rprofmem() below.

  • Once the loop has finished, x has a single name again, so x[[3L]] <- 99 did not copy, and the coercion at the end shows the other thing tracemem() reports: storage.mode<- produced an integer vector from the double one and printed a message.

untracemem(x) clears the bit. Leaving a trace on is harmless in a script, but every copy of a traced object is traced as well, so a trace left on in library code makes every later copy print a message.

What counts as a reference

Not every extra holder of a node counts the same. The next script builds seven situations. In each of them a function receives the vector b, holds it somehow, and returns what it holds (keeper); the script then modifies b and counts the copies, once while keeper still exists and once after it has been replaced by NULL.

# How many copies does the first modification of b make, given something else
# that also holds the vector? tracemem() messages are captured and counted.
copies_on_modify <- function(hold, drop = FALSE) {
  b <- c(1, 2, 3)
  invisible(tracemem(b))
  keeper <- hold(b)
  if (drop) {
    keeper <- NULL
  }
  out <- capture.output(b[[1L]] <- 0)
  sum(startsWith(out, "tracemem[")) + 0L * length(keeper)
}

holders <- list(
  "nothing else" = \(v) NULL,
  "a second name" = \(v) v,
  "a list element" = \(v) list(v),
  "a data frame column" = \(v) data.frame(x = v),
  "an attribute" = \(v) structure(1, extra = v),
  "an environment binding" = \(v) list2env(list(x = v)),
  "a closure's variable" = \(v) {
    force(v)
    \() v
  }
)
data.frame(
  holder = names(holders),
  while_held = vapply(holders, copies_on_modify, numeric(1L)),
  after_dropped = vapply(
    holders, copies_on_modify, numeric(1L),
    drop = TRUE
  ),
  row.names = NULL
)
#>                   holder while_held after_dropped
#> 1           nothing else          0             0
#> 2          a second name          1             0
#> 3         a list element          1             1
#> 4    a data frame column          1             1
#> 5           an attribute          1             1
#> 6 an environment binding          1             1
#> 7   a closure's variable          1             1

Source: R documentation: tracemem

  • With nothing else holding the vector, the modification is in place (0 copies).

  • A second name is a reference: while keeper is bound to the same node the modification copies, and once the second name is removed (here by keeper <- NULL) the count falls back and the modification is in place again. This is the improvement that R 4.0.0 brought; with NAMED the object would have stayed marked as shared.

  • A vector that was stored in a list, in a data frame column, in an attribute, in an environment or in a closure’s frame stays counted after the holder is gone: the modification copies even though keeper is NULL. In this version of R the reference count errs on the side of copying. It can be too high, and then the cost is one extra copy; after that copy b is bound to a node that nothing else refers to, and later modifications are in place. It is not meant to be too low: the design requires that an object is never changed while another holder could still see it.

  • The behaviour of references held by containers is what R 4.6.1 does, observed with tracemem(); the manuals do not promise it, and it can change in a later release. The rule to take away is the safe one: values that have been stored somewhere are treated as shared.

When R modifies in place

Putting the pieces together, a replacement through a built-in function, such as x[[i]] <- v, x$name <- v, names(x) <- v or attr(x, "a") <- v, changes the node in place when the reference count says that the object has a single owner. That is the case when:

  • the value has just been created and bound to one name (the result of c(), numeric() or lapply(), or a function’s return value);

  • the value came out of a function that did not keep a reference to it (a local variable that was returned);

  • the value is a function’s argument and was itself a fresh value in the call (f(c(1, 2, 3))), because nothing else refers to it;

  • the object is an environment, which is never copied, and the same holds for objects built on environments (R6, Reference Classes, data.table with :=), which are modified by reference by design.

It copies when the count is above one: the argument is also bound to a name in the caller, a second name exists, the value is a column or an element that another object still refers to, or the replacement is a function written in R.

# The first modification of an object copies it only when something else can
# still see it. The helper counts the tracemem() messages of one expression.
count_copies <- function(code) {
  sum(startsWith(capture.output(code), "tracemem["))
}

# A value that was just created and bound to one name: modified in place.
fresh <- c(5, 6, 7)
invisible(tracemem(fresh))
count_copies(fresh[[1L]] <- 0)
#> [1] 0

# The result of a function is a value nobody else holds.
make <- function() {
  v <- c(5, 6, 7)
  v
}
made <- make()
invisible(tracemem(made))
count_copies(made[[1L]] <- 0)
#> [1] 0

# An argument that is a fresh value is modified in place inside the function,
# but the same function called with a named object copies it first.
change_first <- function(v) {
  invisible(tracemem(v))
  count_copies(v[[1L]] <- 0)
}
change_first(c(5, 6, 7))
#> [1] 0
named <- c(5, 6, 7)
change_first(named)
#> [1] 1

# A replacement function written in R is different from the built-in `[[<-`:
# it receives the vector as an argument that the caller also holds, so the
# first assignment inside it copies. Each call pays for one copy, even when
# the caller has a single name for the vector.
`second<-` <- function(x, value) {
  x[[2L]] <- value
  x
}
v <- c(1, 2, 3)
invisible(tracemem(v))
second(v) <- 10
#> tracemem[0x... -> 0x...]: second<-
second(v) <- 20
#> tracemem[0x... -> 0x...]: second<-
v[[2L]] <- 30

# An environment is not copied at all: a function can change it in place.
register <- new.env()
register$count <- 0
add_one <- function(env) {
  env$count <- env$count + 1
  invisible(NULL)
}
add_one(register)
add_one(register)
register$count
#> [1] 2

Source: Writing R Extensions: Named objects and copying

  • fresh and made were modified in place (0 copies), and so was the vector passed as change_first(c(5, 6, 7)), while the same function called with named, which is also bound in the caller, made one copy.

  • second<- is a function written in R. Inside it, the parameter x and the caller’s temporary *tmp* both refer to the node, so x[[2L]] <- value copies, on every call, even when the caller has a single name for the vector. The built-in [[<- in the last assignment (v[[2L]] <- 30) copied nothing. The Writing R Extensions section on named objects, in the old words of the NAMED field, says that the optimisation meant for replacement functions cannot be used in user code at present. The practical consequence is that a replacement function in R is for small objects or for code that is not in a hot loop; for a large object keep the work inside a plain [<- or in an environment.

  • The environment was changed by add_one() without any assignment in the caller, twice, and the count in register$count shows both. There is no tracemem() here because tracing an environment is an error.

The same facts can be seen without tracemem(), by listing the large allocations (the section on Rprofmem() below does it for a vector of 80 MB): the replacement function in R allocates a new 80 MB vector on every call, and the built-in one allocates none.

Environments, R6 and data.table: reference semantics on purpose

An environment is never duplicated, so what is stored in it can be updated without copying the environment; the vector in a binding still follows the usual rules. The next script keeps a vector of a million doubles in an environment and in a list.

# An environment is never copied, so what is stored in it can be updated
# without copying the environment. The vector inside follows the usual rules.
big <- numeric(1e6)
big[[1L]] <- 1

cache <- new.env()
cache$values <- big
invisible(tracemem(cache$values))

count_copies <- function(code) {
  sum(startsWith(capture.output(code), "tracemem["))
}

# big and cache$values are the same vector, so the first change copies it.
count_copies(cache$values[[2L]] <- 2)
#> [1] 1
# After that cache owns its vector: later changes are in place.
count_copies(cache$values[[3L]] <- 3)
#> [1] 0
count_copies(cache$values[[4L]] <- 4)
#> [1] 0

# The same sequence with a list in place of the environment: the list is a
# value, so putting it in a function and modifying it there copies the list
# (shallowly) as well as the vector.
registry <- list(values = big)
update_registry <- function(reg) {
  reg$values[[2L]] <- 2
  reg
}
invisible(tracemem(registry))
count_copies(registry <- update_registry(registry))
#> [1] 2

# A counter kept in an environment can be updated from any function.
loans_seen <- new.env()
loans_seen$n <- 0L
count_loan <- function() {
  loans_seen$n <- loans_seen$n + 1L
  invisible(NULL)
}
for (i in 1:3) {
  count_loan()
}
loans_seen$n
#> [1] 3

Source: R documentation: environment

  • big and cache$values were the same vector, so the first modification through the environment copied it (1 copy), and the environment’s own binding then held a private vector: the next two modifications were in place (0 copies).

  • Putting the list through a function that modifies one element copied twice: the list itself (a shallow copy of its pointers) and the vector inside it, because big still referred to the same vector.

  • A counter in an environment can be updated from any function and from any number of places. This is the mechanism behind closures that keep state (see Scoping, Environments and Closures).

R6 objects and Reference Class objects are environments, and so have these reference semantics: Reference Classes and R6 shows $clone() and $copy(), the calls that make an independent object on purpose. data.table goes further: := and the set*() functions modify a table in place, which avoids the copies of a data frame but means that y <- x followed by y[, z := 1] also changes x unless copy() is used (data.table explains the rules). A finalizer, below, can only be attached to an environment or an external pointer, for the same reason: it only makes sense for objects that are never copied.

Loops over tables: a data frame column or a list

The data frame loop is where copy-on-modify costs most in practice. The next script fills one column of a table with the number of days that each shelf loan was out, in seven different ways, and counts the copies of the container (the table of 60 loans is the shelf loans; days_out is NA where a date is missing).

# Counting the copies made by a loop that fills one column of a table, for
# different containers. The loans are the shelf loans; days_out is the loan
# duration in days, NA where a date is missing.
source("scenario.R")

count_copies <- function(code) {
  sum(startsWith(capture.output(code), "tracemem["))
}
days_out <- as.numeric(loans$return_date - loans$loan_date)
rows <- seq_along(days_out)
length(rows)
#> [1] 60

fresh_frame <- function(loan_ids) {
  frame <- data.frame(loan_id = loan_ids, days = NA_real_)
  invisible(tracemem(frame))
  frame
}

frame <- fresh_frame(loans$loan_id)
count_copies(for (i in rows) frame$days[[i]] <- days_out[[i]])
#> [1] 61

frame <- fresh_frame(loans$loan_id)
count_copies(for (i in rows) frame[["days"]][[i]] <- days_out[[i]])
#> [1] 120

frame <- fresh_frame(loans$loan_id)
count_copies(for (i in rows) frame[i, "days"] <- days_out[[i]])
#> [1] 61

# The same loop on a list of columns, which is what a data frame is made of.
columns <- list(loan_id = loans$loan_id, days = rep(NA_real_, length(rows)))
invisible(tracemem(columns))
count_copies(for (i in rows) columns$days[[i]] <- days_out[[i]])
#> [1] 0

# On a matrix, and on a plain vector that is stored back once at the end.
loan_matrix <- matrix(NA_real_, nrow = length(rows), ncol = 2L)
invisible(tracemem(loan_matrix))
count_copies(for (i in rows) loan_matrix[i, 2L] <- days_out[[i]])
#> [1] 0

frame <- fresh_frame(loans$loan_id)
days <- frame$days
count_copies({
  for (i in rows) days[[i]] <- days_out[[i]]
  frame$days <- days
})
#> [1] 2

# No loop at all: one vectorised assignment.
frame <- fresh_frame(loans$loan_id)
count_copies(frame$days <- days_out)
#> [1] 2

Source: R documentation: Extract.data.frame

Loop body Copies for 60 iterations Why

frame$days[[i]] <- v

61

$<-.data.frame is a function written in R that receives the whole table and returns a modified table: one copy of the table per iteration (and one more at the start)

frame[["days"]][[i]] <- v

120

[[<-.data.frame is also written in R, and the double bracket copies twice per iteration

frame[i, "days"] <- v

61

[<-.data.frame is also written in R and copies once

columns$days[[i]] <- v (a list)

0

built-in $<- and [[<- on a list with a single owner modify in place

loan_matrix[i, 2L] <- v (a matrix)

0

one vector with a dim attribute, modified in place

days[[i]] <- v, then frame$days <- days once

2

the loop works on a plain vector (only the table is traced); the table is rebuilt once

frame$days <- days_out (no loop)

2

the same two copies, paid once

The counts are the observation of this version of R, not a promise of the language, but the shape is the point: with a data frame the number of copies grows with the number of iterations, and each copy is a shallow copy of the table (the pointers to the columns) plus a copy of the column that changed. With a list, a matrix or a plain vector it stays at zero. The remedies are the usual ones: compute the whole column with a vectorised expression, fill a plain vector (preallocated, see Vectorisation and Iteration) and assign the column once, or collect the pieces in a list and combine them at the end. Counting copies is the quickest way to see whether a loop has this problem before reaching for a profiler.

Measuring memory

R Extensions lists three ways to profile memory use over time. The first is the sampling profiler Rprof() with memory.profiling = TRUE; the second is Rprofmem(), which writes a stack trace for every large allocation; the third is tracemem(), which follows the copies of one object. The manual notes that the second and third need R built with memory profiling, which is not the default but is used for the macOS and Windows binaries, and that each of the three can mislead (Writing R Extensions: Profiling R code for memory use). object.size() is not on that list because it measures an object, not the use of memory over time. The tools answer different questions:

Tool Question it answers What it misses

object.size(x)

About how many bytes does this one object take?

sharing between elements, environments and what they hold, compact (ALTREP) vectors

tracemem(x)

Is this object being copied, and by whom?

allocations that are not duplications of the traced object; copies of its parts

Rprofmem(file, threshold)

Which calls allocated large vectors, and how large?

memory that is reused rather than allocated; small objects below the threshold

Rprof(memory.profiling = TRUE)

How does memory in use change while the code runs?

which function allocated: the change is attributed to the function running at the end of the sampling interval

gc()

How much is in use now, and what was the peak?

what the memory is used for

object.size()

object.size(x) returns an estimate of the memory that holds x, in bytes, as an object of class object_size. Its help page is frank about the limits: the function gives a rough indication that should be reasonably accurate for atomic vectors, does not detect whether elements of a list are shared, takes sharing of strings inside one character vector into account but not between character vectors, leaves out the symbol-table entry of the name, leaves out associated space such as the environment of a function or what an external pointer points at, and says that sizes of objects that use a compact internal representation "may be over-estimated" (R documentation: object.size).

# object.size() returns an estimate in bytes as an object of class
# "object_size"; format() and print() choose the unit.
source("scenario.R")

x <- numeric(1e6)
object.size(x)
#> 8000048 bytes
class(object.size(x))
#> [1] "object_size"
as.numeric(object.size(x))
#> [1] 8000048
format(object.size(x), units = "auto")
#> [1] "7.6 Mb"
format(object.size(x), units = "MiB")
#> [1] "7.6 MiB"
format(object.size(x), units = "auto", standard = "SI")
#> [1] "8 MB"
print(object.size(x), units = "auto", standard = "IEC", digits = 3L)
#> 7.629 MiB

# Two small vectors and the shelf tables.
object.size(letters)
#> 1712 bytes
format(object.size(books), units = "auto")
#> [1] "5.4 Kb"
format(object.size(loans), units = "auto")
#> [1] "8.9 Kb"

# The strings of a column are stored once each, so repeating a value costs a
# pointer, not a string.
as.numeric(object.size(rep("shelf", 1000L)))
#> [1] 8104
as.numeric(object.size(paste0("shelf", seq_len(1000L))))
#> [1] 71256

# Size of the pieces of a data frame: the columns, the names and the
# row names (compact: stored as c(NA, -n)).
vapply(books, \(column) as.numeric(object.size(column)), numeric(1L))
#>  title author   isbn   year  genre  pages
#>   1296    848   1368    112    712    112
as.numeric(object.size(attributes(books)))
#> [1] 1112

Source: R documentation: object.size

  • print() and format() accept units and standard. With the default standard = "auto" the legacy units are used ("Kb", "Mb", "Gb", binary multiples of 1024), "MiB" and the other IEC names select the IEC standard, and standard = "SI" selects decimal multiples ("kB", "MB"). units = "auto" picks the largest unit in which the result is one or more. digits is the number of decimals. The standard and digits arguments came in R 3.4.0 and the format() method in R 3.1.0.

  • A million doubles is 8000048 bytes: 48 bytes of header and 8 million bytes of data, 7.6 Mb or 8 MB depending on the standard.

  • The 1000-element character vector with one repeated string is 8104 bytes (48, 8000 for the pointers and 56 for the one string node), and the one with 1000 different strings is 71256: the strings are the cost, and a repeated value is only a pointer.

  • A data frame is a list: object.size(books) adds the columns, the names, the class and the compact row.names (stored as c(NA, -n), not as 16 integers).

The sharing caveat is easy to see and easy to forget.

# What object.size() does not see. Each result is a number of bytes.
size_of <- function(x) as.numeric(object.size(x))

# Elements that are the same vector are counted once per element.
x <- numeric(1e6)
c(one = size_of(x), two_elements = size_of(list(x, x)))
#>          one two_elements
#>      8000048     16000160

# An environment is a handle: its contents, and what a function closes over,
# are not included.
store <- new.env()
store$values <- numeric(1e6)
c(environment = size_of(store), contents = size_of(store$values))
#> environment    contents
#>          56     8000048
make_counter <- function() {
  history <- numeric(1e6)
  function() length(history)
}
counter <- make_counter()
size_of(counter)
#> [1] 280

# Names in the global string cache are shared between vectors, but
# object.size() counts the strings of each vector separately.
words <- paste0("word", seq_len(1000L))
c(one = size_of(words), two = size_of(list(words, rev(words))))
#>    one    two
#>  64056 128176

# A compact sequence: the size is what the expanded vector would need.
size_of(1:1e7)
#> [1] 40000048
size_of(1:1e7 + 0L)
#> [1] 40000048

Source: R documentation: object.size

  • A list that holds the same vector twice reports twice the size, 16 million bytes, although only one vector exists. The earlier Design Philosophy and Coming from Other Languages page shows the same warning sign.

  • An environment is a handle of 56 bytes: the million doubles stored in it are not counted. A closure is 280 bytes although its frame holds a million doubles, for the same reason. The size of a closure or an environment is therefore never a measure of the memory it keeps alive.

  • Two character vectors that share their strings are counted separately. The 1000 strings that words and rev(words) share are counted in both.

  • 1:1e7 is a compact sequence that takes next to no memory (the next sections measure it), but object.size() reports the 40 million bytes that the expanded vector would need. The help page’s "may be over-estimated" is this case. The same call on the same numbers after a computation, 1:1e7 + 0L, is a real vector of the same size. For a number you can trust, count allocations instead.

Rprofmem(): which calls allocate

Rprofmem(filename, append = FALSE, threshold = 0) writes the call stack to the file every time R allocates a large vector or a page of memory for small objects. The help page gives the two compile-time constants: a vector is large when it is more than 128 bytes (it is then allocated with malloc), and a page for small objects is 2000 bytes. threshold is a number of bytes: only allocations of large vectors above it are reported. Calling Rprofmem(NULL) switches the profiler off. The help page points out that some of the allocations are of previously used memory and do not increase the total memory use of R, so the file is a list of requests, not of growth. The function is a compile-time option for the same reason as tracemem(): it slows R down even when it is not in use.

The file has one line per event. For a large vector, the line starts with the number of bytes, a space and a colon, and is followed by the calls on the stack, innermost first, each in double quotes; for a page of small objects it starts with new page:. The helper big_allocs() in the next script keeps the byte counts of the allocations of a million bytes or more that happen while its argument is evaluated, which turns the file into a quick test: "did this statement allocate a big vector?"

# Rprofmem() writes one line per large allocation: the number of bytes, then
# the calls that were running. big_allocs() keeps only the sizes of the
# allocations of one million bytes or more made while code is evaluated.
big_allocs <- function(code) {
  path <- tempfile()
  Rprofmem(path, threshold = 1e6)
  force(code)
  Rprofmem(NULL)
  lines <- readLines(path)
  unlink(path)
  lines <- grep("^[0-9]", lines, value = TRUE)
  as.numeric(sub(" :.*$", "", lines))
}

x <- numeric(1e7)
big_allocs(y <- numeric(1e7))
#> [1] 80000048
big_allocs(y[[1L]] <- 1)
#> numeric(0)
big_allocs(z <- x)
#> numeric(0)
big_allocs(z[[1L]] <- 1)
#> [1] 80000048
big_allocs(w <- x + 1)
#> [1] 80000048
big_allocs(w[[1L]] <- 2)
#> numeric(0)
big_allocs(sum(x))
#> numeric(0)
big_allocs(v <- x[1:10])
#> numeric(0)
big_allocs(u <- rev(x))
#> [1] 40000048 80000048

# A replacement function written in R allocates a new vector on every call;
# the built-in `[[<-` does not.
`second<-` <- function(x, value) {
  x[[2L]] <- value
  x
}
big_allocs(second(x) <- 5)
#> [1] 80000048
big_allocs(second(x) <- 6)
#> [1] 80000048
big_allocs(x[[2L]] <- 5)
#> numeric(0)

# The file itself, for one allocation (the call stack follows the colon).
path <- tempfile()
Rprofmem(path, threshold = 1e6)
make_column <- function(n) numeric(n)
column <- make_column(2e6)
Rprofmem(NULL)
cat(readLines(path), sep = "\n")
#> 16000048 :"numeric" "make_column"
unlink(path)

Source: R documentation: Rprofmem

  • y <- numeric(1e7) allocated 80000048 bytes: 8 bytes times ten million, plus the 48-byte header. numeric(0) is what the helper returns when nothing at all was that big.

  • y[[1L]] <- 1 allocated nothing (y has one name); z <- x allocated nothing (a second name); z[[1L]] <- 1 allocated 80000048 bytes (the copy that the second name forces). This is the copy-on-modify rule measured in bytes instead of counted in messages.

  • x + 1 has to allocate a result of the same size; the modification of that result in place does not allocate again; sum(x) and the extraction of ten elements allocate nothing big.

  • rev(x) allocated two vectors, 40000048 and 80000048 bytes: rev.default is x[length(x):1L], so there is the index sequence (a compact sequence that has to be expanded to index with, 10 million integers) and the result.

  • The replacement function second<-, written in R, allocated 80000048 bytes on each of two calls, while the built-in [[<- on the same vector allocated nothing: the allocation counterpart of the tracemem() result in the section on modification in place.

  • The last block shows the file itself for one allocation. make_column(2e6) allocated 16000048 bytes and the stack that followed the colon is "numeric" "make_column": numeric() was called by make_column(). The file path is a temporary file, deleted at the end.

The three profilers are complementary. tracemem() tells you that an object was duplicated and by which calls, without saying how big it was; Rprofmem() tells you how big every large allocation was and which calls asked for it, without saying which allocations were duplications; and Rprof(), next, samples the total over time. The manual’s own example is worth reading: Rprofmem("boot.memprof", threshold = 1000) around a bootstrap shows that, apart from some initial and final work, there are no vector allocations above 1000 bytes, and tracemem() on a data frame in the same example finds fifteen duplications, three for each call, all in one line of code, st$Time <- st$fit + rs[i] (Writing R Extensions: Tracking memory allocations and Tracing copies of an object); the manual’s conclusion is that data frames are slower than matrices and that this is an example of why.

Rprof() with memory.profiling

Rprof(filename, memory.profiling = TRUE) adds the memory in use to every sample of the time profiler, and summaryRprof(filename, memory = "both") reports the change in memory beside the timings; the help page says that the units are MiB and that the change is truncated at zero, and also that the profiler does not track which function actually allocated the memory. R Extensions gives the two reasons why it can mislead: the memory is attributed to the function running at the end of the sampling interval, and a garbage collection can lower the total so that a function looks cheap (Writing R Extensions: Memory statistics from Rprof). The profile below is of the growing-vector idiom from Vectorisation and Iteration. The timings and megabytes depend on the machine, so the script prints the shape of the result and one comparison. Sampling-based profiling in general, Rprof(event = ) (R 4.4.0) and profvis are covered in Performance, Profiling and Benchmarking.

# Rprof(memory.profiling = TRUE) adds the memory in use to every sample. The
# profile is written to a temporary file; only its shape is shown, because the
# timings and megabytes depend on the machine.
grow <- function(n) {
  out <- numeric(0)
  for (i in seq_len(n)) {
    out <- c(out, i)
  }
  out
}

profile_path <- tempfile()
Rprof(profile_path, memory.profiling = TRUE, interval = 0.01)
for (k in 1:3) {
  invisible(grow(30000L))
}
Rprof(NULL)
profile <- summaryRprof(profile_path, memory = "both")
unlink(profile_path)

totals <- profile$by.total
rownames(totals) <- gsub("\"", "", rownames(totals), fixed = TRUE)

names(profile)
#> [1] "by.self"         "by.total"        "sample.interval" "sampling.time"
colnames(totals)
#> [1] "total.time" "total.pct"  "mem.total"  "self.time"  "self.pct"
c("c", "grow") %in% rownames(totals)
#> [1] TRUE TRUE
max(totals$mem.total) > 10
#> [1] TRUE

Source: R documentation: summaryRprof

The result is a list whose by.self and by.total data frames have, in addition to the time columns, a mem.total column. The script prints only the names of the columns, confirms that the profile contains c (the builtin that makes a new vector on every iteration) and grow, and that the memory allocated by the call was more than 10 MiB. In a real run the column holds hundreds of MiB for 30000 iterations: the vector reaches 240 kB, and the sum of the sizes of all the intermediate vectors is quadratic.

The garbage collector and gc()

R does not free objects when their last name goes away. A tracing garbage collector decides which nodes are still reachable from the global environment, the call stack and the other roots, and reclaims the rest, when the memory in use reaches the trigger size. R Internals describes it as generational: the gcgen bit in the header separates the nodes that have survived a collection from the new ones, and there are three levels of collection. Level 0 examines only the youngest generation, level 1 the two youngest, and level 2 everything; after 20 level-0 collections the next one is level 1, after 5 level-1 collections the next is level 2, and if a collection of level n does not free 20% of the nodes (and of the vector heap) the next one is of level n+1. The R-level function gc() performs a level-2 collection. A generational collector must know when an old list is made to point at a young object, so that the young object is not freed: that is the write barrier, kept by making the SEXP opaque to extension code, which can therefore only change a list element through functions such as SET_VECTOR_ELT (R Internals: The write barrier and the garbage collector). The C side of the same rule, PROTECT() and UNPROTECT() for nodes that are not yet reachable, is the subject of Interfacing C, C++ and Fortran.

The report

gc() runs a full collection and returns the memory report as a matrix with rows Ncells and Vcells and six columns. The help page defines them: the first column of each pair is the number of cells currently in use, used, and the second the same in megabytes rounded up to the next 0.1 Mb; gc trigger is the size at which the next automatic collection starts (the Memory help page explains that the initial settings, 350k cons cells and 6 Mb of vector heap, are such trigger sizes, and that R grows and shrinks the areas with usage); and max used is the largest amount used since R started or since the last gc(reset = TRUE). If a limit has been set for either kind, a fifth column with the limit in Mb is added. In the output below the numbers are replaced by <n>.

# gc() runs a full collection and returns the memory report as a matrix.
report <- gc()
class(report)
#> [1] "matrix" "array"
dim(report)
#> [1] 2 6
dimnames(report)
#> [[1]]
#> [1] "Ncells" "Vcells"
#>
#> [[2]]
#> [1] "used"       "(Mb)"       "gc trigger" "(Mb)"       "max used"
#> [6] "(Mb)"
#>
report
#>          used (Mb) gc trigger (Mb) max used (Mb)
#> Ncells <n> <n> <n> <n> <n> <n>
#> Vcells <n> <n> <n> <n> <n> <n>

# Ncells are cons cells of 56 bytes and Vcells are cells of 8 bytes on a 64-bit
# build; the megabyte columns are the cell counts converted and rounded up to
# the next tenth of a megabyte.
mb <- report[, "used"] * c(56, 8) / 1024^2
identical(unname(ceiling(mb * 10) / 10), unname(report[, 2L]))
#> [1] TRUE

# A partial collection looks only at recently allocated objects. The report has
# the same shape.
dim(gc(full = FALSE))
#> [1] 2 6

Source: R documentation: gc

  • The columns come in pairs, a count and its size in megabytes, and the labels of the size columns are all (Mb).

  • A cons cell is 56 bytes and a vector cell 8 bytes on a 64-bit build, as the help page says, so the megabyte columns can be recomputed from the counts: the last identical() is TRUE.

  • gc(full = FALSE) examines only the more recently allocated objects. The argument was added in R 3.5.0. The help page says that an accurate report needs full = TRUE, the default.

The vector cells measure the data of vectors in 8-byte units (the name is a relic of an earlier allocator that used a vector heap, the help page says), and include small vectors in their size classes and large vectors that were allocated individually. The cons-cell count includes the 48-byte headers of vectors, as the previous section explained.

What a collection frees

The following script measures the megabytes in use before and after a 40 Mb vector exists, using the sum of the two (Mb) columns, and prints the differences rounded to the nearest 10 Mb.

# What a collection frees. mb_used() adds the megabytes in use for cons cells
# and for vectors; the differences are rounded to the nearest 10 Mb.
mb_used <- function() sum(gc()[, 2L])

before <- mb_used()
x <- numeric(5e6)
x[[1L]] <- 1
round(mb_used() - before, -1L)
#> [1] 40
rm(x)
round(mb_used() - before, -1L)
#> [1] 0

# "max used" is the high-water mark since R started or since the last
# gc(reset = TRUE): the 40 Mb vector is remembered after it has gone.
invisible(gc(reset = TRUE))
y <- numeric(5e6)
y[[1L]] <- 1
rm(y)
report <- gc()
round(report["Vcells", 6L] - report["Vcells", 2L], -1L)
#> [1] 40

Source: R documentation: gc

  • While the vector is bound to x, about 40 Mb more are in use.

  • After rm(x) the usage is back where it started: mb_used() runs gc() itself, so the object has been collected by the time the number is read. Removing the name made the object unreachable and the collection reclaimed it. Without the explicit call, the memory would have been reclaimed whenever the next automatic collection found it useful.

  • max used remembers the peak: after gc(reset = TRUE), allocating and removing another 40 Mb vector leaves the high-water mark about 40 Mb above the current usage.

The help page notes that calling gc() after a large object has been removed "may prompt R to return memory to the operating system", and also that the primary purpose of calling it is the report. These two statements are the whole case for calling it by hand: to read the report, or to get a number such as "used" that is not inflated by garbage. Automatic collection frees unreachable objects without being asked; a script that calls gc() before every allocation to "make room" only adds the cost of a full collection, and R already collects when it needs room. Memory that has been collected is not necessarily returned to the system, which is the subject of the section on memory that R does not give back, below.

gcinfo(TRUE) makes every automatic collection print a report to the message stream (standard error), and gcinfo(FALSE) switches it off; both return the previous setting. The help page shows the format of a message: a first line with the number of collections of each level and the level of this one, then the memory used for cons cells and for vectors, rounded up to 0.1 Mb and given as a percentage of the current trigger. The script allocates and discards thirty vectors of 8 MB, which is enough to trigger collections; the numbers, and how many reports there are, vary from run to run, so they are shown as <n>.

# gcinfo(TRUE) makes every automatic collection print a report on the message
# stream. Allocating and discarding thirty 8 MB vectors triggers several. The
# number of reports and their numbers vary from run to run.
gcinfo(TRUE)
#> [1] FALSE
for (i in 1:30) numeric(1e6)
#> Garbage collection <n> = <n>+<n>+<n> (level <n>) ...
#> <n> Mbytes of cons cells used (<n>%)
#> <n> Mbytes of vectors used (<n>%)
#> Garbage collection <n> = <n>+<n>+<n> (level <n>) ...
#> <n> Mbytes of cons cells used (<n>%)
#> <n> Mbytes of vectors used (<n>%)
#> Garbage collection <n> = <n>+<n>+<n> (level <n>) ...
#> <n> Mbytes of cons cells used (<n>%)
#> <n> Mbytes of vectors used (<n>%)
#> Garbage collection <n> = <n>+<n>+<n> (level <n>) ...
#> <n> Mbytes of cons cells used (<n>%)
#> <n> Mbytes of vectors used (<n>%)
gcinfo(FALSE)
#> [1] TRUE

# gc.time() returns five timings (user, system, elapsed, and two for child
# processes) spent in the collector; the values are not shown.
length(gc.time())
#> [1] 5
is.numeric(gc.time())
#> [1] TRUE

Source: R documentation: gcinfo

The number after Garbage collection is the count of collections so far, and the three numbers after the equals sign break it down by level (the number of level-0, level-1 and level-2 collections); the level in parentheses is that of the collection that has just run. The percentages are of the trigger, so a value near 100% means that a collection is about to start or has just freed little. gc.time() returns five numbers (user, system and elapsed time in garbage collection, and two for child processes, normally zero) for the time spent collecting while GC timing was enabled, which the call gc.time(on = TRUE) does; the help page warns that, because of timer resolution, the value may be an under-estimate. The values are timings and are not shown.

Finalizers

reg.finalizer(e, f, onexit = FALSE) registers a function to be called when the object e is collected, or when R ends if onexit = TRUE. The object must be an environment or an external pointer, the two kinds of object that are never copied by an assignment (a finalizer only makes sense for an object that stays one object). The help page warns that the interpreter is not re-entrant: a finalizer can be run in the middle of a computation, so there are functions it must not call (options() is the example it gives), and finalizers are scheduled at the collection but run "at a relatively safe time thereafter". Since R 3.0.3 finalizers are marked at a collection and run afterwards (R NEWS: Changes in R 3.0.3), and since R 3.5.0 they are run with interrupts suspended (R NEWS: Changes in R 3.5.0). They are the mechanism behind the cleanup of connections, of temporary files held by objects, and of the $finalize() method of R6 objects.

# A finalizer runs when the garbage collector finds that its environment is
# no longer reachable, not when the last name is removed.
holder <- new.env()
invisible(reg.finalizer(holder, \(env) cat("finalizer ran\n")))
cat("before rm\n")
#> before rm
rm(holder)
cat("after rm, before gc\n")
#> after rm, before gc
invisible(gc())
#> finalizer ran
cat("after gc\n")
#> after gc

# onexit = TRUE also runs the finalizer when R quits, if it was never collected.
survivor <- new.env()
invisible(reg.finalizer(
  survivor, \(env) cat("finalizer ran at exit\n"),
  onexit = TRUE
))
cat("end of script\n")
#> end of script
#> finalizer ran at exit

Source: R documentation: reg.finalizer

The finalizer did not run at rm(holder): removing a name only makes the environment unreachable. It ran at the next collection, which the explicit gc() forced. The second finalizer was never collected during the script, so it ran when R exited; that is the last line of output, printed after the script’s final cat().

Limits on the heap

R has no maximum size for the heap by default: the Memory help page says that the workspace is variable-sized, that there are no longer options to control its maximum size, and that the minimum can be set (the command-line options --min-vsize and --min-nsize, or the environment variables R_VSIZE and R_NSIZE, "for expert use only" in An Introduction to R (An Introduction to R: Invoking R from the command line)). A maximum for the vector heap exists since R 3.5.0, set with the environment variable R_MAX_VSIZE (NEWS 3.5.0: on macOS a default of the larger of 16 GB and the physical memory is applied unless the variable says otherwise, to avoid the R process being killed when macOS over-commits), and since R 3.6.0 it can be queried and changed from R with mem.maxVSize() (in megabytes) and mem.maxNSize() (number of nodes). The help page says that new limits below current usage are ignored, that Inf removes the limit, and that adjusting the vector heap is mostly useful on macOS. The variable R_GC_MEM_GROW, an integer from 0 to 3, controls how aggressively the heap grows when memory fills up (higher values mean fewer collections and more memory); it was added in R 3.0.0. The command-line option --max-ppsize sets the size of the pointer protection stack, which deep recursion and large calculations can exhaust.

# The maximum size of the vector heap can be limited. A request that would go
# past it fails with an error before any memory is taken.
mem.maxVSize()
#> [1] Inf
mem.maxNSize()
#> [1] Inf

mem.maxVSize(100)
#> [1] 100
tryCatch(numeric(2e7), error = conditionMessage)
#> [1] "vector memory limit of 100.0 Mb reached, see mem.maxVSize()"
x <- numeric(1e6)
length(x)
#> [1] 1000000
mem.maxVSize(Inf)
#> [1] Inf
y <- numeric(2e7)
length(y)
#> [1] 20000000

# The limit can also be set when R starts, with the environment variable
# R_MAX_VSIZE (a number of bytes, or a number with a suffix such as M or G). A
# child R process is used so that this session keeps no limit.
rscript <- file.path(R.home("bin"), "Rscript")
with_limit <- function(limit) {
  code <- paste(
    "cat(mem.maxVSize(),",
    "tryCatch(length(numeric(2e7)), error = conditionMessage))"
  )
  system2(
    rscript, c("-e", shQuote(code)),
    stdout = TRUE, env = paste0("R_MAX_VSIZE=", limit)
  )
}
with_limit("100M")
#> [1] "100 vector memory limit of 100.0 Mb reached, see mem.maxVSize()"
with_limit("1G")
#> [1] "1024 20000000"

# The size of a request that cannot be satisfied: no memory is taken, the
# allocation simply fails (here the request is 8 PiB, far beyond any address
# space).
tryCatch(numeric(2^50), error = conditionMessage)
#> [1] "cannot allocate vector of size 8388608.0 Gb"

# memory.limit() and memory.size() are stubs on every platform.
withCallingHandlers(
  memory.limit(),
  warning = \(w) {
    cat("warning:", conditionMessage(w), "\n")
    invokeRestart("muffleWarning")
  }
)
#> warning: 'memory.limit()' is no longer supported
#> [1] Inf

Source: R documentation: mem.maxVSize

  • mem.maxVSize(100) returned the new limit, 100 (Mb). The request for 160 MB then failed before any memory was taken, with the message vector memory limit of 100.0 Mb reached, see mem.maxVSize(); a vector of 8 MB, within the limit, was fine. mem.maxVSize(Inf) removed the limit.

  • The environment variable is read when R starts, so the script starts child R processes to try it. 100M gives the same limit of 100 Mb and the same error; 1G is 1024 Mb and the vector was allocated. A plain number is read as bytes.

  • A request that no memory can satisfy fails in the system allocator, with the other message of this family: cannot allocate vector of size 8388608.0 Gb is the size of 250 doubles. The message appears when malloc fails, so it depends on the limits of the process and the machine: a request that the system agrees to (it may over-commit) and that the machine cannot back with real memory can get the process killed by the operating system instead of producing an error, and that is not something a tryCatch() can handle. The example uses a size that no system can grant.

  • memory.limit(), and memory.size(), which older tutorials use to raise the memory available to R on Windows, are stubs on every platform since R 4.2.0: the help page says they "report infinity" with a warning. The second argument in memory.limit(size = 4000) does nothing.

Memory that R does not give back

A common report is "I removed the objects, ran gc(), and top still shows the R process using a lot of memory". R FAQ 7.42, "Why is R apparently not releasing memory?", explains that it is an effect of the way the operating system allocates memory, not of R. Tools such as ps and top report what the system has reserved for the process, not what the process is using. Most systems allocate memory in two ways. Large chunks are mapped into the address space with mmap, and such a chunk can be released as soon as it is completely free. Small pieces come from the data segment of the process, which is one contiguous area that the system can only shrink from its end, never by punching holes in it. Even one byte still in use at the end of that area stops the system releasing any of it. The FAQ points out that this is more common than it sounds, because a typical computation allocates many intermediate objects, then allocates its result, then removes the intermediates; the result sits at the end. The FAQ adds that this is not necessarily a problem, because the system can page out the unused part, that small objects (strings, pairlists) are the ones affected and large vectors are not, and that on Linux the mallinfo system call (and the mallinfo package) can report the allocator’s layout (R FAQ 7.42: Why is R apparently not releasing memory?).

The next script reproduces both halves of the explanation on Linux, where the resident size of the process is in /proc/self/status; on other systems the if is false and the script prints nothing. It compares sizes instead of printing them, because the sizes depend on the machine and the allocator.

# R FAQ 7.42: freed memory is not always given back to the operating system.
# The script reads the resident set size of the R process, so it works only
# where /proc/self/status exists (Linux). The sizes are compared, not printed:
# the exact numbers depend on the machine and on the allocator.
resident_mb <- function() {
  status <- readLines("/proc/self/status")
  line <- grep("^VmRSS", status, value = TRUE)
  as.numeric(sub("^VmRSS:\\s+([0-9]+) kB$", "\\1", line)) / 1024
}

report <- function(label, outcome) {
  cat(label, outcome, "\n")
}

if (file.exists("/proc/self/status")) {
  start <- resident_mb()

  # One large vector (80 MB) is requested from the system on its own and is
  # returned to it when it is freed.
  big <- numeric(1e7)
  big[] <- 1
  grown_by_big <- resident_mb() - start
  rm(big)
  invisible(gc())
  left_after_big <- resident_mb() - start
  report("large vector, process grew by more than 50 Mb:", grown_by_big > 50)
  report("large vector, back within 10 Mb of the start:", left_after_big < 10)

  # Half a million small vectors live in memory that the allocator keeps in
  # one contiguous area. A result allocated after them sits at the end of it,
  # and the area cannot shrink past the result.
  parts <- lapply(1:5e5, \(i) c(i, 0))
  result <- sum(vapply(parts, sum, numeric(1L)))
  grown_by_small <- resident_mb() - start
  summary_text <- as.character(result)
  rm(parts)
  invisible(gc())
  left_after_small <- resident_mb() - start
  report("small vectors, process grew by more than 30 Mb:", grown_by_small > 30)
  report("small vectors, still 30 Mb above the start:", left_after_small > 30)
}
#> large vector, process grew by more than 50 Mb: TRUE
#> large vector, back within 10 Mb of the start: TRUE
#> small vectors, process grew by more than 30 Mb: TRUE
#> small vectors, still 30 Mb above the start: TRUE

Source: R FAQ 7.42: Why is R apparently not releasing memory?

  • The large vector of 80 MB went back to the system when it was removed and collected: the process shrank to within 10 Mb of where it began.

  • The half million small vectors, freed in the same way, left the process more than 30 Mb larger than it began, although gc() reported the cells as free: the result created after them (summary_text) kept the end of the contiguous area in use. Those 30 Mb are not lost to R: the next allocation of small objects reuses them, which is why a long-running session does not keep growing.

What to do about it follows from the explanation. Do not call gc() hoping to shrink the process; it will report the freed cells, and the system will keep the memory. Watch the used column of gc(), not top, to find out how much R is using. If a job needs to return its memory to the system, run it in a separate R process that exits (the callr package and the parallel workers of Parallel and Asynchronous Computing do this), and avoid keeping millions of small objects when one vector or one data frame would do: a list of 500000 two-element vectors is 500000 vector nodes of 64 bytes each, where a matrix with 500000 rows and two columns holds the same numbers in one vector of 8 MB.

ALTREP: alternative representations

R 3.5.0 introduced the ALTREP framework, which lets a vector keep its data somewhere other than an ordinary array. R NEWS for that release says that it "allows packages to provide alternate representations of basic R objects", that the framework was still experimental, and that arithmetic sequences created by 1:n, seq_along() and the like now use compact representations, and that the coercion of integer and numeric vectors to character is deferred until first use (R NEWS: Changes in R 3.5.0). The alt bit in the header marks such an object. The C interface is declared in R_ext/Altrep.h, and Writing R Extensions classes it as an experimental API, whose entry points may change and whose users should be prepared to adapt (Writing R Extensions: The R API). R 4.3.0 extended the framework to list vectors (NEWS: "ALTREP now supports VECSXP vectors"; R NEWS: Changes in R 4.3.0). For an R user the consequences are in how much memory a vector takes and when the data are really produced.

Compact sequences

1:1e7 does not allocate 40 MB. It allocates a small node that holds the first value, the length and the step, and produces elements on demand. The script measures the allocations of one million bytes or more.

# Compact sequences (ALTREP, R 3.5.0). big_allocs() lists the allocations of
# one million bytes or more that R makes while evaluating its argument.
big_allocs <- function(code) {
  path <- tempfile()
  Rprofmem(path, threshold = 1e6)
  force(code)
  Rprofmem(NULL)
  lines <- grep("^[0-9]", readLines(path), value = TRUE)
  unlink(path)
  as.numeric(sub(" :.*$", "", lines))
}

big_allocs(x <- 1:1e7)
#> numeric(0)
length(x)
#> [1] 10000000
big_allocs(length(x))
#> numeric(0)
big_allocs(sum(x))
#> numeric(0)
big_allocs(max(x))
#> numeric(0)
big_allocs(x[[5e6]])
#> numeric(0)
big_allocs(x[1:3])
#> numeric(0)

# Code that needs the whole vector as an array of integers expands it, and
# the expanded copy is kept: the first call allocates the expansion and the
# result, the second only the result.
big_allocs(y <- x + 0L)
#> [1] 40000048 40000048
big_allocs(y <- x + 0L)
#> [1] 40000048

# Changing one element of a compact sequence makes an ordinary vector.
z <- 1:1e7
big_allocs(z[[1L]] <- 5L)
#> [1] 40000048
big_allocs(z[[2L]] <- 6L)
#> numeric(0)

# object.size() reports the size of the expanded vector, the help page says
# that compact representations may be over-estimated.
as.numeric(object.size(1:1e7))
#> [1] 40000048

Source: Writing R Extensions: Writing compact-representation-friendly code

  • Creating x, taking its length, its sum and its maximum, and extracting an element or three, allocated nothing big: R answers these from the start, the length and the step. length(x) is 10000000.

  • x + 0L needed the data as an array, and Writing R Extensions says that obtaining the data pointer of a compact vector forces it to be fully materialised. The first call allocated 40000048 bytes twice: once to expand x itself (the expanded array is kept inside the compact object, so the next call does not repeat it) and once for the result; the second call allocated only the result. The manual advises C code that iterates over a vector to ask for a data pointer with the *_OR_NULL functions, or to fetch blocks with *_GET_REGION, so that a compact vector is not materialised without need.

  • Modifying one element of a compact sequence allocated a 40 MB ordinary vector; the second modification was in place.

  • object.size() reports 40000048 for the compact vector, as before: it describes what the expanded vector would need, not what is allocated. That is the over-estimate that the help page warns about.

Two more pieces of the same machinery appear in the next script. The conversion of numbers to strings is deferred: as.character(1:1e6) allocated nothing big, and the first use of the vector as a character vector expanded its table of pointers (8 MB) but not the million strings; asking for ten of them then cost next to nothing. A vector that R knows to be sorted carries that fact, so sorting it again and testing it with is.unsorted() were free (R Extensions documents the C functions that query "is sorted" and "has no NA" for such objects, and R 4.1.0 made duplicated() and anyDuplicated() use the information, NEWS: Changes in R 4.1.0), while a descending sort has to produce a vector.

# Other ALTREP vectors: character vectors converted from numbers, and sorted
# vectors. big_allocs() lists the allocations of one million bytes or more.
big_allocs <- function(code) {
  path <- tempfile()
  Rprofmem(path, threshold = 1e6)
  force(code)
  Rprofmem(NULL)
  lines <- grep("^[0-9]", readLines(path), value = TRUE)
  unlink(path)
  as.numeric(sub(" :.*$", "", lines))
}

# as.character() of a long numeric vector is deferred: no string is made
# until an element is used.
big_allocs(chars <- as.character(1:1e6))
#> numeric(0)
# The first use of the vector as a character vector expands its table of
# pointers (8 MB) but not the strings; asking for ten strings then costs
# next to nothing.
big_allocs(chars[[500000L]])
#> [1] 8000048
big_allocs(first_ten <- nchar(chars[1:10]))
#> numeric(0)

# A vector that R knows to be sorted carries that fact, so sorting it again
# and testing it are free.
sorted <- sort(1:1e7)
big_allocs(again <- sort(sorted))
#> numeric(0)
big_allocs(is.unsorted(sorted))
#> numeric(0)
big_allocs(descending <- sort(1:1e7, decreasing = TRUE))
#> [1] 40000048 40000048 40000048

Source: R NEWS: Changes in R 4.1.0

Wrappers: attributes without copying the data

Setting an attribute on a large vector that another name also refers to would, by copy-on-modify, copy the whole vector only to add a label. R 3.6.0 avoids it. Its NEWS says that drop() and unclass() avoid duplicating the data of large enough atomic vectors by returning ALTREP wrapper objects with adjusted attributes, that R-level assignments that change attributes use wrapper objects for larger atomic vectors, so that structure() and unname() do not duplicate data either, and that lists are not yet covered (R NEWS: Changes in R 3.6.0). The C entry point, R_tryWrap(), is documented as "very experimental" in Writing R Extensions.

# Large vectors that are shared are not copied when attributes are changed
# (R 3.6.0): the attribute change is made on a wrapper object that points at
# the same data. big_allocs() lists allocations of one million bytes or more.
big_allocs <- function(code) {
  path <- tempfile()
  Rprofmem(path, threshold = 1e6)
  force(code)
  Rprofmem(NULL)
  lines <- grep("^[0-9]", readLines(path), value = TRUE)
  unlink(path)
  as.numeric(sub(" :.*$", "", lines))
}

x <- numeric(1e7)
x[[1L]] <- 1
y <- x
big_allocs(attr(y, "unit") <- "pages")
#> numeric(0)
big_allocs(class(y) <- "measure")
#> numeric(0)
big_allocs(dim(y) <- c(1e3, 1e4))
#> numeric(0)
big_allocs(z <- unclass(y))
#> numeric(0)
big_allocs(w <- structure(x, label = "pages"))
#> numeric(0)

# Changing a value still copies, and the change is not made in the original.
big_allocs(y[[1L]] <- 0)
#> [1] 80000048
x[[1L]]
#> [1] 1

# A list is copied shallowly: replacing a small element does not copy the
# large vector it holds.
fit <- list(coefficients = x, call = "lm(days ~ pages)", note = "")
big_allocs(fit$note <- "checked")
#> numeric(0)
big_allocs(fit$coefficients[[1L]] <- 2)
#> [1] 80000048

Source: R NEWS: Changes in R 3.6.0

  • Setting an attribute, a class or dim on y, which shares 80 MB with x, allocated nothing: the attribute is on a wrapper that points at the same data. unclass(y) and structure(x, label = ...) also allocated nothing.

  • Changing a value is a different matter: y[[1L]] <- 0 copied the 80 MB, and x[[1L]] shows that the original is untouched.

  • A list is copied shallowly. fit$note <- "checked" allocated no big vector, because the 80 MB coefficients vector was only referred to, not copied; modifying the vector inside the list (fit$coefficients[[1L]] <- 2) copied it, since x still shares it. A loop that updates a small field of a large list is cheap, and one that updates an element of a large vector inside the list is not, unless the list holds the only reference to the vector.

  • This is the situation of the print.svycoxph example in Writing R Extensions, where an assignment to one component, x$call <- x$printcall, makes R copy the whole object. In R 4.6.1 the copy of a list is shallow: the pointers to the elements are copied, not the data they refer to, so a copy costs the size of the pointer array plus the copy of any element that is itself modified. NEWS for R 3.1.0 records changes that avoid duplicating the right-hand side in complex assignments and reduce copying in other places.

Other kinds of ALTREP vectors come from packages. Some data-reading packages return vectors whose data are read from a file or a memory-mapped region only when needed; Reading and Writing Text Data mentions the ones that do it for delimited files. The details that matter for the user are that such a vector behaves as an ordinary vector in R code, and that code that needs the whole vector as an array forces it into memory. Serializing a compact sequence also uses the representation: Serialization: RDS, RData and JSON shows that 1:1e6 serializes to 133 bytes in format version 3 against more than four million in version 2.

Long vectors

Vectors in R 2.x had at most 2^31 - 1 elements, about two billion, because the length was a C int stored in the node. R 3.0.0 added support for longer vectors on 64-bit platforms. R Internals describes the design: a vector with a long length records its length as -1 and keeps the real length as a 64-bit value R_xlen_t at the start of the header; the type is available to packages in Rinternals.h; indexing accepts doubles; and length() returns a double when the length is above 2^31 - 1 (R Internals: Long vectors). Theoretical lengths are 63 bits, but the present limit is 2^52, which exceeds what the operating systems offer and allows lengths to be stored exactly in a double. The support covers raw, logical, integer, double and character vectors, lists and expression vectors; the bytes of one string are still limited to 2^31 - 1. A single vector of 2^31 or more elements takes at least 8 GB if it is integer or logical and 16 GB if it is numeric or character, so the page does not create one. The details of the numeric types and the other long-vector rules (indexing with doubles, seq_len() and seq_along() returning doubles, the functions that still refuse long vectors) are in Atomic Vectors and Types; the script here shows only what can be shown without memory.

# Long vectors: lengths above .Machine$integer.max. Nothing here allocates
# memory: 1:3e9 is a compact sequence, and the other requests fail at once.
.Machine$integer.max
#> [1] 2147483647
x <- c(10, 20, 30)
typeof(length(x))
#> [1] "integer"
is.integer(length(x))
#> [1] TRUE

long <- 1:3e9
typeof(long)
#> [1] "double"
length(long)
#> [1] 3e+09
typeof(length(long))
#> [1] "double"
long[[2.5e9]]
#> [1] 2.5e+09

# The largest length R accepts is 2^52. Requests below it that cannot be
# satisfied fail in the allocator, and larger ones are refused up front.
tryCatch(numeric(2^52), error = conditionMessage)
#> [1] "cannot allocate vector of size 33554432.0 Gb"
tryCatch(numeric(2^52 + 1), error = conditionMessage)
#> [1] "vector size specified is too large"
tryCatch(numeric(-1), error = conditionMessage)
#> [1] "invalid 'length' argument"

# A matrix can hold more than 2^31 - 1 elements, but each dimension is limited
# to 2^31 - 1: the dimension is first converted to an integer.
tryCatch(matrix(0, nrow = 2^31, ncol = 1L), warning = conditionMessage)
#> [1] "NAs introduced by coercion to integer range"
tryCatch(
  suppressWarnings(matrix(0, nrow = 2^31, ncol = 1L)),
  error = conditionMessage
)
#> [1] "invalid 'nrow' value (too large or NA)"

Source: R Internals: Long vectors

  • .Machine$integer.max is 2147483647, and length() of an ordinary vector is an integer: the script prints "integer" and TRUE.

  • 1:3e9 is a compact sequence (the previous section), so it is a long vector that takes next to no memory. Its type is double, because the integers cannot hold its last value, and length(long) is the double 3e+09, printed in scientific notation. Indexing it with a double works. Code that calls as.integer(length(x)) to pass a length to C or Fortran must check for the NA that results for a long vector (R Internals says so), which is also why .C() and .Fortran() do not accept long vectors.

  • The requests at the end show the limit: numeric(2^52) is a legal length, so R tries to allocate 32 PiB (33554432 Gb) and the allocator refuses (cannot allocate vector of size ...); numeric(2^52 + 1) is refused up front (vector size specified is too large); a negative length is invalid 'length' argument.

  • A matrix may have more than 2^31 - 1 elements, provided that each of its dimensions does not: R Internals says that linear algebra done by Fortran with 32-bit INTEGER is the main restriction, and that some complex LAPACK routines overflow silently on matrices that large. A dimension of 2^31 is converted to an integer first, hence the warning NAs introduced by coercion to integer range and the error invalid 'nrow' value (too large or NA). Do not run 1:3e9 + 0L: it expands the sequence, which takes 24 GB.

  • The NEWS for R 3.0.0 lists what works with long vectors (most operations that can sensibly be done, including sort(), order() and sample()) and what answers long vectors not supported yet (functions that use integer indices internally, such as match() and anyDuplicated()); later releases extended the list, for example rank() in R 4.3.3.

Common errors and warnings

The messages below are the real ones from R 4.6.1, collected by the scripts of this page. The wording of cannot allocate vector of size is the one to expect from any 64-bit build; the number is the size of the request, in the unit that fits.

Message What it usually means What to do

cannot allocate vector of size 8388608.0 Gb (the size is the request)

The system allocator refused the request. The process or the machine cannot provide that much memory in one piece, or the request is a mistake (here 250 doubles)

Check the size computation; look for copies with tracemem() and Rprofmem(); process in chunks; free the big objects that are no longer needed; use a machine with more memory

vector memory limit of 100.0 Mb reached, see mem.maxVSize()

A limit on the vector heap is in force: mem.maxVSize(), the environment variable R_MAX_VSIZE, or the default limit on macOS

Raise it with mem.maxVSize() or remove it with mem.maxVSize(Inf); the message prints the limit in Mb

vector size specified is too large

The length is above the largest length R accepts, 2^52

Compute the length again; such a vector could not exist

invalid 'length' argument

A negative length given to numeric(), vector() and the like

Check the arithmetic that produced the length

invalid 'nrow' value (too large or NA), preceded by the warning NAs introduced by coercion to integer range

A matrix dimension above 2^31 - 1; each dimension must fit in an integer even though the matrix may have more elements

Use a different layout (a list of blocks, a database, a memory-mapped file)

long vectors not supported yet

A function that works with integer indices internally was called on a vector longer than 2^31 - 1 (listed in the NEWS for R 3.0.0; not run on this page)

Process the vector in pieces

warnings 'memory.limit()' is no longer supported and 'memory.size()' is no longer supported

Old Windows-only code; the functions are stubs that return Inf

Remove the calls; use operating-system limits, R_MAX_VSIZE or mem.maxVSize()

'tracemem' is not useful for promise and environment objects, argument must not be a function, cannot trace NULL

The object is never duplicated or is not an ordinary object (the messages are collected in Assignment, Names and Values)

Trace an ordinary object

Source: R documentation: Memory-limits

The help page on memory limits adds two more messages that appear when a limit of the build is reached: cannot allocate vector of size for a failure to obtain memory, either because the size exceeded the address-space limit or, more likely, because the system could not provide it; and cannot allocate vector of length for an object that exceeds an individual limit. It also lists the limits that remain: the number of bytes in a character string, and each dimension of an array, are limited to 2^31 - 1.

Pitfalls

  • Growing with c(). out <- c(out, v) allocates a new vector of the new length on every iteration and copies the old one; the profile above shows it as the call that allocates. Growth by assignment past the end is cheaper since R 3.4.0 (the vector is over-allocated), and preallocation is still better.

  • Modifying a data frame column in a loop. The copy count grows with the number of iterations (the table above). Work on a vector or a list and assign once.

  • Reading object.size() as the memory in use. It does not see sharing, environments or what a function closes over, and it over-estimates compact vectors. Add the sizes of objects that share data and you count the data twice.

  • Believing that rm() shrinks the process. rm() removes a name; the collector reclaims the object; the system may keep the memory (R FAQ 7.42).

  • Calling gc() before every allocation. A full collection every time costs more than the memory it saves, and R collects when it needs the room.

  • Leaving tracemem() on. Every copy of a traced object is traced, so a trace that stays on prints a message for every copy made later. Call untracemem(), or run the check in a separate script.

  • Holding on to big objects by accident. A closure keeps the frame it was created in alive; an unforced promise keeps its environment alive (Lazy Evaluation and Promises shows a collection that waits for a promise); a cache in an environment grows until someone removes entries. object.size() will not show any of this; the used column of gc() will.

  • Expecting reference semantics from a list or a data frame. A function that receives a list changes its own copy; it returns the modified value, and the caller must assign it. Environments, R6 objects and data.table tables are the exceptions.

  • Relying on the counts. The number of copies that a loop makes is a property of the version of R. The shape (zero, one per iteration, a constant) is the stable part.

Version notes

Version Change

R 3.0.0

Long vectors on 64-bit platforms: length() returns a double beyond 2^31 - 1, double-valued indices reach beyond it, .C() and .Fortran() do not accept long vectors. The environment variable R_GC_MEM_GROW controls how the heap grows; R_NSIZE and R_VSIZE are restored.

R 3.0.3

Finalizers are marked at a collection and run at a safer later time, because running them during the collection caused problems (for example with options()).

R 3.1.0

object.size() gains a format() method. Complex assignments avoid duplicating the right-hand side where possible, and other changes reduce copying.

R 3.2.0

The memory manager grows the heap more aggressively, which reduces the number of collections.

R 3.4.0

format() and print() for object.size() results get standard and digits. Assignment beyond the end of a vector over-allocates (a growable vector).

R 3.5.0

ALTREP: arithmetic sequences (1:n, seq_along()) are compact and the conversion of numbers to strings is deferred; the object header layout changes (R Internals). gc() gains full. R_MAX_VSIZE limits the vector heap. object.size() works for long vectors. Finalizers run with interrupts suspended.

R 3.6.0

drop(), unclass() and attribute assignments use ALTREP wrappers to avoid duplicating the data of large atomic vectors. mem.maxVSize() and mem.maxNSize() are added.

R 4.0.0

Reference counting replaces NAMED for deciding when objects can be mutated.

R 4.0.3

Accessing a long vector represented by a compact integer sequence no longer segfaults.

R 4.1.0

duplicated() and anyDuplicated() use the sortedness of ALTREP vectors.

R 4.2.0

memory.limit() and memory.size() are stubs on Windows, as on Unix-alikes.

R 4.3.0

ALTREP supports list (VECSXP) vectors.

R 4.3.3

rank() works for long vectors.

R 4.4.0

The internal type S4SXP is renamed OBJSXP; typeof() still reports "S4" for S4 objects.

R 4.6.0

New C functions R_altrep_class_name() and R_altrep_class_package() return the class name and package of an ALTREP object; ALTREP_CLASS() should no longer be used.

Source: R NEWS: Changes in R 4.0.0, 4.0.3, 4.1.0, 4.2.0, 4.3.0, 4.3.3, 4.4.0 and 4.6.0

The entries for R 3.x are in the NEWS file of R 3.x (R NEWS: Changes in R 3.0.0, 3.0.3, 3.1.0, 3.2.0, 3.4.0, 3.5.0 and 3.6.0).

Where older tutorials differ

The memory advice on the web is older than reference counting, ALTREP and the stubs for memory.limit(). Many tutorials show a tool that is not part of R. The next script checks, in the session used for this page, for the two add-on packages that older tutorials name, and shows what the Windows memory functions do now.

# Tools from older tutorials. The add-on packages are not installed here, so
# the page does not show their output.
c(
  pryr = requireNamespace("pryr", quietly = TRUE),
  lobstr = requireNamespace("lobstr", quietly = TRUE)
)
#>   pryr lobstr
#>  FALSE  FALSE

# memory.size() and memory.limit() were Windows-only; they are stubs now.
report_warning <- function(code) {
  withCallingHandlers(
    code,
    warning = \(w) {
      cat("warning:", conditionMessage(w), "\n")
      invokeRestart("muffleWarning")
    }
  )
}
report_warning(memory.size())
#> warning: 'memory.size()' is no longer supported
#> [1] Inf
report_warning(memory.limit(size = 4000))
#> warning: 'memory.limit()' is no longer supported
#> [1] Inf

Source: R documentation: memory.size

Neither pryr nor lobstr was installed, so their output is not shown: pryr::address() and pryr::object_size() are the older names for what lobstr::obj_addr() and lobstr::obj_size() do. The help page of memory.size() says that the two functions are stubs that report infinity with a warning on all platforms, and the script shows the warnings.

Older tutorial Current practice Why Version

“Once an object has two names it is marked shared (NAMED is 2) and the next change copies it, always”

Copy only if the reference count says another holder exists; removing a name lowers the count

R 4.0.0 replaced NAMED by reference counting; the display of .Internal(inspect(x)) shows REF(n)

R 4.0.0

pryr::address(x) and pryr::object_size(x) to find out whether two names share an object and how big it is

tracemem(x) (the identifier is the address) for sharing and copies, object.size() for an estimate, Rprofmem() for allocations; the community package lobstr for exact sizes of shared structure

Base R has the tools; the add-on packages are optional and are not part of the supported interface

current

“Call gc() to free memory”

Do nothing; call gc() only to read the report

R collects when it needs the room; the help page names the report as the main purpose of the call

current

“rm(list = ls()); gc() between steps”

Run the steps in separate R processes, or let the objects go out of scope

rm() removes names, the system may keep the freed memory (R FAQ 7.42), and a fresh process starts clean

current

“memory.limit(size = 8000) raises the memory available to R on Windows”

Use the operating system’s limits, R_MAX_VSIZE or mem.maxVSize()

memory.limit() and memory.size() have been stubs that return Inf since R 4.2.0

R 4.2.0

“Growing a vector in a loop copies it every time”

True for c(x, v); assignment past the end (x[[n + 1]] <- v) is cheaper because the vector is over-allocated, and preallocation is best

R 3.4.0 added growable vectors; the NEWS says that preallocating is still preferred

R 3.4.0

“1:n allocates n integers, so use seq_len() in loops”

Both are compact; the loop over 1:n allocates nothing big. The reason to prefer seq_len() is the case n == 0

ALTREP compact sequences

R 3.5.0

“unclass(x) or attr(x, "a") <- v copy the data”

They do not for large atomic vectors that are shared

ALTREP wrappers

R 3.6.0

“A vector cannot have more than 2^31 - 1 elements”

On 64-bit builds it can; use doubles to index

Long vectors

R 3.0.0

“object.size() tells how much memory the object uses”

It estimates; it ignores sharing and environments and over-estimates compact vectors

The help page lists the limits

current

Source: R NEWS: Changes in R 4.0.0 and 4.2.0

Where to go next

References