Function Factories and Higher-Order Functions
|
This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production. The reference material consulted while preparing these pages is listed in the section bibliography. |
Checked against R 4.6.1 (2026-06-24) on 2026-10-20; R 4.6.2 is scheduled for 2026-10-29. Almost everything on this page is base R (the base, utils and stats packages that ship with R); the one add-on package is purrr 1.2.2, used in a single short example to compare its compose() and partial() with the hand-written versions, and the lintr 3.4.0 default linters were used only to check the scripts. A function in R is a value like any other, so functions can be passed to functions and returned from them. This page is about the second half of that sentence and about the patterns built on it: functions that make functions (function factories), functions that wrap a function to change how it behaves (function operators), the glue that combines functions into pipelines, and the recursion that functional code leans on, with the limits R puts on it and the experimental Tailcall() that lifts one of them.
Every example is a script that was run with Rscript on R 4.6.1 with warnings printed as they happen (options(warn = 1)), and the lines that start with #> are what it printed. The scripts also pass the default lintr linters; a # nolint: comment marks a line where the idiom on show is what a default linter dislikes, so the linter does not hide the lesson. The names books, loans, shelf_today, loan_days and the helper is_valid_isbn13() come from the running library-shelf scenario described on the section index: a script that needs them starts with source("scenario.R"). Where a message contains quotation marks, they are the typographic quotes R prints in a UTF-8 locale. Memory addresses of environments and platform-specific numbers (the size of the C stack, the usage reported in a stack error) never appear in the output on this page; where R prints one, the example prints a class, a name or a generic form and the text says so. The mechanics that factories build on, environments, enclosures and <<-, are the subject of Scoping, Environments and Closures, and the promises behind force() are in Lazy Evaluation and Promises; this page uses them and does not repeat them.
Functions that take and return functions
R’s own documentation calls Reduce(), Filter(), Map(), Find(), Position() and Negate() the "common higher-order functions in functional programming languages" (the title of their shared help page, funprog). A higher-order function is one that takes a function as an argument, returns a function as its value, or both. The community book Advanced R, which is not part of the official documentation, separates three kinds that are worth having names for, and this page follows its vocabulary:
-
A functional takes a function and returns data:
lapply(),vapply(),Reduce(),do.call(). They are the subject of The apply Family and Functionals. -
A function factory takes data and returns a function:
make_limit_check(21L)returns a function that tests loan lengths against 21 days. The returned function is a closure that remembers the arguments of the call that made it. -
A function operator takes one or more functions and returns a function:
Vectorize(),Negate()and the logging, retry and memoising wrappers below. It is a decorator in the vocabulary of other languages.
All three exist because of two facts from the language definition. Functions are first-class objects: a function value can be stored, passed and returned. And a closure carries the environment in which it was created: the manual says that a closure "will contain the environment in which it is created as part of its definition", and that when the function is called its evaluation environment is created with that environment as enclosure, which "is not necessarily the environment of the caller" (The R Language Definition: Lexical environment; the structure of a function object, formals plus body plus environment, is in Function objects). R FAQ 7.12 explains the difference between the enclosing environment (where the function was created) and the parent environment (where it was called from), and R FAQ 3.3.1 lists lexical scoping as the main difference from S (R FAQ 7.12: What are the enclosing and parent environments?, R FAQ 3.3.1: Lexical scoping). A factory is nothing more than a function whose last expression is a function expression: the call that runs the factory has an evaluation frame, the returned function captures it as its enclosure, and the frame stays alive for as long as the returned function does.
Function factories
A function that returns a function
The first script builds two checkers from one factory. over_21 and over_7 have the same code; they differ only in the environment each one carries, and each of those environments holds its own limit. The script also looks at the environments, because they are the whole mechanism: environment() returns the enclosure, ls() lists what it holds, and its parent is the environment in which the factory itself was defined (here the global environment, whose name environmentName() prints; the anonymous call environments have an empty name, and printing one would show a memory address, which this page avoids).
source("scenario.R")
# A function factory: a function whose value is a function.
make_limit_check <- function(limit) {
function(days) days > limit
}
over_21 <- make_limit_check(21L)
over_7 <- make_limit_check(7L)
over_21(c(5L, 21L, 30L))
#> [1] FALSE FALSE TRUE
over_7(c(5L, 21L, 30L))
#> [1] FALSE TRUE TRUE
# Every call of the factory creates a new environment, and that environment
# is the enclosure of the function that call returned.
env_21 <- environment(over_21)
env_7 <- environment(over_7)
identical(env_21, env_7)
#> [1] FALSE
ls(env_21)
#> [1] "limit"
get("limit", envir = env_21)
#> [1] 21
env_7$limit
#> [1] 7
class(env_21)
#> [1] "environment"
environmentName(env_21)
#> [1] ""
# The enclosure of that environment is where the factory was defined.
environmentName(parent.env(env_21))
#> [1] "R_GlobalEnv"
identical(parent.env(env_21), environment(make_limit_check))
#> [1] TRUE
# The returned function is a normal closure: its formals and body are its own.
names(formals(over_21))
#> [1] "days"
body(over_21)
#> days > limit
# On the shelf: one checker per limit, applied to the length of every loan.
loan_length <- as.integer(loans$return_date - loans$loan_date)
sum(over_21(loan_length), na.rm = TRUE)
#> [1] 25
sum(over_7(loan_length), na.rm = TRUE)
#> [1] 47The two checkers are different closures: identical(env_21, env_7) is FALSE, limit is 21 in one and 7 in the other, and body(over_21) shows that the code refers to limit without defining it. That is a free variable, found by lexical scoping in the enclosure, which is the environment of the call make_limit_check(21L). Its only content is limit, the factory’s argument; had the factory created local variables before returning the function, they would be there too, and they would be kept alive with it (the "What a closure keeps alive" part of Scoping, Environments and Closures shows that and how to avoid it). The last two lines apply the checkers to the loan lengths of the shelf scenario: 25 of the returned loans lasted more than 21 days and 47 more than 7.
Two properties explain why factories are used. They give a function configuration that is fixed at creation time, so that a function with the interface f(x) that a caller such as vapply() expects can still depend on a threshold, a connection or a table that the caller knows nothing about. And they make the configuration private to the function: nothing outside can reach limit except through the environment, which is a deliberate act (environment(over_21)$limit), not an accident of naming.
The lazy-evaluation trap and force()
The arguments of the factory are promises (Lazy Evaluation and Promises; the language definition describes them under Argument evaluation), and a promise is evaluated the first time its value is needed. A factory whose closure reads its argument only when the closure is called therefore reads it late, after the caller may have changed the variable that was passed. force(x) evaluates the promise x immediately; the help page says it is useful when an argument will be captured in a closure and later altered by an explicit assignment, or by an implicit one such as the assignment of a loop or an apply function (R documentation: force). It is plain "semantic sugar": evaluating the bare symbol would do the same, and force() states the intent.
The first example uses a changed variable, and then a for loop, where the implicit assignment is the loop variable. The last two blocks show the two apply-style constructors that do not have the problem on R 4.6.1: the NEWS entry of R 3.2.0 says that higher-order functions such as the apply functions and Reduce() now force the arguments to the functions they apply, to eliminate undesirable interactions between lazy evaluation and variable capture in closures (R NEWS: Changes in R 3.2.0). The element that lapply() or Map() hands to the factory is therefore forced, and the loop version of the same code is not.
# The factory keeps its argument n, but n is a promise until it is used.
make_adder <- function(n) {
function(x) x + n
}
n <- 1L
add <- make_adder(n)
n <- 100L
add(1L)
#> [1] 101
# force() evaluates the promise while n is still 1.
make_adder_forced <- function(n) {
force(n)
function(x) x + n
}
n <- 1L
add_forced <- make_adder_forced(n)
n <- 100L
add_forced(1L)
#> [1] 2
# A for loop builds closures that all read the last value of i.
adders <- list()
for (i in 1:3) {
adders[[i]] <- make_adder(i)
}
vapply(adders, \(add) add(10L), integer(1))
#> [1] 13 13 13
# lapply() and Map() force the element they pass (R 3.2.0), so no trap here.
adders <- lapply(1:3, make_adder)
vapply(adders, \(add) add(10L), integer(1))
#> [1] 11 12 13
adders <- Map(make_adder, 1:3)
vapply(adders, \(add) add(10L), integer(1))
#> [1] 11 12 13Source: R documentation: force
add(1L) is 101: the factory returned before n was needed, n was then reassigned to 100, and the promise read the new value when add was first called. After force(n) the same sequence gives 2. The for loop version is the one that appears in the force() help page itself: every closure was created while i was still counting, none of them forced it, and all three read the final 3 afterwards, so the results are 13, 13 and 13. lapply() and Map() give 11, 12 and 13 because of the 3.2.0 change. A for loop is a language construct and not a function, so it is not covered by that change and nothing in R 4.6.1 forces the loop variable for you.
The 3.2.0 change has a limit that is easy to miss: it concerns the element that the apply function passes, not the extra arguments it forwards through the dots. The next script passes a configuration value through the dots of lapply(). The value is still a promise when the factory returns, so the same trap appears even though lapply() is used; force() in the factory fixes it, and Map(), whose dots are the vectors it iterates over and are forwarded to mapply(), forces them.
# What lapply() forces is the element it passes, not the arguments it forwards
# through the dots.
make_offset <- function(i, k) {
function(x) x + k
}
shift <- 1
offsets <- lapply(1:2, make_offset, k = shift)
shift <- 100
offsets[[1L]](0)
#> [1] 100
# Forcing in the factory fixes it for every caller.
make_offset_forced <- function(i, k) {
force(k)
function(x) x + k
}
shift <- 1
offsets <- lapply(1:2, make_offset_forced, k = shift)
shift <- 100
offsets[[1L]](0)
#> [1] 1
# Map() forwards its dots as arguments of mapply(), which forces them.
shift <- 1
offsets <- Map(make_offset, 1:2, k = shift)
shift <- 100
offsets[[1L]](0)
#> [1] 1Source: R documentation: lapply
The practical rule is the one that the language gives anyway: a factory that stores an argument in the closure should call force() on it as its first statement, whatever the caller does. It costs one line, it makes the factory safe for every caller, and a reviewer reading force(limit) knows that the argument was meant to be read now. The cost of leaving it out is a bug that produces plausible numbers (the lazy evaluation page shows the shelf version: the same count three times for three different limits) and that can disappear when the code is run line by line in a debugger, because inspecting a variable there forces the promise.
Stateful functions
Counters with <<-
A closure can also change the variables of its enclosure, and then the factory makes a function with memory. The operator for that is <<-, which assigns not in the current frame but in the first enclosing environment where the name already exists (the details and the lookup rules are in Scoping, Environments and Closures). A counter is the standard example, and the ordinary case on the shelf is an id source that continues the numbering of the loans table. Every counter has its own state, because every call of the factory creates its own environment; environment(counter)$count reads the state from outside. The last function in the script uses <- by mistake: it increments a local copy that dies with each call, so the counter never advances, and no error or warning is produced.
source("scenario.R")
# A counter: the state lives in the environment of the call that made it.
make_counter <- function(start = 0L) {
count <- start
function() {
count <<- count + 1L # nolint: assignment_linter.
count
}
}
counter_a <- make_counter()
counter_b <- make_counter(10L)
counter_a()
#> [1] 1
counter_a()
#> [1] 2
counter_b()
#> [1] 11
counter_a()
#> [1] 3
environment(counter_a)$count
#> [1] 3
environment(counter_b)$count
#> [1] 11
# The next free loan id, continuing the numbering of the loans table.
make_id_source <- function(used) {
last <- max(as.integer(sub("^L", "", used)))
function() {
last <<- last + 1L # nolint: assignment_linter.
sprintf("L%03d", last)
}
}
next_loan_id <- make_id_source(loans$loan_id)
next_loan_id()
#> [1] "L061"
next_loan_id()
#> [1] "L062"
tail(loans$loan_id, 1L)
#> [1] "L060"
# With <- the counter changes a local copy, so it never advances.
make_broken_counter <- function() {
count <- 0L # nolint: object_usage_linter.
function() {
count <- count + 1L
count
}
}
broken <- make_broken_counter()
broken()
#> [1] 1
broken()
#> [1] 1
environment(broken)$count
#> [1] 0Source: An Introduction to R: Scope
The default lintr linters flag <<- through assignment_linter in this configuration, which is why the lines carry # nolint: assignment_linter.: the use is deliberate. R-intro builds its bank-account example on exactly this pattern, as an illustration of lexical scope used to give functions mutable state.
What <<- does when the name is not found
The rule of <<- has a consequence that older tutorials never mention. If the name is not found in any enclosing environment, the assignment creates it in the global environment. A typo in the name of the state variable then does not fail; it silently writes a global variable and leaves the real state untouched. The next script has such a typo (cont for count) and shows the consequence: the function returns 1 every time, and a variable called cont has appeared in the global environment. It then shows two alternatives that do not rely on <<- finding the right name: a state environment that is modified in place, and assign() with an explicit target.
# <<- starts searching in the enclosing environment of the running function
# and walks up the chain. If it finds nothing, it assigns in the global env.
exists("typo_count")
#> [1] FALSE
make_typo_counter <- function() {
count <- 0L
function() {
cont <<- count + 1L # nolint: assignment_linter, object_usage_linter.
cont # nolint: object_usage_linter.
}
}
typo_counter <- make_typo_counter()
typo_counter()
#> [1] 1
typo_counter()
#> [1] 1
exists("cont")
#> [1] TRUE
rm("cont", envir = globalenv())
# A mutable environment needs no <<-: its contents change in place.
make_tally <- function() {
state <- new.env()
state$n <- 0L
function() {
state$n <- state$n + 1L
state$n
}
}
tally <- make_tally()
tally()
#> [1] 1
tally()
#> [1] 2
# assign() with an explicit environment states the target.
make_total <- function() {
total <- 0
function(x) {
assign("total", total + x, envir = parent.env(environment()))
total
}
}
add_to_total <- make_total()
add_to_total(5)
#> [1] 5
add_to_total(7)
#> [1] 12Source: R documentation: assign
An environment is the one mutable container in the language: state$n <- state$n + 1L changes the object that state refers to, so no <<- is needed and a typo in n creates a new entry in state, not a global variable. Reference semantics of this kind are the base of Reference Classes and R6 (Reference Classes and R6), which are environments with methods attached. When the state grows past a counter or two, an R6 class is usually easier to read than a factory that returns a list of five functions.
Private state with local()
A factory function is not the only way to give a function a private environment. local() evaluates an expression in a new environment; the help page says that it is equivalent to evalq() except that its default environment is new and empty, and that this makes it useful for anonymous recursive functions and as a kind of limited namespace (R documentation: eval, evalq, local). If the block ends in a function (or a list of functions), the variables created in the block become that function’s private state, and there is no factory to call twice. It is the right tool for a one-off object, such as a receipt number source for the whole script, and for a small group of functions that share a variable.
source("scenario.R")
# local() evaluates a block in a fresh environment; the block's value is the
# function, and the variables it created stay behind as private state.
next_receipt <- local({
last <- 0L
function() {
last <<- last + 1L # nolint: assignment_linter.
sprintf("R%04d", last)
}
})
next_receipt()
#> [1] "R0001"
next_receipt()
#> [1] "R0002"
exists("last")
#> [1] FALSE
ls(environment(next_receipt))
#> [1] "last"
# Several functions can share the private state: return them as a list.
shelf_stats <- local({
seen <- character()
list(
record = function(isbn) {
seen <<- union(seen, normalise_isbn(isbn)) # nolint: assignment_linter.
invisible(length(seen))
},
distinct = function() length(seen)
)
})
for (isbn in loans$isbn) {
shelf_stats$record(isbn)
}
shelf_stats$distinct()
#> [1] 16
length(unique(loans$isbn))
#> [1] 16
# local() is eval() on a new environment whose parent is the calling frame.
identical(
parent.env(environment(next_receipt)),
globalenv()
)
#> [1] TRUE
f <- local({
x <- 1
function() x
})
x <- 2
f()
#> [1] 1The variable last does not exist in the global environment (exists("last") is FALSE) but lives in the environment of next_receipt, as ls(environment(next_receipt)) shows. The second block returns a list of two functions that share seen; after the loop over loans$isbn it has seen 16 distinct values, which is what length(unique(loans$isbn)) says. The parent of the environment that local() created is the global environment, because that is where local() was called from. The last lines repeat a point from the lazy-evaluation page in a different form: f captured x <- 1 of the block, and the later global x <- 2 does not reach it.
Memoisation
A memoised function remembers the results it has already computed, keyed by its arguments, and returns the stored value instead of computing it again. It is a function operator that needs a mutable cache, and the natural cache in R is an environment: it is a hash table keyed by strings, it is modified in place, and it lives in the closure. new.env(hash = TRUE, parent = emptyenv()) creates it (hash = TRUE is the default; the empty parent is a second safety net, because with the default inherits = TRUE the functions exists() and get() follow the enclosure chain and would find a key such as "sum" in the base package), exists(key, envir = cache, inherits = FALSE) asks whether a key is present, and assign() and get() write and read it (R documentation: environment, exists, assign, get). The keys are names, so the cache converts the argument with as.character().
The classic demonstration is the Fibonacci function, which in its naive form recomputes the same values exponentially often. The script counts the calls of the naive version and then memoises the same definition. Two details matter. The memoised function has to call itself through the memoised name (fib calls fib), because the recursive calls inside the body are what fill the cache; a body that used Recall() would call the unwrapped function and the cache would not help. And the counters hits and misses live in the same closure as the cache, so they can be read from outside with environment().
source("scenario.R")
# memoise(): a function operator that caches results in an environment.
memoise <- function(f) {
force(f)
cache <- new.env(hash = TRUE, parent = emptyenv())
hits <- 0L
misses <- 0L
function(x) {
key <- as.character(x)
if (exists(key, envir = cache, inherits = FALSE)) {
hits <<- hits + 1L # nolint: assignment_linter.
return(get(key, envir = cache, inherits = FALSE))
}
misses <<- misses + 1L # nolint: assignment_linter.
value <- f(x)
assign(key, value, envir = cache)
value
}
}
# The naive Fibonacci function repeats the same sub-problems.
calls <- 0L
fib_plain <- function(n) {
calls <<- calls + 1L # nolint: assignment_linter.
if (n < 2) n else fib_plain(n - 1) + fib_plain(n - 2)
}
fib_plain(20)
#> [1] 6765
calls
#> [1] 21891
# The memoised version must call itself through the memoised name.
fib <- memoise(function(n) if (n < 2) n else fib(n - 1) + fib(n - 2))
fib(20)
#> [1] 6765
fib(50)
#> [1] 12586269025
environment(fib)$misses
#> [1] 51
environment(fib)$hits
#> [1] 49
sort(as.integer(ls(environment(fib)$cache)))[1:5]
#> [1] 0 1 2 3 4
length(environment(fib)$cache)
#> [1] 51Source: R documentation: exists
The naive fib_plain(20) makes 21891 calls for the value 6765. The memoised fib(50) needs 51 evaluations in total (one for each value from 0 to 50, with fib(20) already cached from the first call) and answers 49 further requests from the cache, and the keys are the argument values as text, which ls() returns in alphabetical order, hence the as.integer() before sort(). The value 12586269025 is exact because it is below 2^53, the limit up to which a double represents every integer exactly; fib(80) would not be (see Numbers and Arithmetic).
Memoisation helps when a function is expensive, pure and called repeatedly with the same arguments. The shelf has such a case only in a mild form: the loans table has 60 rows but 16 distinct ISBNs, so validating the ISBN of every loan with a memoising wrapper around is_valid_isbn13() calls the checker 16 times instead of 60. The checker is cheap, so this is a demonstration and not a recommendation; for a function this cheap the lookup costs as much as the work. The script also shows the two edge cases a cache needs to think about: a key that is not a legal name, and a function that is not pure.
source("scenario.R")
memoise <- function(f) {
force(f)
cache <- new.env(hash = TRUE, parent = emptyenv())
hits <- 0L
misses <- 0L
function(x) {
key <- as.character(x)
if (exists(key, envir = cache, inherits = FALSE)) {
hits <<- hits + 1L # nolint: assignment_linter.
return(get(key, envir = cache, inherits = FALSE))
}
misses <<- misses + 1L # nolint: assignment_linter.
value <- f(x)
assign(key, value, envir = cache)
value
}
}
# Check every ISBN of every loan, asking the checker once per distinct value.
check_isbn <- memoise(is_valid_isbn13)
valid_loan <- vapply(loans$isbn, check_isbn, logical(1), USE.NAMES = FALSE)
table(valid_loan)
#> valid_loan
#> FALSE TRUE
#> 3 57
c(
loans = nrow(loans),
distinct = length(unique(loans$isbn)),
misses = environment(check_isbn)$misses,
hits = environment(check_isbn)$hits
)
#> loans distinct misses hits
#> 60 16 16 44
# Keys are variable names, so an empty string is not a valid key.
tryCatch(check_isbn(""), error = \(e) conditionMessage(e))
#> [1] "invalid first argument"
# Only pure functions are safe to cache: the result may depend on x alone.
loan_count <- function(isbn) sum(loans$isbn == isbn)
count_cached <- memoise(loan_count)
first <- loans$isbn[[1L]]
count_cached(first)
#> [1] 3
loans <- loans[loans$isbn != first, ]
loan_count(first)
#> [1] 0
count_cached(first)
#> [1] 3Source: R documentation: exists
The 57 TRUE and 3 FALSE results are the loans whose ISBN passes and fails the check: the three failures are the one-digit-short value planted in the data and two loans of the title whose check digit is wrong in the books table. The counters show 16 misses and 44 hits. The empty string cannot be a key: exists("") fails with invalid first argument, and assign("", ...) would say attempt to use zero-length variable name, so a general-purpose memoiser has to map the argument to a key that is always a legal name, for instance by pasting a prefix in front of it. The last part is the real danger of memoisation. loan_count() reads the global loans, so its result is not a function of its argument alone; after the table changed, loan_count(first) is 0 but the cached function still says 3. Memoise only functions whose value depends on their arguments and nothing else, and give the cache a way to be emptied (a function in the same closure that runs rm(list = ls(cache), envir = cache)) if the data can change.
Function operators
A function operator takes a function and returns a function with changed behaviour. The returned function has to accept whatever the original does, which is why the wrappers on this page take ... and pass it on; it has to be created with force() on the function and on any option it will read later (the trap above applies to every argument of a factory, and f is an argument like the others); and it should return what the original returns, so that it can replace the original at the call site without anything else changing. Memoising is one such operator. The next ones are those that come up in practice: logging, timing, delaying, retrying and checking arguments.
Logging, timing and delay
with_logging() prints what it is about to call and what came back. Its label defaults to deparse(substitute(f)), the name the caller used, which works because substitute() still sees the expression of the promise after f has been forced. The timing wrapper keeps the elapsed seconds in the closure; to keep this page free of timings the script prints only whether the value is a number, and uses on.exit() so that the time is recorded even if f signals an error. with_delay() waits before every call, which is how a client keeps under a rate limit; a delay of 0 seconds keeps the script fast.
source("scenario.R")
# A function operator takes a function and returns a (decorated) function.
with_logging <- function(f, label = deparse(substitute(f))) {
force(f)
force(label)
function(...) {
cat("calling ", label, " with ", ...length(), " argument(s)\n", sep = "")
result <- f(...)
cat(label, " returned ", length(result), " value(s)\n", sep = "")
result
}
}
logged_valid <- with_logging(is_valid_isbn13)
logged_valid(books$isbn[1:3])
#> calling is_valid_isbn13 with 1 argument(s)
#> is_valid_isbn13 returned 3 value(s)
#> [1] TRUE TRUE TRUE
# The dots pass every argument through, named or not.
logged_round <- with_logging(round)
logged_round(3.14159, digits = 2)
#> calling round with 2 argument(s)
#> round returned 1 value(s)
#> [1] 3.14
# Timing: keep the elapsed seconds in the closure; print no timings.
with_timing <- function(f) {
force(f)
elapsed <- NA_real_ # nolint: object_usage_linter.
wrapper <- function(...) {
start <- proc.time()[["elapsed"]]
on.exit({
stop_at <- proc.time()[["elapsed"]]
elapsed <<- stop_at - start # nolint: assignment_linter.
})
f(...)
}
wrapper
}
timed_valid <- with_timing(is_valid_isbn13)
invisible(timed_valid(books$isbn))
is.numeric(environment(timed_valid)$elapsed)
#> [1] TRUE
environment(timed_valid)$elapsed >= 0
#> [1] TRUE
# Delay: wait before every call (here 0 seconds, so the script is fast).
with_delay <- function(f, seconds) {
force(f)
force(seconds)
function(...) {
Sys.sleep(seconds)
f(...)
}
}
slow_valid <- with_delay(is_valid_isbn13, seconds = 0)
sum(slow_valid(books$isbn))
#> [1] 14Source: R documentation: Sys.sleep
logged_valid(books$isbn[1:3]) prints the two log lines around the call and returns the three results unchanged. logged_round(3.14159, digits = 2) shows why the wrapper takes the dots: the named argument digits reaches round() as written, and the log counts two arguments. Both wrappers have the interface function(...), which is the price of generality; the section on match.call() below shows how to keep the formals of the original when that matters.
Retry
A retry operator calls a function again when it fails, up to a number of times. The test below uses a deterministic stand-in for an unreliable service: make_flaky() is itself a factory with a counter, and it fails a fixed number of times before it succeeds. The wrapper catches the condition with tryCatch() (Conditions and Error Handling), logs the failure, and finally re-signals the last condition object with stop(result), so the caller sees the original message.
# A function that fails a given number of times and then works.
make_flaky <- function(failures) {
calls <- 0L
function(x) {
calls <<- calls + 1L # nolint: assignment_linter.
if (calls <= failures) {
stop("temporary failure number ", calls, call. = FALSE)
}
x * 2
}
}
with_retry <- function(f, times = 3L) {
force(f)
force(times)
function(...) {
for (attempt in seq_len(times)) {
result <- tryCatch(f(...), error = \(e) e)
if (!inherits(result, "error")) {
return(result)
}
cat("attempt", attempt, "failed:", conditionMessage(result), "\n")
}
stop(result)
}
}
sturdy <- with_retry(make_flaky(failures = 2L))
sturdy(21)
#> attempt 1 failed: temporary failure number 1
#> attempt 2 failed: temporary failure number 2
#> [1] 42
hopeless <- with_retry(make_flaky(failures = 5L), times = 2L)
tryCatch(hopeless(21), error = \(e) conditionMessage(e))
#> attempt 1 failed: temporary failure number 1
#> attempt 2 failed: temporary failure number 2
#> [1] "temporary failure number 2"Source: R documentation: tryCatch
sturdy fails twice and succeeds on the third attempt, so the result is 42 after two log lines. hopeless is allowed two attempts for a function that fails five times, logs both failures and passes the second error to the caller, which prints its message. In real code a retry should also wait between attempts (compose it with with_delay()), catch only the error classes that are worth repeating and not every error, and give up quickly on errors that cannot go away, such as a bad argument.
Checking arguments
A decorator can also hold preconditions, so that the function it wraps stays focused on its job. with_checks() takes a named list of predicates, whose names are the messages, and tests the first argument against each of them before the call. The function it returns has the formals (x, ...), not those of is_valid_isbn13(); the last two lines show the difference, which is the cost of this style: the original function’s argument names are no longer visible with args() or in the editor’s tooltip. The defensive-programming page (Defensive Programming and Input Validation) covers checking inside the function with stopifnot() and friends, which is usually simpler when the checks belong to one function and are not shared.
source("scenario.R")
# Argument checking as a decorator: the checks live outside the function.
with_checks <- function(f, ...) {
force(f)
checks <- list(...)
function(x, ...) {
for (name in names(checks)) {
if (!isTRUE(checks[[name]](x))) {
stop("argument x must satisfy: ", name, call. = FALSE)
}
}
f(x, ...)
}
}
valid_isbn <- with_checks(
is_valid_isbn13,
"be a character vector" = is.character,
"have no missing values" = \(x) !anyNA(x)
)
valid_isbn(books$isbn[1:2])
#> [1] TRUE TRUE
tryCatch(valid_isbn(123), error = \(e) conditionMessage(e))
#> [1] "argument x must satisfy: be a character vector"
tryCatch(valid_isbn(c("9780306406157", NA)), error = \(e) conditionMessage(e))
#> [1] "argument x must satisfy: have no missing values"
# The wrapper has the formals of its own definition, not those of f.
names(formals(valid_isbn))
#> [1] "x" "..."
names(formals(is_valid_isbn13))
#> [1] "x"Source: R documentation: stopifnot
match.call() and sys.function() in a wrapper
Two tools from Functions help wrappers behave well. match.call() returns the call to the function that is running, with the arguments matched to the formals; inside a wrapper it describes the call to the wrapper, which is what the user typed, so an error message can show it (R documentation: match.call). sys.function() returns the function that is running, which is the wrapper itself, so a wrapper can read attributes that were attached to it, or refer to itself without a name (R documentation: sys.function). The last part of the script keeps the formals of the original function: formals(wrapper) <- formals(f) gives the wrapper the right interface; the body must then use those names, for instance by forwarding with match.call(), do.call() or explicit arguments.
# match.call() inside a wrapper reports the call to the wrapper, so an error
# can name what the user typed.
with_context <- function(f) {
force(f)
function(...) {
call <- match.call()
tryCatch(
f(...),
error = \(e) {
stop("in ", deparse(call), ": ", conditionMessage(e), call. = FALSE)
}
)
}
}
safe_log <- with_context(\(x) if (x < 0) stop("negative input") else log(x))
safe_log(1)
#> [1] 0
tryCatch(safe_log(-1), error = \(e) conditionMessage(e))
#> [1] "in safe_log(-1): negative input"
# sys.function() is the running function itself, so a wrapper can read its own
# attributes or call itself without a name.
with_name <- function(f, label) {
force(f)
force(label)
structure(
function(...) {
me <- sys.function()
cat("running", attr(me, "label"), "\n")
f(...)
},
label = label
)
}
named_sqrt <- with_name(sqrt, "square root")
named_sqrt(16)
#> running square root
#> [1] 4
attr(named_sqrt, "label")
#> [1] "square root"
# A wrapper that keeps the formals of f, so args() and help stay useful.
keep_formals <- function(wrapper, f) {
formals(wrapper) <- formals(f)
wrapper
}
wrapper <- function(...) "wrapped"
names(formals(keep_formals(wrapper, function(title, author = NA) NULL)))
#> [1] "title" "author"Source: R documentation: sys.function
Composing functions
Composition by hand
To compose two functions is to build one that applies the first and passes the result to the second. R has no operator for it in base R 4.6.1; the definition is one line, and the order of the arguments is a choice that you document: this page uses compose(f, g) to mean "f first, then g", which reads in the order of the calls. As in every factory, f and g must be forced. The script composes normalise_isbn() and is_valid_isbn13() and then shows what happens when they are not forced: a composed function whose parts are reassigned before its first call uses the new parts (a double that has become x - 100 gives -98 for an input of 1 where 4 was expected). The worst case is the trap that every tutorial author falls into once: f <- lazy_compose(f, f) makes the promise f refer to the variable f that has just been overwritten by the composed function itself, and calling it recurses until R reports a stack overflow. The script checks only the class of that error with inherits(failure, "stackOverflowError"), because the message depends on which stack runs out first.
source("scenario.R")
# Composition by hand: compose(f, g) runs f first, then g.
compose <- function(f, g) {
force(f)
force(g)
\(x) g(f(x))
}
check_isbn <- compose(normalise_isbn, is_valid_isbn13)
check_isbn(c("978-0-306-40615-7", "978-0-306-40615-8"))
#> [1] TRUE FALSE
# Without force(), the composed function reads f and g when it is first called.
lazy_compose <- \(f, g) \(x) g(f(x))
add_one <- \(x) x + 1
double <- \(x) x * 2
h <- lazy_compose(add_one, double)
double <- \(x) x - 100
h(1)
#> [1] -98
# The classic way to hit it: reassigning the name that was composed.
f <- \(x) x + 1
f <- lazy_compose(f, f)
failure <- tryCatch(f(1), error = \(e) e)
inherits(failure, "stackOverflowError")
#> [1] TRUE
# With force() the old f is captured and the reassignment is harmless.
f <- \(x) x + 1
f <- compose(f, f)
f(1)
#> [1] 3Composing many functions with Reduce()
Reduce(f, x) folds a list with a binary function. When the elements are functions and the binary function is compose, the result is the composition of all of them in the order of the list (R documentation: funprog). Reduce() forces the functions it applies since R 3.2.0, so the inline form Reduce(\(f, g) \(x) g(f(x)), fs) is safe even without force(); a named compose() that forces its arguments is still the clearer version. A pipeline that applies the functions to a value, without building the composed function, is a Reduce() where the accumulator is the value: Reduce(\(acc, step) step(acc), steps, x). With accumulate = TRUE the intermediate values are kept, which makes it easy to inspect where a pipeline went wrong.
source("scenario.R")
compose <- function(f, g) {
force(f)
force(g)
\(x) g(f(x))
}
# Reduce() folds a list of functions into one: the first runs first.
clean_title <- Reduce(
compose,
list(trimws, tolower, \(s) gsub("[[:space:]]+", " ", s))
)
clean_title(c(" The Quiet Streets ", "THE LONG TABLE"))
#> [1] "the quiet streets" "the long table"
# The same fold with the composition written inline.
pipeline <- Reduce(\(f, g) \(x) g(f(x)), list(trimws, toupper, nchar))
pipeline(c(" ab ", "xyz"))
#> [1] 2 3
# A variadic version: feed each function the result of the previous one.
pipe_through <- function(...) {
steps <- list(...)
\(x) Reduce(\(acc, step) step(acc), steps, x)
}
known <- \(y) y[!is.na(y)]
decade <- \(y) y %/% 10L * 10L
label <- \(d) paste0(d, "s")
decade_labels <- pipe_through(known, decade, label, unique)
decade_labels(books$year)
#> [1] "1970s" "1810s" "2010s" "2020s" "2000s" "1990s"
# The same steps at the call site, with the native pipe.
books$year |> known() |> decade() |> label() |> unique()
#> [1] "1970s" "1810s" "2010s" "2020s" "2000s" "1990s"
# Reduce(accumulate = TRUE) keeps every intermediate result.
Reduce(
\(acc, step) step(acc),
list(trimws, toupper, nchar),
" ab ",
accumulate = TRUE,
simplify = FALSE
)
#> [[1]]
#> [1] " ab "
#>
#> [[2]]
#> [1] "ab"
#>
#> [[3]]
#> [1] "AB"
#>
#> [[4]]
#> [1] 2
#>Source: R documentation: funprog
The native pipe |> is the composition operator at the call site: x |> f() |> g() is parsed as g(f(x)), a syntax transformation that creates no function at all (Pipes and the Native Pipe). Use the pipe when the steps are known where the data is, and a composed function when the pipeline has to be stored, named, passed to vapply() or reused with different data. The two give the same labels for the shelf’s decades in the script above.
Partial application, do.call() and match.fun()
Partial application fixes some arguments of a function now and leaves the others for later. The hand-written version stores the fixed arguments in the closure and builds the call with do.call(), which takes a function (or its name) and a list of arguments (R documentation: do.call); match.fun() turns a function or a character string into a function and fails otherwise, so a helper can accept either (R documentation: match.fun). Negate() is the small function operator that base R provides next to Reduce() and Filter(): it returns the opposite predicate.
source("scenario.R")
# Partial application by hand: fix some arguments now, supply the rest later.
partial <- function(f, ...) {
force(f)
fixed <- list(...)
function(...) do.call(f, c(fixed, list(...)))
}
round_1 <- partial(round, digits = 1)
round_1(3.14159)
#> [1] 3.1
round_1(c(2.55, 7.349))
#> [1] 2.5 7.3
pad_id <- partial(sprintf, "L%03d")
pad_id(7L)
#> [1] "L007"
# match.fun() accepts a function or its name, so a helper can take either.
apply_twice <- function(f, x) {
f <- match.fun(f)
f(f(x))
}
apply_twice("sqrt", 16)
#> [1] 2
apply_twice(\(x) x + 1, 1)
#> [1] 3
# do.call() builds a call from a function (or name) and a list of arguments.
do.call("paste", list("L", "060", sep = "-"))
#> [1] "L-060"
args <- list(shelf_today, loans$loan_date[[1L]])
do.call(difftime, c(args, units = "days"))
#> Time difference of 532 days
fs <- list(min = min, max = max, mean = mean)
vapply(fs, \(f) f(books$pages, na.rm = TRUE), numeric(1))
#> min max mean
#> 88.0000 432.0000 268.2143
vapply(fs, do.call, numeric(1), args = list(books$pages, na.rm = TRUE))
#> min max mean
#> 88.0000 432.0000 268.2143
# Negate() is a function operator in base R: it returns the opposite predicate.
is_old <- \(year) !is.na(year) & year < 2000L
Negate(is_old)(c(1990L, 2010L))
#> [1] FALSE TRUE
sum(vapply(books$year, Negate(is.na), logical(1)))
#> [1] 15Source: R documentation: do.call
do.call("paste", list("L", "060", sep = "-")) is the programmatic form of paste("L", "060", sep = "-"), useful when the arguments are only known as a list. The vapply() over a list of functions in the middle of the script is the idiomatic way to call several summary functions on the same data; the last vapply() shows do.call itself used as the function that is applied, with the argument list passed as a named argument.
Packages: purrr
The purrr package has compose() and partial() (labelled package documentation: purrr package documentation: compose and purrr package documentation: partial). purrr::compose() composes from the right by default, as in mathematical notation, so compose(inc, dbl)(3) is inc(dbl(3)), which is 7; .dir = "forward" reverses the order. This differs from the compose(f, g) of this page, which runs f first: a good reason to document the direction whenever a helper like this is defined. purrr::partial() returns a function of class purrr_function_partial, and the package goes beyond what the hand-written version above does. The script uses purrr 1.2.2.
library(purrr)
# purrr::compose() composes right to left by default, like f(g(x)).
inc <- \(x) x + 1
dbl <- \(x) x * 2
compose(inc, dbl)(3)
#> [1] 7
compose(inc, dbl, .dir = "forward")(3)
#> [1] 8
# purrr::partial() fixes arguments; the result prints as <partialised>.
round_1 <- partial(round, digits = 1)
round_1(3.14159)
#> [1] 3.1
class(round_1)
#> [1] "purrr_function_partial" "function"
# The base R equivalents of the two calls above.
Reduce(\(f, g) \(x) g(f(x)), list(inc, dbl))(3)
#> [1] 8
packageVersion("purrr") >= "1.2.2"
#> [1] TRUEUpcoming in R-devel: Compose(), Partial() and Funcall()
The development version of R (4.7.0, not released) lists three new base functions. The R-devel NEWS entry reads: "New higher-order functions Compose(), Funcall() and Partial()." (R NEWS: Changes in R-devel). They are upcoming: they do not exist in R 4.6.1, the release this page was checked against, and no example on this page calls them. The first script below confirms that exists() is FALSE for each name on 4.6.1. The release NEWS has no entry for them and the release help page of Reduce() does not document them, so their exact arguments should be taken from the 4.7.0 documentation when it is released; the entry may also change before then. Until a release that has them is the version you can rely on, the hand-written compose() and partial() above, or purrr::compose() and purrr::partial() (purrr 1.2.2), are the current equivalents. The name Funcall already appears in the help page of Reduce() in R 4.6.1, where its examples define it by hand as function(f, ...) f(...) and use it with Reduce(right = TRUE) to apply a list of functions one after the other; the second script runs the same idea under a lower-case name so that it cannot be confused with the upcoming function.
# Compose(), Partial() and Funcall() are listed in the R-devel NEWS only.
# On R 4.6.1 none of them exists.
vapply(c("Compose", "Partial", "Funcall"), exists, logical(1))
#> Compose Partial Funcall
#> FALSE FALSE FALSE
getRversion() < "4.7.0"
#> [1] TRUESource: R NEWS: Changes in R-devel
# What the upcoming Funcall() name suggests, written by hand: apply a function
# to arguments. Reduce() can then fold a list of functions into one result.
funcall <- function(f, ...) f(...)
funcall(sum, 1, 2, 3)
#> [1] 6
# log(exp(acos(cos(0)))), applied from the right.
Reduce(funcall, list(log, exp, acos, cos), 0, right = TRUE)
#> [1] 0
# A function operator: iterate(f, n) applies f n times.
iterate <- function(f, n = 1L) {
force(f)
force(n)
\(x) Reduce(funcall, rep.int(list(f), n), x, right = TRUE)
}
add_week <- \(d) d + 7L
iterate(add_week, 3L)(as.Date("2026-06-30"))
#> [1] "2026-07-21"
# Fixed points: a few steps of the Babylonian square root of 2.
step <- \(t) (t + 2 / t) / 2
iterate(step, 5L)(1)
#> [1] 1.414214
sqrt(2)
#> [1] 1.414214Source: R documentation: funprog
Vectorize() and Map()
Vectorize() is a function operator in base R: it takes a function written for scalar arguments and returns one that maps it over vectors by calling mapply() (R documentation: Vectorize). Map() does the same job with an explicit call and returns a list. Neither makes the code faster: they hide a loop, and a function that works on whole vectors, such as the vectorised substr() itself, is better. Both are described in more depth in The apply Family and Functionals, and the difference between a vectorised function and a loop in Vectorisation and Iteration.
source("scenario.R")
# Vectorize() turns a function of scalars into one that maps over arguments.
short_title <- \(title, width) substr(trimws(title), 1L, width)
vshort <- Vectorize(short_title)
vshort(books$title[1:3], 8L)
#> The Dispossessed Pride and Prejudice the quiet streets
#> "The Disp" "Pride an" "the quie"
vshort(books$title[1:3], c(3L, 5L, 7L))
#> The Dispossessed Pride and Prejudice the quiet streets
#> "The" "Pride" "the qui"
# Map() does the same job with an explicit function and returns a list.
Map(short_title, books$title[1:2], 4L) |> unlist(use.names = FALSE)
#> [1] "The " "Prid"
# Vectorize() is a function operator: it returns a closure around mapply().
class(vshort)
#> [1] "function"
names(formals(vshort))
#> [1] "title" "width"Source: R documentation: Vectorize
Recursion
A recursive function calls itself on a smaller part of its input until a base case stops it. R-intro says that functions may be recursive and may themselves define functions within themselves, and its worked example is a recursive numerical integration (An Introduction to R: Recursive numerical integration). Recursion fits data that is itself nested, such as a tree, a directory or the result of parsing JSON: the function mirrors the shape of the data, and a loop would need an explicit stack. For a flat vector a vectorised function or a loop is shorter and faster; the question of when to recurse and when to iterate is taken further in Vectorisation and Iteration.
Walking a tree
The shelf as a tree: the genres of books (trimmed and lower-cased), each split by decade, with the titles as leaves. A node is either a character vector (a leaf) or a list of child nodes, so every walker has the same shape: handle the leaf, otherwise combine the results of the children. vapply() over the children keeps the combination type-safe, and the recursive call is count_titles itself, passed by name, which is how a recursive function is normally handed to an apply function.
source("scenario.R")
# A tree of the shelf: genre -> decade -> titles.
shelf <- books
shelf$genre <- tolower(trimws(shelf$genre))
shelf$decade <- ifelse(
is.na(shelf$year),
"unknown",
paste0(shelf$year %/% 10L * 10L, "s")
)
tree <- lapply(split(shelf, shelf$genre), \(g) {
lapply(split(g, g$decade), \(d) trimws(d$title))
})
names(tree)
#> [1] "classic" "fiction" "mystery" "non-fiction"
#> [5] "poetry" "science fiction"
names(tree$classic)
#> [1] "1810s" "1990s"
# A leaf is a character vector; everything else is a list of children.
count_titles <- function(node) {
if (is.character(node)) {
return(length(node))
}
sum(vapply(node, count_titles, integer(1)))
}
count_titles(tree)
#> [1] 16
vapply(tree, count_titles, integer(1))
#> classic fiction mystery non-fiction poetry
#> 2 4 2 3 2
#> science fiction
#> 3
depth <- function(node) {
if (is.character(node)) {
return(0L)
}
1L + max(vapply(node, depth, integer(1)))
}
depth(tree)
#> [1] 2
# A walker that prints the tree, one indentation level per recursion level.
show_tree <- function(node, label = "shelf", indent = 0L) {
pad <- strrep(" ", indent)
if (is.character(node)) {
cat(pad, label, ": ", length(node), " title(s)\n", sep = "")
return(invisible(NULL))
}
cat(pad, label, "\n", sep = "")
for (name in names(node)) {
show_tree(node[[name]], name, indent + 1L)
}
invisible(NULL)
}
show_tree(tree[c("classic", "poetry")])
#> shelf
#> classic
#> 1810s: 1 title(s)
#> 1990s: 1 title(s)
#> poetry
#> 2000s: 1 title(s)
#> unknown: 1 title(s)
# The recursion on the base case: an empty list has no children.
count_titles(list())
#> [1] 0count_titles() finds all 16 titles; mapped over the top level it gives the number per genre; depth() is 2, because the leaves are two levels down. show_tree() is a walker with a side effect: it prints one line per node and uses the recursion level (indent) as the only state. The empty list returns 0 because vapply() over nothing is integer(0) and its sum is 0, which is the base case for a node without children. For trees with an unknown shape the recursive functions rapply() (apply a function to the leaves of a nested list) and unlist() (flatten it) do part of this work, see Lists.
Recall() and sys.function()
Recall() calls the function in which it appears, whatever its current name; the help page describes it as a placeholder for the name of the function in which it is called, which lets a recursive function keep working after it is renamed (R documentation: Recall). That is its entire use, and it is rare in practice: functions are almost always called by their name, and a name that is visible in the file is easier to read than a placeholder. The help page also carries a caveat: Recall() does not work correctly when it is passed as a function argument, for instance to the apply family. The script shows the working case, the renaming case that Recall() was invented for, the failure inside vapply() with its real message, and the alternative that works in an apply call, sys.function(), which returns the running function itself (R documentation: sys.function). The help page also points to local() as another way to write anonymous recursive functions.
# Recall() calls the function it appears in, whatever its current name.
fact <- function(n) if (n <= 1) 1 else n * Recall(n - 1)
fact(5)
#> [1] 120
factorial_of <- fact
rm(fact)
factorial_of(5)
#> [1] 120
# A recursive function that names itself breaks when it is renamed.
fact_named <- function(n) if (n <= 1) 1 else n * fact_named(n - 1)
fact_copy <- fact_named
rm(fact_named)
tryCatch(fact_copy(5), error = \(e) conditionMessage(e))
#> [1] "could not find function \"fact_named\""
# Recall is rarely needed, because a function is usually called by its name.
# It does not work when handed to another function as an argument.
depth <- function(node) {
if (!is.list(node)) {
return(0L)
}
1L + max(0L, vapply(node, Recall, integer(1)))
}
tryCatch(depth(list(1, list(2))), error = \(e) conditionMessage(e))
#> [1] "argument \"FUN\" is missing, with no default"
# sys.function() names the running function without a global name.
depth_sys <- function(node) {
if (!is.list(node)) {
return(0L)
}
self <- sys.function()
1L + max(0L, vapply(node, self, integer(1)))
}
depth_sys(list(1, list(2, list(3))))
#> [1] 3
# Outside any function, Recall has nothing to call.
tryCatch(Recall(1), error = \(e) conditionMessage(e))
#> [1] "'Recall' called from outside a closure"Source: R documentation: Recall
fact_copy() fails because fact_named() looks for the name it was defined under, and that name was removed. Recall() has no such dependency. Passed to vapply() the call Recall is made by vapply() and not by the function, and R reports argument "FUN" is missing, with no default; in this situation name the function or capture sys.function() in a variable first, as depth_sys() does. Outside any function Recall() fails with 'Recall' called from outside a closure.
Stack limits
Every call of a closure creates an evaluation frame, and R keeps several stacks and limits in order to survive runaway recursion. The help page of Cstack_info() says that the "evaluation depth" is the number of nested R expressions currently under evaluation, and that this has a limit controlled by options("expressions") (R documentation: Cstack_info). The same help page documents the other values it reports: size and current (the size and the estimated usage of the C stack in bytes, or NA if unknown), direction and eval_depth. The sizes depend on the platform and on how R was started, so this page prints only the names. The expressions option sets a limit on the number of nested expressions that will be evaluated, with valid values from 25 to 500000 and a default of 5000; the help page warns that raising it may cause a segfault from overflow of the C stack, and that once the limit is reached an error is thrown (R documentation: options).
# Cstack_info() reports the C stack and the evaluation depth. The sizes are
# platform-specific, so only the names are shown here.
names(Cstack_info())
#> [1] "size" "current" "direction" "eval_depth"
Cstack_info()[["eval_depth"]] > 0L
#> [1] TRUE
getOption("expressions")
#> [1] 5000
count_down <- function(n) if (n == 0) 0 else 1 + count_down(n - 1)
count_down(1000)
#> [1] 1000
# Too deep for the default limit of 5000 nested expressions.
too_deep <- tryCatch(count_down(10000), error = \(e) e)
conditionMessage(too_deep)
#> [1] "evaluation nested too deeply: infinite recursion / options(expressions=)?"
class(too_deep)
#> [1] "expressionStackOverflowError" "stackOverflowError"
#> [3] "error" "condition"
# A lower limit makes the same call fail earlier; restore it afterwards.
old <- options(expressions = 100)
low <- tryCatch(count_down(1000), error = \(e) conditionMessage(e))
options(old)
low
#> [1] "evaluation nested too deeply: infinite recursion / options(expressions=)?"
getOption("expressions")
#> [1] 5000Source: R documentation: Cstack_info
The recursion count_down(1000) works, count_down(10000) hits the default limit of 5000 nested expressions, and the error text is evaluation nested too deeply: infinite recursion / options(expressions=)?. Lowering expressions to 100 makes the thousand-level call fail with the same message; the example restores the option afterwards with the value returned by options(), which is the pattern to use whenever a script changes an option temporarily. How many levels of recursion fit under 5000 depends on the function and on whether it has already been byte-compiled by the JIT compiler (the evaluation depth that Cstack_info() reports for the same call can differ between the first calls and later ones), which is why the example does not state a depth.
Four stack overflow errors
Since R 4.2.0 the stack errors have classes (the NEWS entry says that stack-overflow errors are now signalled as errors inheriting from class stackOverflowError), documented on one help page (R documentation: stackOverflowError; R NEWS: Changes in R 4.2.0). They can be caught with an exiting handler from tryCatch(), and the help page warns that calling handlers from withCallingHandlers() may fail because there may not be enough stack left to run them.
| Class | Raised when | Message seen on R 4.6.1 |
|---|---|---|
|
the evaluation depth exceeds |
|
|
the node stack used by the byte code engine overflows |
|
|
the C stack overflows; the |
|
|
the pointer protection stack overflows (see |
not reproduced on this page |
Raising expressions does not make the C stack or the node stack bigger, so past a point a different error comes first. The next script raises the limit to its maximum of 500000 and recurses again: the first call fails with the node stack error, and a recursion nested through lapply() (which uses C stack for every level) fails with the C stack error. The number inside the C stack message is the usage in bytes. It is platform-specific, so the script replaces every digit sequence with <n> and says so; cstack$usage holds the same number as a field of the condition object.
count_down <- function(n) if (n == 0) 0 else 1 + count_down(n - 1)
# Raising the limit does not make the stack bigger: another error comes first.
old <- options(expressions = 500000)
node <- tryCatch(count_down(1e5), error = \(e) e)
conditionMessage(node)
#> [1] "node stack overflow"
class(node)
#> [1] "nodeStackOverflowError" "stackOverflowError" "error"
#> [4] "condition"
# Nesting through lapply() consumes C stack for every level.
nest <- function(n) lapply(n, \(i) if (i == 0) 0 else nest(i - 1))
cstack <- tryCatch(nest(1e5), error = \(e) e)
options(old)
class(cstack)
#> [1] "CStackOverflowError" "stackOverflowError" "error"
#> [4] "condition"
# The number in the message is the stack usage in bytes; it varies by platform.
gsub("[0-9]+", "<n>", conditionMessage(cstack))
#> [1] "C stack usage <n> is too close to the limit"
is.numeric(cstack$usage)
#> [1] TRUEWhich of the errors a given recursion produces depends on the function, on whether it has been byte-compiled and on the platform’s stack size, so the examples on this page that show one print its message and the code should not rely on the specific class. The only portable conclusion is the one the error message itself gives: infinite recursion is a bug in the function, or the problem is too deep for recursion on a stack of this size, and the loop or the tail-call version of the next section is the remedy.
Tail calls: Tailcall() and Exec() (experimental)
A call is a tail call when it is the last thing a function does: its value is returned as the value of the caller, with nothing left to compute. A language that eliminates tail calls can run a tail-recursive function in constant stack space. R does not do this by itself. R 4.4.0 added two functions to make it possible, and the NEWS entry describes them like this: "New experimental functions Tailcall() and Exec() to support writing stack-space-efficient recursive functions." (R NEWS: Changes in R 4.4.0). They are experimental: the help page says they "may be changed or dropped in future released versions of R", so code that uses them for production work should be isolated behind one function and tested on every R upgrade. They are documented on one help page (R documentation: Tailcall and Exec).
Tailcall(FUN, ...) evaluates a call to FUN (a function or the name of one) with the arguments in the current environment. If the Tailcall() expression is in tail position in an R function and no on.exit() expression is set, the evaluation context of the new call replaces the currently executing one instead of being added to the stack. The help page adds two consequences: because of lazy evaluation it may be necessary to force some arguments to avoid accumulating deferred evaluations, and stack traces from traceback() or sys.calls() show only the call given to Tailcall(), not the call whose stack entry was replaced.
A tail-recursive sum
The first script sums the integers from 1 to n with an accumulator. The plain recursive version works for a thousand levels and fails for 100000 with a stack error (here the node stack message). The Tailcall() version computes the same sum for a hundred thousand and for a million levels, and its result for 1e5 is the Gauss formula n(n+1)/2 = 5000050000. Note the line acc <- acc + n before the call: without it the argument acc + n would be passed as an unevaluated promise, each promise referring to the previous one, and the chain of 100000 promises would be forced at once at the end, which overflows the stack even though the calls themselves took no space. The last part of the script shows exactly that, under the expressions error. This is the "force some arguments" caveat from the help page, and it is the same lesson as the function-factory trap above, in a different place.
# Plain recursion: one evaluation frame per element.
sum_plain <- function(n, acc = 0) {
if (n == 0) {
return(acc)
}
sum_plain(n - 1, acc + n)
}
sum_plain(1000)
#> [1] 500500
tryCatch(sum_plain(1e5), error = \(e) conditionMessage(e))
#> [1] "node stack overflow"
# Tailcall() (R 4.4.0, experimental) replaces the running call with the new one.
sum_tail <- function(n, acc = 0) {
if (n == 0) {
return(acc)
}
acc <- acc + n # evaluate the accumulator now, not as a chain of promises
Tailcall(sum_tail, n - 1, acc)
}
sum_tail(1000)
#> [1] 500500
sum_tail(1e5)
#> [1] 5000050000
sum_tail(1e6)
#> [1] 500000500000
# The first argument can also be the name of the function.
count_to <- function(n, i = 0L) {
if (i >= n) {
return(i)
}
Tailcall("count_to", n, i + 1L)
}
count_to(50000L)
#> [1] 50000
# Lazy evaluation: passing acc + n unevaluated builds a chain of promises
# that is forced all at once, at the very end.
sum_lazy <- function(n, acc = 0) {
if (n == 0) {
return(acc)
}
Tailcall(sum_lazy, n - 1, acc + n)
}
sum_lazy(1000)
#> [1] 500500
tryCatch(sum_lazy(1e5), error = \(e) conditionMessage(e))
#> [1] "evaluation nested too deeply: infinite recursion / options(expressions=)?"A tail-recursive walk over the loans
The second script is the same idea on the shelf. longest() walks a vector one element at a time, carrying the best value so far, and ends with a Tailcall(); the loan lengths of the loans table are 60 elements, and the same function over the table repeated 500 times is 30000 recursive calls, which a function written with plain recursion cannot do. Both versions give 37 days for the table itself, which is max(loan_length, na.rm = TRUE). In real code max() is the answer for this particular job, since it is vectorised; the point is the shape of the recursion.
source("scenario.R")
# The longest loan in a vector, walking one element per recursive call.
loan_length <- as.integer(loans$return_date - loans$loan_date)
longest <- function(x, i = 1L, best = NA_integer_) {
if (i > length(x)) {
return(best)
}
if (!is.na(x[[i]]) && (is.na(best) || x[[i]] > best)) {
best <- x[[i]]
}
Tailcall(longest, x, i + 1L, best)
}
longest(loan_length)
#> [1] 37
max(loan_length, na.rm = TRUE)
#> [1] 37
# 60 loans repeated 500 times make 30000 calls; Tailcall() does not grow
# the stack.
longest(rep(loan_length, 500L))
#> [1] 37
# The same function written with plain recursion exhausts the expressions limit.
longest_plain <- function(x, i = 1L, best = NA_integer_) {
if (i > length(x)) {
return(best)
}
if (!is.na(x[[i]]) && (is.na(best) || x[[i]] > best)) {
best <- x[[i]]
}
longest_plain(x, i + 1L, best)
}
longest_plain(loan_length)
#> [1] 37
tryCatch(
longest_plain(rep(loan_length, 500L)),
error = \(e) conditionMessage(e)
)
#> [1] "node stack overflow"When the context is not reused
The reuse of the frame is a property of the call, not of the function name. The script below counts the frames on the stack at the bottom of a 50-level recursion: 51 for plain recursion (the call from the top level plus fifty more), 1 for Tailcall(), which is the stack-trace remark of the help page in numbers. It then adds an on.exit() expression to the function, which turns the reuse off; the frame count goes back to 51 and a deep call fails with the C stack error. A Tailcall() that is not in tail position, such as 1 + Tailcall(...), is evaluated in the ordinary way and gives the correct value for a small input (10), but it grows the stack and fails for 1e5.
# The number of frames on the call stack at the bottom of the recursion.
probe_plain <- function(n) {
if (n == 0) length(sys.calls()) else probe_plain(n - 1)
}
probe_tail <- function(n) {
if (n == 0) length(sys.calls()) else Tailcall(probe_tail, n - 1)
}
probe_plain(50)
#> [1] 51
probe_tail(50)
#> [1] 1
# An on.exit() expression stops the context from being reused.
probe_exit <- function(n) {
on.exit(invisible(NULL))
if (n == 0) length(sys.calls()) else Tailcall(probe_exit, n - 1)
}
probe_exit(50)
#> [1] 51
generic <- \(e) gsub("[0-9]+", "<n>", conditionMessage(e))
tryCatch(probe_exit(1e5), error = generic)
#> [1] "C stack usage <n> is too close to the limit"
# Not in tail position: the addition happens after the call returns.
not_tail <- function(n) {
if (n == 0) {
return(0)
}
1 + Tailcall(not_tail, n - 1)
}
not_tail(10)
#> [1] 10
tryCatch(not_tail(1e5), error = generic)
#> [1] "C stack usage <n> is too close to the limit"
# The target must be a function or the name of one.
bad_target <- function() Tailcall("no_such_function", 1)
tryCatch(bad_target(), error = \(e) conditionMessage(e))
#> [1] "could not find function \"no_such_function\""The last line of the script shows what happens when the first argument names a function that does not exist: could not find function "no_such_function", as for any other call.
Exec()
Exec(expr, envir) evaluates a call expression expr in the environment envir (by default the environment from which Exec() is called), replacing the current call context under the same rules as Tailcall(). The help page suggests it for simplifying the call stack of functions that build an expression and then evaluate it, and its example is a simplified do.call(). The script uses the stack depth to show the effect (a function called through Exec() has one frame fewer) and adapts the help page’s docall() so that the frames of do.call() and of docall() can be listed: the do.call() frame stays on the stack (it is called runner in the output because that is the name it was called under) and the Exec()-based version leaves nothing behind.
# Exec() evaluates a call expression in place of the current call context.
inner <- function() length(sys.calls())
via_call <- function() inner()
via_exec <- function() Exec(quote(inner()), environment())
via_call()
#> [1] 2
via_exec()
#> [1] 1
# A simplified do.call() written with Exec() (adapted from the help page):
# its own frame is not left on the call stack.
docall <- function(what, args) {
Exec(as.call(c(list(substitute(what)), args)), parent.frame())
}
frames <- function() {
vapply(sys.calls(), \(cl) as.character(cl[[1L]])[[1L]], character(1))
}
run <- function(runner) runner("frames", list())
run(do.call)
#> [1] "run" "runner" "frames"
run(docall)
#> [1] "run" "frames"Trampolines: the alternative before R 4.4.0
Before Tailcall(), the way to run a deeply tail-recursive computation in constant stack space was a trampoline: the recursive function does not call itself but returns a function (a thunk) that would make the next step, and a driver loop keeps calling the thunks until a plain value comes back. The stack never holds more than the driver and one step. It works on every R version, it needs no experimental feature, and it asks the author to write the recursion in the thunk-returning style; the acc <- acc + n line is needed here for the same reason as above. Today a plain loop is shorter than either a trampoline or Tailcall() for a computation like this, and recursion remains the right shape only when the problem is recursive (the tree above) and shallow.
# A trampoline: each step returns a function (a thunk) instead of calling
# itself, and a loop keeps calling thunks until a plain value comes back.
trampoline <- function(f, ...) {
result <- f(...)
while (is.function(result)) {
result <- result()
}
result
}
sum_thunk <- function(n, acc = 0) {
if (n == 0) {
return(acc)
}
acc <- acc + n
\() sum_thunk(n - 1, acc)
}
trampoline(sum_thunk, 1e5)
#> [1] 5000050000
trampoline(sum_thunk, 10)
#> [1] 55Common errors
The messages below were captured from R 4.6.1. The ones about stacks are in the sections above, and the misspelled name in <<- is in the counter section, because it is not an error at all.
msg <- \(expr) tryCatch(expr, error = \(e) conditionMessage(e))
# A factory that returns a value instead of a function.
make <- function(n) n
msg(make(1)(2))
#> [1] "attempt to apply non-function"
# match.fun() and do.call() with something that is not a function.
msg(match.fun("no_such_function"))
#> [1] "object 'no_such_function' of mode 'function' was not found"
msg(match.fun(1))
#> [1] "'1' is not a function, character or symbol"
msg(do.call(1, list()))
#> [1] "'what' must be a function or character string"
msg(do.call("sum", 1))
#> [1] "second argument must be a list"
msg(Reduce(1, 1:3))
#> [1] "'1' is not a function, character or symbol"
# A closure called with too many arguments.
make_id <- function() \(x) x
msg(make_id()(1, 2))
#> [1] "unused argument (2)"
# A vapply() over closures: FUN.VALUE says what a result may be.
msg(vapply(1:2, \(i) \() i, numeric(1)))
#> [1] "values must be type 'double',\n but FUN(X[[1]]) result is type 'closure'"
# A memo cache: a missing key, an empty key.
cache <- new.env()
msg(get("k", envir = cache))
#> [1] "object 'k' not found"
cache[["k"]]
#> NULL
msg(exists("", envir = cache))
#> [1] "invalid first argument"
msg(assign("", 1, envir = cache))
#> [1] "attempt to use zero-length variable name"
# Recall outside a function, and Map with lengths that do not recycle.
msg(Recall(1))
#> [1] "'Recall' called from outside a closure"
tryCatch(
Map(\(x, y) x + y, 1:3, 1:2),
warning = \(w) conditionMessage(w)
)
#> [1] "longer argument not a multiple of length of shorter"Source: R documentation: match.fun
| Message | Usual cause | Fix |
|---|---|---|
|
a factory returned a value, not a function, and the result was called |
end the factory with a function expression, and check |
|
|
pass the function, not its result ( |
|
the first argument of |
pass the function object or its name |
|
the arguments of |
|
|
the closure returned by a factory has fewer parameters than the call supplies |
check the formals with |
|
|
use |
|
|
test with |
|
|
call the function by name |
|
|
pass equal lengths, or recycle explicitly |
Source: R documentation: do.call
Pitfalls collected
| Pitfall | Why it happens | What to do |
|---|---|---|
All closures from a |
the factory argument is an unforced promise on the loop variable, which has its final value when the closures run |
|
A value passed to |
|
|
A composed function recurses forever after |
the argument |
|
A counter never increases |
|
|
A global variable appears after calling a closure |
|
check the spelling, or keep state in an environment |
The memoised function returns stale values |
it is not pure, or its data changed |
cache only pure functions, and provide a reset |
|
an empty string is not a legal name |
map arguments to keys that are always legal names |
|
|
name the function, or use |
Deep recursion fails with a stack error |
frames, nodes or C stack run out before the base case |
a loop, a vectorised function, a trampoline or |
|
it is not in tail position, or an |
put it in tail position and remove |
A lazy accumulator with |
|
compute the new value before the call, as the help page does |
Version notes
| Version | Change | Source |
|---|---|---|
R 3.2.0 |
Higher-order functions such as the apply functions and |
NEWS 3.2.0 |
R 4.1.0 |
|
NEWS 4.1.0 |
R 4.2.0 |
Stack-overflow errors inherit from the class |
NEWS 4.2.0, |
R 4.4.0 |
|
NEWS 4.4.0, |
R-devel (4.7.0, under development) |
|
R-devel NEWS |
purrr 1.2.2 (CRAN) |
|
purrr documentation |
The R-devel entries are in the development NEWS and may change before 4.7.0 is released. For the full list of changes by release see What’s New and Migrating Older Code.
Where older tutorials differ
| Older tutorial | Current practice | Why | Version |
|---|---|---|---|
A counter kept in a global variable that a function changes with |
A factory that returns the counter function, so the state lives in the closure |
two counters need two independent states, and the global can be changed from anywhere |
always |
|
Call the function by name; |
|
always |
|
|
a misspelled name silently creates a global variable |
always |
Factories that store their argument unforced |
|
the closure reads the argument when it first runs, not when it is made |
R 3.2.0 for the apply family; loops still need it |
|
A cache environment inside the memoising closure, with |
the global list can be changed from anywhere, shares state between functions, and cannot be created per function |
always |
|
|
a declared result type fails loudly; |
R 4.1.0 |
"R cannot do tail calls, so write a loop" |
A loop is still the first choice; |
R 4.4.0 adds experimental |
R 4.4.0 |
|
|
the pipe is a syntax transformation, and a composed function is a value |
R 4.1.0 |
A helper package imported only for |
A two-line |
keep the direction of composition documented; the development version of R lists its own |
R-devel |
|
|
the function object cannot be hidden or masked by a different function of the same name at call time |
always |
Where to go next
-
The apply Family and Functionals for
lapply(),vapply(),Map(),Reduce(),Filter()andrapply(), the functionals that factories feed. -
Scoping, Environments and Closures for environments, enclosures,
<<-and what a closure keeps alive. -
Lazy Evaluation and Promises for promises,
force()and the shelf version of the factory trap. -
Functions for argument matching, the dots,
on.exit(),sys.function(),match.call()andRecall(). -
Pipes and the Native Pipe for composition at the call site.
-
Vectorisation and Iteration for loops, preallocation and when not to recurse.
-
Conditions and Error Handling for
tryCatch(), the basis of the retry operator. -
Reference Classes and R6 for mutable state that outgrows a closure.
-
Debugging for
traceback()and the call stack thatTailcall()shortens. -
Performance, Profiling and Benchmarking for measuring whether memoisation or a loop is faster.
References
-
The R Language Definition: Function objects, Lexical environment, Argument evaluation
-
An Introduction to R: Scope, Writing your own functions, Recursive numerical integration
-
R FAQ 3.3.1: Lexical scoping
-
R FAQ 7.12: What are the enclosing and parent environments?
-
R NEWS: Changes in R 3.2.0; R NEWS: Changes in R 4.1.0, 4.2.0 and 4.4.0; R-devel NEWS