Evaluation and Non-Standard Evaluation

In standard evaluation a function uses the value of each argument: R wraps the argument in a promise, which holds the expression and the caller’s environment, and evaluates it in that environment the first time the function needs it (The R Language Definition: Argument evaluation), so the function sees a number, a string or a data frame. In non-standard evaluation the function looks at the expression the caller typed, through substitute() (see Lazy Evaluation and Promises, which shows that the expression lives in the promise), and then does something else with it: it prints it as a label, evaluates it in a place of its own choosing, or never evaluates it. The Language Definition introduces the building blocks in its chapter on computing on the language (The R Language Definition: Computing on the language), and eval() is described there as the means of evaluating call objects, with the environment as the part that often has to be set to something else (The R Language Definition: More on evaluation).

Source: R documentation: substitute

The calls in the last block do five different things with their arguments. label() turns the expression into text and never evaluates it. library(stats) and library("stats") do the same because library() looks at its argument with substitute() and accepts either a name or a string (the argument character.only decides). data.frame(pages, years) takes the column names from the expressions of its arguments, which is why years * 2 becomes the awkward name years...2. A formula quotes both sides. And with() and subset() evaluate the expression inside the data frame, which is the subject of most of this page.

The motivation is that data analysis is mostly about columns, and that writing books$pages three times in an expression is noise. The rlang documentation traces data masking back to the S language, which introduced attach() for data scopes in 1988 and data-masked formulas for modelling functions in 1993; it credits Peter Dalgaard’s frametools package of 1997 as the origin of base::transform() and base::subset(), and Luke Tierney’s change that made formulas remember their environment (published in R 1.1.0) as the step that made hygienic data masking possible. with() followed in 2001 (rlang package documentation: data masking). Thomas Lumley’s note "Standard Nonstandard Evaluation Rules", cited from the ?with help page, is the closest thing to an official statement of the conventions (R Development: Standard Nonstandard Evaluation Rules). The price of the convenience is described in every one of those sources in the same terms: the function has to be told where to look, and the user can no longer tell from the call alone which names are columns and which are variables.

The base functions that use the pattern, and what each one does with the expression, are collected here; the ones that are not obvious are taken apart below.

Source: R documentation: eval, evalq, local

Every non-standard evaluation function answers three questions, and a reader who asks them of an unfamiliar function can predict its behaviour. What does it capture, and when? Where does it evaluate what it captured: in which environment, and with which data in front of it? And what does it return, given that assignments made while evaluating usually vanish with the temporary environment? The sections that follow keep coming back to the second question, because every pitfall on this page is a case of the function and its user giving different answers to it.

eval(expr, envir = parent.frame(), enclos = if (is.list(envir) || is.pairlist(envir)) parent.frame() else baseenv()) evaluates the object expr in the environment described by envir. The help page lists what expr may be: a call, an expression vector, a name (looked up, and its binding evaluated), a promise, or any basic type such as a vector, a function or an environment, which are returned unchanged. evalq(expr, envir, enclos) is eval(quote(expr), ...); the help page adds that eval evaluates its first argument in the current scope before passing it to the evaluator, and that evalq avoids this. eval.parent(expr, n) is a shorthand for eval(expr, parent.frame(n)), and local() is evalq() with a new environment as the default (R documentation: eval, evalq, local). The Language Definition says that by default the evaluation frame is the one from which eval is called, and that very often the relevant frame is that of the parent of the current one, because an object that came out of substitute() contains variables that make sense to the caller only (The R Language Definition: More on evaluation).

Source: The R Language Definition: More on evaluation

The last two evaluations in the block show a mistake that is easy to make. A string is a constant: eval("x * 2") returns the string, and only parse(text = ) or str2lang() turn text into something eval() can run (str2lang() was added in R 3.6.0 as a streamlined parse(text = , keep.source = FALSE) for a single expression, see the version notes). And eval() is an ordinary function, so its argument is evaluated before eval() sees it.

Source: R documentation: eval, evalq, local

eval(expr) evaluates the call stored in expr. eval(quote(expr)) evaluates the symbol expr, whose value is that call, so it returns the call itself, unevaluated; one more eval() runs it. And eval(x * 2) is not a quotation at all: R computes 20 first and eval() evaluates the constant 20. When the argument is already the value you wanted, the extra eval() is harmless; when the argument has side effects, they happen once for the argument and the result is evaluated again, which is why a function that receives its code unevaluated uses substitute() and not the value.

An expression vector (see Language Objects and Quoting) is evaluated element by element and the value is the value of the last one. The Language Definition points out the subtle difference from a block: a call nested in another call is evaluated as a subexpression, while an expression object stays quoted unless it is passed directly to eval() (The R Language Definition: Evaluation of expression objects).

Source: The R Language Definition: Evaluation of expression objects

eval(ex) ran both elements and returned 7. Substituting a call for x gave "numeric", because the call 2 + 2 was evaluated before mode() saw it; substituting an expression vector gave "expression", because the evaluator does not evaluate an expression object recursively. The two objects e and ee print identically, which the manual calls out as a source of confusion, and class() is the way to tell them apart.

envir can be an environment, which is used as it is; a list or a data frame, which is copied into a temporary environment; a pairlist, which is treated like a list; NULL, which is read as an empty list; or an integer, read as in sys.call(). The help page spells out the consequence of the copy: if the expression changes a component named in the list, the change is lost. Anything else is an error with a message of its own.

Source: R documentation: eval, evalq, local

The integer form reads a frame number. Non-negative numbers count from the global environment, which is frame 0, so 1L was the outermost function (outer_fn()) and 2L the next one. A negative number counts back from the frame of eval() itself, which is why -1L gave the function that called eval(), here inner_fn(), and not its caller; sys.frame(-1L) called from the same function would count back from inner_fn() and give outer_fn() (see the call stack below). The two errors at the end are the messages of a character string and of a vector of length two used as envir.

Source: R documentation: sys.call, sys.frame, sys.function and parent.frame

When envir is a list or a data frame, R builds a temporary environment from it, and enclos is that environment’s parent: the place where a name that is not a column is looked up next. The default is parent.frame(). NULL is read as the base environment (baseenv(), which holds the base package but not stats or utils), and an environment can be given. When envir is itself an environment, enclos is ignored, because an environment already has its own parent.

Source: R documentation: eval, evalq, local

The temporary environment is a real environment: ls() lists the columns, and parent.env() returns the enclos that was passed. That is all there is to a data mask in base R. The Language Definition describes the call form for the case it was designed for, the evaluation of the terms of a model formula in a data argument with the caller as the enclosure: eval(expr, data, sys.frame(sys.parent())) (The R Language Definition: More on evaluation).

The expression handed to a function was written by the caller, so the variables it mentions are the caller’s. The default of enclos is parent.frame(), and a default argument is evaluated inside the function it belongs to; the sys.parent help page notes that these functions look at the call stack at the time they are evaluated, and that when they provide default values they are evaluated in the called function and count frames accordingly, naming the envir argument of eval as the example (R documentation: sys.call, sys.frame, sys.function and parent.frame). Inside eval(), then, the default parent.frame() is whoever called eval(). When a function calls eval(substitute(expr), data) without a third argument, that is the function itself, and the caller’s variables are out of reach. The help page of eval says as much in its note: when evaluating in a data frame that was passed to a function, the relevant enclosure is often the caller’s environment, so one needs eval(x, data, parent.frame()). Written as an argument of eval() in the body of the function, parent.frame() is a promise that is evaluated in the function’s frame, and so it gives the function’s caller: the right place.

Source: R documentation: eval, evalq, local

count_long(bad_with, books) fails with object 'n' not found even though n is a formal argument of count_long(): the expression pages > n was evaluated in the temporary environment of bad_with(), whose parent is the frame of bad_with(), then the global environment, and the frame of count_long() is not on that path. The second symptom is subtler and does not raise an error: the arguments and local variables of the wrong function become visible to the expression. bad_with(books, class(data)) found data, the formal argument of bad_with() (the data frame itself), and not the string the caller had defined, and a typo that happens to match a local variable of the function is silently accepted. This is the most common bug in hand-written non-standard evaluation functions, and the cure is always the same third argument.

Evaluating in a list evaluates in a copy, so assignments are lost. Evaluating in an environment changes that environment. And eval(N <- 4, env) is a classic trap: the argument is evaluated first, in the caller, so the assignment happens in the caller and eval() receives the value 4. evalq() quotes its argument, and the assignment goes where it was meant to. The Language Definition remarks that evaluating in a given environment may change it, most obviously with an assignment, and that the same is true of lists except that the original does not change because one is working on a copy.

Source: R documentation: eval, evalq, local

The first eval() shows the copy: cutoff is still 300 after quote(cutoff <- 1) ran in a list, and the data frame shelf got no double column. The third block is the one to remember. eval(n_copies <- 4, shelf_env) printed 4 (the value of the assignment, visible because it was the argument) and changed the global n_copies, leaving the variable in shelf_env at 27; evalq(n_copies <- 5, shelf_env) changed the one in shelf_env.

eval.parent() is the tool for the opposite need, a function that is supposed to change a variable of its caller. It evaluates in the frame of the caller of the function that calls it, because inside it parent.frame(n + 1) counts the extra frame of eval.parent() itself, as the printed body shows.

Source: The R Language Definition: More on evaluation

bump(visits) captured the symbol visits with substitute(), built the call visits <- visits + 1 and ran it in the caller. Inside set_visits() the same function changed the local variable and left the global one alone. A function that rewrites its caller’s variables is unusual in R, where functions work on copies and return the new value, and it is worth a comment when it is used (the replacement functions of Functions are the idiomatic route).

The word "data mask" comes from the tidyverse documentation, but the thing is the temporary environment of the previous section: an environment made from a data frame, in front of the caller’s environment, so that columns hide variables of the same name. The rlang documentation describes data masking as resting on three language features, defusing code with substitute() (or enquo()), first-class environments into which lists and data frames can be turned, and explicit evaluation with eval() (rlang package documentation: data masking). In base R the whole of with() fits in one line, which the first block prints before writing the same function by hand.

Source: R documentation: with, within

Four things are visible in that block. The column pages hides the variable pages <- 1000, so my_with(books, round(mean(pages, na.rm = TRUE), 1)) is not affected by it. The mask is a copy, so reassigning pages inside the block changed the result (864) and not the data frame (still 432 as its maximum). The enclosure of the mask is the global environment, because that is where my_with() was called from, which is exactly what parent.frame() in the third argument buys. And a list or an environment works in place of a data frame.

with(data, expr) evaluates expr in a mask made from data; the help page says the mask has the caller’s environment as its parent, and that when data is already an environment it is used with its existing parent. Assignments made inside take place in the mask, not in the workspace. within(data, expr) examines the mask after the evaluation and returns a modified copy of the data. The help page warns that this can fail for a data frame if the block creates objects that cannot be stored in a data frame, and presents within() as an alternative to transform(). Two details trip people up. New variables are appended in the reverse of the order in which they were created, because within() reads them back from the environment, and a variable removed with rm() inside the block is removed from the result.

Source: R documentation: with, within

The help page carries a warning that belongs next to the first rule of this page: for interactive use with() is effective and nice to read, but in a function variables in data "may accidentally override local variables", and with modelling or graphics functions that have a data argument that argument should be preferred to with(data, ...). The last two lines are the accident: the local year is hidden by the column and the comparison is of the column with itself. The list method of within() gained a keepAttrs argument in R 3.5.0 (see the version notes).

subset(x, subset, select) is the textbook non-standard evaluation function. Its two expressions are evaluated differently, and the source printed below shows both. subset is evaluated in the data frame, with parent.frame() as the enclosure, and must produce a logical vector; rows where it is NA are dropped, which is the difference from indexing with [, where an NA index gives an all-NA row. select is evaluated in a list that maps each column name to its position, so title:year is the range of positions, -c(isbn, author) drops columns, and c(title, year) picks them (the help page describes this as first replacing the column names with the column numbers and then indexing).

Source: R documentation: subset

The call subset(books, year > 2015) returned eight rows and books[books$year > 2015, ] returned nine, one of them a row of NA values for the book whose year is missing. The condition must be logical: subset(books, year) fails with 'subset' must be logical, and since R 4.3.0 the data frame method warns about extra arguments, which catches the common slip of writing = for ==.

Source: R documentation: subset

The text of that warning starts with In subset.data.frame(...) : because chkDots(), the helper that raises it, builds the message from sys.call() (the class of the condition is chkDotsWarning), which is one more use of the call stack described below. The help page of subset() ends with a warning about its own design, in the same words as the pages of transform(): it is a convenience function intended for interactive use, and for programming it is better to use the standard subsetting functions like [, because the non-standard evaluation of the subset argument can have unanticipated consequences. The rest of the page shows what those consequences are.

transform(_data, ...) takes tagged expressions (tag = value) and evaluates all of them in the data frame at once: tags that match an existing column replace it and the others are appended. Because the list(...) is evaluated as one call, an argument cannot use a column created by an earlier argument, which is a real difference from within(), from a for loop over [[<- and from dplyr::mutate().

Source: R documentation: transform

The error object 'age' not found is the whole lesson: both arguments saw the original data, in which there is no age. The printed source shows why: a single call to eval() evaluates substitute(list(...)) in the data frame _data, in one step, before any column is replaced. within(), which evaluates a block in order, accepts the second assignment. The help page of transform() repeats the warning of subset() about interactive use, and adds a remark that is worth believing: if some of the values are not vectors of the appropriate length, you get whatever you deserve.

The functions that fit models use the same pattern with the formula as the carrier of the environment. A formula remembers where it was created, and model.frame() evaluates the variables of the formula, and the subset and weights arguments, in data first and then in the environment of the formula, not in the frame of the function that called lm().

Source: R documentation: model.frame

The formula pages ~ year written inside fit_pages() has the frame of that function as its environment, and cutoff is found there. The formula f written in the global environment does not see the argument cutoff of fit_with(): the error is the same object 'cutoff' not found that bad_with() produced, for the same reason, and it disappears when a global cutoff exists. R 1.1.0 made formulas keep their environment, as the history above says, and the Language Definition lists the sys.frame(sys.parent()) evaluation of the matched call as the way lm() evaluates its model.frame() call in the caller (The R Language Definition: Manipulation of function calls). The environment of a formula, terms() and the rest of the machinery are the subject of Formulas.

A subset()-like function has three steps: capture the expression with substitute(), evaluate it in the data with the caller as the enclosure, and use the result. The capture must come first, before the argument is touched, because substitute() only returns the expression of a promise that has not been replaced by an assignment (Lazy Evaluation and Promises shows the cases).

Source: R documentation: substitute

shelf_subset(books, pages > 400) found pages in the data. longer_than() calls it with an expression that mentions n, a variable of longer_than(), and it was found because shelf_subset() evaluated in parent.frame(), the frame of longer_than(). And trace_cond() shows that nothing is evaluated until eval() is called, and then once. The line data[rows & !is.na(rows), , drop = FALSE] is the same treatment of NA that subset() applies. These are the three habits that every function of this family needs. The pitfalls start when one of them is missing, or when two such functions meet.

A variable named like a column. The mask comes first, so a parameter called year is invisible inside year > year, which compares the column with itself. The result is not an error but an empty data frame. The cure is a parameter name that cannot be a column name, or an explicit pronoun (see tidy evaluation below).

A column that does not exist, and a variable that does. If the name is not in the data, R continues to the enclosure. When no variable has that name the error is the clear object 'price' not found, but when the caller’s environment happens to contain a variable with the same name, the expression is silently evaluated with it: the result of publisher == "Nobody" is an empty data frame, not an error, because the global string publisher was used. The longer the workspace and the more generic the names, the more likely this is, and it is the reason scripts that rely on non-standard evaluation break when a variable is added far away.

Forwarding an argument. A function that receives a condition and passes it on to another non-standard evaluation function gives the second one the symbol cond, not the expression that the user typed. The second function evaluates cond in the data, does not find a column of that name, looks in the frame of the first function, finds a promise there, and forces it in the global environment, where pages does not exist. The user sees an error about a name they never wrote, in a function they never called.

Source: R documentation: subset

The first call returned no rows at all; the second, with the parameter renamed, returned eight. The global string publisher made an impossible condition evaluate silently to nothing, while the misspelled price was reported. And the forwarding call failed with object 'pages' not found, although pages is a column of books.

The reliable way out of the forwarding problem is to separate the two jobs. A function that takes the expression and the environment as ordinary arguments does the work and can be called from anywhere; the non-standard function is a two-line wrapper that captures and passes on. The convention has a history: dplyr 0.3 (2014) gave every function that does non-standard evaluation a standard-evaluation version ending in an underscore, powered by the lazyeval package, and dplyr 0.7.0 (2017) replaced that approach with tidy evaluation (dplyr package documentation: NEWS); the trailing underscore is still the clearest name for a twin in plain base R. A third argument env = parent.frame() carries the environment that the caller means.

Source: R documentation: do.call

long_books() forwards its own argument with the same two lines the original used, and finds min_pages in count_longer() even though the function that does the evaluation is two levels away. The twin also makes it possible to run a list of conditions, which a purely non-standard function cannot do, and do.call() or bquote() can build the call for the non-standard function when only that is available. do.call(shelf_subset, list(books, cond)) works because the call that do.call() builds contains the call object pages > 400 as an argument, and substitute() inside shelf_subset() returns it; the help page of do.call() warns that the behaviour of substitute() under do.call() is undefined and subject to change, so the pattern is fine in a script and not something a library should rest on.

A column used as a bare name looks, to a static checker, like a reference to a variable that nobody defined. R CMD check runs codetools over the code of a package and reports no visible binding for global variable as a NOTE; lintr reports the same with its object_usage_linter, and an IDE shows a squiggle. The checker is right in general and cannot know that this function is going to be evaluated in a data mask. The declaration that tells the check which names are defined elsewhere is documented on the help page of globalVariables() (R documentation: globalVariables); the check is the work of codetools (codetools package documentation: checkUsage).

Source: codetools package documentation: checkUsage

codetools::checkUsage() printed the report that R CMD check turns into a NOTE for long_books(): the name of the function, then no visible binding for global variable ‘pages’. There are four ways to deal with it, in order of preference. Use ordinary evaluation inside package code (data$pages or data[["pages"]], as in long_books_std()), where nothing needs declaring. In functions that must use the data mask, tell the checker the names: utils::globalVariables(c("pages", "year")) at the top level of the package, once, which the help page describes as naming objects that should be regarded as defined globally when the check is applied (a package-wide declaration, so a typo in one of those names is no longer reported anywhere). With tidy evaluation, use the .data pronoun, .data$pages, imported from rlang, which makes the reference explicit (see below). And do not assign pages <- NULL in the function body to silence the note: it works against the checker and creates a real local variable, and in long_books_null() above a misspelled column would not raise object 'pages' not found: the expression would silently see the NULL.

A column name arrives in a function either as a bare name (mean_of(books, pages)) or as a string (mean_of(books, "pages")). Standard R code uses strings and [[, which needs no special treatment; a non-standard function converts the bare name with substitute(). Supporting both in one function looks friendly and is fragile, because the function cannot tell whether the symbol it received is the name of a column or the name of a variable that holds a string.

Source: R documentation: parse, str2lang, str2expression

mean_of_either(books, column) returned NA with a warning, because column is a variable that holds "pages": the function treated the symbol as a column name, did not find one, and evaluated it as a variable holding a string. The two APIs should not be mixed in one argument. When a string must become an expression, str2lang() parses exactly one expression and fails with parsing result not of length one, but 2 for more, and parse(text = ) produces an expression vector. Both execute whatever the text says, so they are for text the program built itself, not for text a user typed into a form; and in nearly every other case an index with [[ does the job without evaluating anything (R FAQ 7.21: How can I turn a string into a variable? gives that advice for the variable-name case: get() or assign() at most, and preferably a list indexed by name, or substitute() with as.name() when a call has to be built; it is applied in the table at the end of this page).

If a function must run code that varies, build the call as an object rather than as text. bquote() writes the call and splices values into it with .(), call() and as.call() assemble it from parts, and substitute() with a list fills in the names of an expression. The four calls in the block below are the same call, and a call is a list whose first element is the function.

Source: R documentation: bquote

call_1[[3L]][[3L]] <- 400 changed the threshold inside the call, because a call is a list-like object whose elements can be replaced, as the manual does with the result of match.call() in its treatment of function calls (The R Language Definition: Manipulation of function calls). Reduce() joined a list of conditions with &, which is the base R counterpart of what the tidy !!! splice does for filter(). The standard-evaluation escape hatches of this section and the previous ones fit in one table.

Source: R FAQ 7.21: How can I turn a string into a variable?

The structure of calls is described in Language Objects and Quoting, and the whole subject of why eval(parse(text = )) is a smell comes back in the table of older habits at the end.

parent.frame(n) is shorthand for sys.frame(sys.parent(n)), the help page says, and it follows the chain of callers, not the numbers. A function that is called by another function that was called by a third has parent.frame(2) as the environment of the third, if the chain is linear. The chain is not linear when a function is invoked through another: lapply() is a function with a frame of its own, so the caller of the function it applies is the frame of lapply(); do.call() evaluates the call in the envir you give it, so that is the parent frame in that case. And code evaluated by eval() sits deeper in the stack than the function that called eval(), because the stack holds both the call to the eval() closure and the context that the evaluator creates for the evaluation, and both print as the call to eval().

Source: R documentation: sys.call, sys.frame, sys.function and parent.frame

In the first function whoami() returned the frame of direct() as expected. Through lapply(), parent.frame() was not the frame of via_lapply() but the frame of lapply(), and parent.frame(2L) was. Through do.call() with an envir, the parent was exactly that environment. The code in frames_inside_eval() ran at frame 3, while the function itself ran at frame 1, and sys.calls() shows the two eval() entries. This is why the help page cautions that internal functions may or may not set contexts and so may not be counted, that S3 methods can do surprising things, and that, as an effect of lazy evaluation, the stack functions look at the stack at the time they are evaluated and not at the time they are called.

The same fact explains a trap in the rule to evaluate a default in the callee. parent.frame() as a default argument is evaluated when the argument is first used, inside the function, so it counts frames from there and gives the function’s caller (a function f(env = parent.frame()) is correct). The help page advises against passing calls to the sys. functions as arguments to other functions, because they still look at the stack and count from the frame of the function evaluation they were called from. When the environment must be captured at a particular moment, capture it in a variable at the top of the function: caller <- parent.frame().

The list is turned into a call whose arguments are the values themselves, and everything that reports the call reports the values: error messages, traceback(), sys.call(), match.call() and substitute(). For a small vector the call is short. For a data frame it runs to dozens of lines, and a function given as an object instead of a name puts its whole definition in the call, which is what the traceback of an error shows. The help page warns that the behaviour of substitute() and similar functions under do.call() is undefined and subject to change.

Source: R documentation: do.call

do.call("check_input", list(books)) built a call whose deparse() is 21 lines long, and the model that was fitted through do.call("lm", ...) stored the whole data frame in its call element (22 lines), where lm(pages ~ year, data = books) stores one short line. Everything that prints or deparses the call later, including summary() of the model and update(), pays for it, and update() would re-evaluate a call that contains a copy of the data instead of the name books. Three habits avoid it. Pass the function by name when it is a string. Pass the data as a symbol with quote = TRUE or with as.name(), as the last call in the block did, so that the call contains data = books. And do not build calls with large values when you can build them with names. The cost is not only readability: R 4.1.2 made try() considerably faster when an error has a long call, such as one made by do.call(), which shows that long calls were a practical problem (see the version notes).

In an NSE call four environments are in play, and most mistakes are a confusion between two of them. The frame of the function that does the evaluation is where its own variables live; its caller’s frame (parent.frame()) is where the user’s variables live; the closure environment (environment(f)) is where the function was created, which is where free variables of its body are found; and the data mask is a temporary environment between the other two. The scoping page treats the first three in depth (Scoping, Environments and Closures).

Source: R documentation: environment

For the call to probe(data) made from caller_fn(), the frame of probe() is of course not the caller’s frame, parent.frame() is exactly the caller’s frame, the function is defined in the global environment (environment(sys.function())), and the parent of the data mask is the caller’s frame, because eval() was given parent.frame() as its third argument. sd lives in the namespace of stats and mean in base, and the counter’s own environment holds the state that its closure updates.

The first difference is what is captured. substitute() returns an expression and parent.frame() returns the environment, as two separate pieces of information that the programmer has to keep together. rlang::enquo() captures both in one object, a quosure. A quosure carries an expression, which quo_get_expr() returns, and an environment, which quo_get_env() returns; it is hygienic because it evaluates in the environment it carries, and maskable because, when evaluated in a data mask, the mask comes first (rlang package documentation: What are quosures and when are they needed?). The documentation points out that quosures are inspired by the environment-tracking mechanism of formulas and that they resemble promises, with one difference: a promise is evaluated once and cached, while a quosure can be evaluated repeatedly, in different masks, which is what a grouped computation needs. The class vector c("quosure", "formula") and the printed ^ in front of the expression are the visible signs.

Source: rlang package documentation: enquo, enquos

quo(pages > 300) printed as <quosure> with expr: ^pages > 300 and env: global, the environment shown by name when it is a well-known one (a quosure created inside a function prints the address of that function’s frame, which these pages leave out). capture() returned the same expression that substitute() returns in capture_base(), and quo_get_env() is the same environment as parent.frame(). The final two lines show the other direction: a quosure is not a call, so base eval() cannot evaluate it, but its two parts can be passed separately and give the same logical vector as eval_tidy(). The rlang page states the limitation in these terms: for historical reasons, base::eval() does not support quosure evaluation, and quosures currently require eval_tidy().

eval_tidy(expr, data) is eval() with a data mask that also understands quosures and the .data pronoun. A function built on it has the same forwarding problem as the base version, because enquo() captures one level up: whatever the user typed, as seen by the function that is called directly. The solution in tidy evaluation is to forward the quosure, and the embrace operator is the way to write it: {{ cond }} captures the argument cond of the current function, with the environment of the user, and injects it into the call. The rlang documentation describes it as combining enquo() and !! in one step (rlang package documentation: Embrace operator).

Source: rlang package documentation: embrace operator

The first report_rows() failed with object 'pages' not found, exactly the failure of long_books() earlier, for the same reason: count_rows() captured the symbol cond, evaluated it in the frame of report_rows() and forced the original promise in the global environment, where there is no pages. With {{ cond }} the second version worked and the quosure that reached count_rows() held the user’s expression and the user’s environment. count_longer() shows what the environment is for: min_pages is a variable of that function, the data mask is books, and the expression pages > min_pages is evaluated with both, as in the base pattern with parent.frame(), but across any number of function boundaries. The peek_via() call returns the expression typed by the user, not the symbol cond.

The verbs of dplyr already defuse their arguments, so inside a function the only job is to forward the argument: embrace it where the verb expects a data-masked expression. The same argument can be embraced in as many places as needed, and an expression works as well as a name.

Source: dplyr package documentation: Programming with dplyr

The two functions mean_embrace() and mean_enquo() are equal: {{ var }} is a shorthand for !!enquo(var). The tibble printed by mean_by() is the standard print method of tibbles, with the column types under the names and the numbers shown to three significant digits. The dplyr article recommends embracing, and says to embrace each of several user-supplied expressions separately. It also notes that the data frame argument of a verb is neither data-masked nor tidy-selected, so a function that only passes a data frame on needs no special treatment.

Data masking has the ambiguity that every function on this page had: a column and a variable with the same name, with the column winning. The tidy remedy is a pair of pronouns. .data$year is the column and .env$year is the variable of the environment (the rlang documentation says that .data retrieves data-variables from the data frame and .env retrieves env-variables from the environment, so the lookup is explicit and there is no ambiguity; rlang package documentation: .data and .env pronouns). .data[[col]] reads a column whose name is in a string, and it never confuses col with a column called col: the variable inside the double brackets is always an env-variable. The opposite mistake is silent: filter(books, col > 300) compares the text "pages" with the number 300, which is true, and keeps every row without an error.

Source: rlang package documentation: .data and .env pronouns

The block shows five facts. With a global year <- 2020, filter(books, year > year) returned no rows, as subset() did; the same condition with pronouns returned the three books published after 2020. The function books_after(data, year) is safe with the pronouns even though its parameter has the name of a column. filter(books, col > 300) returned all sixteen rows, which is the silent failure described above, while .data[[col]] > 300 returned the six long books, and the loop at the end summarised two columns chosen by name. When no pronoun is used the data wins, even over a local variable set to zero in the function (mean_pages() returned the mean of the column). And .data is not a data frame: names(.data) is NULL, so length(names(.data)) is zero. The rlang documentation notes the same: .data is a pronoun, not a real data frame, so you cannot take its names or map a function over its contents, and .env is not an actual environment.

Two practical notes close the topic. In a package, .data appears to the checker as an undefined global just like a bare column name did, so it is imported from rlang or declared with utils::globalVariables(".data"), as the rlang page recommends; unlike the bare names it is a single, uniform declaration. And rlang::.data is a "fake" pronoun: write the unqualified .data in data-masked code, not rlang::.data. The lintr object-usage check also objects to .data and .env in the scripts of this page, as the nolint comment shows.

Embracing covers the common case. The general mechanism underneath is injection, also called quasiquotation: changing a piece of code before R evaluates it. !!x injects the value or expression x into a defused expression, and !!!x splices a list of values or expressions as separate arguments. The rlang documentation says that !! works only in functions that take defused and data-masked arguments (including those defused with {{ }} or the en-prefixed operators like enquo()) and inside inject(), and that with base functions inject() has to be used to enable it; it also says that in tidyverse code !! is no longer a common pattern, because the .env pronoun solves the ambiguity problem more naturally and the embrace operator makes defuse-and-inject easier to learn (rlang package documentation: Injection operator). The splice operator !!! provides the same functionality as do.call(), as that page puts it, turning a list of inputs into distinct arguments (rlang package documentation: Splice operator).

The base counterparts are bquote() with .() for a value and with ..() and splice = TRUE for a list (the splicing form was added to bquote() in R 4.0.0), as.call() on a list that starts with the function, and substitute() with a list; the two families are compared at the end of this section.

Source: rlang package documentation: inject

The first group compares a free variable, a value injected with !! and the same value injected with bquote(): expr(pages > !!min_pages) and bquote(pages > .(min_pages)) both printed pages > 300, so the number is part of the expression and no longer depends on the environment, which is why injecting a value is a way to settle the column-versus-variable ambiguity. sym("pages") is as.name("pages"), and syms() makes a list of symbols; injecting them with !! and !!! is the string-to-column bridge. !! binds like unary minus, so !!a + b is (!!a) + b and !!a * b + 3 is (!!a) * b + 3, as the rlang NEWS says of version 0.2.0 (rlang package documentation: NEWS). Outside a function that supports injection, !! is the double negation of base R: !!TRUE is TRUE and !!c(0, 1) is FALSE TRUE, which is why a stray !! in ordinary code is a silent mistake. And inject(), available since rlang 0.4.9, turns the operators on for any function: inject(subset(books, pages > !!min_pages)) and inject(rbind(!!!list(...))) use subset() and rbind() from base R.

Dots need no special syntax: a function that passes ... to a data-masking verb passes the expressions untouched, and the verb defuses them. enquos(...) captures the dots as a list of quosures when the function needs to look at them, and the base counterpart is substitute(list(...)) or match.call(expand.dots = FALSE)$.... Names built from the data or from a variable use the := operator with glue syntax on the left: "{{ var }}_mean" := ... takes the name from the argument, and "{prefix}_pages" takes it from a variable. Selecting columns is a second mini-language, tidy selection, and it needs a different bridge for each kind of indirection: embrace an argument, wrap a character vector in all_of() (or any_of(), which ignores missing names). The dplyr article summarises this as two forms of indirection: a data-variable in a function argument is embraced, and an env-variable holding a character vector is indexed with .data[[ ]] for data masking and wrapped in all_of() for tidy selection.

Source: dplyr package documentation: across

count_by(books, genre) counted without any special syntax, because group_by(...) received the dots. capture_dots() returned quosures labelled by as_label(); base_dots() returned the same expressions as a call to list built by substitute(), which is how data.frame() finds the names. mean_named() produced pages_mean and year_mean columns from the argument, and across(all_of(cols), ...) computed the mean of two columns whose names were in a character vector, while mean_of() embraced a tidy-selection argument and accepted c(pages, year) and where(is.numeric). across() was added to dplyr in version 1.0.0, as the rlang page on the splice operator remarks when it says that splicing symbolised column names is no longer as useful since dplyr 1.0 and across().

The equivalences claimed in this section can be checked. The block runs each pair and compares the results with identical(); the last pair is the complete job, the tidy filter() function written with an embrace and the base one written with substitute() and eval().

Source: rlang package documentation: data masking

Every comparison printed TRUE (the splice was checked against both bquote() and as.call()), the two labels are the same text, and the two filter functions returned the same titles. The table collects the translations used on this page; each row names the base mechanism that does the job, and the rlang page on data masking lists the same three ingredients (defusal with substitute(), environments, explicit evaluation with eval()) as the base of the whole system.

Source: dplyr package documentation: Programming with dplyr