Testing with testthat
|
This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production. The reference material consulted while preparing these pages is listed in the section bibliography. |
Checked against R 4.6.1 (2026-06-24) on 2026-10-20; R 4.6.2 is scheduled for 2026-10-29. This page uses the add-on package testthat 3.3.2 (the CRAN release of 2026-01-11) and the packages that go with it in the examples: withr 3.0.3 (temporary state), covr 3.6.5 (coverage), devtools 2.5.2 (the devtools::test() wrapper, which loads the package with pkgload 1.5.3), usethis 3.2.2 (calls to use_testthat() and use_test()), lintr 3.4.0 (one call to lint_package()), rlang 1.3.0 (one custom expectation calls rlang::enquo()) and tinytest 1.4.3 (one run of its test runner and one call to setup_tinytest()). testthat itself compares objects with waldo 0.6.2 and prints with cli 3.6.6; R CMD build rebuilt the vignette of the package with knitr 1.52 and rmarkdown 2.32. The shelf package is the one built for the Creating Packages page, version 0.1.0. The package DT, which covr::report() needs, is not installed in the image used here, so that one call is shown failing and nothing else is claimed about its output. A GitHub Actions workflow is shown as a fragment that was not run, and Codecov is only named.
Testing is the habit of writing down what a piece of code is supposed to do as code that checks it, so that the checking can be repeated after every change. R itself ships only the minimal machinery for this: the tests directory of a package, whose .R files R CMD check runs, with optional reference output next to them. Almost every package on CRAN goes through testthat instead, an add-on package that gives that directory a structure: a test is a named block, a test contains expectations, expectations say what the result must look like and produce readable failure reports, and a few special files hold shared helpers and fixtures. This page explains that structure with the tests of the shelf package, the tools that run and report them, the features around them (snapshots, mocking, skipping, fixtures, the third edition), and the base-R and tinytest alternatives. It ends with coverage measurement with covr and a CI fragment.
Every example is a script that was run with Rscript on R 4.6.1 with warnings printed as they happen (options(warn = 1)), and the lines that start with #> are what it printed. The scripts also pass the default lintr linters (whose line limit is 80 characters). The names books, loans and shelf_today and the book() constructor come from the running library-shelf scenario described on the section index: a script that needs them starts with source("scenario.R"), or calls shelf:: functions when the built package is used. The shelf package shown here was created, documented, tested, built and checked for real with these tools; its files are shown as the tools wrote them. The experiments that change the package (a failing test file, a changed print method, a parallel setting) ran in a throwaway copy of the package directory and each one puts everything back; the transcripts show the package path as ~/shelf (paths shortened). The default testthat reporters finish a passing run with a random emoji or a short word of praise, and print a spinner line that a terminal overwrites in place but that a captured log keeps as separate lines. The scripts therefore silence the praise where they can (StopReporter$new(praise = FALSE)), and the few transcripts that still contain a random line say which one was left out.
This page builds on Creating Packages, which made the package, and on Documentation with Rd, roxygen2 and Vignettes; the conditions that expect_error() and its relatives catch are explained in Conditions and Error Handling, and the checks a function should make of its own arguments in Defensive Programming and Input Validation. Style checking of the tests is the subject of Style, Linting and Static Analysis, and running the whole check before a release of Publishing to CRAN.
Where tests live and how testthat fits
A package keeps its tests in the tests directory, which Writing R Extensions describes as the place for package-specific test code: .R files whose results are written to a .Rout file, which R CMD check compares with a .Rout.save file when one exists. The directory is copied to the check area, the tests run with that copy as the working directory, in a vanilla R session without a random-number seed, with R_LIBS set so that library(shelf) finds the copy of the package installed for the check. R does not say how the tests should be written. testthat is a convention on top of that mechanism, and a package opts in with three things: Suggests: testthat in DESCRIPTION (a package used only to run tests is a suggestion, not an import), a one-line driver in tests/testthat.R, and the test files in tests/testthat/.
The vocabulary has four levels, and the names matter because the failure reports and the reporters use them.
| Level | What it is | Where it is written |
|---|---|---|
Expectation |
One check of one result: |
a call of an |
Test |
A named group of expectations that share local objects and are reported together. |
|
File |
The tests of one source file or one topic; the unit that snapshots, filters and the progress display work on. |
|
Suite |
Every file, plus the helper and setup files that prepare them. |
|
Each test runs in a fresh environment whose parent is the package namespace when the tests belong to a package, so a test can call internal functions such as shelf_today() without :::. The expectations are ordinary functions that signal an expectation condition; the reporter listening for those conditions counts them and prints the failures. Because a test is just code that runs in a controlled environment, everything the other pages teach about functions, scoping and conditions applies inside it.
The layout of the tests directory
tests/testthat.R
usethis::use_testthat(3) creates the directory and the driver. The script below runs it on a throwaway package in a temporary directory (not on shelf, which already has the result) and prints the messages of the call, the files that exist afterwards, the driver it wrote and the two lines of DESCRIPTION that mention testthat:
# A throwaway package in a temporary directory (nothing of shelf is touched).
dir <- file.path(tempdir(), "tmpkg")
invisible(capture.output(
suppressMessages(usethis::create_package(dir, open = FALSE))
))
# Keep the messages of use_testthat(), without the line that names the path.
msgs <- character()
withCallingHandlers(
usethis::with_project(dir, usethis::use_testthat(3)),
message = \(m) {
msgs <<- c(msgs, conditionMessage(m))
invokeRestart("muffleMessage")
}
)
cat(grep("active project", msgs, value = TRUE, invert = TRUE), sep = "")
#> ✔ Adding testthat to 'Suggests' field in DESCRIPTION.
#> ✔ Adding "3" to 'Config/testthat/edition'.
#> ✔ Creating 'tests/testthat/'.
#> ✔ Writing 'tests/testthat.R'.
#> ☐ Call `usethis::use_test()` to initialize a basic test file and open it for
#> editing.
list.files(dir, recursive = TRUE)
#> [1] "DESCRIPTION" "NAMESPACE" "tests/testthat.R"
cat(readLines(file.path(dir, "tests", "testthat.R")), sep = "\n")
#> # This file is part of the standard setup for testthat.
#> # It is recommended that you do not modify it.
#> #
#> # Where should you do additional test configuration?
#> # Learn more about the roles of various files in:
#> # * https://r-pkgs.org/testing-design.html#sec-tests-files-overview
#> # * https://testthat.r-lib.org/articles/special-files.html
#>
#> library(testthat)
#> library(tmpkg)
#>
#> test_check("tmpkg")
grep("testthat", readLines(file.path(dir, "DESCRIPTION")), value = TRUE)
#> [1] " testthat (>= 3.0.0)" "Config/testthat/edition: 3"
The argument 3 selects the third edition of testthat (see The third edition), which the call records in DESCRIPTION. The fragment of the DESCRIPTION of shelf that matters for testing is:
Suggests:
knitr,
rmarkdown,
testthat (>= 3.0.0),
withr
Config/testthat/edition: 3
Depends:
R (>= 4.1.0)
The Suggests field lists the packages that tests, examples and vignettes use but that the package does not need in order to work; withr is there because the tests call withr::local_options() with ::, and testthat (>= 3.0.0) is the version that introduced the third edition. Writing R Extensions says that a package used by tests must be listed in Depends, Imports or Suggests, that code using a suggested package must be conditional on it being there, and that this includes the packages that manage test suites; its example is testthat 1.0.0, which once contained code that did not compile on standards-compliant platforms. testthat 3.3.0 took one half of this burden off the test authors: on CRAN, test_that() now skips a test automatically when a package that the test needs is not installed.
The list is checked. With the line testthat (>= 3.0.0), deleted from Suggests and R CMD check --as-cran run on the result (the output is trimmed to the lines about the tests), the check reports the undeclared dependency. --as-cran also turns on _R_CHECK_SUGGESTS_ONLY_, which R Internals documents as true for the CRAN submission checks and which runs examples, tests and vignettes with a temporary library holding only the declared packages, so the run then fails at the first line of tests/testthat.R:
* checking for unstated dependencies in ‘tests’ ... WARNING
'library' or 'require' call not declared from: ‘testthat’
* checking tests ...
Running ‘testthat.R’
ERROR
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> # This file is part of the standard setup for testthat.
> # It is recommended that you do not modify it.
> #
> # Where should you do additional test configuration?
> # Learn more about the roles of various files in:
> # * https://r-pkgs.org/testing-design.html#sec-tests-files-overview
> # * https://testthat.r-lib.org/articles/special-files.html
>
> library(testthat)
Error in library(testthat) : there is no package called 'testthat'
Execution halted
Status: 1 ERROR, 1 WARNING
Source: R Internals: Tools
The file tests/testthat.R is what R CMD check actually runs. This is the file use_testthat() wrote, unchanged:
# This file is part of the standard setup for testthat.
# It is recommended that you do not modify it.
#
# Where should you do additional test configuration?
# Learn more about the roles of various files in:
# * https://r-pkgs.org/testing-design.html#sec-tests-files-overview
# * https://testthat.r-lib.org/articles/special-files.html
library(testthat)
library(shelf)
test_check("shelf")
library(testthat) attaches the expectation functions and library(shelf) the package being tested (the same call that prints the startup message shelf 0.1.0: loan period is 21 days); test_check("shelf") then finds tests/testthat/ relative to the check directory, runs every file in it with the check reporter and, if anything failed, ends with an error that makes R CMD check report the test run as failed. The comment header, which testthat says not to modify, points to the article on the special files. Everything else about the suite goes into tests/testthat/, not into this file.
tests/testthat/
The files in tests/testthat/ have different jobs, decided by their names. Only names matter, not contents:
| File or directory | Role |
|---|---|
|
Test files. Run in alphabetical order; each should pass when run alone or in any order. The convention is one test file for each file in |
|
Helper code: functions and constants shared by the tests, custom expectations, skip helpers. Sourced by |
|
Global preparation for a run, such as turning off interactive behaviour. Sourced by |
|
The legacy place for cleanup, no longer recommended. Run after the tests. |
|
Data files for the tests. Files that testthat does not know are ignored by it, which makes the directory a safe place for persistent test data; reach it with |
|
Snapshot files written and read by |
|
Written in a check directory when a test fails under |
The listing of the tests directory of shelf, as find prints it (the order is the collation order of sort):
tests
tests/testthat
tests/testthat/fixtures
tests/testthat/fixtures/books-small.csv
tests/testthat/helper-shelf.R
tests/testthat.R
tests/testthat/_snaps
tests/testthat/_snaps/book.md
tests/testthat/test-book.R
tests/testthat/test-clean.R
tests/testthat/test-loans.R
usethis::use_test("name") creates tests/testthat/test-name.R with a one-test template and reports the file it wrote. The template for a name that does not match an existing file in R/ is:
usethis::with_project(
".",
usethis::use_test("scratch", open = FALSE),
quiet = TRUE
)
cat(readLines("tests/testthat/test-scratch.R"), sep = "\n")
unlink("tests/testthat/test-scratch.R")
#> test_that("multiplication works", {
#> expect_equal(2 * 2, 4)
#> })
The usethis::with_project(".", ..., quiet = TRUE) wrapper is only there so that the script prints the template and not usethis’s own messages. The script is a fragment to run from the package directory: it reads back the file that use_test() wrote and deletes it. In an interactive session the call is simply usethis::use_test("name"), and with the file R/name.R open usethis::use_test() picks the name itself.
The tests of the shelf package
The suite of shelf has three test files, one helper file and one fixture. They are package test files, run by devtools::test() or test_check(), not scripts to run with Rscript; the blocks below are the files exactly as they are in ~/shelf/tests/testthat/, and each is explained in the sections that follow. They pass the default lintr linters with one finding, explained below. Together they hold 52 expectations: devtools::test() reports [ FAIL 0 | WARN 0 | SKIP 0 | PASS 52 ].
helper-shelf.R
The helper is loaded before every test file. It defines the one fixed date that the tests share and a function that reads the small catalogue fixture through test_path().
# Loaded before every test file: a fixed "today" and a helper that reads the
# small catalogue fixture.
shelf_fixed_today <- as.Date("2026-06-30")
read_fixture_books <- function() {
read.csv(test_path("fixtures", "books-small.csv"))
}
Linting this file in a fresh session finds one thing, and the reason is a general property of lintr and not a problem of the helper. object_usage_linter asks the R session which functions exist, and a plain Rscript -e 'lintr::lint_package()' has not attached testthat, so test_path() is "not visible". The recorded build of the package linted it after devtools::test() had attached testthat and found nothing. The second call below attaches testthat first:
# A fresh session does not know test_path(): lintr cannot see where it is from.
lintr::lint_package()
# With testthat attached the same file is clean.
library(testthat)
lintr::lint_package()
The first call prints the lint (the file, line and column, the linter and its message, the line of code and a marker under the name); the second prints the message No lints found.:
tests/testthat/helper-shelf.R:6:12: warning: [object_usage_linter] no visible global function definition for 'test_path'
read.csv(test_path("fixtures", "books-small.csv"))
^~~~~~~~~
ℹ No lints found.
The remedies are testthat::test_path() called with the package name in the helper, or a # nolint: object_usage_linter. comment on the line, or listing testthat among the globals in .lintr; the file is shown as written.
test-book.R
The tests of the book class: normalising input, optional fields, the error messages of validate_book(), the type check of new_book(), two snapshot tests, and a describe() block for is_valid_isbn13().
test_that("book() trims the text fields and normalises the ISBN", {
b <- book(" the quiet streets ", "Mara Quill", "978-0-00-100010-0",
"2019", "Fiction ", 214)
expect_s3_class(b, "book")
expect_identical(b$title, "the quiet streets")
expect_identical(b$author, "Mara Quill")
expect_identical(b$isbn, "9780001000100")
expect_identical(b$year, 2019L)
expect_identical(b$genre, "Fiction")
expect_type(b$pages, "integer")
})
test_that("book() accepts missing optional fields", {
b <- book("Untitled Draft")
expect_true(is.na(b$author))
expect_true(is.na(b$isbn))
expect_identical(format(b), "Untitled Draft (NA, n.d.)")
})
test_that("validate_book() rejects bad values with a clear message", {
expect_error(book(""), "a book needs a title")
expect_error(book("X", isbn = "9780001000101"), "invalid ISBN-13")
expect_error(book("X", year = 2200), "implausible year")
expect_error(
validate_book(new_book(title = c("a", "b"))),
"every field of a book must have length 1"
)
})
test_that("new_book() checks types only", {
expect_error(new_book(title = 1), "is.character(title) is not TRUE",
fixed = TRUE)
b <- new_book(title = "x", author = "y", isbn = "z", year = 1L,
genre = "g", pages = 2L)
expect_identical(class(b), "book")
expect_identical(class(unclass(b)), "list")
})
test_that("a book prints and summarises as expected", {
b <- book("Salt and Static", "Mara Quill", "978-0-01-400145-3", 2018,
"Mystery", 244)
expect_snapshot(print(b))
expect_snapshot(summary(b))
expect_s3_class(summary(b), "summary.book")
expect_output(print(book("Paper Moons", "Noor Haddad")), "n.d.")
})
describe("is_valid_isbn13()", {
it("accepts hyphenated and bare forms of a valid ISBN", {
expect_identical(
is_valid_isbn13(c("978-0-00-100010-0", "9780001000100")),
c(TRUE, TRUE)
)
})
it("rejects wrong check digits, short strings, empty strings and NA", {
expect_identical(
is_valid_isbn13(c("9780001000101", "978030640615", "", NA)),
c(FALSE, FALSE, FALSE, FALSE)
)
})
it("returns a logical of the same length, including length zero", {
expect_identical(is_valid_isbn13(character()), logical())
})
})
The two snapshot expectations write their results to tests/testthat/_snaps/book.md, a file named after the test file and committed with the package:
# a book prints and summarises as expected
Code
print(b)
Output
<book> Salt and Static (Mara Quill, 2018)
---
Code
summary(b)
Output
Book: Salt and Static
ISBN known: TRUE
Pages: 244
test-clean.R
The tests of the cleaning functions read tests/testthat/fixtures/books-small.csv through the helper:
"title","author","isbn","year","genre","pages"
" the quiet streets ","Mara Quill","9780001000100",2019,"Fiction ",214
"Midnight Ledger","Iris Fenwick","978-0-00-300031-3",2016,"Mystery",301
"Rivers of Glass","Tomas Arden","",2012,"Non-fiction",352
test_that("clean_books() tidies the fixture catalogue", {
cleaned <- clean_books(read_fixture_books())
expect_identical(cleaned$title[[1L]], "the quiet streets")
expect_identical(cleaned$author[[3L]], "Tomas Arden")
expect_identical(cleaned$genre, c("fiction", "mystery", "non-fiction"))
expect_identical(cleaned$isbn, c("9780001000100", "9780003000313", NA))
})
test_that("clean_books() keeps the shape of the packaged catalogue", {
cleaned <- clean_books(books)
expect_identical(dim(cleaned), dim(books))
expect_identical(names(cleaned), names(books))
expect_identical(sum(is.na(cleaned$isbn)), 2L)
expect_setequal(
unique(cleaned$genre),
c("science fiction", "classic", "fiction", "mystery", "non-fiction",
"poetry")
)
})
test_that("clean_loans() parses dates and repairs impossible values", {
raw <- data.frame(
loan_id = c("L1", "L2", "L3"),
isbn = rep("9780001000100", 3L),
member = c("Ana Ruiz", "ana ruiz ", ""),
loan_date = c("2026-01-10", "2026-02-30", "2026-03-01"),
return_date = c("2026-01-05", "", "2026-03-20")
)
cleaned <- clean_loans(raw)
expect_s3_class(cleaned$loan_date, "Date")
expect_identical(cleaned$member, c("Ana Ruiz", "ana ruiz", NA))
expect_true(is.na(cleaned$loan_date[[2L]]))
expect_true(is.na(cleaned$return_date[[1L]]))
expect_identical(cleaned$return_date[[3L]], as.Date("2026-03-20"))
})
test_that("clean_loans() leaves Date columns alone", {
expect_identical(clean_loans(loans)$loan_date, loans$loan_date)
})
The fixture is a three-row catalogue with one row of every kind of mess that clean_books() repairs: a title with blanks and lower case, a hyphenated ISBN, an empty ISBN and a double space in an author. The expected values in the test are small enough to be written out, which is what makes the test a good specification. The third test builds its input in the test itself, as a three-row data frame, because the impossible date and the early return belong to the case being described and a reader should see them next to the expectation.
test-loans.R
The tests of the loan analyses use the shared fixed date, replace the package’s clock in one test, change an option in another and skip one test on CRAN.
test_that("overdue_loans() counts late returns and loans still out", {
out <- overdue_loans(loans, today = shelf_fixed_today)
expect_true(all(out$days_out > 21L))
expect_identical(sum(is.na(out$return_date)), 7L)
expect_identical(nrow(out), 32L)
expect_false(any(is.na(out$loan_date)))
})
test_that("overdue_loans() uses the current date by default", {
local_mocked_bindings(shelf_today = function() shelf_fixed_today)
expect_identical(
overdue_loans(loans),
overdue_loans(loans, today = shelf_fixed_today)
)
})
test_that("the loan period comes from the shelf.loan_days option", {
withr::local_options(shelf.loan_days = 60L)
out <- overdue_loans(loans, today = shelf_fixed_today)
expect_true(all(out$days_out > 60L))
expect_lt(nrow(out), 32L)
})
test_that("loan_summary() reports one row per genre", {
cleaned <- clean_books(books)
s <- loan_summary(loans, cleaned, today = shelf_fixed_today)
expect_named(s, c("genre", "loans", "overdue", "overdue_rate"))
expect_identical(s$genre, sort(unique(cleaned$genre)))
expect_true(all(s$overdue <= s$loans))
expect_equal(s$overdue_rate, s$overdue / s$loans)
})
test_that("loans_per_month() returns months in order", {
m <- loans_per_month(loans)
expect_identical(m$month, sort(m$month))
expect_identical(sum(m$loans), sum(!is.na(loans$loan_date)))
})
test_that("a long check is skipped on CRAN", {
skip_on_cran()
big <- loans[rep(seq_len(nrow(loans)), 50L), ]
expect_identical(
nrow(overdue_loans(big, today = shelf_fixed_today)),
32L * 50L
)
})
All of the tests use the packaged books and loans, which the package exports as lazy-loaded data, so no test reads a file from the working directory. That is deliberate: a test may not rely on the working directory, a network or the current date, which are the three things that differ between a laptop and a check machine.
A first test: test_that() and expectations
test_that(desc, code) takes a description and a block of code. The description should finish the sentence "test that …": the failure report quotes it, and a suite of well-chosen descriptions reads as a specification. The block is evaluated in its own environment with the reporter listening: every expect_*() call inside it registers one result, a failed expectation does not stop the block (the later expectations still run and are reported), and an error that is not caught by an expectation ends the block and is reported as an error against the test. A block without any expectation is reported as skipped, with the reason empty test. test_that() also calls local_test_context(), which sets the environment variable TESTTHAT and, in the third edition, makes the output reproducible (see Reproducible output); it does not need a context() call.
A test file can also be run as a script, which is the easiest way to see the mechanics. Outside a package there is no DESCRIPTION to say which edition applies, so the scripts select the third edition with the environment variable TESTTHAT_EDITION (the third-edition section explains why). In a script, test_that() uses the stop reporter: a passing test prints nothing here (with praise on, it prints Test passed and a random emoji), and a failing test prints its report and then stops the script with an error, so the script catches it to go on.
library(testthat)
source("scenario.R")
set_reporter(StopReporter$new(praise = FALSE))
# A test is a description plus a block of expectations.
test_that("addition works", {
expect_equal(1 + 1, 2)
expect_true(is.numeric(1L))
})
# A test with a failing expectation stops the script; catch it to continue.
result <- tryCatch(
test_that("a failing expectation", {
expect_equal(1 + 1, 3)
}),
error = \(e) conditionMessage(e)
)
#> ── Failure: a failing expectation ──────────────────────────────────────────────
#> Expected `1 + 1` to equal 3.
#> Differences:
#> 1/1 mismatches
#> [1] 2 - 3 == -1
result
#> [1] "Test failed with 1 failure and 0 successes."
# An error inside the block is also reported against the test.
tryCatch(
test_that("an error inside a test", {
books$title[[100L]]
}),
error = \(e) conditionMessage(e)
)
#> ── Error: an error inside a test ───────────────────────────────────────────────
#> <subscriptOutOfBoundsError/error/condition>
#> Error in `books$title[[100L]]`: subscript out of bounds
#> [1] "Test failed with 1 failure and 0 successes."
# A test without expectations is skipped as empty.
test_that("nothing is checked here", {
x <- 1 + 1
})
#> ── Skip: nothing is checked here ───────────────────────────────────────────────
#> Reason: empty test
The report of the failure shows the three parts that every failure has: the location (here the test description; in a file, the file and line), the sentence that says what the expectation expected, and the differences. Since testthat 3.3.0 every expectation states what it expected and what it saw in the same shape, and returns the value of its first argument whether it succeeds or fails, so chains of expectations work, as the last line of the script in the section on types and classes shows.
The rest of this section goes through the families of expectations. Every example here is a script in which a failed expectation raises an error; the one-line helper failure() catches that error and prints its message, which is exactly the text that appears in a test report. An expectation that succeeds prints nothing.
Equality: expect_equal() and expect_identical()
expect_identical(object, expected) demands that the two objects are the same, the way identical() does for the base data types. expect_equal(object, expected) demands that they are the same up to numerical error: it differs from expect_identical() only in setting tolerance = testthat_tolerance(), so small floating-point differences are ignored, and as a consequence 1 and 1L are equal. Both compare with waldo::compare() in the third edition; in the second edition expect_identical() used identical() and expect_equal() used all.equal(), whose tolerance rules are different for different code paths. The tolerance is the one of all.equal(): relative to the size of the numbers, absolute when the differences are very small. The rule for choosing is simple. Use expect_identical() for what is exact by construction (strings, integers, logicals, names, classes, NA values) and expect_equal() for results of floating-point arithmetic. For the arithmetic itself, see Numbers and Arithmetic.
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
source("scenario.R")
# Outside test_that() a failed expectation is an error: catch it to read it.
failure <- \(code) tryCatch(code, error = \(e) cat(conditionMessage(e), "\n"))
# Floating point: equal within tolerance, but not identical.
x <- 0.1 + 0.2
expect_equal(x, 0.3)
failure(expect_identical(x, 0.3))
#> Expected `x` to be identical to 0.3.
#> Differences:
#> `actual`: 0.300000000000000044
#> `expected`: 0.299999999999999989
#>
# Integer vs double: identical() is strict about the type.
expect_equal(2L, 2)
failure(expect_identical(2L, 2))
#> Expected 2L to be identical to 2.
#> Differences:
#> `actual` is an integer vector (2)
#> `expected` is a double vector (2)
#>
# The default tolerance is the one of all.equal(), and it is relative.
testthat_tolerance()
#> [1] 1.490116e-08
failure(expect_equal(100, 101))
#> Expected 100 to equal 101.
#> Differences:
#> `actual`: 100.0
#> `expected`: 101.0
#>
expect_equal(100, 101, tolerance = 0.05)
# waldo describes where two objects differ.
b1 <- book("Salt and Static", "Mara Quill", year = 2018)
b2 <- book("Salt and Static", "Mara Quill", year = 2019)
failure(expect_equal(b1, b2))
#> Expected `b1` to equal `b2`.
#> Differences:
#> `actual$year`: 2018
#> `expected$year`: 2019
#>
# Attributes and names matter unless you ignore them.
failure(expect_equal(c(a = 1, b = 2), c(1, 2)))
#> Expected `c(a = 1, b = 2)` to equal `c(1, 2)`.
#> Differences:
#> `names(actual)` is a character vector ('a', 'b')
#> `names(expected)` is absent
#>
expect_equal(c(a = 1, b = 2), c(1, 2), ignore_attr = TRUE)
# Two data frames of different length: every differing column is reported.
failure(expect_equal(head(books[, 1:3], 2), head(books[, 1:3], 3)))
#> Expected `head(books[, 1:3], 2)` to equal `head(books[, 1:3], 3)`.
#> Differences:
#> `attr(actual, 'row.names')`: 1 2
#> `attr(expected, 'row.names')`: 1 2 3
#>
#> actual vs expected
#> title author isbn
#> actual[1, ] The Dispossessed Ursula K. Le Guin 978-0-306-40615-7
#> actual[2, ] Pride and Prejudice Jane Austen 978-0-14-143951-8
#> + expected[3, ] the quiet streets Mara Quill 9780001000100
#>
#> actual$title | expected$title
#> [1] "The Dispossessed" | "The Dispossessed" [1]
#> [2] "Pride and Prejudice" | "Pride and Prejudice" [2]
#> - " the quiet streets " [3]
#>
#> `actual$author`: "Ursula K. Le Guin" "Jane Austen"
#> `expected$author`: "Ursula K. Le Guin" "Jane Austen" "Mara Quill"
#>
#> `actual$isbn`: "978-0-306-40615-7" "978-0-14-143951-8"
#> `expected$isbn`: "978-0-306-40615-7" "978-0-14-143951-8" "9780001000100"
#>
The waldo reports are the main reason to use the third edition. The report for b1 and b2 is actual$year against expected$year: it names the element that differs and shows both values. For a data frame with a row missing waldo prints a side-by-side table of the differing rows and then one comparison for each column that differs, which is more than the message all.equal() would give but still points at the first thing to look at. The extra arguments of expect_equal() go to waldo::compare(): ignore_attr = TRUE ignores attributes such as names (the old expect_equivalent()), and ignore_function_env and ignore_formula_env ignore the environments of functions and formulas, which differ between runs when tests compare fitted models.
Logical, type and class expectations
expect_true(x) and expect_false(x) require a single TRUE or FALSE, not something that merely behaves like one: expect_true(1) fails, because 1 is a double, and expect_true(c(TRUE, FALSE)) fails because the length is two. expect_null() checks NULL. The type and class expectations check the object system explicitly, which is why the older expect_is() is deprecated in the third edition: expect_type(x, "integer") checks typeof(), expect_s3_class(x, "book") checks that an S3 object inherits from a class (exact = TRUE requires the class vector to be exactly that), expect_s4_class() an S4 class, and expect_s7_class() and expect_r6_class() the S7 and R6 equivalents. Passing NA as the class asserts that the object is not an S3 (or S4) object at all. Two newer helpers check the shape in one call: expect_vector(x, ptype, size) and expect_shape(x, nrow, ncol, dim).
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
source("scenario.R")
failure <- \(code) tryCatch(code, error = \(e) cat(conditionMessage(e), "\n"))
# Logical expectations.
expect_true(is_valid_isbn13("978-0-306-40615-7"))
expect_false(is_valid_isbn13(""))
failure(expect_true(c(TRUE, FALSE)))
#> Expected `c(TRUE, FALSE)` to be TRUE.
#> Differences:
#> `actual`: TRUE FALSE
#> `expected`: TRUE
#>
failure(expect_true(1))
#> Expected 1 to be TRUE.
#> Differences:
#> `actual` is a double vector (1)
#> `expected` is a logical vector (TRUE)
#>
expect_null(NULL)
# Types: the typeof() of the object.
b <- book("Salt and Static", "Mara Quill", year = 2018)
expect_type(b, "list")
expect_type(b$year, "integer")
failure(expect_type(b$year, "double"))
#> Expected `b$year` to have type "double".
#> Actual type: "integer"
# S3 classes: expect_s3_class() checks inherits(), exact = TRUE the whole class.
expect_s3_class(b, "book")
expect_s3_class(summary(b), "summary.book")
failure(expect_s3_class(b, "data.frame"))
#> Expected `b` to inherit from "data.frame".
#> Actual class: "book".
failure(expect_s3_class(unclass(b), "book"))
#> Expected `unclass(b)` to be an S3 object.
#> Actual OO type: none.
expect_s3_class(books, "data.frame")
expect_s3_class(books, "data.frame", exact = TRUE)
expect_s3_class(unclass(b), NA)
# S4 classes (and S7 with expect_s7_class()) have their own expectations.
setClass("Shelf", slots = c(books = "character"))
s <- new("Shelf", books = "A")
expect_s4_class(s, "Shelf")
failure(expect_s4_class(b, "Shelf"))
#> Expected `b` to be an S4 object.
#> Actual OO type: S3.
# The vector helpers check type, length and shape in one call.
expect_vector(books$year, ptype = integer(), size = nrow(books))
expect_shape(books, dim = c(16L, 6L))
failure(expect_shape(books, nrow = 20L))
#> Expected `books` to have 20 rows.
#> Actual rows: 16.
# Since testthat 3.3.0 an expectation returns its first argument: chain them.
books |> expect_s3_class("data.frame") |> expect_length(6L)
The class of b is "book", so expect_s3_class(b, "data.frame") fails and reports the actual class; the unclassed list is Actual OO type: none. See S3 Classes and Generics for what the class attribute is, and S4 Classes and Methods for setClass().
Length, names, sets, comparisons and text
These expectations exist because expect_equal(length(x), 6L) and expect_true(all(a %in% b)) fail with messages that say nothing about the objects. The dedicated forms say what they saw:
-
expect_length(x, n)andexpect_named(x, names, ignore.order, ignore.case); -
expect_setequal(x, y): the same elements, in any order and with any duplication;expect_mapequal(x, y): the same names and the same value for each name, in any order; -
expect_contains(x, y): every element ofyis inx, andexpect_in(x, y): every element ofxis iny(the two directions of subset, added in testthat 3.1.9);expect_disjoint()the opposite; -
expect_lt(),expect_lte(),expect_gt()andexpect_gte()compare numbers, dates and times, and show both values on failure; -
expect_match(x, regexp)andexpect_no_match()match text against a regular expression (see Strings and Regular Expressions), andexpect_all_true(),expect_all_false()andexpect_all_equal()check every element at once (testthat 3.3.0) and say which elements differ, whereexpect_true(all(...))can only sayFALSE.
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
source("scenario.R")
failure <- \(code) tryCatch(code, error = \(e) cat(conditionMessage(e), "\n"))
genres <- unique(tolower(trimws(books$genre)))
# Length and names.
expect_length(books, 6L)
failure(expect_length(genres, 5L))
#> Expected `genres` to have length 5.
#> Actual length: 6.
expect_named(books, c("title", "author", "isbn", "year", "genre", "pages"))
failure(expect_named(books, c("title", "author")))
#> Expected `books` to have names `c("title", "author")`.
#> Differences:
#> `actual`: "title" "author" "isbn" "year" "genre" "pages"
#> `expected`: "title" "author"
#>
expect_named(books, rev(names(books)), ignore.order = TRUE)
# Sets: order and duplicates do not matter.
expect_setequal(genres, rev(genres))
failure(expect_setequal(genres, c("fiction", "poetry", "cooking")))
#> Expected `genres` to have the same values as `c("fiction", "poetry", "cooking")`.
#> Actual: "science fiction", "classic", "fiction", "mystery", "non-fiction", "poetry"
#> Expected: "fiction", "poetry", "cooking"
#> Needs: "science fiction", "classic", "mystery", "non-fiction"
#> Absent: "cooking"
# Named lists: the same names and values, in any order.
expect_mapequal(list(a = 1, b = 2), list(b = 2, a = 1))
failure(expect_mapequal(list(a = 1, b = 2), list(a = 1, b = 3)))
#> Expected `list(a = 1, b = 2)` to contain the same values as `list(a = 1, b = 3)`.
#> Differences:
#> `actual$b`: 2.0
#> `expected$b`: 3.0
#>
# Membership: expect_contains() and expect_in() are the two directions.
expect_contains(genres, c("poetry", "classic"))
expect_in(c("poetry", "classic"), genres)
failure(expect_contains(genres, "cooking"))
#> Expected `genres` to contain all values in "cooking".
#> Actual: "science fiction", "classic", "fiction", "mystery", "non-fiction", "poetry"
#> Expected: "cooking"
#> Missing: "cooking"
failure(expect_in("cooking", genres))
#> Expected "cooking" to only contain values from `genres`.
#> Actual: "cooking"
#> Expected: "science fiction", "classic", "fiction", "mystery", "non-fiction", "poetry"
#> Invalid: "cooking"
# Numeric comparisons and text matches.
long <- books[books$pages > 400 & !is.na(books$pages), ]
expect_lt(nrow(long), 5L)
expect_gt(mean(books$year, na.rm = TRUE), 1900)
failure(expect_gt(nrow(books), 16L))
#> Expected `nrow(books)` > 16L.
#> Actual comparison: 16 <= 16
#> Difference: 0 <= 0
expect_gte(nrow(books), 16L)
expect_match(books$title[[1L]], "^The ")
failure(expect_match(books$title[[1L]], "^A "))
#> Expected `books$title[[1L]]` to match regexp "^A ".
#> Actual text:
#> ✖ │ The Dispossessed
expect_no_match(books$title[[1L]], "cooking")
# Every element at once, with a better message than expect_true(all(...)).
expect_all_true(nchar(books$title) > 0L)
failure(expect_all_true(!is.na(books$pages)))
#> Expected every element of `!is.na(books$pages)` to equal TRUE.
#> Differences:
#> `actual[3:9]`: TRUE TRUE TRUE FALSE TRUE TRUE TRUE
#> `expected[3:9]`: TRUE TRUE TRUE TRUE TRUE TRUE TRUE
#>
#> `actual[12:16]`: TRUE TRUE TRUE FALSE TRUE
#> `expected[12:16]`: TRUE TRUE TRUE TRUE TRUE
#>
failure(expect_true(all(!is.na(books$pages))))
#> Expected `all(!is.na(books$pages))` to be TRUE.
#> Differences:
#> `actual`: FALSE
#> `expected`: TRUE
#>
The last two calls show why expect_all_true() exists: for the same vector it prints the two stretches of the vector (actual[3:9] and actual[12:16]) around the two FALSE elements, so the missing page counts can be found, and expect_true(all(...)) can only say that actual is FALSE. A few of these are easy to mix up. expect_setequal() is about elements and ignores order; expect_equal() of two vectors does not. expect_contains(genres, "poetry") has the larger set first and expect_in("poetry", genres) the smaller, so the two read in opposite orders. For a named list, expect_mapequal() compares the content and not the order of the names.
Conditions: errors, warnings and messages
expect_error(), expect_warning(), expect_message() and the general expect_condition() check that the code signals the condition. The arguments are a regular expression for the message (regexp), a condition class (class), fixed = TRUE to match the text literally, and inherit to match across the parent errors of chained errors. They return the captured condition, so it can be examined, and they are the tools to test the paths that a function rejects (see Defensive Programming and Input Validation and Conditions and Error Handling).
In the third edition each of them captures at most one condition, the first one that matches the regexp and class, and lets every other condition bubble up. A condition that does not match is therefore not reported as "the message differs" but as the raw error, warning or message that it is; in a test, an unmatched error becomes an error of the test. The message is a regular expression, which is the usual cause of surprises with parentheses and dots (fixed = TRUE turns that off), and matching a message makes a test as fragile as the message. Prefer class = where the condition has one. If what you want to check is that the message is readable, use expect_snapshot(), which shows it in full.
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
source("scenario.R")
failure <- \(code) tryCatch(code, error = \(e) cat(conditionMessage(e), "\n"))
# expect_error() takes a regular expression for the message...
expect_error(book(""), "a book needs a title")
# ...fixed = TRUE when the message holds regular-expression characters...
expect_error(
new_book(title = 1), "is.character(title) is not TRUE", fixed = TRUE
)
# ...and a class, which is much less fragile than the text.
expect_error(stop("no such shelf"), class = "simpleError")
# A message that does not match.
failure(expect_error(book(""), "a book needs an author"))
#> a book needs a title
# Code that does not fail, or fails in a different way.
failure(expect_error(book("A title")))
#> Expected `book("A title")` to throw a error.
failure(expect_error(stop("something else"), "a book needs a title"))
#> something else
# Warnings and messages.
expect_warning(as.integer("x"), "NAs introduced by coercion")
expect_message(message("loading"), "loading")
failure(expect_warning(1 + 1))
#> Expected `1 + 1` to throw a warning.
# A custom condition class is matched by class =.
shelf_condition <- structure(
class = c("shelf_overdue", "warning", "condition"),
list(message = "3 loans are overdue", call = NULL)
)
expect_warning(warning(shelf_condition), class = "shelf_overdue")
expect_condition(warning(shelf_condition), "overdue")
# The expectation returns the condition, so you can inspect it.
cnd <- expect_error(book(""), "title")
class(cnd)
#> [1] "simpleError" "error" "condition"
conditionMessage(cnd)
#> [1] "a book needs a title"
# No condition at all.
expect_no_error(book("A title"))
expect_no_warning(as.integer("1"))
expect_no_message(1 + 1)
failure(expect_no_error(book("")))
#> Expected `book("")` not to throw any errors.
#> Actually got a <simpleError> with message:
#> a book needs a title
failure(expect_no_warning(as.integer("x")))
#> Expected `as.integer("x")` not to throw any warnings.
#> Actually got a <simpleWarning> with message:
#> NAs introduced by coercion
Read the three failing examples carefully. In the first, expect_error(book(""), "a book needs an author"), the error that the code really signals does not match the pattern, so it escapes the expectation: the helper prints the message of the real error, a book needs a title, and not a report that the pattern differed. The second, expect_error(book("A title")), reports Expected ... to throw a error. because the code returns normally (the article "a" before "error" is the real testthat text). In the third, the error something else escapes in the same way. The value returned by expect_error() is the condition (class(cnd) is "simpleError" "error" "condition"), so conditionMessage(cnd) is available. The last group is expect_no_error(), expect_no_warning(), expect_no_message() and expect_no_condition(), which assert that the code runs without that condition and print the condition when it does not.
The expect_no_*() expectations (experimental when added in testthat 3.1.5, stable since 3.2.0) are the replacements for expect_error(code, NA). Two cautions apply to them. Writing R Extensions tells package authors not to test for the absence of warnings, because later versions of R or of other packages may add warnings (some deprecation notices are meant to stay for a long time) and the tests should not turn them into failures; and it advises against testing the exact text of messages produced by R or by other packages, which can change and be translated. Both are good reasons to expect_no_warning() only about code that you own, and to test messages of your own functions, not R’s.
Output, silence and visibility
expect_output(code, regexp) checks what the code prints to the standard output. regexp = NA asserts that nothing is printed. expect_silent() asserts that the code prints nothing, signals no message and no warning; expect_visible() and expect_invisible() check whether the value of the call is printed at the console. Printing is done by print() methods and by cat(), so these are the tools for testing a print method such as print.book(), with the caveat that a test of the exact layout of a print method is more convenient as a snapshot.
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
source("scenario.R")
failure <- \(code) tryCatch(code, error = \(e) cat(conditionMessage(e), "\n"))
b <- book("Paper Moons", "Noor Haddad")
# Printed output: a regular expression, or NA for "no output at all".
expect_output(print(b), "n.d.")
failure(expect_output(print(b), "2018"))
#> Expected output from `print(b)` to match regexp "2018".
#> Actual output:
#> ✖ │ <book> Paper Moons (Noor Haddad, n.d.)
expect_output(invisible(b), NA)
failure(expect_output(print(b), NA))
#> Expected `print(b)` to produce no output.
#> Actual output:
#> <book> Paper Moons (Noor Haddad, n.d.)
# capture.output() gives the text that expect_output() matches against.
out <- capture.output(print(b))
out
#> [1] "<book> Paper Moons (Noor Haddad, n.d.) "
expect_identical(out, "<book> Paper Moons (Noor Haddad, n.d.) ")
# expect_silent(): no output, no messages, no warnings.
expect_silent(format(b))
failure(expect_silent(print(b)))
#> Expected `print(b)` to run silently.
#> Actual noise: output.
failure(expect_silent(as.integer("x")))
#> Expected `as.integer("x")` to run silently.
#> Actual noise: warnings.
# Visible and invisible results.
expect_invisible(invisible(1))
expect_visible(format(b))
failure(expect_invisible(format(b)))
#> Expected `format(b)` to return invisibly.
#> Actual visibility: visible.
Your own expectations: pass(), fail() and succeed()
An expectation that a package needs several times can be written as a function. It has four parts: capture the value and its label with quasi_label(rlang::enquo(object)), call fail(message) for each way the value can violate the expectation, call pass() when it does not, and return the input invisibly so that the expectation can be piped. The message has the same shape as the built-in ones: the first element says what was expected, the following ones what was seen. testthat 3.3.0 added pass(), for use in place of succeed(), which still works and just records a success with a message. Put such functions in a helper-*.R file, so that every test file has them.
Because an expectation is a function it can have bugs, and it must signal exactly one success or exactly one failure. expect_success() and expect_failure() check that, and expect_snapshot_failure() records the failure message of an expectation in a snapshot. The script uses the built package through shelf:: calls and installs the stop reporter without praise.
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
set_reporter(StopReporter$new(praise = FALSE))
books <- shelf::books
# A custom expectation: capture the label, fail or pass, return the input.
expect_valid_isbn <- function(object) {
act <- quasi_label(rlang::enquo(object))
ok <- shelf::is_valid_isbn13(act$val)
if (!all(ok)) {
fail(c(
sprintf("Expected every element of %s to be a valid ISBN-13.", act$lab),
sprintf("Invalid: %s", toString(dQuote(act$val[!ok], FALSE)))
))
} else {
pass()
}
invisible(act$val)
}
# Expectations are functions: test them with expect_success()/expect_failure().
test_that("expect_valid_isbn() reports one success or one failure", {
expect_success(expect_valid_isbn(books$isbn[1:5]))
expect_failure(expect_valid_isbn(books$isbn))
})
# And look at the message that a user would see.
tryCatch(
test_that("the failure message names the bad ISBNs", {
expect_valid_isbn(books$isbn)
}),
error = \(e) invisible(NULL)
)
#> ── Failure: the failure message names the bad ISBNs ────────────────────────────
#> Expected every element of `books$isbn` to be a valid ISBN-13.
#> Invalid: "", "978-0-01-100112-9"
The families, summarised:
| Question | Expectation | Note |
|---|---|---|
Same value? |
|
waldo diffs in the third edition; |
|
|
a length-1 logical, not "truthy" |
Which type or class? |
|
replace |
How long, which names? |
|
|
Which elements? |
|
sets ignore order |
Bigger or smaller? |
|
numbers, dates, times |
Text? |
|
regular expressions |
A condition? |
|
one captured condition; |
Printed output? |
|
|
A recorded result? |
|
see Snapshot tests |
Something else? |
|
custom expectations and skips |
Reading a failure
A test suite earns its keep when something breaks, so it is worth looking at what a break looks like. The experiment below adds a scratch test file to a copy of the package with one problem of each kind: a number that is off by one, a string that differs in case, an expect_error() whose regular expression does not match the real message, an error raised by the code under test, and a warning raised while an expectation evaluates its argument. The file is tests/testthat/test-failing.R, written by a script and deleted again at the end:
# A scratch test file with one problem of each kind (removed again at the end).
writeLines(
trimws(r"---(
test_that("overdue_loans() finds 31 late loans", {
out <- overdue_loans(loans, today = shelf_fixed_today)
expect_equal(nrow(out), 31L)
})
test_that("book() lower-cases the title", {
expect_identical(book("Salt and Static")$title, "salt and static")
})
test_that("a bad ISBN is rejected", {
expect_error(book("X", isbn = "9780001000101"), "invalid isbn")
})
test_that("clean_books() accepts a partial catalogue", {
cleaned <- clean_books(books[, c("title", "author")])
expect_identical(nrow(cleaned), 16L)
})
test_that("a plain coercion warns", {
expect_identical(as.integer("x"), NA_integer_)
})
)---", whitespace = "\n"),
"tests/testthat/test-failing.R"
)
devtools::test(filter = "failing")
unlink("tests/testthat/test-failing.R")
The script writes the file with writeLines() (the long string in the middle of the script is the content of test-failing.R), then runs only that file with devtools::test(filter = "failing"), and deletes the file. The filter is a regular expression matched against the part of the file name between test- and .R. The output is the real report of testthat 3.3.2, from the start of the run to the end of the failures; the spinner lines that a terminal overwrites, the second copy of the failures that the reporter prints under the Failed tests heading, and the random word of encouragement that closes a failing run are left out:
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
✔ | F W S OK | Context
✖ | 4 1 1 | failing
────────────────────────────────────────────────────────────────────────────────
Failure ('test-failing.R:3:3'): overdue_loans() finds 31 late loans
Expected `nrow(out)` to equal 31L.
Differences:
`actual`: 32
`expected`: 31
Failure ('test-failing.R:7:3'): book() lower-cases the title
Expected `book("Salt and Static")$title` to be identical to "salt and static".
Differences:
`actual`: "Salt and Static"
`expected`: "salt and static"
Error ('test-failing.R:11:3'): a bad ISBN is rejected
Error: invalid ISBN-13: 9780001000101
Backtrace:
▆
1. ├─testthat::expect_error(book("X", isbn = "9780001000101"), "invalid isbn") at test-failing.R:11:3
2. │ └─testthat:::expect_condition_matching_(...)
3. │ └─testthat:::quasi_capture(...)
4. │ ├─testthat (local) .capture(...)
5. │ │ └─base::withCallingHandlers(...)
6. │ └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
7. └─shelf::book("X", isbn = "9780001000101")
8. └─shelf::validate_book(...) at shelf/R/book.R:28:3
Error ('test-failing.R:15:3'): clean_books() accepts a partial catalogue
Error in ``$<-.data.frame`(`*tmp*`, genre, value = character(0))`: replacement has 0 rows, data has 16
Backtrace:
▆
1. └─shelf::clean_books(books[, c("title", "author")]) at test-failing.R:15:3
2. ├─base::`$<-`(`*tmp*`, genre, value = `<chr>`) at shelf/R/clean.R:23:3
3. └─base::`$<-.data.frame`(`*tmp*`, genre, value = `<chr>`) at shelf/R/clean.R:23:3
Warning ('test-failing.R:20:3'): a plain coercion warns
NAs introduced by coercion
Backtrace:
▆
1. └─testthat::expect_identical(as.integer("x"), NA_integer_) at test-failing.R:20:3
2. └─testthat::quasi_label(enquo(object), label)
3. └─rlang::eval_bare(expr, quo_get_env(quo))
────────────────────────────────────────────────────────────────────────────────
══ Results ═════════════════════════════════════════════════════════════════════
[ FAIL 4 | WARN 1 | SKIP 0 | PASS 1 ]
The line that starts with a cross, ✖ | 4 1 1 | failing, is the progress line of the file: the columns are the failures (F), warnings (W), skips (S) and passes (OK) so far, which is why the empty column is the skips. Each problem then gets a block that starts with its kind and its place, Failure ('test-failing.R:3:3'), which is the file, the line and the column of the expectation (a clickable link in a terminal that supports hyperlinks), followed by the description of the test.
-
The first failure is
expect_equal(nrow(out), 31L): testthat names the expression it evaluated,nrow(out), shows both values underactualandexpectedand does not say anything else, because there is nothing else to say about two numbers. The difference to a hand-writtenexpect_true(nrow(out) == 31L), whose report would benrow(out) == 31L is not TRUE, is the point of the dedicated expectations. -
The second is the same for strings; the waldo display lines the strings up under each other.
-
The third is the pitfall of the previous section. The pattern
"invalid isbn"is case-sensitive and the real message isinvalid ISBN-13: 9780001000101, so the expectation does not match; in the third edition the real error is not swallowed but escapes, and the test is reported as anError, with the message of the error under the heading and a backtrace of the calls. The backtrace shows where the error was raised (shelf::validate_book()) and, because devtools loads the package from source with source references, the file and line (shelf/R/book.R:28:3, relative to the directory that holds the package; paths shortened). -
The fourth is a bug that the test found in the code: calling
clean_books()with only two columns fails withreplacement has 0 rows, data has 16, which is a symptom and not a diagnosis: the function assignstolower(squish(books$genre))tobooks$genre, and for a catalogue without agenrecolumn that value has length zero. The report prints the call stack with the source location of the assignment inshelf/R/clean.R:23:3. A caller deserves a message that names the missing column (see Defensive Programming and Input Validation). -
The last block is a warning,
NAs introduced by coercion, which comes from evaluatingas.integer("x")as the argument of the expectation. The expectation itself passed (PASS 1); warnings are counted separately and reported, and a package whose tests warn is a package whose check logs are noisy. Wrap the call insuppressWarnings()or test the warning withexpect_warning().
The totals at the end, [ FAIL 4 | WARN 1 | SKIP 0 | PASS 1 ], are the line that every reporter finishes with, and devtools::test() returns normally even when tests fail (its argument stop_on_failure defaults to FALSE), unlike test_check(), which signals an error.
The same failure in R CMD check
Under R CMD check a failing test produces the report that a maintainer sees in the check log. To see it, one failing test file was added to a copy of the package, with the unchanged suite beside it. This is the whole file tests/testthat/test-failing.R; its expectation is false because the packaged loans contain 32 overdue loans for the fixed date, not 31:
test_that("a failing check", {
expect_equal(nrow(overdue_loans(loans, today = shelf_fixed_today)), 31L)
})
The package was then built and checked with the commands of the section on R CMD check (R CMD build shelf, then R CMD check --as-cran --no-manual on the tarball). The platform header of the log and the checks that are not about tests are trimmed from the excerpt. The check log shows the last 13 lines of the test output, which is the default of the environment variable _R_CHECK_TESTS_NLINES_, and says where the whole output went: a file testthat.Rout.fail in the check directory.
* checking tests ...
Running ‘testthat.R’
ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
══ Skipped tests (2) ═══════════════════════════════════════════════════════════
• On CRAN (2): 'test-book.R:39:1', 'test-loans.R:40:3'
══ Failed tests ════════════════════════════════════════════════════════════════
── Failure ('test-failing.R:2:3'): a failing check ─────────────────────────────
Expected `nrow(overdue_loans(loans, today = shelf_fixed_today))` to equal 31L.
Differences:
`actual`: 32
`expected`: 31
[ FAIL 1 | WARN 0 | SKIP 2 | PASS 49 ]
Error:
! Test failures.
Execution halted
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes ... OK
* checking re-building of vignette outputs ... OK
* checking for non-standard things in the check directory ... OK
* checking for detritus in the temp directory ... OK
* DONE
Status: 1 ERROR
Source: R Internals: Tools
The two skips on CRAN in the excerpt are explained in Skipping tests, and Error: ! Test failures. followed by Execution halted is how test_check() ends a failing run: the check reports the test run as ERROR, and the status of the whole check is 1 ERROR. Since testthat 3.3.0, a test run under R CMD check also writes a reprex for each failing expectation into tests/testthat/_problems/ of the check directory, so that the failure can be reproduced without a copy of the whole package. (The line Saving _problems/test-failing-2.R in the full output is that happening; the 2 is the line of the failing expectation.) After the failing check, the files of the test directory of the check and the reprex file were:
shelf.Rcheck/tests:
startup.Rs
testthat
testthat.R
testthat.Rout.fail
shelf.Rcheck/tests/testthat/_problems:
test-failing-2.R
# Extracted from test-failing.R:2
# setup ------------------------------------------------------------------------
library(testthat)
test_env <- simulate_test_env(package = "shelf", path = "..")
attach(test_env, warn.conflicts = FALSE)
# test -------------------------------------------------------------------------
expect_equal(nrow(overdue_loans(loans, today = shelf_fixed_today)), 31L)
simulate_test_env() builds the environment that a test would run in, with the package namespace as parent and the helper files loaded, and attach(test_env, ...) makes it the search path, so that the one expectation can be run from a plain R session. Setting the environment variable TESTTHAT_PROBLEMS=false turns the files off.
To investigate a failure in your own session, the usual route is not the reprex but devtools::load_all() followed by running the test body line by line: the helper file is attached by load_all(), and everything in the package, internal functions included, is visible. The commands of Debugging (browser(), traceback(), options(error = recover)) work inside a test as they do anywhere. testthat::local_test_context() reproduces the options and environment variables that a test sets before running, when the failure depends on them.
Describing behaviour with describe() and it()
describe() and it() are a second way of writing the same tests, in the style of behaviour-driven development tools such as RSpec for Ruby and Mocha for JavaScript. A describe() block names a component, an it() inside it names one specification, and the pair reads as a sentence: "is_valid_isbn13() accepts hyphenated and bare forms of a valid ISBN". The interface has been in testthat since version 0.9 (2014), and it() was exported in 3.1.8 so that it can be run interactively. Each it() is evaluated in its own environment like a test_that() block, describe() blocks can be nested, and a specification without code, it("handles NA"), is reported as a skipped test with the reason empty test, which makes it a to-do list that the suite shows. The help page says to use describe() to verify that you implement the right things and test_that() to ensure you do the things right: in practice most packages use test_that() and keep describe() for functions with several specifications.
The shelf suite has one describe() block, in test-book.R:
describe("is_valid_isbn13()", {
it("accepts hyphenated and bare forms of a valid ISBN", {
expect_identical(
is_valid_isbn13(c("978-0-00-100010-0", "9780001000100")),
c(TRUE, TRUE)
)
})
it("rejects wrong check digits, short strings, empty strings and NA", {
expect_identical(
is_valid_isbn13(c("9780001000101", "978030640615", "", NA)),
c(FALSE, FALSE, FALSE, FALSE)
)
})
it("returns a logical of the same length, including length zero", {
expect_identical(is_valid_isbn13(character()), logical())
})
})
The names of the three tests in a report or in the data frame that the runner returns are the descriptions joined with /: is_valid_isbn13() / accepts hyphenated and bare forms of a valid ISBN. test_file() can run one test of a file with its desc argument (testthat 3.2.0), and since testthat 3.3.0 desc can be a character vector that names the path through nested blocks, so one it() of a describe() is selected by the two descriptions. The first row of the result is the specification that ran, and the second the enclosing describe(), which contains no expectation of its own:
library(testthat)
# Run one specification of the describe() block of test-book.R by its path.
res <- test_file(
"tests/testthat/test-book.R",
desc = c(
"is_valid_isbn13()",
"rejects wrong check digits, short strings, empty strings and NA"
),
package = "shelf", load_package = "source", reporter = "silent"
)
results <- as.data.frame(res)
results$test
results$nb
#> shelf 0.1.0: loan period is 21 days
#> [1] "is_valid_isbn13() / rejects wrong check digits, short strings, empty strings and NA"
#> [2] "is_valid_isbn13()"
#> [1] 1 0
Snapshot tests
Some expected results are awkward to write as code: output with many quotes and line breaks, output too long to put in a test file, a print method or an error message whose point is that a human finds it readable. For these testthat has snapshot tests, also called golden tests: instead of describing the expected result in code, the expectation runs the code, records what happened in a separate Markdown file under tests/testthat/_snaps/, and on later runs compares the new result with the recorded one. They were added in testthat 3.0.0 and expect_snapshot() stopped being experimental in 3.1.0. Snapshots work only in the third edition (in the second, expect_snapshot() stops with an error saying that expect_snapshot() requires the 3rd edition, as the section on common errors shows) and are stored only when a whole test file runs, with test_file(), test_dir() or one of the functions built on them; run at the console, a snapshot expectation only displays the result, and the testthat article shows it saying that it can’t save or compare to a reference when testing interactively.
The first run: recording
expect_snapshot(code) runs code as if typed at the console and records the code, the printed output, the messages, the warnings and the errors. The first time it runs there is nothing to compare with, so testthat records the result and reports it as a warning, so that you read it before you commit it. The experiment below deletes the _snaps directory of the copy of the package and runs the tests twice. The spinner lines of devtools::test() are left out:
# Throw the stored snapshots away to see what a first run looks like.
unlink("tests/testthat/_snaps", recursive = TRUE)
devtools::test()
list.files("tests/testthat/_snaps")
devtools::test()
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
✔ | F W S OK | Context
✔ | 2 24 | book
────────────────────────────────────────────────────────────────────────────────
Warning ('test-book.R:42:3'): a book prints and summarises as expected
Adding new snapshot:
Code
print(b)
Output
<book> Salt and Static (Mara Quill, 2018)
Warning ('test-book.R:43:3'): a book prints and summarises as expected
Adding new snapshot:
Code
summary(b)
Output
Book: Salt and Static
ISBN known: TRUE
Pages: 244
────────────────────────────────────────────────────────────────────────────────
✔ | 14 | clean
✔ | 14 | loans
══ Results ═════════════════════════════════════════════════════════════════════
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 52 ]
[1] "book.md"
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
✔ | F W S OK | Context
✔ | 24 | book
✔ | 14 | clean
✔ | 14 | loans
══ Results ═════════════════════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 52 ]
In the first run the two warnings, WARN 2, are Adding new snapshot: followed by the code and the output that were recorded, and the file is back: list.files() shows book.md. In the second run the same two expectations pass without any report and the totals are the ones of the normal run. The recorded file is the _snaps/book.md shown earlier. Its format is a subset of Markdown on purpose, so that a reviewer can read changes in a pull request without running anything. Each test is a level-one heading; each snapshot is indented by four spaces as a code block; snapshots of the same test are separated by ---; inside one, the labels Code, Output, Message and Condition say what the lines below are. The file is named after the test file, test-book.R giving _snaps/book.md. A test that is deleted loses its snapshot the next time the whole suite runs, and a snapshot file with no snapshots left is deleted.
On a continuous integration server a new snapshot is a failure and not a warning (see Snapshots on CI). A snapshot is part of the code and goes in version control; reviewing the diff of _snaps/ is the review of the behaviour change.
When a snapshot changes
If the output changes, the expectation fails, shows a diff of the old and the new, and writes the new result to book.new.md next to the recorded file. The experiment below changes print.book() so that it prints <Book> instead of <book>, runs the tests of that file, accepts the change with snapshot_accept(), shows what changed in book.md, and restores the source and the snapshot:
# Remember the originals so that the script can put them back at the end.
book_r <- readLines("R/book.R")
snap <- readLines("tests/testthat/_snaps/book.md")
# Change what print.book() prints: "<book>" becomes "<Book>".
changed <- sub('cat("<book>"', 'cat("<Book>"', book_r, fixed = TRUE)
writeLines(changed, "R/book.R")
# The snapshot no longer matches: the test fails and a .new.md file appears.
devtools::test(filter = "book")
list.files("tests/testthat/_snaps")
# The change is deliberate: accept it, which replaces book.md by book.new.md.
testthat::snapshot_accept("book")
list.files("tests/testthat/_snaps")
setdiff(readLines("tests/testthat/_snaps/book.md"), snap)
# Clean up: restore the source and the snapshot.
writeLines(book_r, "R/book.R")
writeLines(snap, "tests/testthat/_snaps/book.md")
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
✔ | F W S OK | Context
✖ | 1 23 | book
────────────────────────────────────────────────────────────────────────────────
Failure ('test-book.R:42:3'): a book prints and summarises as expected
Snapshot of code has changed:
old vs new
Code
print(b)
Output
- <book> Salt and Static (Mara Quill, 2018)
+ <Book> Salt and Static (Mara Quill, 2018)
* Run `testthat::snapshot_accept("book")` to accept the change.
* Run `testthat::snapshot_review("book")` to review the change.
────────────────────────────────────────────────────────────────────────────────
══ Results ═════════════════════════════════════════════════════════════════════
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 23 ]
[1] "book.md" "book.new.md"
Updating snapshots: 'book.md'.
[1] "book.md"
[1] " <Book> Salt and Static (Mara Quill, 2018) "
The report is a diff in which - lines are the recorded result and + lines the new one, with the lines Code and print(b) as context, and the two lines at the end of the report name the remedies. snapshot_accept("book") replaces book.md by book.new.md, which is the right step only when the change is the one intended; if it is not, fix the code and the test passes again, and the .new.md file goes away. snapshot_review("book") opens a Shiny application (it needs the shiny package) that shows each pending change and lets you accept or reject it; snapshot_reject() deletes the .new.md files (testthat 3.3.0). The recorded change is a single line, as setdiff() of the two versions of book.md shows, with the leading spaces of the code-block indentation.
Snapshots on CI
A snapshot that has no stored version is only a warning on a developer’s machine, so that the first run records it. On a continuous-integration server it is a failure, because a missing snapshot there almost always means that the snapshot file was never committed (testthat 3.3.0). The server is recognised by the environment variable CI, which hosted services set to true. The experiment sets it by hand, after remembering the stored snapshot and deleting it, runs the tests of test-book.R, and puts the snapshot back:
# Remember the stored snapshot, then throw it away to simulate a forgotten file.
snap <- readLines("tests/testthat/_snaps/book.md")
unlink("tests/testthat/_snaps", recursive = TRUE)
# CI services set CI=true (GitHub Actions does): a new snapshot now fails.
Sys.setenv(CI = "true")
devtools::test(filter = "book")
# Put the stored snapshot back.
dir.create("tests/testthat/_snaps", showWarnings = FALSE)
writeLines(snap, "tests/testthat/_snaps/book.md")
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
✔ | F W S OK | Context
✖ | 2 24 | book
────────────────────────────────────────────────────────────────────────────────
Failure ('test-book.R:42:3'): a book prints and summarises as expected
Adding new snapshot:
Code
print(b)
Output
<book> Salt and Static (Mara Quill, 2018)
Failure ('test-book.R:43:3'): a book prints and summarises as expected
Adding new snapshot:
Code
summary(b)
Output
Book: Salt and Static
ISBN known: TRUE
Pages: 244
────────────────────────────────────────────────────────────────────────────────
══ Results ═════════════════════════════════════════════════════════════════════
[ FAIL 2 | WARN 0 | SKIP 0 | PASS 24 ]
The two blocks are the same as the two warnings of the first run, but they are reported as Failure and the totals are [ FAIL 2 | WARN 0 | SKIP 0 | PASS 24 ] (PASS 24 counts the passes in test-book.R alone, since the filter selected that file). The content that was going to be recorded is shown, which is what you need to compare with the file you meant to commit. The remedy is to run the tests locally, read the snapshot, and commit tests/testthat/_snaps/. On a machine where the stored file exists but some of its snapshots differ the report is the diff of the previous section, whatever the value of CI. (The random line of encouragement that the progress reporter prints after a failing run is left out of the transcript.)
Other snapshot expectations
The six tests below, again in a scratch file of the copy of the package, show the other forms. They run once with the reporter silenced, and the script prints the files that the first run created:
# A scratch test file that uses the other snapshot expectations.
writeLines(
trimws(r"---(
test_that("an invalid book is snapshotted with its error", {
expect_snapshot(book("X", isbn = "9780001000101"), error = TRUE)
})
test_that("messages and warnings are recorded with the code", {
expect_snapshot({
x <- as.integer("x")
message("loaded ", nrow(books), " books")
summary(book("Paper Moons", "Noor Haddad", pages = 120))
})
})
test_that("a value is snapshotted in a readable form", {
expect_snapshot_value(loans_per_month(loans)[1:2, ], style = "deparse")
})
test_that("a file is snapshotted as a whole", {
path <- withr::local_tempfile(fileext = ".csv")
write.csv(loans_per_month(loans)[1:3, ], path, row.names = FALSE)
expect_snapshot_file(path, "months.csv")
})
test_that("variable text is cleaned with transform", {
path <- withr::local_tempfile(fileext = ".csv")
expect_snapshot(
cat("saved to", path, "\n"),
transform = \(lines) gsub(path, "<path>", lines, fixed = TRUE)
)
})
test_that("a snapshot can depend on the R version", {
variant <- paste0("r", getRversion()$major, ".", getRversion()$minor)
expect_snapshot(print(loans_per_month(loans)[1:2, ]), variant = variant)
})
)---", whitespace = "\n"),
"tests/testthat/test-snapshots.R"
)
# The first run creates every snapshot (the reporter is silenced here).
invisible(devtools::test(filter = "snapshots", reporter = "silent"))
list.files("tests/testthat/_snaps", recursive = TRUE)
cat(readLines("tests/testthat/_snaps/snapshots.md"), sep = "\n")
cat(readLines("tests/testthat/_snaps/snapshots/months.csv"), sep = "\n")
cat(readLines("tests/testthat/_snaps/r4.6/snapshots.md"), sep = "\n")
# Clean up the scratch test and its snapshots.
unlink("tests/testthat/test-snapshots.R")
unlink(
file.path("tests/testthat/_snaps", c("snapshots.md", "snapshots", "r4.6")),
recursive = TRUE
)
list.files("tests/testthat/_snaps")
The script prints the files that the first run created, and then the contents of three of them. The files are book.md (the snapshots of the real suite, untouched), r4.6/snapshots.md (the variant), snapshots.md (the test file) and, for the whole-file snapshot, snapshots/months.csv; list.files(recursive = TRUE) shows where each lives. The text printed afterwards is snapshots.md, then months.csv, then r4.6/snapshots.md; the last line is the list of the snapshot directory after the clean-up:
[1] "book.md" "r4.6/snapshots.md" "snapshots.md"
[4] "snapshots/months.csv"
# an invalid book is snapshotted with its error
Code
book("X", isbn = "9780001000101")
Condition
Error:
! invalid ISBN-13: 9780001000101
# messages and warnings are recorded with the code
Code
x <- as.integer("x")
Condition
Warning:
NAs introduced by coercion
Code
message("loaded ", nrow(books), " books")
Message
loaded 16 books
Code
summary(book("Paper Moons", "Noor Haddad", pages = 120))
Output
Book: Paper Moons
ISBN known: FALSE
Pages: 120
# a value is snapshotted in a readable form
structure(list(month = c("2025-01", "2025-02"), loans = c(4L,
2L)), row.names = 1:2, class = "data.frame")
# variable text is cleaned with transform
Code
cat("saved to", path, "\n")
Output
saved to <path>
"month","loans"
"2025-01",4
"2025-02",2
"2025-03",3
# a snapshot can depend on the R version
Code
print(loans_per_month(loans)[1:2, ])
Output
month loans
1 2025-01 4
2 2025-02 2
[1] "book.md"
-
error = TRUEis required to record an error.expect_snapshot(book("X", isbn = "9780001000101"))without it fails, because an unexpected error is a failure (even on CRAN); with it, the error is recorded as aConditionand the test passes, and an expected error that does not occur is a failure. The error argument checks that at least one expression in the block errors, not that every one does. The display shows the call and the message the way R prints them at the console; it does not include the class of the condition unlesscnd_class = TRUE, because classes are added by upstream packages and R itself (testthat 3.1.1 introduced the argument and 3.2.0 made the class-free display the default for every package). -
One
expect_snapshot()can hold several statements, as the second test does: the assignment with a warning, a message, and a call whose printed value is recorded. Each statement is recorded under its ownCodeheading, followed byOutput,MessageorConditionfor what it produced. -
expect_snapshot_value(x, style)records the value instead of the printed text. The styles are"json"(the default; the simplest output, for simple objects, written with the jsonlite package),"json2"(more verbose, for a wider range of objects),"deparse"(a depiction of the object as R code) and"serialize"(a binary serialization that works for nearly any object and is opaque);tolerancemakes the comparison numerical, likeexpect_equal(). In the example,deparserecords two lines of R code that rebuild the data frame. -
expect_snapshot_file(path, name)snapshots a whole file, such as a plot or a written CSV, in_snaps/<test file>/<name>, one file for each expectation. A changed file cannot be diffed automatically: the new version is saved asmonths.new.csvandsnapshot_review()shows both. The function is usually called through a wrapper that skips the test on platforms where the output is not reproducible, and a wrapper should callannounce_snapshot_file()so that testthat does not delete a snapshot that was not regenerated because the test was skipped. For plots testthat points to the vdiffr package, which is not installed here and is named only. -
transform = \(lines) ...rewrites the lines before they are recorded, which is how to remove temporary paths, versions, times and other things that vary. Here the temporary path fromwithr::local_tempfile()is replaced by<path>; without the transform, the snapshot would change on every run, and the test would fail on every run after the first. -
variant = "r4.6"stores the result in_snaps/r4.6/snapshots.mdinstead of_snaps/snapshots.md, so that one test can record the output that differs by operating system, R version or dependency version, each in its own file. testthat calls variants an advanced feature: once they exist, you are responsible for deleting the ones that are no longer used, and for having a way to get changed snapshots out of the CI system and back into the repository.
Snapshots and CRAN
expect_snapshot() has the argument cran = FALSE: by default the snapshot expectations are not verified on CRAN, because snapshot tests rely on details of the output that dependencies may change (the help page gives that reason). On CRAN they return without recording or comparing anything. In testthat 3.3.0 they stopped skipping the rest of the test (as skip_on_cran() would) and began returning right away, neither passing nor failing, and the test that contains them is reported as skipped on CRAN. That is what the real R CMD check --as-cran of the package shows: SKIP 2, the snapshot test in test-book.R and the skip_on_cran() test in test-loans.R, and PASS 49 instead of the PASS 52 of devtools::test() (see Skipping tests). Do not set cran = TRUE for text that depends on other packages or on R’s own messages. Writing R Extensions warns against exactly that: R’s messages change between versions and are translated.
When to use a snapshot
A snapshot is a record of what the code did, which is not the same as a statement of what it should do. A reviewer who accepts a changed snapshot without reading it has turned the test off. Use a snapshot where a human is the judge (a print method, a cli message, the wording of an error), where the output is long and only its stability matters, or as a first test of legacy code. Use expect_equal() and its relatives for values and structure, which a snapshot hides in text, and expect_error(class = ) where an error has a class; in the shelf suite, describe() and expect_identical() test the logic, and only the two print methods are snapshots. Keep the code in a snapshot small, so that a failure diff points at one thing.
Fixtures and withr
A test should leave the world as it found it. The state after test number i is the starting state of test i + 1, so a test that sets an option, an environment variable or the working directory and forgets to undo it makes the next test depend on it, and the failures that result appear in a different place from the cause. The test-fixtures article of testthat lists the things a test typically changes: a file or a directory, a resource on an external system, an R option, an environment variable, the working directory, or the state of the package under test. A fixture in this sense is a piece of code that sets one of them up for the length of a test and removes it afterwards.
testthat does not contain the tools for this itself; it relies on the package withr (3.0.3 here, an add-on package, listed under Suggests in shelf because the tests call it with ::). Its functions come in pairs: with_*() changes something for the evaluation of one expression and local_*() changes it until the surrounding function, or the surrounding test_that() block, ends. The five that cover most tests are:
| Change | withr function |
|---|---|
Create a temporary file that is deleted afterwards |
|
Create a temporary directory |
|
Set an R option |
|
Set an environment variable |
|
Change the working directory |
|
withr has many more: local_seed() and with_seed() for the random-number generator, local_timezone(), local_locale(), local_language() and local_collate() for the things that differ between machines, local_path(), local_libpaths(), local_connection(), local_db_connection(), local_package() and the graphics devices (local_png(), local_pdf(), local_svg()). The script below uses the most common ones inside tests and prints, after each test, that the state is back. It sets the stop reporter without praise so that a passing test prints nothing.
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
set_reporter(StopReporter$new(praise = FALSE))
source("scenario.R")
# A temporary file that is deleted when the test ends.
test_that("local_tempfile() cleans up after the test", {
path <- withr::local_tempfile(fileext = ".csv")
write.csv(head(books, 3L), path, row.names = FALSE)
expect_true(file.exists(path))
expect_identical(nrow(read.csv(path)), 3L)
assign("kept_path", path, envir = globalenv())
})
file.exists(kept_path)
#> [1] FALSE
# An option, an environment variable and the working directory.
test_that("options, environment variables and directories are restored", {
withr::local_options(digits = 3L, shelf.loan_days = 14L)
withr::local_envvar(SHELF_ENV = "test")
dir <- withr::local_tempdir()
withr::local_dir(dir)
expect_identical(Sys.getenv("SHELF_ENV"), "test")
expect_identical(getOption("shelf.loan_days"), 14L)
expect_identical(format(pi), "3.14")
expect_identical(list.files(), character())
})
c(Sys.getenv("SHELF_ENV", "<unset>"), format(pi), basename(getwd()))
#> [1] "<unset>" "3.141593" "shelf"
is.null(getOption("shelf.loan_days"))
#> [1] TRUE
# A seed that does not disturb the caller's random number stream.
set.seed(1)
before <- runif(1)
set.seed(1)
test_that("with_seed() makes a random sample reproducible", {
s1 <- withr::with_seed(42, sample(nrow(books), 3L))
s2 <- withr::with_seed(42, sample(nrow(books), 3L))
expect_identical(s1, s2)
})
identical(runif(1), before)
#> [1] TRUE
# defer() runs a cleanup when the test finishes, last in, first out.
test_that("defer() registers cleanups that run in reverse order", {
withr::defer(cat("cleanup 1\n"))
withr::defer(cat("cleanup 2\n"))
cat("test body\n")
expect_true(TRUE)
})
#> test body
#> cleanup 2
#> cleanup 1
# A helper that sets the option for its caller: pass .local_envir through.
local_loan_days <- function(days, env = parent.frame()) {
withr::local_options(shelf.loan_days = days, .local_envir = env)
invisible(days)
}
test_that("a local_ helper changes state for one test only", {
local_loan_days(7L)
expect_identical(getOption("shelf.loan_days"), 7L)
})
getOption("shelf.loan_days", "<unset>")
#> [1] "<unset>"
Read the output from the top.
-
The first test writes three rows of
booksto a temporary file and reads them back. The path is saved in a variable of the global environment only to look at it afterwards:file.exists(kept_path)isFALSE, becauselocal_tempfile()registered the deletion of the file for the end of the test. The file lives in the session’s temporary directory, which is the only place CRAN policy allows a package to write during checks (apart from the user-specific directories oftools::R_user_dir(), which tests should not touch). -
The second test changes an option (
digits), a package option (shelf.loan_days), an environment variable and the working directory (to a temporary directory, which is itself deleted at the end). Inside the test the new values are in force (format(pi)is"3.14"withdigits = 3). Outside, the linec(Sys.getenv("SHELF_ENV", "<unset>"), format(pi), basename(getwd()))shows the environment variable unset,piprinted with seven digits again, and the working directory back where it was (its last component,shelf, is the directory the script is run from);is.null(getOption("shelf.loan_days"))isTRUEbecause the option did not exist before. -
The third test shows
with_seed(). It runs a block with a given seed and restores the random-number stream afterwards, sorunif(1)after the test continues the stream the caller had (identical(runif(1), before)isTRUE). The seed does not leak out of the test. This is the right way to test code that uses random numbers; Writing R Extensions points out that tests in a package run in a vanilla session without a seed. -
The fourth test calls
withr::defer()twice. The deferred actions run when the test ends, in the reverse order of registration:cleanup 2is printed beforecleanup 1, after the line printed by the body of the test. Last in, first out is what makes teardown correct: a file that was created inside a temporary directory must be removed before the directory. -
The last test defines a helper,
local_loan_days(), that changes an option for its caller. The helper takes anenv = parent.frame()argument and passes it on as.local_envir; without that, the option would be restored when the helper itself returns, which is immediately. The finalgetOption("shelf.loan_days", "<unset>")prints<unset>again.
on.exit() and defer()
Base R’s on.exit() is the primitive behind all of this: it runs an expression when the current function exits, whether normally or through an error. In a test the habit is on.exit(options(old), add = TRUE, after = FALSE), where the two arguments make sure that a second on.exit() call adds to the first instead of replacing it, and that the cleanup runs in reverse order. The test-fixtures article names three drawbacks: the arguments are easy to forget, it does nothing useful at the top level of a script (the code is never run), and it cannot be put in a helper, because it is bound to the function that calls it. withr::defer() has none of them: its default behaviour is the right one, it can be told which environment to bind the cleanup to (a helper passes its caller’s environment), and at the top level of a script or the console it records the action and you run it with withr::deferred_run(). The local_*() functions are defer() applied to the most common setup actions. See Scoping, Environments and Closures for what parent.frame() is.
Source: R documentation: on.exit
The name local_ is chosen on purpose: the change is local to the function or test. The article adds that a local_ call can be scoped to a smaller part of a test with local({ ... }).
Where a fixture is applied: test, file, package
A fixture can be applied at three scopes, and the choice is a trade-off between duplication and debuggability.
-
Inside a
test_that()block, which is the default. The change lasts for that test. A failing test can be reproduced in an interactive session by running the body line by line. -
At the top of a test file, outside any
test_that(). It then lasts to the end of the file, and so affects every test after it. This saves repetition, but a test that fails cannot be reproduced without first running the lines at the top. The testthat article leans towards copying the fixture into each test for exactly that reason. -
Package-wide, in a setup file in
tests/testthat/. The code of a file whose name starts withsetup(by conventionsetup.R) runs once before any test file. Cleanup belongs in the same file, registered withwithr::defer(..., teardown_env())orwithr::local_options(..., .local_envir = teardown_env()):teardown_env()is a special environment that stays alive until all the tests have run. The older approach, ateardown-*.Rfile, still works but is no longer recommended, because setup and cleanup in different files drift apart.
A helper file (helper-*.R) is different: it only defines things, such as the shelf_fixed_today constant and read_fixture_books() of the shelf suite, custom expectations and local_* helpers. The difference between the two matters when the tests are run interactively. Helper files are sourced by devtools::load_all() as well as by test_check() and its relatives, so the objects exist when you work on a failing test. Setup files are sourced only by the functions that run tests, not by load_all(), so after load_all() whatever a setup file would have created, such as an option, is missing. The experiment below adds a scratch setup file and a scratch test to a copy of the package, runs them, and then calls load_all() outside the tests to see which of the two is available:
# A scratch setup file and a test file that depends on it.
writeLines(
trimws(r"---(
withr::local_options(shelf.quiet = TRUE, .local_envir = teardown_env())
message("setup-scratch.R: the setup file ran")
withr::defer(message("setup-scratch.R: the teardown ran"), teardown_env())
)---", whitespace = "\n"),
"tests/testthat/setup-scratch.R"
)
writeLines(
trimws(r"---(
test_that("the setup file has run before the tests", {
expect_true(getOption("shelf.quiet"))
expect_identical(shelf_fixed_today, as.Date("2026-06-30"))
expect_true(file.exists(test_path("fixtures", "books-small.csv")))
message("test-scratch.R: inside the test")
})
)---", whitespace = "\n"),
"tests/testthat/test-scratch.R"
)
devtools::test(filter = "scratch")
# Outside a test, load_all() sources the helpers but not the setup files.
devtools::load_all(quiet = TRUE)
shelf_fixed_today
is.null(getOption("shelf.quiet"))
testthat::test_path("fixtures", "books-small.csv")
unlink(c("tests/testthat/setup-scratch.R", "tests/testthat/test-scratch.R"))
This is a script for a package directory, run with Rscript from the package; its transcript, with the progress spinner’s redrawn lines taken out (the message that the test prints appears at the end of such a line in a captured log) is:
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
setup-scratch.R: the setup file ran
✔ | F W S OK | Context
test-scratch.R: inside the test
✔ | 3 | scratch
══ Results ═════════════════════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 3 ]
setup-scratch.R: the teardown ran
shelf 0.1.0: loan period is 21 days
[1] "2026-06-30"
[1] TRUE
[1] "tests/testthat/fixtures/books-small.csv"
The setup file ran first (setup-scratch.R: the setup file ran, a message printed before the progress table), the test saw the option it set (expect_true(getOption("shelf.quiet"))) and the constant from the helper (shelf_fixed_today), found the fixture through test_path(), and the teardown message (the teardown ran) was printed after the results, when teardown_env() was cleaned up. After the run, load_all() makes shelf_fixed_today available ("2026-06-30"), but the option that the setup file set is not there (is.null(getOption("shelf.quiet")) is TRUE), and test_path() resolves the fixture relative to the tests directory. Keep setup files small for that reason: whatever they create has to be recreated by hand before a failing test can be debugged.
Data files for tests go in the tests/testthat/fixtures/ directory (any name works, since testthat ignores directories it does not know), which the shelf suite uses for books-small.csv. Reach them with test_path("fixtures", "books-small.csv"): the function returns a path that is correct whether the tests are run from the package root, from tests/testthat/ or by R CMD check. A helper that reads the file, as read_fixture_books() does, keeps the path in one place. Anything a test writes goes to withr::local_tempfile(), never into the tests directory.
Finding the test that leaks: set_state_inspector()
When a suite becomes slow to debug because some test leaves a mess, set_state_inspector() (testthat 3.1.9) registers a function that testthat calls before and after every test; it reports any difference as a warning against the test that caused it. The function returns the pieces of state to watch. The scratch file below has one polite test and two that leak, an option and an environment variable:
library(testthat)
# Ask testthat to compare these pieces of global state before and after a test.
set_state_inspector(function() {
list(
digits = getOption("digits"),
shelf_env = Sys.getenv("SHELF_ENV", NA)
)
})
writeLines(
trimws(r"---(
test_that("a polite test restores what it changes", {
withr::local_options(digits = 3L)
expect_true(TRUE)
})
test_that("a test that leaks an option", {
options(digits = 4L)
expect_true(TRUE)
})
test_that("a test that leaks an environment variable", {
Sys.setenv(SHELF_ENV = "leak")
expect_true(TRUE)
})
)---", whitespace = "\n"),
"tests/testthat/test-leaky.R"
)
devtools::test(filter = "leaky")
unlink("tests/testthat/test-leaky.R")
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
✔ | F W S OK | Context
✔ | 2 3 | leaky
────────────────────────────────────────────────────────────────────────────────
Warning ('test-leaky.R:6:1'): a test that leaks an option
Global state has changed:
`before$digits`: 7
`after$digits`: 4
Warning ('test-leaky.R:11:1'): a test that leaks an environment variable
Global state has changed:
`before$shelf_env`: NA
`after$shelf_env`: "leak"
────────────────────────────────────────────────────────────────────────────────
══ Results ═════════════════════════════════════════════════════════════════════
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 3 ]
Each warning carries the location of the test (test-leaky.R:6:1, the line of the test_that() call), the word Global state has changed and the difference found by waldo, with the value before and after. The polite test, which used withr::local_options(), is not reported. A test file that is run in parallel inherits the state of the previous file in the same worker, so the inspector is also the tool for the stochastic failures that parallel testing can expose.
Mocking with local_mocked_bindings()
Mocking means temporarily replacing a function by a fake that behaves in a way the test controls. The reasons are the ones in the mocking article of testthat: to reach a failure that is hard to produce on purpose (what if a package is missing?), to make a test faster (replace a slow computation), and to make a test reliable (replace the clock, the network, the contents of a file). The same article is frank about the cost: mocking is an advanced technique that can make tests brittle, or let them hide problems, and should be used only when a simpler technique does not work.
In shelf there is exactly one mock, and it is a good example of the situation. overdue_loans() takes the reference date as an argument, today, which makes nearly all the tests deterministic without any mock. Its default, shelf_today(), is a one-line internal function that returns the current date: the code that reads the clock is in one place, behind a name that a test can replace.
The function is defined in R/loans.R with a comment that says why it exists ("kept as a function so that tests can replace it"), and its body is a single call of the base function Sys.Date.
The test that uses it is the second one of test-loans.R (shown in full above):
test_that("overdue_loans() uses the current date by default", {
local_mocked_bindings(shelf_today = function() shelf_fixed_today)
expect_identical(
overdue_loans(loans),
overdue_loans(loans, today = shelf_fixed_today)
)
})
local_mocked_bindings(shelf_today = function() shelf_fixed_today) replaces the binding shelf_today in the namespace of the package under test, for the length of the test; at the end of the test the original is put back. No .package argument is needed here: when a whole package’s tests are run, or when one package is loaded with devtools::load_all(), testthat knows which package is meant. Outside a package, in a script, it must be named.
How the replacement works
local_mocked_bindings() changes the binding in the package’s namespace environment, where the package’s own functions find their free variables (see Packages, Libraries and Namespaces for what a namespace is and Scoping, Environments and Closures for lexical scoping). Code in the package that calls shelf_today() sees the replacement; code elsewhere is not affected, which is the property testthat calls hygiene. Three consequences follow from that, and each shows up in the script below:
-
An unexported function cannot be reached as
shelf::shelf_today(); the::operator reports that it is not exported, and:::reaches it. The script prints both. -
Only a binding that already exists can be replaced. The namespace of a package is locked once it is loaded, so a new binding cannot be added, and
local_mocked_bindings()reports a missing binding. -
A function that belongs to base R, such as
Sys.Date, has no binding in the package namespace and so cannot be replaced there. The remedies are either to put a placeholder binding for the base function in the package (interactive <- NULLis the example of the help page), or, asshelfdoes, to wrap the call in a function of the package, which is also better design. A function that the package calls asotherpackage::fun()is not read from the package namespace either; import it or call it through a wrapper that can be mocked.
The binding can be anything, not only a function (since testthat 3.2.1). Bindings in the imports of the package are also replaced, and so are S3 methods (since 3.1.8). with_mocked_bindings(code, ...) is the same replacement for one expression instead of the rest of the current scope.
library(testthat)
set_reporter(StopReporter$new(praise = FALSE))
loans <- shelf::loans
# shelf_today() is internal: shelf:: cannot reach it, shelf::: can.
tryCatch(shelf::shelf_today(), error = \(e) conditionMessage(e))
#> [1] "'shelf_today' is not an exported object from 'namespace:shelf'"
inherits(shelf:::shelf_today(), "Date")
#> [1] TRUE
# Which environment owns the function? The package, until it is replaced.
owner <- \() environmentName(environment(shelf:::shelf_today))
owner()
#> [1] "shelf"
# Outside the package's own tests, name the package whose binding to mock.
test_that("shelf_today() can be replaced for one test", {
local_mocked_bindings(
shelf_today = \() as.Date("2026-06-30"),
.package = "shelf"
)
expect_identical(shelf:::shelf_today(), as.Date("2026-06-30"))
print(owner())
expect_identical(nrow(shelf::overdue_loans(loans)), 32L)
})
#> [1] ""
# After the test the original function is back.
owner()
#> [1] "shelf"
# The same replacement for a block of code only.
with_mocked_bindings(
nrow(shelf::overdue_loans(loans)),
shelf_today = \() as.Date("2026-06-30"),
.package = "shelf"
)
#> [1] 32
# Only bindings that already exist can be mocked.
tryCatch(
local_mocked_bindings(not_a_function = \() 1, .package = "shelf"),
error = \(e) conditionMessage(e)
)
#> [1] "Can't find binding for `not_a_function`"
# Base functions such as Sys.Date are not bindings of the package namespace.
tryCatch(
local_mocked_bindings(
Sys.Date = \() as.Date("2026-06-30"),
.package = "shelf"
),
error = \(e) conditionMessage(e)
)
#> [1] "Can't find binding for `Sys.Date`"
# A mock that returns different values in turn, such as answers to a prompt.
test_that("mock_output_sequence() returns its values one at a time", {
answer <- mock_output_sequence("3", "n")
expect_identical(answer(), "3")
expect_identical(answer(), "n")
expect_error(answer(), "Can't find value for 3th iteration")
})
The first call prints the real text of the error that :: gives for an internal function; the next line shows ::: working. owner() asks which environment the function belongs to: before the test it is the package ("shelf"). Inside the test, shelf:::shelf_today() returns the fixed date, owner() prints an empty name (the replacement is a function created inside the test, whose environment has no name), and overdue_loans(loans), called with no today argument, finds the 32 overdue loans that today = shelf_fixed_today found in the real test. After the test owner() is "shelf" again: the original function is back. with_mocked_bindings() returns the value of its expression (32). The next two calls are the real messages for the two mistakes above. The last test shows mock_output_sequence(), which creates a function that returns its arguments one at a time, for code that asks for several answers (a prompt, a retried request); asking for more values than it holds is an error, caught by expect_error().
Other mocks, and alternatives
testthat 3.3.0 added local_mocked_s3_method(), local_mocked_s4_method() and local_mocked_r6_class() for methods and classes, and 3.3.2 let the S3 and S4 versions mock methods that do not exist yet or remove one temporarily. The mocking article says to reach for them once the basic idea is clear and not before.
The older with_mock() and local_mock() replaced functions by modifying the namespace of a package. The NEWS of testthat deprecated them in 3.0.0, recommending the mockr or mockery packages instead, and made them defunct in 3.3.0, because the technique that made them work is no longer permitted in R 4.5.0; the message that the script of the third-edition section printed for with_mock() is "with_mock() was deprecated in testthat 3.2.0 and is now defunct" followed by the advice to use with_mocked_bindings(). Those two packages, which were the recommendation before local_mocked_bindings() existed, are not used on this page; neither is installed in the image used here.
Before mocking, three designs remove the need for it, and tests written that way are simpler to read:
-
Make the state an argument.
overdue_loans(loans, today = ...)andloan_summary(loans, books, today, loan_days)take the date and the loan period as parameters with defaults, so a test passes the values it wants. -
Make the global state a package option read at call time, as
shelf.loan_daysis, and change it for one test withwithr::local_options(). -
Separate computation from input and output, so the part under test is a pure function (see Writing code that is easy to test).
Mocking is the right tool when none of those fits: a failure that cannot be provoked (the network is down), a slow or non-deterministic dependency, an interactive prompt, a platform check. Keep the replacement as small as possible, and test the real thing too wherever you can.
Skipping tests
A skipped test is neither a pass nor a failure: it is a test that cannot or should not run in the current circumstances, and the reporter says why. The cases the skipping article lists are the usual ones. The test depends on a web service that sometimes fails, or on credentials that exist only on some machines. It uses a feature that some operating systems lack (Windows, for example, has had limited UTF-8 support). It needs a version of R or of a dependency that not every test machine has. Or it is simply too slow for the CRAN check farm, whose policy asks that checking a package should take as little CPU time as possible, that long-running tests may be made optional, and that a package which uses Internet resources should fail gracefully, with an informative message and without a check warning or error, when the resource is unavailable.
Source: CRAN Repository Policy
skip(message) signals a condition of class skip; test_that() catches it, abandons the rest of the block and reports the reason. The helper functions call skip() for you when a condition holds:
| Function | Skips when |
|---|---|
|
The condition is |
|
The package is not installed or cannot be loaded, or is older than |
|
The running R does not satisfy the specification (testthat 3.3.0). |
|
The tests count as running on CRAN: |
|
The operating system ( |
|
The environment variable |
|
covr is measuring coverage (the environment variable |
|
The check runs on a Bioconductor build machine ( |
|
There is no internet connection; needs the curl package, and itself calls |
|
The message identified by |
The script catches the skip condition with tryCatch() (a skip is a condition, see Conditions and Error Handling) and prints the reason that each helper would have given. The helper reason() returns "not skipped" when the code ran to the end:
library(testthat)
# A skip is a condition of class "skip": catch it to read the reason.
reason <- \(code) {
tryCatch(
{
code
"not skipped"
},
skip = \(s) conditionMessage(s)
)
}
reason(skip("the catalogue server is down"))
#> [1] "Reason: the catalogue server is down"
reason(skip_if(TRUE, "RUN_SLOW_TESTS is not set"))
#> [1] "Reason: RUN_SLOW_TESTS is not set"
reason(skip_if_not(FALSE))
#> [1] "Reason: FALSE is not TRUE"
reason(skip_if_not(file.exists("secure-token.json"), "no token"))
#> [1] "Reason: no token"
reason(skip_if_not_installed("nosuchpackage"))
#> [1] "Reason: {nosuchpackage} is not installed"
reason(skip_if_not_installed("testthat", minimum_version = "99.0.0"))
#> [1] "Reason: Installed testthat is version 3.3.2; but 99.0.0 is required"
reason(skip_on_os("windows"))
#> [1] "not skipped"
reason(skip_unless_r(">= 99.0"))
#> [1] "Reason: Current R version (4.6.1) does not satisfy requirement (>= 99.0)"
reason(skip_on_cran())
#> [1] "Reason: On CRAN"
reason(skip_on_ci())
#> [1] "not skipped"
# skip_on_cran() looks at NOT_CRAN, which devtools and testthat set to "true".
Sys.getenv("NOT_CRAN", "<unset>")
#> [1] "<unset>"
Sys.setenv(NOT_CRAN = "false")
reason(skip_on_cran())
#> [1] "Reason: On CRAN"
Sys.setenv(NOT_CRAN = "true")
reason(skip_on_cran())
#> [1] "not skipped"
# local_on_cran() pretends to be on CRAN until the current scope ends.
local({
local_on_cran(TRUE)
reason(skip_on_cran())
})
#> [1] "Reason: On CRAN"
reason(skip_on_cran())
#> [1] "not skipped"
# skip_on_ci() looks at CI.
Sys.setenv(CI = "true")
reason(skip_on_ci())
#> [1] "Reason: On CI"
Notice what each line says. skip_if_not(FALSE) has no message of its own, and the reason is the generated FALSE is not TRUE. skip_if_not_installed() writes the package name in braces and, for a version requirement, both versions. skip_unless_r() prints the running version of R next to the requirement. The three helpers whose answer depends on the environment (skip_on_cran(), skip_on_ci(), skip_on_os()) give different answers on different machines, which is the point, but also the reason to be careful. In a plain Rscript session NOT_CRAN is not set, so skip_on_cran() already reports On CRAN, as it does on the check farm; the variable NOT_CRAN=true is what turns it off, and devtools::test() and testthat::test_local() set it when it is not set already (the table in Running tests shows which test function does what). Setting NOT_CRAN to "false" skips, "true" does not, and local_on_cran(TRUE) pretends to be on CRAN until the end of its scope, which is how you rehearse a CRAN run from an interactive session.
What a skip looks like in a report
The reporters count the skips and, since testthat 3.0.0, list them by reason at the end, so that a suite which silently skips half of its tests is visible. The scratch file below has four tests that skip for four reasons, one that has no expectation at all (which test_that() reports as a skip with the reason empty test), and one that runs:
# A scratch test file that skips for several reasons.
writeLines(
trimws(r"---(
test_that("a test that needs a service", {
skip("the catalogue server is down")
expect_true(FALSE)
})
test_that("a test that needs a suggested package", {
skip_if_not_installed("nosuchpackage")
expect_true(FALSE)
})
test_that("a test for newer versions of R only", {
skip_unless_r(">= 99.0.0")
expect_true(FALSE)
})
test_that("a test that runs everywhere", {
skip_on_os("windows")
expect_true(TRUE)
})
test_that("a test that has not been written yet", {
x <- 1
})
)---", whitespace = "\n"),
"tests/testthat/test-skips.R"
)
devtools::test(filter = "skips")
unlink("tests/testthat/test-skips.R")
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
✔ | F W S OK | Context
✔ | 4 1 | skips
══ Results ═════════════════════════════════════════════════════════════════════
── Skipped tests (4) ───────────────────────────────────────────────────────────
• {nosuchpackage} is not installed (1): 'test-skips.R:7:3'
• Current R version (4.6.1) does not satisfy requirement (>= 99.0.0) (1):
'test-skips.R:12:3'
• empty test (1): 'test-skips.R:21:1'
• the catalogue server is down (1): 'test-skips.R:2:3'
[ FAIL 0 | WARN 0 | SKIP 4 | PASS 1 ]
The summary after the results has one line for each reason, with the number of tests and the places in the file; the progress line for the file has 4 in the skip column and 1 in the passes. The skipped test with the reason empty test is the one that has no expectations.
shelf has two tests that skip on CRAN: the long check at the end of test-loans.R, which calls skip_on_cran(), and the snapshot test of test-book.R, which skips itself because snapshot expectations return without checking when NOT_CRAN is not true (see Snapshots and CRAN). The effect on a real check is shown in Tests under R CMD check: [ FAIL 0 | WARN 0 | SKIP 2 | PASS 49 ] without NOT_CRAN and [ FAIL 0 | WARN 0 | SKIP 0 | PASS 52 ] with it.
Writing skip helpers, and when not to skip
A skip condition that several tests share belongs in a helper whose name starts with skip_, in a helper-*.R file. The fragment below is an example and not a script, and shelf has no such file:
skip_if_no_token <- function() {
skip_if_not(file.exists("secure-token.json"), "no token")
}
Another technique from the same article is to put a skip() inside a package function that cannot work without some tool, guarded by testthat::is_testing() (or the environment variable TESTTHAT, set to true while tests run), so that the function raises its normal error for users and a skip for the tests; the authors say they have mixed feelings about it, because it mixes code and tests.
Skipping is a decision with a cost. A skipped test is not a tested feature, and the usual way to fool yourself is to skip so much on CRAN or CI that the part of the suite that does run proves little. The advice in the article is to keep an eye on the skip summary. Three more points follow from this page. Do not skip because a suggested package is not installed: since testthat 3.3.0, test_that() does that by itself on CRAN, and the article advises assuming suggested packages are present except where they are hard to install. Do not use skip_on_os() to avoid a portability bug; fix the code. And if a test is slow, think about a smaller input before skip_on_cran(): the shelf test that is skipped on CRAN multiplies the loans fifty times to make the input large, and a smaller multiple would test the same thing.
The third edition: Config/testthat/edition and local_edition()
testthat has a very large number of reverse dependencies, and changing the behaviour of expect_equal() or of expect_warning() would turn passing test suites on CRAN into failing ones. testthat 3.0.0 (2020-10-31) therefore introduced the idea of an edition: a bundle of behaviours that a package must choose explicitly. A package that does not choose keeps the second edition, the behaviour of testthat 2.x; a package that chooses the third edition gets the current recommendations. The authors say that a new edition will be rare and will come with a new major version, so a fourth edition would arrive with testthat 4.0.0, and they explain why a new edition is not a new major version number: CRAN checks every package that uses testthat with the latest testthat, so incrementing the major version alone would not protect anyone from failures.
The edition is chosen in three places, from the widest to the narrowest.
-
In
DESCRIPTION, for every test of the package:Config/testthat/edition: 3.usethis::use_testthat(3)writes this field, together with the version requirementtestthat (>= 3.0.0)inSuggests, as the listing of theshelfpackage earlier on this page shows. -
In a test file or a test, with
local_edition(3)(orlocal_edition(2)to go back for one test while upgrading a suite). The change lasts until the current scope ends.edition_get()returns the edition in force. -
With the environment variable
TESTTHAT_EDITION, which testthat 3.0.3 added: handy for scripts that run outside a package, because outside a package there is noDESCRIPTIONto read, and the answer is then the second edition. This is why the scripts of this page that call third-edition behaviour start withSys.setenv(TESTTHAT_EDITION = "3"), and whyedition_get()in the script below prints2.
The third edition makes three kinds of change, and the scripts show each of them side by side with the second edition.
-
Equality is compared by waldo.
expect_equal()andexpect_identical()callwaldo::compare()instead ofall.equal()andidentical(). The difference is in what a failure tells you, as the first part of the script shows with two factors that have different levels. The second edition reportsAttributes: < Component “levels”: Lengths (2, 3) differ (string compare on first 2) >, which is the message ofall.equal(); the third reports the two level vectors one under the other with the nameslevels(actual)andlevels(expected). Besides the display, three behaviours differ: a single definition of tolerance everywhere (the old code mixedall.equal()and testthat’s owncompare()), the environments of functions and formulas are compared unlessignore_function_envorignore_formula_envis set (this matters when testing fitted models), and a time zone that is set implicitly differs from one that is set explicitly. -
Warnings and messages are captured one at a time and nothing is swallowed. In the second edition
expect_warning()swallowed every warning in the code, matching or not. In the third it captures at most one warning, the first that matchesregexpandclass, and every other warning bubbles up, so the second warning ofwarn_twice()is reported in the test (with the backtrace in the output). The same holds forexpect_error(),expect_message()andexpect_condition(), which use the same logic. Messages are no longer silenced automatically: the textreading the catalogueprinted bychatty()appears in the output of the third-edition test and not in that of the second. Silence them withsuppressMessages(), or test them withexpect_message(). -
Outdated functions are deprecated. They still run, but each one warns, so the warnings tell you what to change; they do not make
R CMD checkfail.
library(testthat)
set_reporter(StopReporter$new(praise = FALSE))
# Outside a package there is no DESCRIPTION, so the 2nd edition is active.
edition_get()
#> [1] 2
# Equality: all.equal() in the 2nd edition, waldo in the 3rd.
a <- factor(c("fiction", "poetry"))
b <- factor(c("fiction", "poetry"), levels = c("fiction", "poetry", "classic"))
compare_in <- \(edition) {
tryCatch(
test_that(paste("factors in edition", edition), {
local_edition(edition)
expect_equal(a, b)
}),
error = \(e) invisible(NULL)
)
}
compare_in(2)
#> ── Failure: factors in edition 2 ───────────────────────────────────────────────
#> Expected `a` to equal `b`.
#> Differences:
#> Attributes: < Component “levels”: Lengths (2, 3) differ (string compare on first 2) >
compare_in(3)
#> ── Failure: factors in edition 3 ───────────────────────────────────────────────
#> Expected `a` to equal `b`.
#> Differences:
#> `levels(actual)`: "fiction" "poetry"
#> `levels(expected)`: "fiction" "poetry" "classic"
#>
# Warnings: the 2nd edition swallows every warning, the 3rd captures one.
warn_twice <- \() {
warning("first")
warning("second")
}
test_that("warnings in edition 2", {
local_edition(2)
expect_warning(warn_twice(), "first")
})
test_that("warnings in edition 3", {
local_edition(3)
expect_warning(warn_twice(), "first")
})
#> ── Warning: warnings in edition 3 ──────────────────────────────────────────────
#> second
#> Backtrace:
#> ▆
#> 1. ├─testthat::expect_warning(warn_twice(), "first")
#> 2. │ └─testthat:::expect_condition_matching_(...)
#> 3. │ └─testthat:::quasi_capture(...)
#> 4. │ ├─testthat (local) .capture(...)
#> 5. │ │ └─base::withCallingHandlers(...)
#> 6. │ └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
#> 7. └─global warn_twice()
# Messages: silenced in the 2nd edition, shown in the 3rd.
chatty <- \() {
message("reading the catalogue")
16L
}
test_that("messages in edition 2", {
local_edition(2)
expect_equal(chatty(), 16L)
})
test_that("messages in edition 3", {
local_edition(3)
expect_equal(chatty(), 16L)
})
#> reading the catalogue
The script ran with the second edition in force, because outside a package nothing selects the third. Each comparison is made inside a test_that() block, with local_edition(edition) as its first line. The failure of the second-edition factor test, the same failure in the third edition, the warning second that escapes the third-edition expect_warning() (with its backtrace, which shows that it was raised by the call warn_twice() in the test), and the message reading the catalogue that appears only once are all the real output of the script. The order of the lines in a captured log can differ from the order in which the text was produced, because messages and warnings go to the error stream and results to the standard output.
The deprecated functions
The functions that the third edition deprecates are the ones that its own article lists. Run in the third edition, each one warns with the replacement to use, and the script below catches the warning to print its text:
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
source("scenario.R")
# In the 3rd edition the old helpers still run, but each one warns.
deprecation <- \(code) {
tryCatch({
code
"no warning"
}, warning = \(w) conditionMessage(w))
}
b <- book("Salt and Static", "Mara Quill", year = 2018)
deprecation(expect_is(b, "book"))
#> [1] "`expect_is()` was deprecated in the 3rd edition.\nℹ Use `expect_type()`, `expect_s3_class()`, or `expect_s4_class()` instead"
deprecation(expect_equivalent(c(a = 1), 1))
#> [1] "`expect_equivalent()` was deprecated in the 3rd edition.\nℹ Use expect_equal(ignore_attr = TRUE)"
deprecation(context("book"))
#> [1] "`context()` was deprecated in the 3rd edition."
deprecation(setup(NULL))
#> [1] "`setup()` was deprecated in the 3rd edition.\nℹ Please use test fixtures instead see vignette('test-fixtures') for details"
deprecation(teardown(NULL))
#> [1] "`teardown()` was deprecated in the 3rd edition.\nℹ Please use test fixtures instead see vignette('test-fixtures') for details"
deprecation(expect_that(1, equals(1)))
#> [1] "`expect_that()` was deprecated in the 3rd edition."
deprecation(expect_known_value(1, tempfile()))
#> [1] "`expect_known_value()` was deprecated in the 3rd edition.\nℹ Please use `expect_snapshot_value()` instead"
# with_mock() and local_mock() were removed in testthat 3.3.0.
tryCatch(with_mock(sqrt = \(x) 0, sqrt(4)), error = \(e) conditionMessage(e))
#> [1] "`with_mock()` was deprecated in testthat 3.2.0 and is now defunct.\nℹ Please use `with_mocked_bindings()` instead."
The replacements, in the order of the output, are: expect_type(), expect_s3_class() or expect_s4_class() for expect_is() (it forces you to say which object system you mean); expect_equal(ignore_attr = TRUE) for expect_equivalent(); nothing for context(), because the file name already serves as the context; test fixtures instead of setup() and teardown() (see Fixtures and withr); plain expectations instead of the very old expect_that(x, equals(y)) syntax; and snapshots (expect_snapshot_value() and its relatives) for expect_known_value(), expect_known_output(), expect_known_hash() and expect_equal_to_reference(). The last line of the output is not a warning but an error: with_mock() and local_mock(), deprecated for a long time, are defunct since testthat 3.3.0, and the message points to with_mocked_bindings() (see Mocking). Some other changes of the third edition are described elsewhere on this page: the condition expectations return the condition, not the value of the code (testthat 3.1.0, a change that applies to the third edition only), expect_snapshot() requires the third edition, and parallel testing requires it as well.
Reproducible output
test_that() and it() call local_test_context() at the start of each test. It sets the environment variable TESTTHAT to true (in both editions) and, in the third edition only, calls local_reproducible_output(), which makes printed output the same on every machine: it turns off colour and cli Unicode bullets, sets the console width to 80, the collation to C, the language to C (to disable translations; this was "en" before testthat 3.2.2), max.print to its default, and so on. These are the arguments of the function and their defaults: width = 80, crayon = FALSE, unicode = FALSE, rstudio = FALSE, hyperlinks = FALSE, lang = "C". A test that needs something else asks for it with local_reproducible_output(width = 40). The script prints the values outside a test, inside, and inside a test that changes the width:
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
set_reporter(StopReporter$new(praise = FALSE))
# Outside a test.
c(Sys.getenv("TESTTHAT", "<unset>"), getOption("width"), is_testing())
#> [1] "<unset>" "80" "FALSE"
# Inside test_that() testthat makes the output reproducible.
test_that("tests run in a controlled context", {
cat("TESTTHAT:", Sys.getenv("TESTTHAT"), "\n")
cat("width:", getOption("width"), "\n")
cat("unicode:", getOption("cli.unicode"), "\n")
cat("collation:", Sys.getlocale("LC_COLLATE"), "\n")
expect_true(is_testing())
expect_false(is_checking())
expect_false(is_snapshot())
})
#> TESTTHAT: true
#> width: 80
#> unicode: FALSE
#> collation: C
# local_reproducible_output() lets one test ask for something else.
test_that("a test can change the console width", {
local_reproducible_output(width = 40L)
cat("width:", getOption("width"), "\n")
expect_identical(getOption("width"), 40L)
})
#> width: 40
Outside a test the width is 80, R’s default; Sys.getenv("TESTTHAT") is <unset> and is_testing() is FALSE. Inside the test TESTTHAT is true, cli.unicode is FALSE and the collation is C, whatever the machine’s locale is. The three predicates is_testing(), is_checking() and is_snapshot() (testthat 3.1.9 for the last two) tell code whether it runs in tests, in a test run by R CMD check, or inside a snapshot. is_parallel() tells whether the files are run in worker processes. They exist for the few cases where a package function really has to behave differently under test.
Upgrading a suite from the second edition
The upgrade has four steps, in the order of the third-edition article. Activate the edition (usethis::use_testthat(3) does it). Replace or remove the deprecated functions. If the output became noisy, capture or suppress the warnings and messages that were silently swallowed before. Then inspect the tests that fail because the objects are no longer "all equal"; use local_edition(2) in a test that needs more time, and remove it afterwards. Tests that call expect_equal() on models, formulas or functions are the ones most likely to need a change.
Running tests
The same test files can be run in several ways, and the ways differ in what they load, which reporter they use, whether they assume the tests are not on CRAN and whether a failure is an error. The package source is loaded with pkgload::load_all() (pkgload 1.5.3), the function behind devtools::load_all(), in the development workflows, and the installed package is used in the checking workflows.
| Function | Purpose | Loads the package | Defaults |
|---|---|---|---|
|
The everyday command in a package directory. Saves open files, runs |
from source, with all functions visible ( |
progress reporter; a failing test does not raise an error ( |
|
Runs the test file that is open in the editor (or the test file that belongs to the open source file) |
from source |
compact reporter for a single file |
|
|
from source ( |
progress reporter; a failure is an error |
|
The low-level function behind the others. Runs every |
|
progress reporter, or the parallel one; a failure is an error |
|
One file, or one test of it |
as |
compact reporter |
|
What |
the installed package, attached with |
check reporter; a failure is an error |
|
The tests of an installed package, if the tests were installed with it |
installed |
check reporter |
Three arguments are worth knowing. filter of test_dir() and its relatives is a regular expression matched against the file name after test- and .R have been removed: filter = "loans|clean" runs test-loans.R and test-clean.R, and a pattern that matches nothing is an error, No test files found.. desc of test_file() (testthat 3.2.0) runs one test of a file, selected by its description; since 3.3.0 it can be a character vector that names the path through nested describe() blocks. And shuffle = TRUE (testthat 3.3.0) randomly reorders the top-level expressions of each file, which is a way to find tests that depend on the order in which they run.
From the console: test_file(), test_dir() and test_local()
The script runs parts of the suite of shelf in five ways, with the check reporter for the first four so that the output is compact. It must be run from the package directory. Two things in it deserve attention. test_file() and test_dir() do not load the package by themselves, so the calls give package = "shelf" and load_package = "source". And test_dir() does not set NOT_CRAN, so in the third call the test that calls skip_on_cran() is skipped, whereas test_local() in the fourth call assumes a development session and runs it.
library(testthat)
# One file: load the package from source first (test_file() does not).
test_file(
"tests/testthat/test-clean.R",
package = "shelf", load_package = "source", reporter = "check"
)
# One test of a file (testthat 3.2.0 and later), found by its description.
test_file(
"tests/testthat/test-book.R",
desc = "validate_book() rejects bad values with a clear message",
package = "shelf", load_package = "source", reporter = "check"
)
# A directory, filtered by the part of the file name after "test-".
# test_dir() does not set NOT_CRAN, so the skip_on_cran() test is skipped.
test_dir(
"tests/testthat", filter = "loans|clean",
package = "shelf", load_package = "source", reporter = "check"
)
# test_local() is devtools::test() without devtools: it assumes not on CRAN.
test_local(".", filter = "loans|clean", reporter = "check")
# The value is a testthat_results list that converts to a data frame.
res <- test_local(".", filter = "book", reporter = "silent")
class(res)
head(as.data.frame(res)[, c("file", "nb", "failed", "skipped")], 3L)
shelf 0.1.0: loan period is 21 days
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 14 ]
shelf 0.1.0: loan period is 21 days
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 4 ]
shelf 0.1.0: loan period is 21 days
[ FAIL 0 | WARN 0 | SKIP 1 | PASS 27 ]
══ Skipped tests (1) ═══════════════════════════════════════════════════════════
• On CRAN (1): 'test-loans.R:40:3'
[ FAIL 0 | WARN 0 | SKIP 1 | PASS 27 ]
shelf 0.1.0: loan period is 21 days
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 28 ]
shelf 0.1.0: loan period is 21 days
[1] "testthat_results"
file nb failed skipped
1 test-book.R 7 0 FALSE
2 test-book.R 3 0 FALSE
3 test-book.R 4 0 FALSE
Read the totals in order. The first run, of test-clean.R, has 14 passing expectations, as the file has (PASS 14). The second runs the single test selected by its desc: PASS 4. The third runs the files whose names match loans|clean: 27 expectations pass and 1 test is skipped, with the report of the skip (On CRAN, with the location of the test in the file). The fourth, test_local(), runs the same files and nothing is skipped: PASS 28. The last lines show what the functions return: an object of class testthat_results, a list that as.data.frame() turns into one row per test with the file, the number of expectations (nb), whether any failed or was skipped, the warnings and the timings. The script prints only the file, nb, failed and skipped columns, because the timing columns differ from run to run.
Reporters
A reporter is the object that receives the results and decides what to print. Every function that runs several tests has a reporter argument that takes a name ("minimal") or an object (MinimalReporter$new()). The reference page of testthat lists these:
| Reporter | What it does |
|---|---|
|
The default of |
|
The default when the files run in parallel. |
|
The default of |
|
For |
|
A summary of failures for interactive use; one character per expectation, |
|
The reporter of a |
|
Machine-readable output: the Test Anything Protocol, jUnit XML (needs the xml2 package; read by CI dashboards such as Jenkins) and TeamCity messages. |
|
Collect the results without printing; do nothing visible; run several reporters at once; throw an error at the end if any test failed. |
|
Reports the time of each test and filters out the fast ones (default threshold 1 second); testthat 3.3.0. |
|
Call |
The default can be changed with the options testthat.default_reporter, testthat.default_parallel_reporter, testthat.default_compact_reporter and testthat.default_check_reporter. The script runs one small file with four reporters and prints a few lines of what each gives; the JUnit file is written to a temporary file and only the number of <testcase> elements is printed:
library(testthat)
run <- \(reporter) {
test_local(".", filter = "clean", reporter = reporter)
}
# The one-line-per-expectation dot reporter.
invisible(run("minimal"))
# Where each expectation is, handy to find a hanging test.
out <- capture.output(invisible(run("location")))
head(out[nzchar(out)], 6L)
# What R CMD check shows: the counts, plus details only for problems.
invisible(run("check"))
# Machine-readable output: TAP, and JUnit XML for CI servers.
out <- capture.output(invisible(run("tap")))
head(out, 4L)
xml <- tempfile(fileext = ".xml")
invisible(run(JunitReporter$new(file = xml)))
sum(grepl("<testcase ", readLines(xml), fixed = TRUE))
shelf 0.1.0: loan period is 21 days
..............
shelf 0.1.0: loan period is 21 days
[1] "Start test: clean_books() tidies the fixture catalogue"
[2] " 'test-clean.R:3:3' [success]"
[3] " 'test-clean.R:4:3' [success]"
[4] " 'test-clean.R:5:3' [success]"
[5] " 'test-clean.R:6:3' [success]"
[6] "End test: clean_books() tidies the fixture catalogue"
shelf 0.1.0: loan period is 21 days
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 14 ]
shelf 0.1.0: loan period is 21 days
[1] "1..14"
[2] "ok 1 clean_books() tidies the fixture catalogue"
[3] "ok 2 clean_books() tidies the fixture catalogue"
[4] "ok 3 clean_books() tidies the fixture catalogue"
shelf 0.1.0: loan period is 21 days
[1] 14
The minimal reporter printed 14 dots, one per expectation. The location reporter printed Start test: and End test: lines around the location of every expectation, as 'test-clean.R:3:3' [success]; the first six lines are shown. The check reporter printed only the totals, because nothing went wrong. The TAP reporter’s first lines are the plan 1..14, then one ok line per expectation with the description of the test. The JUnit file has 14 test cases. The line shelf 0.1.0: loan period is 21 days before each run is the startup message of the package, printed when load_all() attaches it.
Stopping early: set_max_fails()
A suite in which the same mistake breaks hundreds of expectations produces more output than anybody reads. The progress reporter stops a run after 10 failures; the limit is the option testthat.progress.max_fails or the environment variable TESTTHAT_MAX_FAILS, which set_max_fails() sets (for the current R process and any process it starts), and set_max_fails(Inf) removes it. The scratch file below sorts first, because its name starts with aaa, and fails twice:
# A scratch file that sorts before the others and fails twice.
writeLines(
trimws(r"---(
test_that("first failure", expect_identical(1L, 2L))
test_that("second failure", expect_identical("a", "b"))
)---", whitespace = "\n"),
"tests/testthat/test-aaa-failing.R"
)
# Stop after the first failing file instead of running everything.
testthat::set_max_fails(1L)
Sys.getenv("TESTTHAT_MAX_FAILS")
devtools::test()
# Inf lifts the limit again.
testthat::set_max_fails(Inf)
unlink("tests/testthat/test-aaa-failing.R")
[1] "1"
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
✔ | F W S OK | Context
✖ | 2 0 | aaa-failing
────────────────────────────────────────────────────────────────────────────────
Failure ('test-aaa-failing.R:1:1'): first failure
Expected 1L to be identical to 2L.
Differences:
`actual`: 1
`expected`: 2
Failure ('test-aaa-failing.R:2:1'): second failure
Expected "a" to be identical to "b".
Differences:
`actual`: "a"
`expected`: "b"
────────────────────────────────────────────────────────────────────────────────
Maximum number of failures exceeded; quitting.
ℹ Increase this number with (e.g.) `testthat::set_max_fails(Inf)`
The first line of the script’s output, "1", is the environment variable after set_max_fails(1L). Both failures of the scratch file are reported, because the progress reporter stops at the end of the file in which the limit is exceeded (since testthat 3.1.0), not in the middle of it, and the message Maximum number of failures exceeded; quitting. names the remedy. The remaining test files are not run, which is what the limit is for, and also why a failing run with a low limit says nothing about the tests that did not run.
Running in parallel
Test files can run in separate R processes. The conditions are the third edition and the line Config/testthat/parallel: true in DESCRIPTION. By default testthat uses getOption("Ncpus", 2) processes (the number of processes can be set with the environment variable TESTTHAT_CPUS, best put in .Renviron), and TESTTHAT_PARALLEL=false turns parallel running off temporarily, which is what to do for print debugging, because the output of print(), cat() and message() from the test files is lost when they run in a worker. A field Config/testthat/start-first lists glob patterns of test files to start first, so that the slowest ones begin early. Each worker loads testthat and the package, runs the setup files and then receives test files one at a time; state therefore persists between the files that one worker runs, which is the reason files must clean up after themselves (see Fixtures and withr). Packages with many fast test files gain little, because starting the workers and relaying the results costs time: the article gives the tests of testthat itself as an example, 10 seconds in serial and 8 in parallel. A test_dir() call on a single file runs in serial since testthat 3.3.2. Some reporters cannot work in parallel (the debug, JUnit and location reporters and the stop reporter), and testthat then runs the files serially.
# Switch parallel testing on in DESCRIPTION (and put it back at the end).
description <- readLines("DESCRIPTION")
writeLines(c(description, "Config/testthat/parallel: true"), "DESCRIPTION")
tail(readLines("DESCRIPTION"), 2L)
# Two worker processes; the "check" reporter prints only the totals.
Sys.setenv(TESTTHAT_CPUS = "2")
devtools::test(reporter = "check")
# TESTTHAT_PARALLEL=false runs the files in this process again.
Sys.setenv(TESTTHAT_PARALLEL = "false")
devtools::test(reporter = "check")
writeLines(description, "DESCRIPTION")
[1] "LazyData: true" "Config/testthat/parallel: true"
ℹ Testing shelf
Starting 2 test processes.
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 52 ]
ℹ Testing shelf
shelf 0.1.0: loan period is 21 days
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 52 ]
The first line is the end of DESCRIPTION with the new field, then the real message Starting 2 test processes. appears, and the totals are the same PASS 52 as before. The second run, with TESTTHAT_PARALLEL set to false, runs in this process and prints the package startup message that the workers do not. The script restores DESCRIPTION at the end.
Environment variables and options
The behaviour of the test runners is controlled by a handful of environment variables. All are read by testthat or by the check tools of R, and the table says where each is documented.
| Variable or option | Effect | Documented in |
|---|---|---|
|
|
testthat: |
|
|
testthat: |
|
|
testthat: Skipping tests |
|
Selects the edition without a |
testthat NEWS 3.0.3 |
|
|
testthat: Running tests in parallel |
|
Failures after which the progress reporter stops (default 10). |
testthat: |
|
|
testthat NEWS 3.3.0 |
|
Number of trailing lines of test output kept in the check log (default 13; 0 keeps all lines except the R preamble). |
R Internals: Tools |
|
Whether examples, tests and vignettes run with a temporary library holding only the declared packages (false by default, true for CRAN submission checks); whether a suggested package that is not available is an error (true by default, false for CRAN submission checks). |
R Internals: Tools |
Source: R Internals: Tools
Tests under R CMD check
Everything above ran the tests from a development session. The tests of a package are also run, without any of devtools, by R CMD check, which first builds the source package into a tarball and then checks it. The two commands for shelf are below; --no-manual is there because the image has no TeX installation, and the environment variable switches off the checks that need the network, since R CMD check --as-cran would otherwise contact CRAN while checking the package name (see Publishing to CRAN for that check):
R CMD build shelf
_R_CHECK_CRAN_INCOMING_REMOTE_=false R CMD check --as-cran --no-manual shelf_0.1.0.tar.gz
The log is long. Its first lines (version of R, platform, compiler, session charset and the time) are trimmed here, and so are the lines of the checks that are not about tests. Two steps matter for this page: checking for unstated dependencies in ‘tests’ (every package that the files under tests/ load with library() or :: must be declared in DESCRIPTION), and checking tests itself:
* checking for unstated dependencies in ‘tests’ ... OK
* checking tests ...
Running ‘base-check.R’
Comparing ‘base-check.Rout’ to ‘base-check.Rout.save’ ... OK
Running ‘testthat.R’
OK
* checking for unstated dependencies in vignettes ... OK
* DONE
Status: OK
Each file that is directly in tests/ is run in a separate R process, in a copy of the directory, with the installed package available; its output is written next to it as <name>.Rout. The test run is ERROR if a script stops with an error, and OK otherwise. Here two files ran. The first, base-check.R, is the plain-R test file of the section on base R below, which was added to the copy of the package for this run; its output was compared with a reference file. The second, testthat.R, is the driver of the suite. Its output is in shelf.Rcheck/tests/testthat.Rout, where the check directory keeps it, and the part that test_check() produced is:
sed -n '/^> test_check/,/^> proc.time/p' shelf.Rcheck/tests/testthat.Rout
> test_check("shelf")
[ FAIL 0 | WARN 0 | SKIP 2 | PASS 49 ]
══ Skipped tests (2) ═══════════════════════════════════════════════════════════
• On CRAN (2): 'test-book.R:39:1', 'test-loans.R:40:3'
[ FAIL 0 | WARN 0 | SKIP 2 | PASS 49 ]
>
The totals are not those of devtools::test(). [ FAIL 0 | WARN 0 | SKIP 2 | PASS 49 ] means that two tests were skipped on CRAN, and the report names them: test-book.R:39:1, the test that contains the two snapshot expectations, and test-loans.R:40:3, the one that calls skip_on_cran(). The NOT_CRAN environment variable is not set in a check run, and R CMD check --as-cran is what CRAN does, so testthat acts as it would there. The 49 are the 52 expectations of the development run minus the expectation in the skipped long test and the two snapshot expectations, which returned without checking anything (the other two expectations of the snapshot test did run). A maintainer who wants the whole suite to run in the local check sets the variable. Everything else in the log is the same:
NOT_CRAN=true _R_CHECK_CRAN_INCOMING_REMOTE_=false R CMD check --as-cran --no-manual shelf_0.1.0.tar.gz
> test_check("shelf")
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 52 ]
>
With NOT_CRAN=true the same suite reports SKIP 0 and PASS 52, both the snapshots and the long test run, and the snapshot file _snaps/book.md is compared with the output. This is the check to run before a release (and what the continuous-integration workflow of the section on coverage does), and the other, without the variable, is the check CRAN runs. Both must pass. A suite that passes only with NOT_CRAN=true has hidden a problem in a skipped test; one that passes only without it has a test that depends on something the check farm does not have.
Three more things happen in a check run, all visible in the documentation of R and of testthat.
-
The number of lines kept. The check log of a failing test shows only the last 13 lines of the output, which is the default of
_R_CHECK_TESTS_NLINES_(0 keeps everything but the R preamble). The check reporter of testthat was designed so that those 13 lines are useful. The full output of a failed run is intests/testthat.Rout.failin the check directory. The section on reading a failure shows the log of a failing check. -
Only declared packages. With
--as-cranthe variable_R_CHECK_SUGGESTS_ONLY_is true, so tests run with a library that contains only the packages declared inDESCRIPTION. A test that uses a package that is not listed inSuggestsfails in the check although it worked in the development session, where all your packages are installed (the first section of this page shows the real output of that case). -
The time limit and the CPU time. The CRAN Repository Policy asks that checking take as little CPU time as possible, and that long-running tests be made optional. Parallel testing must respect the limit of two cores that the policy sets for a package that uses several threads or cores; testthat’s default of
getOption("Ncpus", 2)workers keeps within it.
Source: CRAN Repository Policy
The tests of an installed package can also be run later with tools::testInstalledPackage(), the function that covr uses to run the tests under instrumentation, and R CMD check can take its tests from a non-standard directory with the option --test-dir: Writing R Extensions suggests inst/slowTests for slow tests, and names inst/testWithOracle and inst/randomTests as other examples.
Base R alternatives: tests/*.R, .Rout.save and stopifnot
testthat is a convention on top of a mechanism that R itself provides, and the mechanism works without it. Writing R Extensions describes it in one paragraph. The tests directory holds .R files; the results of running a file are written to a .Rout file; if there is a .Rout.save file with the same name, the two are compared, and differences are reported but do not cause an error. The directory is copied to the check area and the tests run with that copy as the working directory, with R_LIBS set so that library(shelf) finds the package installed for the check. The tests run in a vanilla R session and without a random-number seed, so tests that use random numbers must set one. An error in a script, such as a failed stopifnot(), makes the test run fail. That is the whole contract: a test is a script that stops with an error when something is wrong.
The script below writes such a file into the copy of the package, tests/base-check.R, and lists the tests directory. The file loads the package, asserts with stopifnot() that a known ISBN is valid, builds a book, prints it and its summary, and prints two numbers about the overdue loans. It is the style of test that many older packages, and the packages of R itself, use: a script that exercises the code, with the expected output stored next to it.
# A base-R test: a plain script in tests/, run by R CMD check.
writeLines(
trimws(r"---(
library(shelf)
stopifnot(is_valid_isbn13("978-0-306-40615-7"))
b <- book("Salt and Static", "Mara Quill", "978-0-01-400145-3", 2018,
"Mystery", 244)
print(b)
summary(b)
overdue <- overdue_loans(loans, today = as.Date("2026-06-30"))
nrow(overdue)
range(overdue$days_out)
)---", whitespace = "\n"),
"tests/base-check.R"
)
list.files("tests")
[1] "base-check.R" "testthat" "testthat.R"
The reference output is made by running the file the way R CMD check will, with R CMD BATCH --vanilla, and keeping the result under the name that the check looks for. This is the file that the command writes, without the start-up banner of R (which carries the version and the platform) and without the proc.time() call and the timings that R CMD BATCH appends:
cd shelf/tests
R CMD BATCH --vanilla base-check.R base-check.Rout.save
Source: R documentation: BATCH
> library(shelf)
shelf 0.1.0: loan period is 21 days
>
> stopifnot(is_valid_isbn13("978-0-306-40615-7"))
>
> b <- book("Salt and Static", "Mara Quill", "978-0-01-400145-3", 2018,
+ "Mystery", 244)
> print(b)
<book> Salt and Static (Mara Quill, 2018)
> summary(b)
Book: Salt and Static
ISBN known: TRUE
Pages: 244
>
> overdue <- overdue_loans(loans, today = as.Date("2026-06-30"))
> nrow(overdue)
[1] 32
> range(overdue$days_out)
[1] 22 84
Source: R documentation: Rdiff
When R CMD check compares base-check.Rout with base-check.Rout.save, it uses tools::Rdiff(), which removes the startup banner, the timings that R CMD BATCH appends and the lines about loading packages from both files, maps fancy quotes to plain ones and compares the rest line by line. Text between the markers > ## IGNORE_RDIFF_BEGIN and > ## IGNORE_RDIFF_END is ignored. Writing R Extensions adds the cautions: the comparison is verbatim, so anything that varies between runs (version numbers of other packages that are printed when they load, numbers printed with too many digits, timings, rounding errors that are really zero) makes the check report a difference, and the comparison is made in the end user’s locale, so reference output should be ASCII. The check for shelf with the matching file was OK in the log above. If the expected value in the reference file is edited from 32 to 31 (an experiment on the copy of the package), the check reports the difference and carries on; the status becomes a NOTE, not an error:
* checking tests ...
Running ‘base-check.R’
Comparing ‘base-check.Rout’ to ‘base-check.Rout.save’ ...
19c19
< [1] 32
---
> [1] 31
Running ‘testthat.R’
NOTE
Status: 1 NOTE
The lines 19c19, < [1] 32 and > [1] 31 are the output of diff: line 19 of the new output has 32, line 19 of the reference has 31. Reference files are fragile, and that is the weak point of the approach: every change in a print method means regenerating the file.
stopifnot(), all.equal() and the tools package
For assertions inside the script, base R has stopifnot() and all.equal(), and the tools package has assertError() and assertWarning(). The script below runs them against the shelf scenario:
source("scenario.R")
# stopifnot(): silent when everything is TRUE, an error otherwise.
stopifnot(is_valid_isbn13("978-0-306-40615-7"), nrow(books) == 16L)
caught <- \(code) tryCatch(code, error = \(e) conditionMessage(e))
caught(stopifnot(nrow(books) == 20L))
#> [1] "nrow(books) == 20L is not TRUE"
caught(stopifnot("every ISBN must be valid" = all(is_valid_isbn13(books$isbn))))
#> [1] "every ISBN must be valid"
caught(stopifnot(all.equal(1, 1.1)))
#> [1] "1 and 1.1 are not equal:\n Mean relative difference: 0.1"
# all.equal() describes the difference; identical() only says yes or no.
all.equal(0.1 + 0.2, 0.3)
#> [1] TRUE
all.equal(1, 1.1)
#> [1] "Mean relative difference: 0.1"
all.equal(c(a = 1), c(b = 1))
#> [1] "Names: 1 string mismatch"
identical(0.1 + 0.2, 0.3)
#> [1] FALSE
isTRUE(all.equal(100, 100.0000001))
#> [1] TRUE
# The tools package has assertions for conditions.
tools::assertError(book(""))
tools::assertWarning(as.integer("x"))
caught(tools::assertError(book("A title")))
#> [1] "Failed to get error in evaluating book(\"A title\")"
caught(tools::assertWarning(book("A title")))
#> [1] "Failed to get warning in evaluating book(\"A title\")"
Source: R documentation: stopifnot
-
stopifnot(...)is silent when every argument isTRUE; otherwise it stops with a message built from the failing expression:nrow(books) == 20L is not TRUE. Since R 4.0.0 an argument can be named, and the name is the message ("every ISBN must be valid" = all(...)), which makes the failure readable without a message in the code (R NEWS: Changes in R 4.0.0). Since R 3.5.0 the expressions can also be given in one braced block asexprs =.stopifnot(all.equal(1, 1.1))shows a pitfall thatstopifnot()softens:all.equal()returns eitherTRUEor a character string that describes the difference, so an expression such asif (all.equal(x, y))is an error waiting to happen.stopifnot()has recognised a call toall.equal()since R 3.4.0 and reports its message (1 and 1.1 are not equal:followed byMean relative difference: 0.1); anywhere else, writeisTRUE(all.equal(x, y)). -
all.equal()compares two objects with a tolerance (by defaultsqrt(.Machine$double.eps), about1.5e-8, relative to the size of the numbers) and says how they differ:Names: 1 string mismatch.identical()says only yes or no, and is exact:identical(0.1 + 0.2, 0.3)isFALSE. These are the base R counterparts ofexpect_equal()andexpect_identical(); in the third edition testthat uses waldo and not these two functions. -
tools::assertError(expr)andtools::assertWarning(expr)evaluateexprand signal an error if no error (respectively no warning) occurred, with the messageFailed to get error in evaluating .... Since R 4.0.0 they accept aclassesargument to require a specific condition class. They are the base R counterparts ofexpect_error()andexpect_warning().
The table puts the three approaches side by side:
| Question | Base R (tests/*.R) |
testthat | tinytest |
|---|---|---|---|
Assert something is true |
|
|
|
Equal within tolerance |
|
|
|
Exactly identical |
|
|
|
An error with a message |
|
|
|
Printed output |
|
|
|
Grouping and names |
none (a script) |
|
none (a script) |
On failure |
the script stops at the first error |
all expectations of all tests run, one report |
all expectations of a file run, one report |
Source: R documentation: all.equal
Base-R tests are enough for a small package, they have no dependency, and they are also what to reach for in a package that must not suggest testthat. Their weakness is the failure report: a script stops at the first error with a one-line message, with no summary and no way to run one test, and the reference output compares text and not values. testthat adds the structure, and tinytest takes a middle way.
tinytest in brief
tinytest 1.4.3 (an add-on package, installed in the scratch library of these examples) is a small testing framework built on the idea that a test file should be a plain script. In its words, a test file is an R script in which some or all statements express an expectation; there is no test_that() block. It imports only the base packages parallel and utils. Its expectation functions have the names of the testthat ones (expect_equal(), expect_error(), expect_true()) with different arguments: the regular expression of expect_error() is called pattern, and the equality uses all.equal(). A test file runs from top to bottom, every expectation is recorded and nothing stops at a failure; run_test_file() returns an object of class tinytests that all_pass() and summary() inspect. The script below writes a four-expectation test file to a temporary directory and runs it. (Because the examples of this page run in a package-less script, shelf:: calls stand in for the package that a real test file would load.)
# A tinytest file is a plain script of expect_*() calls: no test_that() blocks.
file <- file.path(tempdir(), "test_isbn.R")
writeLines(r"---(
expect_true(shelf::is_valid_isbn13("978-0-306-40615-7"))
expect_false(shelf::is_valid_isbn13(""))
expect_equal(shelf::normalise_isbn("978-0-306-40615-7"), "9780306406157")
expect_error(shelf::book(""), pattern = "a book needs a title")
)---", file)
out <- tinytest::run_test_file(file, verbose = 0L)
tinytest::all_pass(out)
#> [1] TRUE
summary(out)
#>
#> File Results fails passes
#> test_isbn.R 4 0 4
#> Total 4 0 4
tinytest::setup_tinytest(pkgdir) prepares a package: it creates inst/tinytest/ with an example test file, creates tests/tinytest.R so that R CMD check runs the tests, and adds tinytest to Suggests. The experiment below runs it on a copy of the shelf package in a temporary directory and prints what it wrote; the real tests/testthat.R and its suite are not touched:
# Let tinytest set up a COPY of the package (the real one stays untouched).
copy <- file.path(tempdir(), "shelf")
dir.create(copy)
files <- list.files(all.files = TRUE, no.. = TRUE)
invisible(file.copy(files, copy, recursive = TRUE))
tinytest::setup_tinytest(copy, verbose = FALSE)
cat(readLines(file.path(copy, "tests", "tinytest.R")), sep = "\n")
grep("tinytest", readLines(file.path(copy, "DESCRIPTION")), value = TRUE)
list.files(file.path(copy, "inst"), recursive = TRUE)
if (requireNamespace("tinytest", quietly = TRUE)) {
tinytest::test_package("shelf")
}
[1] " withr, tinytest"
[1] "CITATION" "tinytest/test_shelf.R"
The driver tests/tinytest.R is the three lines printed: it calls tinytest::test_package("shelf") when tinytest is installed. The test files live in inst/tinytest/ and so are installed with the package, which lets a user run tinytest::test_package("shelf") after installation; authors who prefer not to install them can put them in a directory under tests/ and name it in the testdir argument. The grep line shows tinytest added to Suggests (the line is the end of the field). Choose tinytest over testthat when a light dependency matters, when you want the test files to be installed with the package, or when the script style suits the team; the price is the smaller set of expectations, no snapshot tests, no mocking helpers and less tooling around it.
Test coverage with covr
Coverage answers a narrow question: which lines of the package did the tests run? It does not say whether the lines were tested well, only that they were reached, so it finds the code that no test touches and says nothing about the quality of the tests that do. covr (3.6.5, an add-on package) is the standard tool. It modifies the code of the package, adding a tracking call to each call of the source, installs the instrumented copy in a temporary library and runs the tests with it, using tools::testInstalledPackage(). The covr documentation explains why it cannot run inside R CMD check: the modification may change behaviour in edge cases, and for compiled code the package is compiled without optimisation.
covr::package_coverage() takes the path of the package (the working directory by default) and the type of code to run: "tests" (the default), "vignettes", "examples", "all" or "none". The script runs it on the shelf package and prints the result in the three forms that matter.
cov <- covr::package_coverage()
cov
covr::percent_coverage(cov)
# Lines that no test ran.
covr::zero_coverage(cov)
# Per file, as numbers.
round(covr::coverage_to_list(cov)$filecoverage, 2L)
shelf Coverage: 93.81%
R/zzz.R: 40.00%
R/loans.R: 96.43%
R/book.R: 100.00%
R/clean.R: 100.00%
R/isbn.R: 100.00%
[1] 93.80531
filename functions line value
49 R/loans.R shelf_today 37 0
1 R/zzz.R .onLoad 2 0
2 R/zzz.R .onLoad 3 0
3 R/zzz.R .onLoad 4 0
4 R/zzz.R .onLoad 5 0
5 R/zzz.R .onLoad 6 0
6 R/zzz.R .onLoad 8 0
R/book.R R/clean.R R/isbn.R R/loans.R R/zzz.R
100.00 100.00 100.00 96.43 40.00
The package coverage is 93.81 per cent. The summary lists the files from the least to the most covered: R/zzz.R 40 per cent, R/loans.R 96.43, and 100 for R/book.R, R/clean.R and R/isbn.R. percent_coverage() gives the number (93.80531), and zero_coverage() the lines that no test ran, as a data frame with the file, the function and the line. Two groups of lines are listed. In R/loans.R, line 37 of shelf_today: its body is the one call that reads the clock, and no test runs it, because the tests either pass today explicitly or replace the function with a mock (see Mocking). In R/zzz.R, lines 2 to 6 and line 8 of .onLoad, the package hook that sets the default of the shelf.loan_days option; covr reports that the hook was not run although it runs whenever the package is loaded (the startup message shelf 0.1.0: loan period is 21 days that the hook .onAttach prints is built from the option that .onLoad sets, and it appears in the test log); the cause of the zero was not investigated here, and the number is shown as measured. The last form, coverage_to_list(), gives the same numbers per file for use in code.
The other output formats of covr are for other tools. covr::report() writes an HTML page with the source annotated line by line and opens it, which is the most useful way to work through the uncovered lines. It needs the packages DT and htmltools; DT is not installed in the image used here, and the call fails with the real message below.
# covr::report() writes an HTML report and needs the DT and htmltools packages.
cov <- covr::package_coverage()
tryCatch(covr::report(cov), error = \(e) conditionMessage(e))
[1] "The `DT` and `htmltools` packages must be installed to use `covr::report()`"
covr::codecov() runs the coverage and uploads it to the Codecov service (named here, not run: it needs an upload token, read from the environment variable CODECOV_TOKEN or from a codecov.yml file in the package), and covr has similar functions for Coveralls, GitLab and Azure, and to_cobertura() and to_sonarqube() that write the XML formats other tools read. Lines that should not count, such as defensive branches that cannot be reached, are excluded in the source with the comment # nocov (one line) or a # nocov start and # nocov end pair, by the line_exclusions and function_exclusions arguments of package_coverage(), or with a .covrignore file; the patterns are the options covr.exclude_pattern, covr.exclude_start and covr.exclude_end. A test that should not run under coverage (a slow one, a test of timing) calls skip_on_covr().
A continuous-integration workflow
Coverage and the checks of the previous sections are usually run on every push by a continuous-integration service. The usethis function use_github_action() copies the maintained workflows of the r-lib/actions repository into .github/workflows/ (use_github_action("check-standard"), use_github_action("test-coverage")), and covr’s own documentation recommends it for setting up coverage. The fragment below shows the shape of such a workflow, with the actions pinned to @v2 and a placeholder package; it is a fragment and was not run, because it needs a hosted runner:
name: R-CMD-check
on:
push:
branches: [main]
pull_request:
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: r-lib/actions/setup-r@v2
with:
use-public-rspm: true
- uses: r-lib/actions/setup-r-dependencies@v2
with:
extra-packages: any::rcmdcheck
needs: check
- uses: r-lib/actions/check-r-package@v2
coverage:
runs-on: ubuntu-latest
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
steps:
- uses: actions/checkout@v4
- uses: r-lib/actions/setup-r@v2
with:
use-public-rspm: true
- uses: r-lib/actions/setup-r-dependencies@v2
with:
extra-packages: any::covr
needs: coverage
- name: Test coverage
run: covr::codecov(quiet = FALSE)
shell: Rscript {0}
Two things in a workflow like this change what the tests see. Continuous-integration services set the environment variable CI (the documentation of skip_on_ci() names GitHub Actions among them), so skip_on_ci() tests skip, and a snapshot that has no stored file is a failure and not a warning (see Snapshots on CI), which is why the snapshot files must be committed. And whether the skip_on_cran() tests run in the R CMD check step depends on whether NOT_CRAN is set in the environment of the job, which the sections above showed to make the difference between SKIP 2 and SKIP 0; read the skip summary in the log, and set NOT_CRAN: true in the env of the job when you want those tests to run. The workflow files themselves were not run here.
Writing code that is easy to test
A test is easy to write when the function under test takes what it needs as arguments and returns what it produces. It is hard to write when the function reads a file, asks for the current date, prints and returns nothing, all in one place. Most techniques of this page (fixtures, mocks, snapshots, skips) exist to cope with code of the second kind, and writing tests is a good way to push code towards the first kind. The shelf package follows this: overdue_loans() takes today, loan_summary() takes today and loan_days, clean_books() and clean_loans() take a data frame and return one, and the code that reads the clock is a one-line internal function.
The script shows the same report written both ways. The first version, report_hard(), reads a file and the clock and prints. The second is three functions: a pure function that counts the loans still out and the late ones from a data frame and a date, a function that formats the counts, and a thin shell that reads the file and prints. The tests then need no file and no clock for the first two, and use the fixed shelf_today of the scenario for the third:
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
set_reporter(StopReporter$new(praise = FALSE))
source("scenario.R")
# The one place that reads the clock.
current_date <- Sys.Date
# Hard to test: it reads a file, asks the clock and prints, all in one place.
report_hard <- function(path = "loans.csv") {
loans <- read.csv(path, colClasses = "character")
loans$loan_date <- as.Date(loans$loan_date, format = "%Y-%m-%d")
out <- current_date() - loans$loan_date[is.na(loans$return_date)]
cat("Still out:", length(out), "\n")
}
# Easier: a pure function that takes the date and the data as arguments...
count_out <- function(loans, today) {
still_out <- loans[is.na(loans$return_date), ]
c(
out = nrow(still_out),
late = sum(today - still_out$loan_date > 21L, na.rm = TRUE)
)
}
# ...a function that only formats...
report_line <- function(counts) {
sprintf("Still out: %d (%d late)", counts[["out"]], counts[["late"]])
}
# ...and a thin shell that does the input and the output.
report <- function(path = "loans.csv", today = current_date()) {
loans <- read.csv(path, colClasses = "character")
loans$loan_date <- as.Date(loans$loan_date, format = "%Y-%m-%d")
loans$return_date <- as.Date(loans$return_date, format = "%Y-%m-%d")
cat(report_line(count_out(loans, today)), "\n")
}
test_that("count_out() depends only on its arguments", {
counts <- count_out(
data.frame(
loan_date = as.Date(c("2026-06-01", "2026-06-25", "2026-05-01")),
return_date = as.Date(c(NA, NA, "2026-05-10"))
),
today = shelf_today
)
expect_identical(counts, c(out = 2L, late = 1L))
})
test_that("report_line() formats the counts", {
expect_identical(
report_line(c(out = 7L, late = 5L)),
"Still out: 7 (5 late)"
)
})
test_that("report() works end to end with a fixed date", {
expect_output(report(today = shelf_today), "Still out: 7 \\(\\d+ late\\)")
})
report(today = shelf_today)
#> Still out: 7 (7 late)
Notice what each test needs. count_out() is tested with a data frame of three rows written in the test, so the expected values (out = 2L, late = 1L) can be checked by reading the test. report_line() is a string function. Only the third test touches the file system and uses expect_output() with a regular expression, because the thin shell has little logic left. The last line prints the same report, Still out: 7 (7 late), for the fixed date.
The habits that follow from this, in the order a reader would meet them:
-
Pass the date, the random seed, the paths and the options as arguments, with defaults that read the real thing (
today = shelf_today()in the package), as the test-fixtures article suggests when it says that making external state an explicit argument is often worthwhile because it shows which inputs determine the outputs. -
Return values and let the caller print. A function that returns a data frame can be tested with
expect_equal()andexpect_named(); one that only prints needsexpect_output()or a snapshot. -
Keep one idea in a function. The tests in the
shelfsuite are short becausesquish(),normalise_isbn()andis_valid_isbn13()are separate functions that can be tested alone. -
Test the edges, not only the middle.
is_valid_isbn13()has a specification for a wrong check digit, a short string, an empty string,NAand a vector of length zero; the loan tests have an impossible date and an early return. The data of the scenario already contain the awkward cases on purpose. -
When you find a bug, write the test that fails first, then fix it. The test stays as a record. The failure reported in the section on reading a failure (
clean_books()of a partial catalogue) is exactly that kind of finding. -
Test behaviour, not implementation. A test that fixes the exact text of an error message or the exact layout of a print method breaks when those change for good reasons; test the
classof the condition where there is one, and use a snapshot, whose diff a human reads, for text that is meant to be read. -
Do not test R, or other people’s packages.
expect_equal(sum(1:3), 6L)tests nothing in your package. -
Let the file layout help. Name the tests of
R/loans.Rtests/testthat/test-loans.R;usethis::use_test()(whose template is shown with the layout of thetestsdirectory) creates the file with that name for the active source file, anddevtools::test_active_file()runs the test file that goes with the open source file.
The layout of a test suite is also documentation. A new contributor reads the descriptions of the tests as a list of what the code is supposed to do, which is a reason to write them as the sentence "test that …". The failure reports then read as sentences too: Failure ('test-failing.R:3:3'): overdue_loans() finds 31 late loans.
Common errors and warnings
The messages below were captured while preparing this page; the script that produced the first group is shown, and the others come from the experiments above. It prints the class of the first condition that each statement signals and its message. A mistake in the call of an expectation is usually an error of class rlang_error, with the argument named in backticks, and a mistake in the test code is a failure of the test.
library(testthat)
Sys.setenv(TESTTHAT_EDITION = "3")
source("scenario.R")
# Print the class and the message of the first condition the code signals.
show <- \(code) {
text <- tryCatch(
{
code
"no condition"
},
condition = \(cnd) {
paste0("<", class(cnd)[[1L]], "> ", conditionMessage(cnd))
}
)
cat(text, "\n", sep = "")
}
# A misspelt argument is a warning, not an error.
show(expect_equal(1, 1, tollerance = 1))
#> <rlang_warning> Unused arguments (tollerance = 1)
# An extra argument of a condition expectation reaches grepl().
show(expect_error(stop("x"), "x", foo = 1))
#> <rlang_error> Failed to compare error to `regexp`.
#> Caused by error in `grepl()`:
#> ! unused argument (foo = 1)
# A forgotten argument.
show(expect_equal(1))
#> <rlang_error> argument `expected` is missing, with no default.
# The description of a test is a single string.
show(test_that(1, expect_true(TRUE)))
#> <rlang_error> `description` must be a single string, not the number 1.
# Arguments are checked: a length must be a whole number.
show(expect_length(1:3, "3"))
#> <rlang_error> `n` must be a whole number, not the string "3".
show(expect_match(1, "a"))
#> <rlang_error> `object` must be a character vector, not the number 1.
# Only editions 2 and 3 exist.
show(local_edition(5))
#> <rlang_error> `x` must be a whole number between 2 and 3, not the number 5.
# A test file that does not exist.
show(test_file("no-such-file.R"))
#> <rlang_error> `path` does not exist.
# A directory without test files.
empty <- file.path(tempdir(), "empty")
dir.create(empty)
show(test_dir(empty))
#> <rlang_error> No test files found.
# Mocking needs to know which package to change.
show(local_mocked_bindings(x = 1))
#> <rlang_error> No packages loaded with pkgload
# Snapshots exist only in the 3rd edition.
show(local({
local_edition(2)
expect_snapshot(1)
}))
#> <rlang_error> `expect_snapshot()` requires the 3rd edition.
Reading them in the order of the output: a misspelt argument of expect_equal() is only a warning (Unused arguments (tollerance = 1)), so the expectation runs with the default tolerance and the typo goes unnoticed in a log full of output; an extra argument of a condition expectation goes to grepl(), which is why the error says that it failed to compare the error to regexp; the message "argument expected is missing, with no default." is the plain R error for an omitted argument; the argument checks of testthat 3.3.0 name the argument and the type that was expected (for expect_length(1:3, "3") the message says that n must be a whole number, not the string "3"); local_mocked_bindings() called with no package loaded by pkgload says so, and a snapshot expectation in the second edition says which edition it needs.
The table adds the messages of the other sections, with the section that explains them.
| Message | Where it comes from | What to do |
|---|---|---|
|
A script that uses testthat without attaching it |
|
"there is no package called 'testthat'" in the first line of |
testthat is not in |
Add |
|
|
Call it as |
|
|
Check the spelling; for a base function, add a placeholder binding or a wrapper |
|
The first run of an |
Read the recorded text and commit |
|
The same, on a continuous-integration server, where |
Commit the snapshot file that you generated locally |
|
The output of snapshotted code differs from the stored one |
Fix the code, or |
|
|
The code did not fail; check the call |
|
|
As above, in a base-R test |
|
A state inspector found a test that leaked an option or a variable (see Fixtures and withr) |
Use |
|
The progress reporter reached |
Fix the first failures, or |
|
|
Read the report above it; the full log is |
|
Old functions in a third-edition suite |
Use the function that the message names |
|
An expression evaluated as the argument of an expectation warned (see Reading a failure) |
|
Two of the messages are worth remembering because they are misleading at first sight. Expected ... to throw a error. has the article "a" in front of the word error, as testthat writes it, and the other part of the sentence is the name of the expression that did not fail. And the warning NAs introduced by coercion in a test report is not about the expectation at all: it is the warning of the code that was passed to the expectation, reported against the test that evaluated it, with a backtrace that starts at the expectation.
Pitfalls
The pitfalls below are collected from what the experiments of this page showed. Each is followed by the cure.
-
Testing with
expect_true(a == b). When it fails the report says thata == bis notTRUEand nothing else.expect_equal(a, b)andexpect_identical(a, b)show both values and the difference. The rule for choosing is that values built by floating-point arithmetic go toexpect_equal()and exact things (strings, integers, names, classes) go toexpect_identical(). R FAQ 7.31 explains why0.1 + 0.2is not0.3to the last bit;1Land1differ forexpect_identical()and not forexpect_equal(). -
A regular expression where a literal was meant.
expect_error(f(), "is.character(title) is not TRUE")treats the parentheses and the dots as regular-expression syntax; addfixed = TRUE(as theshelftest does) or match onclass =. Matching on the text of a message also makes the test fragile: the message of an error raised by R or by another package can change between versions and is translated in other languages, and Writing R Extensions advises against testing it. -
Expecting
expect_error()to report that the pattern differed. In the third edition an error that does not match escapesexpect_error()and the test reports the original error, with its backtrace (the third failure of the failing-file experiment). -
A test that depends on the clock, the locale, the time zone, the working directory, the network or a random number. These differ between your laptop and the check machine, which is why the test passes in one and fails in the other. Pass the value in (
today = shelf_fixed_today), replace the source with a mock (shelf_today()), fix the state withwithr::local_envvar(),withr::local_timezone()orwithr::with_seed(), and let the third edition make the output reproducible. Tests that use random numbers need a seed in any case, because R CMD check runs them in a vanilla session without one. -
State that leaks from one test into the next. An option set with
options(), an environment variable, a changed working directory or a file written to the package directory makes the next test, or the next run, depend on this one. Usewithr::local_*()andwithr::defer(), write only totempdir()(CRAN Repository Policy: a package should not write in the user’s home filespace, nor anywhere else on the file system apart from the session’s temporary directory, and such usage should be cleaned up), and useset_state_inspector()to find the culprit. -
Tests that Writing R Extensions lists as bad practice. It names tests of the time a command takes (you cannot know how fast or how loaded the platform is; at best a ratio of times, and even that is not advisable), tests of the exact format of R messages (they change and are translated), tests for the absence of warnings (later versions of R or of other packages can add warnings), comparisons of file paths without canonicalising them (
normalizePath()), and numerical tests with no tolerance (only test accuracy after a formal error analysis, and always with a tolerance). -
Test files that depend on each other. They run in alphabetical order in a serial run, and in an order that depends on which worker is free in a parallel one. Run them with
shuffle = TRUE(random order of the top-level expressions in a file) and in parallel now and then to find out. -
Reading a green check as proof when tests were skipped.
SKIP 2in a check log is two tests that did not run. Look at the skip summary, run the check once withNOT_CRAN=true, and keepskip_on_cran()for tests that are slow or depend on a service, not for tests that are inconvenient. -
Snapshots accepted without reading.
snapshot_accept()makes the new output the truth. Read the diff first. Snapshot only what a human is the judge of, keep the code in a snapshot small, usetransformto remove paths, times and versions, and do not snapshot text that other packages or R itself produce (the messages of R change between versions and are translated). Committests/testthat/_snaps/, or the tests fail on CI. -
Relying on a snapshot on CRAN. The snapshot expectations return without checking on CRAN (
cran = FALSE), so any behaviour that is checked only by a snapshot is unchecked there. Keep an ordinary expectation for what must hold. -
Expecting
devtools::test()to fail a script. It returns normally when tests fail (stop_on_failure = FALSE). For an exit status, usedevtools::test(stop_on_failure = TRUE),testthat::test_local()orR CMD check;test_check()raises an error, and so doestest_dir()by default. -
A
test_that()block with no expectation. It is reported as a skipped test,empty test, not as a passing one. -
Setup that works in
test_check()but not inload_all(). Setup files are not sourced bydevtools::load_all(). Put what a debugging session needs in a helper file or in the test, and keep the setup file small. -
Using
Sys.setenv()oroptions()in a test file at the top level, outside a test. The change lasts to the end of the process, not the end of the file, and in parallel runs it persists in the worker. Usewithr::local_*()at the top of the file (it lasts to the end of the file) or inside the test. -
Forgetting that internal functions are visible in tests. Tests run in an environment whose parent is the package namespace, so
shelf_today()is visible without:::; the same function from a script outside the package needsshelf:::shelf_today(). Test the exported behaviour first, and write tests of internals only for the logic that needs them. -
A package used in tests but not declared. Tests that call
library(somepackage)orsomepackage::fun()for a package missing fromDESCRIPTIONfail the check (checking for unstated dependencies in ‘tests’), and tests that use a suggested package must be skipped when it is missing. Since testthat 3.3.0test_that()does the latter on CRAN by itself. -
Printing noise. A test that prints is hard to read in a log. Use
expect_output(),capture.output()or a snapshot, and note that the output ofprint(),cat()andmessage()is lost in parallel runs.
Version notes
testthat’s own changes are recorded in its NEWS file (the testthat changelog); the entries below are those that matter for the topics of this page, with the dates of the CRAN releases.
| Version | CRAN release | Change |
|---|---|---|
0.9 |
2014-09-22 |
The BDD interface ( |
3.0.0 |
2020-10-31 |
Editions ( |
3.0.3 |
2021-06-16 |
The edition can be set by the environment variable |
3.1.0 |
2021-10-04 |
|
3.1.1 |
2021-12-03 |
Condition expectations match across the ancestry of chained errors ( |
3.1.5 |
2022-10-08 |
|
3.1.7 |
2023-03-12 |
|
3.1.8 |
2023-05-04 |
|
3.1.9 |
2023-06-15 |
|
3.2.0 |
2023-10-06 |
|
3.2.1 |
2023-12-02 |
|
3.2.2 |
2024-12-10 |
|
3.3.0 |
2025-11-13 |
Requires R 4.1. All failure messages rewritten, and all expectations return their first argument; |
3.3.1 |
2025-11-25 |
The |
3.3.2 |
2026-01-11 |
OpenTelemetry traces; |
The facts about R itself that this page used are these.
| R version | Change |
|---|---|
3.4.0 |
|
3.5.0 |
|
4.0.0 |
|
4.1.0 |
The base environment and its namespace are locked, so bindings can no longer be added to them or removed from them. The native pipe and |
4.4.0 |
The Rd macro |
The packages used on this page, with their versions: testthat 3.3.2 (CRAN release 2026-01-11), withr 3.0.3, covr 3.6.5, tinytest 1.4.3, devtools 2.5.2, usethis 3.2.2, pkgload 1.5.3, lintr 3.4.0, rlang 1.3.0, waldo 0.6.2, cli 3.6.6, knitr 1.52 and rmarkdown 2.32. The R side is R 4.6.1.
Where older tutorials differ
Many tutorials on testing in R were written for testthat 1.x or 2.x, or for a package workflow of the Travis CI era. They are not wrong about the principle, but several of the functions in them now warn or no longer exist.
| Older material says | Now | See |
|---|---|---|
|
The file name is the context; |
|
|
|
|
|
|
|
|
Plain expectations such as |
|
|
|
|
|
The |
|
|
|
|
|
|
|
|
|
|
Tests stored in |
|
|
Silent messages in tests |
Messages bubble up in the third edition; use |
|
|
|
|
Coverage with |
covr; the covr page lists both older tools as no longer supported |
|
|
|
|
The |
Still possible (tinytest’s |
Where to go next
The package that the tests belong to is built step by step in Creating Packages, and its help pages and vignette in Documentation with Rd, roxygen2 and Vignettes. The next steps of the same workflow are Style, Linting and Static Analysis, which checks the tests as it checks the code, and Publishing to CRAN, which runs the checks that decide whether a package is accepted. For the conditions that tests catch, read Conditions and Error Handling and Defensive Programming and Input Validation; for finding out why a test fails, Debugging. A script that is run from the command line is tested differently from a package, with its exit status as the thing to check: Scripts and the Command Line and, in the section on command-line tools for coding agents, Testing CLIs. The package side of the same topics is in Packages, Libraries and Namespaces (what a namespace is) and S3 Classes and Generics (the book class that the tests exercise). The community book R Packages (second edition) has the chapters Testing basics, Designing your test suite and Advanced testing techniques; the testthat package website has the reference and the articles listed in the references below.
References
-
Writing R Extensions: Package subdirectories (the
testsdirectory), Checking packages, Suggested packages and Writing portable packages -
R Internals: Tools (the
_R_CHECK_*environment variables) -
R NEWS (Changes in R 4.0.0, 4.1.0 and 4.4.0) and R 3.x NEWS (Changes in R 3.4.0 and 3.5.0)
-
R documentation: stopifnot, all.equal, on.exit, assertError and assertWarning, Rdiff and BATCH
-
testthat package documentation: Reference, testthat 3e, Snapshot tests, Test fixtures, Mocking, Skipping tests, Special files, Running tests in parallel, Custom expectations and Changelog
-
withr package documentation: Reference
-
covr package documentation: covr
-
tinytest package documentation: run_test_file, setup_tinytest and test_package
-
usethis package documentation: use_testthat, use_test and use_github_action; devtools package documentation: test; lintr package documentation: lint_package