tidyr, purrr and stringr
|
This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production. The reference material consulted while preparing these pages is listed in the section bibliography. |
Checked against R 4.6.1 (2026-06-24) on 2026-10-20; R 4.6.2 is scheduled for 2026-10-29. This page uses add-on packages, none of which is part of R: tidyr 1.3.2, purrr 1.2.2, stringr 1.6.0, forcats 1.0.1, lubridate 1.9.5 and readr 2.2.0, together with dplyr 1.2.1 and tibble 3.3.1 wherever a data frame has to be filtered, joined or counted, the tidyverse 2.0.0 meta-package for its start-up message, and lifecycle 1.0.5 (the package behind the function-stage labels, whose version is printed once). They were installed from CRAN, and the lintr 3.4.0 default linters were used only to check the scripts. Base R already has a tool for every job below, and the Data Frames, The apply Family and Functionals, Strings and Regular Expressions, Factors and Dates and Times pages teach those first. This page is about the tidyverse spelling of the same work: what each package adds, which base function it replaces, where its answers differ, and the real error and warning text of the places where it is stricter. The packages are community-maintained by Posit (the tidyverse team) and documented on their own pkgdown sites, which this page cites for package behaviour.
Every example is a script that was run with Rscript on R 4.6.1 with warnings printed as they happen (options(warn = 1)), and the lines that start with #> are what it printed. The scripts also pass the default lintr linters. Each block stands alone: a block that needs the cleaned shelf data repeats the few setup lines that make it, so any block can be pasted into a fresh session and run. The names books, loans, shelf_today, loan_days and the book() constructor come from the running library-shelf scenario described on the section index: a script that needs them starts with source("scenario.R"), and the CSV files books.csv and loans.csv sit in the working directory. The books data frame has messy title, author, isbn and genre columns, and loans has one impossible date (2025-02-30) and one return date before its loan date, which is what the cleaning examples work on. Tibbles are printed with the pillar package, whose output uses the typographic characters of a UTF-8 locale. Where a message contains quotation marks or bullets, they are the ones R and the cli package print in such a locale.
This page is the second of the three data-manipulation pages. The tidyverse and dplyr covers the data-frame verbs (filter(), mutate(), summarise(), joins) that the shelf examples use without explaining them, and data.table is the other large ecosystem. Dates, strings and factors are first taught with base R on their own pages; here lubridate, stringr and forcats are shown next to the base function they correspond to, not re-taught. Parallel iteration with in_parallel() belongs to Parallel and Asynchronous Computing.
The packages and what they replace
The six packages answer six separate jobs. They share conventions (the data comes first, so they work with the native pipe |> of Pipes and the Native Pipe; names are lower-case with underscores; arguments after the required ones are named; errors are formatted by the rlang and cli packages), but each can be used alone.
| Package | Version | What it is for | Base R that overlaps |
|---|---|---|---|
tidyr |
1.3.2 |
reshaping ( |
|
purrr |
1.2.2 |
|
|
stringr |
1.6.0 |
|
|
forcats |
1.0.1 |
|
|
lubridate |
1.9.5 |
parsers named for the order of the parts ( |
|
readr |
2.2.0 |
|
|
Source: The tidyverse: Packages
The versions come from the installed packages, and the minimum R version is part of each package’s DESCRIPTION. tidyr, purrr, stringr, forcats and readr all declare R 4.1 or later, which is the release that introduced the native pipe and the \(x) function shorthand, so the examples on this page use both and none of the formula shorthand (~ .x + 1) that older tutorials use.
# The add-on packages used on this page, and the versions checked.
pkgs <- c(
"tidyr", "purrr", "stringr", "forcats", "lubridate", "readr",
"dplyr", "tibble", "tidyverse", "lifecycle"
)
vapply(pkgs, \(p) as.character(packageVersion(p)), character(1))
#> tidyr purrr stringr forcats lubridate readr dplyr tibble
#> "1.3.2" "1.2.2" "1.6.0" "1.0.1" "1.9.5" "2.2.0" "1.2.1" "3.3.1"
#> tidyverse lifecycle
#> "2.0.0" "1.0.5"
# The minimum R version is part of each package's DESCRIPTION.
vapply(
c("tidyr", "purrr", "stringr", "forcats", "readr"),
\(p) packageDescription(p)$Depends,
character(1)
)
#> tidyr purrr stringr forcats readr
#> "R (>= 4.1.0)" "R (>= 4.1)" "R (>= 4.1.0)" "R (>= 4.1)" "R (>= 4.1)"Source: R NEWS: Changes in R 4.1.0
Loading the packages
The tidyverse meta-package attaches nine core packages with one call and prints what it did. The first block of the message is the list of packages with their versions; the second is the list of conflicts, that is functions that now mask a function of another package that was attached earlier (here filter() and lag() of stats).
library(tidyverse)
#> ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
#> ✔ dplyr 1.2.1 ✔ readr 2.2.0
#> ✔ forcats 1.0.1 ✔ stringr 1.6.0
#> ✔ ggplot2 4.0.3 ✔ tibble 3.3.1
#> ✔ lubridate 1.9.5 ✔ tidyr 1.3.2
#> ✔ purrr 1.2.2
#> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
#> ✖ dplyr::filter() masks stats::filter()
#> ✖ dplyr::lag() masks stats::lag()
#> ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errorsThe scripts on this page attach the packages one at a time instead, so that a script names the packages it depends on, and a package that is only needed for one call is called with :: (tibble::tibble()). Attaching dplyr and lubridate prints the lists of masked objects on the console; the scripts pass warn.conflicts = FALSE to library() to keep the output short, which is a documented argument and changes nothing else. The messages are worth reading once:
library(dplyr)
#> Attaching package: ‘dplyr’
#>
#> The following objects are masked from ‘package:stats’:
#>
#> filter, lag
#>
#> The following objects are masked from ‘package:base’:
#>
#> intersect, setdiff, setequal, union
library(lubridate)
#> Attaching package: ‘lubridate’
#>
#> The following objects are masked from ‘package:base’:
#>
#> date, intersect, setdiff, unionSource: R documentation: library
Both packages mask intersect(), setdiff() and union() of base R; dplyr’s versions also accept data frames, so setdiff() of two data frames works once dplyr is attached. A call that depends on which of two functions of the same name is found first is a source of confusing results, and the rule is the one of Packages, Libraries and Namespaces: the package attached last is searched first, and base::setdiff() or dplyr::setdiff() says which one is meant.
Reading data with readr
readr reads delimited text into a tibble. read_csv() is read_delim() with a comma, and the same family has read_csv2() (semicolon, decimal comma), read_tsv() and read_delim() for other separators. Since readr 2.0.0 (2021-07-20) the engine is vroom, which reads files lazily and uses several threads; readr 2.1.0 turned the lazy reading back off by default because it kept files locked on Windows. The differences from read.csv() that matter on the shelf data are in the first block: text fields are trimmed (trim_ws = TRUE), so " the quiet streets " loses its blanks while read.csv() keeps them; empty fields are NA in every column, text included (na = c("", "NA")), where read.csv() leaves an empty string in a character column; and whole numbers are read as doubles (year is <dbl>), where read.csv() gives integers. The column types are guessed from the first guess_max rows, at most 1000.
library(readr)
# read_csv() guesses the column types from the first rows. Text fields are
# trimmed (trim_ws = TRUE), and empty fields are NA in every column.
books <- read_csv("books.csv", show_col_types = FALSE)
books[1:5, c("title", "isbn", "year", "genre")]
#> # A tibble: 5 × 4
#> title isbn year genre
#> <chr> <chr> <dbl> <chr>
#> 1 The Dispossessed 978-0-306-40615-7 1974 Science fiction
#> 2 Pride and Prejudice 978-0-14-143951-8 1813 Classic
#> 3 the quiet streets 9780001000100 2019 Fiction
#> 4 Loud Roofs 978-0-00-200029-1 2021 fiction
#> 5 Midnight Ledger 978-0-00-300031-3 2016 Mystery
spec(books)
#> cols(
#> title = col_character(),
#> author = col_character(),
#> isbn = col_character(),
#> year = col_double(),
#> genre = col_character(),
#> pages = col_double()
#> )
sum(is.na(books$isbn))
#> [1] 1
sum(read.csv("books.csv")$isbn == "", na.rm = TRUE)
#> [1] 1
books$title[3]
#> [1] "the quiet streets"
read.csv("books.csv")$title[3]
#> [1] " the quiet streets "col_types fixes the types instead of guessing them, and cols() builds the specification (spec() printed one above). Dates are col_date() with an optional format. A value that cannot be parsed becomes NA and readr records where: the call gives a warning and problems() returns a tibble with the row, the column, what was expected and the text that was found. The impossible date 2025-02-30 of the shelf data is such a value. The problems() result also has a file column with the path that was read, which is left out of the output below.
library(readr)
# Explicit types: the date columns of loans.csv. The impossible date 2025-02-30
# becomes NA, with a warning and a record of the problem.
loan_types <- cols(
.default = col_character(),
loan_date = col_date(),
return_date = col_date()
)
tryCatch(
read_csv("loans.csv", col_types = loan_types),
warning = \(w) cat(conditionMessage(w), "\n")
)
#> One or more parsing issues, call `problems()` on your data frame for details,
#> e.g.:
#> dat <- vroom(...)
#> problems(dat)
loans <- suppressWarnings(read_csv("loans.csv", col_types = loan_types))
problems(loans)[c("row", "col", "expected", "actual")]
#> # A tibble: 1 × 4
#> row col expected actual
#> <int> <int> <chr> <chr>
#> 1 21 4 date in ISO8601 2025-02-30
loans[19:21, c("loan_id", "loan_date", "return_date")]
#> # A tibble: 3 × 3
#> loan_id loan_date return_date
#> <chr> <date> <date>
#> 1 L019 2025-07-02 2025-07-25
#> 2 L020 NA 2025-07-15
#> 3 L021 2025-07-05 2025-07-21Row 21 of the file is the 20th data row (the header is row 1), which is why loans[20, ] is the row with the NA date.
readr 2.2.0 changed one thing that older tutorials get wrong: the first argument of read_csv() and its relatives is a path or connection, and a string containing a newline used to be accepted as the data itself. A bare literal string now gives a deprecation warning, and the data must be wrapped in I(). write_csv() writes no row names, quotes only where needed and writes NA as an empty field.
library(readr)
# Since readr 2.2.0, literal text must be wrapped in I(): a bare string is
# treated as a path, and the old behaviour is deprecated.
read_csv(I("title,pages\nMidnight Ledger,288\n"), show_col_types = FALSE)
#> # A tibble: 1 × 2
#> title pages
#> <chr> <dbl>
#> 1 Midnight Ledger 288
tryCatch(
read_csv("title,pages\nMidnight Ledger,288\n", show_col_types = FALSE),
warning = \(w) cat(conditionMessage(w), "\n")
)
#> The `file` argument of `read_csv()` should use `I()` for literal data as of
#> readr 2.2.0.
#>
#> # Bad (for example):
#> read_csv("x,y\n1,2")
#>
#> # Good:
#> read_csv(I("x,y\n1,2"))
# Writing: write_csv() never writes row names and writes NA as an empty field.
path <- tempfile(fileext = ".csv")
write_csv(head(read_csv("books.csv", show_col_types = FALSE), 3), path)
readLines(path)
#> [1] "title,author,isbn,year,genre,pages"
#> [2] "The Dispossessed,Ursula K. Le Guin,978-0-306-40615-7,1974,Science fiction,387"
#> [3] "Pride and Prejudice,Jane Austen,978-0-14-143951-8,1813,Classic,432"
#> [4] "the quiet streets,Mara Quill,9780001000100,2019,Fiction,214"
file.remove(path)
#> [1] TRUEreadr reads and writes text files and is covered here only as far as the reshaping examples need it. Reading text in general (encodings, fixed-width files, connections) is in Reading and Writing Text Data and Connections and the File System.
Reshaping with tidyr
tidyr starts from an idea about the shape of a table. In tidy data each variable is a column, each observation is a row and each value is a cell (tidyr package documentation: Tidy data). A report that shows genres down the side and months across the top is wide: the months are the values of one variable, month, spread over several columns. Analysis code (grouping, plotting, modelling) usually wants the long form, with one row per genre and month and the count in one column, and a report wants the wide one. Base R converts between them with reshape(), stack() and table(), which Data Frames explains with the same data; reshape() takes the names of the id, time and value columns in several arguments and builds the new column names itself, which is the part many people find hard to remember. tidyr 1.0.0 (2019-09-11) added pivot_longer() and pivot_wider() as redesigned replacements for gather() and spread(), with the argument names saying which columns the names and the values come from.
| tidyr | Base R | Difference worth knowing |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
- |
superseded by |
The data for the next two blocks is the number of loans per month and genre in 2026: the ISBNs of the loans are matched with the (cleaned) ISBNs of the books, the genre is trimmed and put in lower case, and count() gives one row for every month-and-genre pair that occurs.
pivot_wider(): long to wide
pivot_wider() takes a long table and makes one new column for each distinct value of names_from, filled from values_from. The columns that are neither are the identifiers (id_cols, by default all of them), and one output row is made for every distinct combination of them. A combination that does not occur in the input is NA in the output, unless values_fill gives a value. Since tidyr 1.3.0 the optional arguments must be named (... sits right after data). names_prefix goes in front of the new names, names_sep (default "_") joins the parts of a new name (the value name and the names_from value when values_from has several columns), names_glue builds the names from a template, names_sort and names_vary control the column order, names_expand and id_expand (tidyr 1.2.0) make the implicit factor combinations explicit, and unused_fn says how to summarise the columns that are not used.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
source("scenario.R")
# Loans per month and genre in 2026, one row per month-and-genre that occurs.
books$genre <- tolower(trimws(books$genre))
books$isbn <- gsub("[^0-9X]", "", books$isbn)
monthly <- loans |>
filter(loan_date >= as.Date("2026-01-01")) |>
inner_join(books[c("isbn", "genre")], by = "isbn") |>
count(month = format(loan_date, "%Y-%m"), genre)
monthly
#> month genre n
#> 1 2026-01 classic 1
#> 2 2026-01 fiction 2
#> 3 2026-01 science fiction 1
#> 4 2026-02 fiction 1
#> 5 2026-02 science fiction 2
#> 6 2026-03 non-fiction 1
#> 7 2026-03 science fiction 2
#> 8 2026-04 fiction 2
#> 9 2026-04 poetry 1
#> 10 2026-04 science fiction 1
#> 11 2026-05 fiction 2
#> 12 2026-05 science fiction 1
# Long to wide: genres down, months across. A missing combination is NA ...
wide <- monthly |>
pivot_wider(names_from = month, values_from = n)
wide
#> # A tibble: 5 × 6
#> genre `2026-01` `2026-02` `2026-03` `2026-04` `2026-05`
#> <chr> <int> <int> <int> <int> <int>
#> 1 classic 1 NA NA NA NA
#> 2 fiction 2 1 NA 2 2
#> 3 science fiction 1 2 2 1 1
#> 4 non-fiction NA NA 1 NA NA
#> 5 poetry NA NA NA 1 NA
# ... unless values_fill says otherwise. names_glue builds the new names, and
# id_cols names the identifying columns.
monthly |>
pivot_wider(
id_cols = genre, names_from = month, values_from = n,
values_fill = 0L, names_glue = "n_{month}"
)
#> # A tibble: 5 × 6
#> genre `n_2026-01` `n_2026-02` `n_2026-03` `n_2026-04` `n_2026-05`
#> <chr> <int> <int> <int> <int> <int>
#> 1 classic 1 0 0 0 0
#> 2 fiction 2 1 0 2 2
#> 3 science fiction 1 2 2 1 1
#> 4 non-fiction 0 0 1 0 0
#> 5 poetry 0 0 0 1 0
# Several value columns: one new column per value and name.
monthly |>
mutate(share = round(n / sum(n), 2), .by = month) |>
pivot_wider(
names_from = month, values_from = c(n, share),
names_vary = "slowest"
) |>
names()
#> [1] "genre" "n_2026-01" "share_2026-01" "n_2026-02"
#> [5] "share_2026-02" "n_2026-03" "share_2026-03" "n_2026-04"
#> [9] "share_2026-04" "n_2026-05" "share_2026-05"pivot_longer(): wide to long
pivot_longer() is the reverse. cols selects the columns to gather, names_to is the name of the new column that receives their names and values_to the name of the one that receives their values. The selection uses the tidyselect language (!genre means every column but genre, starts_with(), matches() and the rest), and a selection that matches nothing is an error. Two arguments turn the information in the names into columns: names_sep splits at a separator and names_pattern takes the pieces from the groups of a regular expression; names_to then has one name for each piece, and the special name .value makes a piece the name of a value column, which is how several measures are gathered at once. names_transform and values_transform convert the new columns (a single function applies to all of them), names_ptypes and values_ptypes check their types, values_drop_na = TRUE drops the rows whose value is missing, names_prefix removes a prefix and cols_vary (tidyr 1.3.0) sets the order of the output rows.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
source("scenario.R")
# The wide report of the previous script.
books$genre <- tolower(trimws(books$genre))
books$isbn <- gsub("[^0-9X]", "", books$isbn)
wide <- loans |>
filter(loan_date >= as.Date("2026-01-01")) |>
inner_join(books[c("isbn", "genre")], by = "isbn") |>
count(month = format(loan_date, "%Y-%m"), genre) |>
pivot_wider(names_from = month, values_from = n, values_fill = 0L)
# Wide to long: cols selects the columns to gather.
wide |>
pivot_longer(cols = !genre, names_to = "month", values_to = "n") |>
head(5)
#> # A tibble: 5 × 3
#> genre month n
#> <chr> <chr> <int>
#> 1 classic 2026-01 1
#> 2 classic 2026-02 0
#> 3 classic 2026-03 0
#> 4 classic 2026-04 0
#> 5 classic 2026-05 0
# names_transform parses the names; values_drop_na drops the NA values.
wide |>
pivot_longer(
cols = starts_with("2026"), names_to = "month", values_to = "n",
names_transform = \(x) as.Date(paste0(x, "-01")),
values_transform = \(x) replace(x, x == 0L, NA),
values_drop_na = TRUE
) |>
head(4)
#> # A tibble: 4 × 3
#> genre month n
#> <chr> <date> <int>
#> 1 classic 2026-01-01 1
#> 2 fiction 2026-01-01 2
#> 3 fiction 2026-02-01 1
#> 4 fiction 2026-04-01 2
# names_pattern with several names_to: columns like "n_2026_01".
by_part <- wide |>
rename_with(\(x) paste0("n_", gsub("-", "_", x)), starts_with("2026"))
by_part |>
pivot_longer(
cols = starts_with("n_"),
names_to = c("year", "month"),
names_pattern = "n_(\\d{4})_(\\d{2})",
names_transform = list(year = as.integer, month = as.integer),
values_to = "n"
) |>
head(3)
#> # A tibble: 3 × 4
#> genre year month n
#> <chr> <int> <int> <int>
#> 1 classic 2026 1 1
#> 2 classic 2026 2 0
#> 3 classic 2026 3 0The result of the pivot_longer() pipeline in the second call is the counts that occur, with the month as a date, which is the form group_by() and ggplot() want. The third call parses the numbers out of names such as n_2026_01.
Duplicates, types and empty selections
The pivots refuse to guess in three situations, and in two of them the failure is the useful part. If the identifier columns and names_from do not identify one row uniquely, pivot_wider() cannot put two values in one cell: it warns and returns list-columns, and the warning lists the ways out (values_fn = list keeps the list, values_fn = sum or another summary reduces the duplicates, and the code it prints finds them). If the gathered columns have different types, pivot_longer() stops because a column has one type; values_transform = as.character converts them first.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
source("scenario.R")
# Duplicates: two rows for the same month and genre cannot fit one cell.
dup <- tibble::tibble(
genre = c("fiction", "fiction", "poetry"),
month = c("2026-01", "2026-01", "2026-01"),
n = c(2L, 1L, 1L)
)
tryCatch(
pivot_wider(dup, names_from = month, values_from = n),
warning = \(w) cat(conditionMessage(w), "\n")
)
#> Values from `n` are not uniquely identified; output will contain list-cols.
#> • Use `values_fn = list` to suppress this warning.
#> • Use `values_fn = {summary_fun}` to summarise duplicates.
#> • Use the following dplyr code to identify duplicates.
#> {data} |>
#> dplyr::summarise(n = dplyr::n(), .by = c(genre, month)) |>
#> dplyr::filter(n > 1L)
# values_fn says how to combine them, and the result is a plain column.
pivot_wider(dup, names_from = month, values_from = n, values_fn = sum)
#> # A tibble: 2 × 2
#> genre `2026-01`
#> <chr> <int>
#> 1 fiction 3
#> 2 poetry 1
# Mixed types cannot share one value column.
mixed <- tibble::tibble(id = 1:2, pages = 214L, title = "Ten Winters")
tryCatch(
pivot_longer(mixed, !id),
error = \(e) cat(conditionMessage(e), "\n")
)
#> Can't combine `pages` <integer> and `title` <character>.
pivot_longer(mixed, !id, values_transform = as.character)
#> # A tibble: 4 × 3
#> id name value
#> <int> <chr> <chr>
#> 1 1 pages 214
#> 2 1 title Ten Winters
#> 3 2 pages 214
#> 4 2 title Ten Winters
# A selection that matches nothing is an error, not an empty result.
tryCatch(
pivot_longer(mixed, starts_with("zzz")),
error = \(e) cat(conditionMessage(e), "\n")
)
#> `cols` must select at least one column.The text of the duplicates warning is the one tidyr 1.3.2 prints; tidyr 1.3.1 changed the suggested code to use the .by argument and the native pipe. A third situation is a new column name that collides with an identifier column: names_repair = "check_unique" (the default) stops with Names must be unique. and the positions of the duplicated names, and the names_repair argument chooses another strategy.
Splitting and combining columns
A column that holds several pieces of information in one string, such as "Le Guin, Ursula K." (a surname, a comma and a given name) or "978-0-306-40615-7" (five parts of an ISBN), is split into columns by the separate_wider_*() family, which tidyr 1.3.0 (2023-01-24) added. The functions are separate_wider_delim() (split at a delimiter), separate_wider_position() (split at fixed widths) and separate_wider_regex() (take the pieces from a vector of regular expressions); the long counterparts separate_longer_delim() and separate_longer_position() split into rows instead. They replace separate(), extract() and separate_rows(), which the tidyr reference marks as superseded: they stay in the package and receive only critical bug fixes. The tidyr NEWS describes them as thorough refreshes with better performance, a more consistent interface and a new approach to handling problems: a mismatch between the number of pieces and the number of names is an error that says how to diagnose it.
separate_wider_delim() and separate_wider_position()
separate_wider_delim(data, cols, delim, names = , names_sep = ) splits each value of the selected column at the literal delim, which is not a regular expression. names gives the new column names in order; names_sep names the pieces <column>_1, <column>_2, and so on, when the number of pieces is not known. An NA in names drops that piece. The original column is removed unless cols_remove = FALSE. separate_wider_position(data, cols, widths) takes a named integer vector: each width is the number of characters of a piece, and an unnamed width is skipped.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
source("scenario.R")
# separate_wider_delim(): split on a fixed delimiter into named columns.
# "Le Guin, Ursula K." has the surname first, so this row splits into two.
lamp <- books[books$author == "Le Guin, Ursula K.", c("title", "author")]
lamp |>
separate_wider_delim(author, delim = ", ", names = c("surname", "given"))
#> # A tibble: 1 × 3
#> title surname given
#> <chr> <chr> <chr>
#> 1 The Lamp at Noon Le Guin Ursula K.
# Without names, names_sep numbers the pieces: member_1, member_2, ...
members <- tibble::tibble(member = c("Ana Ruiz", "Dev Patel", "Eva Lindqvist"))
members |>
separate_wider_delim(member, delim = " ", names_sep = "_")
#> # A tibble: 3 × 2
#> member_1 member_2
#> <chr> <chr>
#> 1 Ana Ruiz
#> 2 Dev Patel
#> 3 Eva Lindqvist
# The five parts of a hyphenated ISBN-13.
isbns <- tibble::tibble(isbn = c("978-0-306-40615-7", "978-0-14-143951-8"))
isbns |>
separate_wider_delim(
isbn, delim = "-",
names = c("ean", "group", "publisher", "title_no", "check")
)
#> # A tibble: 2 × 5
#> ean group publisher title_no check
#> <chr> <chr> <chr> <chr> <chr>
#> 1 978 0 306 40615 7
#> 2 978 0 14 143951 8
# separate_wider_position(): fixed widths. A loan id is a letter and a number.
ids <- loans[1:3, "loan_id", drop = FALSE]
ids |>
separate_wider_position(loan_id, widths = c(prefix = 1, number = 3))
#> # A tibble: 3 × 2
#> prefix number
#> <chr> <chr>
#> 1 L 001
#> 2 L 002
#> 3 L 003
# An unnamed width is skipped, and cols_remove = FALSE keeps the input.
ids |>
separate_wider_position(
loan_id, widths = c(1, number = 3), cols_remove = FALSE
)
#> # A tibble: 3 × 2
#> number loan_id
#> <chr> <chr>
#> 1 001 L001
#> 2 002 L002
#> 3 003 L003separate_wider_regex()
separate_wider_regex(data, cols, patterns) takes a character vector of regular expressions that must match the whole value, in order, with nothing left over. A named element becomes a column; an unnamed element has to match but is dropped, which is how literal separators are written. It is the replacement of extract() and the way to split strings whose pieces do not have one delimiter. The patterns are ICU regular expressions, as everywhere in stringr (see stringr below), and the third call of the block shows that a value that does not match is an error that points to too_few.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
source("scenario.R")
# separate_wider_regex(): a character vector of patterns, in order. A named
# element becomes a column; an unnamed one must match but is dropped.
isbn_pattern <- c(
ean = "\\d{3}", "-", group = "\\d+", "-", publisher = "\\d+", "-",
title_no = "\\d+", "-", check = "\\d"
)
tibble::tibble(isbn = books$isbn[c(1, 2, 4)]) |>
separate_wider_regex(isbn, patterns = isbn_pattern)
#> # A tibble: 3 × 5
#> ean group publisher title_no check
#> <chr> <chr> <chr> <chr> <chr>
#> 1 978 0 306 40615 7
#> 2 978 0 14 143951 8
#> 3 978 0 00 200029 1
# A label with the year in brackets: two named pieces and literal text.
tibble::tibble(label = c("The Dispossessed (1974)", "Ten Winters (2022)")) |>
separate_wider_regex(
label,
patterns = c(title = ".+", " \\(", year = "\\d{4}", "\\)")
) |>
mutate(year = as.integer(year))
#> # A tibble: 2 × 2
#> title year
#> <chr> <int>
#> 1 The Dispossessed 1974
#> 2 Ten Winters 2022
# The whole value must match. A bare ISBN and an empty one do not.
isbns <- tibble::tibble(isbn = books$isbn[c(1, 3, 9)])
tryCatch(
separate_wider_regex(isbns, isbn, patterns = isbn_pattern),
error = \(e) cat(conditionMessage(e), "\n")
)
#> Expected each value of `isbn` to match the pattern, the whole pattern,
#> and nothing but the pattern.
#> ! 2 values have problems.
#> ℹ Use `too_few = "debug"` to diagnose the problem.
#> ℹ Use `too_few = "align_start"` to silence this message.
separate_wider_regex(
isbns, isbn, patterns = isbn_pattern, too_few = "align_start"
)
#> # A tibble: 3 × 5
#> ean group publisher title_no check
#> <chr> <chr> <chr> <chr> <chr>
#> 1 978 0 306 40615 7
#> 2 978 <NA> <NA> <NA> <NA>
#> 3 <NA> <NA> <NA> <NA> <NA>Too few and too many pieces
The most useful part of the new design is what happens when a value does not have the expected number of pieces. By default every function stops and says how many values were wrong and which argument to set. too_few has the options "error" (the default), "debug" and "align_start", plus "align_end" for the delimiter function; too_many (delimiter and position functions) has "error", "debug" and "drop", plus "merge" for the delimiter function. "debug" adds three columns, named after the input column: _ok is TRUE for the values that fit, _pieces is the number of pieces found (_matches for the regex function) and _remainder is the text that was left over; it also warns that it did so, because debug mode is meant for diagnosis and not for production code.
The block splits every author of the shelf data on ", ". Only one of the 16 books has an author with the surname first; the other 15 are in the form "Ursula K. Le Guin", so the default stops with the number of values that were too short, and "debug" shows which ones. "align_start" would fill the missing pieces with NA at the end, which is rarely what you want for names. The last calls give an ISBN too few names: "error" again, "merge" keeps the rest in the last column, and "drop" discards it.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
source("scenario.R")
# Splitting every author on ", " fails: only one of 16 has the surname first.
authors <- tibble::tibble(title = books$title, author = books$author)
split_author <- \(df, ...) {
separate_wider_delim(
df, "author", delim = ", ", names = c("surname", "given"), ...
)
}
tryCatch(split_author(authors), error = \(e) cat(conditionMessage(e), "\n"))
#> Expected 2 pieces in each element of `author`.
#> ! 15 values were too short.
#> ℹ Use `too_few = "debug"` to diagnose the problem.
#> ℹ Use `too_few = "align_start"/"align_end"` to silence this message.
# too_few = "debug" keeps the input and adds author_ok, author_pieces and
# author_remainder. It also warns that it did so.
tryCatch(
split_author(authors, too_few = "debug"),
warning = \(w) cat(conditionMessage(w), "\n")
)
#> Debug mode activated: adding variables `author_ok`, `author_pieces`, and
#> `author_remainder`.
debugged <- suppressWarnings(split_author(authors, too_few = "debug"))
count(debugged, author_ok)
#> # A tibble: 2 × 2
#> author_ok n
#> <lgl> <int>
#> 1 FALSE 15
#> 2 TRUE 1
debugged[debugged$author_ok, c("author", "surname", "given")]
#> # A tibble: 1 × 3
#> author surname given
#> <chr> <chr> <chr>
#> 1 Le Guin, Ursula K. Le Guin Ursula K.
# Too many pieces: an ISBN has five parts, and we only name two.
isbn <- tibble::tibble(isbn = "978-0-306-40615-7")
tryCatch(
separate_wider_delim(isbn, isbn, delim = "-", names = c("ean", "group")),
error = \(e) cat(conditionMessage(e), "\n")
)
#> Expected 2 pieces in each element of `isbn`.
#> ! 1 value was too long.
#> ℹ Use `too_many = "debug"` to diagnose the problem.
#> ℹ Use `too_many = "drop"/"merge"` to silence this message.
separate_wider_delim(
isbn, isbn, delim = "-", names = c("ean", "group", "rest"),
too_many = "merge"
)
#> # A tibble: 1 × 3
#> ean group rest
#> <chr> <chr> <chr>
#> 1 978 0 306-40615-7
separate_wider_delim(
isbn, isbn, delim = "-", names = c("ean", "group"), too_many = "drop"
)
#> # A tibble: 1 × 2
#> ean group
#> <chr> <chr>
#> 1 978 0The correct repair for the authors is not an option of separate_wider_delim() but a cleaning step first: put every author in one form with str_replace() (see stringr below), and then split. The rule in all of these functions is the same: decide what a malformed value means, and say so in the argument, instead of finding out from the data.
Splitting into rows, and putting columns together
separate_longer_delim() and separate_longer_position() split one value into several rows, repeating the other columns: a cell with "crime;noir;city" becomes three rows. unite(data, col, ..., sep = "_", remove = TRUE, na.rm = FALSE) is the inverse of the wider functions: it pastes the selected columns into one. A missing value is pasted as the text "NA" unless na.rm = TRUE, the same trap as paste() in base R. The old separate_rows() is the superseded form of separate_longer_delim().
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
# One cell with several values becomes several rows.
tags <- tibble::tibble(
title = c("Midnight Ledger", "Ten Winters"),
tags = c("crime;noir;city", "family;winter")
)
tags |>
separate_longer_delim(tags, delim = ";")
#> # A tibble: 5 × 2
#> title tags
#> <chr> <chr>
#> 1 Midnight Ledger crime
#> 2 Midnight Ledger noir
#> 3 Midnight Ledger city
#> 4 Ten Winters family
#> 5 Ten Winters winter
# unite() is the inverse of separate_wider_*(): it pastes columns together.
authors <- tibble::tibble(surname = "Le Guin", given = "Ursula K.")
authors |>
unite("author", given, surname, sep = " ")
#> # A tibble: 1 × 1
#> author
#> <chr>
#> 1 Ursula K. Le Guin
# NA is pasted as the text "NA" unless na.rm = TRUE.
partial <- tibble::tibble(surname = "Quill", given = NA_character_)
partial |>
unite("author", given, surname, sep = " ")
#> # A tibble: 1 × 1
#> author
#> <chr>
#> 1 NA Quill
partial |>
unite("author", given, surname, sep = " ", na.rm = TRUE)
#> # A tibble: 1 × 1
#> author
#> <chr>
#> 1 QuillMissing values and implicit gaps
A long table only has the rows that occur, so a month in which no poetry was lent is not a row at all: the missing value is implicit. complete() adds a row for every combination of the columns it is given and fills the new rows from fill; expand() and crossing() list the combinations without the data; drop_na() drops rows with an NA in the chosen columns (the analogue of na.omit() and complete.cases()); replace_na() replaces NA with a value for each column, which is cast to the type of the column, so a value that cannot be cast, such as 1.5 for an integer column, is an error since tidyr 1.2.0; and fill() carries the last non-missing value down a column, by group when .by is given (new in tidyr 1.3.2).
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
source("scenario.R")
# Loans per genre and month, only for the combinations that occur.
books$genre <- tolower(trimws(books$genre))
books$isbn <- gsub("[^0-9X]", "", books$isbn)
monthly <- loans |>
filter(loan_date >= as.Date("2026-01-01")) |>
inner_join(books[c("isbn", "genre")], by = "isbn") |>
count(month = format(loan_date, "%Y-%m"), genre)
nrow(monthly)
#> [1] 12
# complete() makes the implicit gaps explicit: every month with every genre.
full <- monthly |>
complete(month, genre, fill = list(n = 0L))
nrow(full)
#> [1] 25
full |>
filter(genre == "poetry")
#> # A tibble: 5 × 3
#> month genre n
#> <chr> <chr> <int>
#> 1 2026-01 poetry 0
#> 2 2026-02 poetry 0
#> 3 2026-03 poetry 0
#> 4 2026-04 poetry 1
#> 5 2026-05 poetry 0
# drop_na() and replace_na() deal with explicit NA values.
books_na <- tibble::as_tibble(books[c("title", "year", "pages")])
nrow(drop_na(books_na, year, pages))
#> [1] 13
replace_na(books_na, list(year = 0L, pages = 0L))[6:7, ]
#> # A tibble: 2 × 3
#> title year pages
#> <chr> <int> <int>
#> 1 A Map of Salt 2008 0
#> 2 Garden of Small Fires 0 96
# fill() carries the last non-missing value down; .by (new in tidyr 1.3.2)
# restarts the filling in each group.
gaps <- tibble::tibble(
shelf = c("A", "A", "A", "B", "B"),
genre = c("fiction", NA, NA, NA, "poetry")
)
fill(gaps, genre)
#> # A tibble: 5 × 2
#> shelf genre
#> <chr> <chr>
#> 1 A fiction
#> 2 A fiction
#> 3 A fiction
#> 4 B fiction
#> 5 B poetry
fill(gaps, genre, .by = shelf)
#> # A tibble: 5 × 2
#> shelf genre
#> <chr> <chr>
#> 1 A fiction
#> 2 A fiction
#> 3 A fiction
#> 4 B <NA>
#> 5 B poetryThe explicit zeros of complete() are what table() gives for free in base R. The difference matters when the table is then used as data: a model or a plot of a long table without the zeros silently leaves them out.
Nesting and rectangling
A data frame column can be a list, and when it is a list of data frames the table is nested: one row per group, with the group’s rows inside a cell. Nesting is the tidyverse’s answer to "do something for each group and keep the results next to the group", the job that split() plus lapply() does in base R (see Data Frames). Rectangling is the opposite problem: deeply nested lists, usually from JSON, are turned into a table.
nest() and unnest()
nest(.data, .by = ) groups by the columns of .by and puts the remaining columns, as a tibble, in a list-column called data (.key renames it). The other form names the list-column and the columns to nest, nest(history = c(loan_id, loan_date)), and groups by whatever is left; since tidyr 1.3.0 .key is used whenever ... is not given. unnest() reverses it, repeating the other columns for every row. The rows of a list-column can be summarised with vapply() or with purrr::map_int() and its relatives (see purrr below); dplyr also has nest_by() for row-wise work on nested data (see The tidyverse and dplyr). chop() and unchop() do the same for vectors instead of data frames, and pack() and unpack() make and unmake data-frame columns.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
source("scenario.R")
# Nesting: one row per book, and the loans of that book in a list-column.
loans <- tibble::as_tibble(
loans[!is.na(loans$loan_date), c("loan_id", "isbn", "member", "loan_date")]
)
by_book <- loans |>
nest(.by = isbn)
head(by_book, 3)
#> # A tibble: 3 × 2
#> isbn data
#> <chr> <list>
#> 1 9780141439518 <tibble [3 × 3]>
#> 2 9780306406157 <tibble [7 × 3]>
#> 3 9780002000291 <tibble [6 × 3]>
by_book$data[[1]]
#> # A tibble: 3 × 3
#> loan_id member loan_date
#> <chr> <chr> <date>
#> 1 L001 Hugo Silva 2025-01-14
#> 2 L028 Ana Ruiz 2025-09-18
#> 3 L046 Ana Ruiz 2026-01-09
# The other form names the list-column and the columns to nest.
loans |>
nest(history = c(loan_id, loan_date)) |>
head(2)
#> # A tibble: 2 × 3
#> isbn member history
#> <chr> <chr> <list>
#> 1 9780141439518 Hugo Silva <tibble [1 × 2]>
#> 2 9780306406157 Ben Okafor <tibble [1 × 2]>
# unnest() is the inverse: the rows come back, with the key repeated.
by_book |>
unnest(data) |>
head(3)
#> # A tibble: 3 × 4
#> isbn loan_id member loan_date
#> <chr> <chr> <chr> <date>
#> 1 9780141439518 L001 Hugo Silva 2025-01-14
#> 2 9780141439518 L028 Ana Ruiz 2025-09-18
#> 3 9780141439518 L046 Ana Ruiz 2026-01-09
nrow(unnest(by_book, data)) == nrow(loans)
#> [1] TRUE
# A list-column is a list, so base R can summarise it too.
by_book |>
mutate(n_loans = vapply(data, nrow, integer(1))) |>
select(isbn, n_loans) |>
arrange(desc(n_loans)) |>
head(3)
#> # A tibble: 3 × 2
#> isbn n_loans
#> <chr> <int>
#> 1 9780001000100 8
#> 2 9780306406157 7
#> 3 9780013001331 7Rectangling nested lists
Data that comes from a web API or a JSON file is a list of records, each a list that can contain other lists and vectors, and the records do not all have the same fields. tidyr turns it into a table in two steps: put the list in a column of a tibble, then unnest it one level at a time. unnest_wider() makes each named element a column (a record that lacks the element gets NA), unnest_longer() makes each element of a vector a row, and hoist() pulls out the pieces named by a path, the way purrr::pluck() does, and keeps the rest. Each level of nesting is a decision, wider or longer, and the result of the first call shows the structure that is still to be unpacked: <named list [2]> is a record, <chr [2]> a vector, <NULL> an empty one.
The record in the block has a nested member (a record of its own, in which one member has no card) and a renewals vector (empty for one loan). names_sep prefixes the unnested names, which avoids collisions; unnest_longer() drops the records whose vector is empty unless keep_empty = TRUE, a behaviour that tidyr 1.3.0 made consistent for NULL and for empty vectors.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
# JSON-like data, built in code: a list of loan records, each with a nested
# member and a vector of renewal dates. Real JSON read with
# jsonlite::fromJSON(simplifyVector = FALSE) has the same shape.
records <- list(
list(
loan_id = "L001", book = "Pride and Prejudice",
member = list(name = "Hugo Silva", card = 7L),
renewals = c("2025-01-28", "2025-02-11")
),
list(
loan_id = "L002", book = "The Dispossessed",
member = list(name = "Ben Okafor", card = 12L),
renewals = character()
),
list(
loan_id = "L003", book = "Ten Winters",
member = list(name = "Dev Patel"),
renewals = "2025-02-03"
)
)
shelf <- tibble::tibble(record = records)
shelf
#> # A tibble: 3 × 1
#> record
#> <list>
#> 1 <named list [4]>
#> 2 <named list [4]>
#> 3 <named list [4]>
# unnest_wider(): each named element becomes a column.
wide <- shelf |>
unnest_wider(record)
wide
#> # A tibble: 3 × 4
#> loan_id book member renewals
#> <chr> <chr> <list> <list>
#> 1 L001 Pride and Prejudice <named list [2]> <chr [2]>
#> 2 L002 The Dispossessed <named list [2]> <NULL>
#> 3 L003 Ten Winters <named list [1]> <chr [1]>
# The nested member is a list-column again: unnest it with a prefix.
wide |>
unnest_wider(member, names_sep = "_")
#> # A tibble: 3 × 5
#> loan_id book member_name member_card renewals
#> <chr> <chr> <chr> <int> <list>
#> 1 L001 Pride and Prejudice Hugo Silva 7 <chr [2]>
#> 2 L002 The Dispossessed Ben Okafor 12 <NULL>
#> 3 L003 Ten Winters Dev Patel NA <chr [1]>
# unnest_longer(): each element of a vector becomes a row. Records without
# renewals disappear unless keep_empty = TRUE.
wide |>
select(loan_id, renewals) |>
unnest_longer(renewals)
#> # A tibble: 3 × 2
#> loan_id renewals
#> <chr> <chr>
#> 1 L001 2025-01-28
#> 2 L001 2025-02-11
#> 3 L003 2025-02-03
wide |>
select(loan_id, renewals) |>
unnest_longer(renewals, keep_empty = TRUE, values_to = "renewed")
#> # A tibble: 4 × 2
#> loan_id renewed
#> <chr> <chr>
#> 1 L001 2025-01-28
#> 2 L001 2025-02-11
#> 3 L002 <NA>
#> 4 L003 2025-02-03hoist() is the shortcut when only a few fields are wanted from deep inside the records. Each argument is new_name = path, where the path is a character vector of names, an integer vector of positions, or a list that mixes both (list("renewals", 1L) is the first renewal). A component that is absent is NA; hoist() has no .default. .remove = FALSE keeps the extracted components in the list, .ptype checks the type of each new column and .transform converts each component as it is extracted. Elements that have no names cannot become columns without names_sep, and the error says so.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
records <- list(
list(
loan_id = "L001", book = "Pride and Prejudice",
member = list(name = "Hugo Silva", card = 7L),
renewals = c("2025-01-28", "2025-02-11")
),
list(
loan_id = "L002", book = "The Dispossessed",
member = list(name = "Ben Okafor", card = 12L),
renewals = character()
),
list(
loan_id = "L003", book = "Ten Winters",
member = list(name = "Dev Patel"),
renewals = "2025-02-03"
)
)
shelf <- tibble::tibble(record = records)
# hoist() pulls out the pieces you name, by path, and keeps the rest.
shelf |>
hoist(
record,
"loan_id",
member_name = c("member", "name"),
card = list("member", "card"),
first_renewal = list("renewals", 1L)
)
#> # A tibble: 3 × 5
#> loan_id member_name card first_renewal record
#> <chr> <chr> <int> <chr> <list>
#> 1 L001 Hugo Silva 7 2025-01-28 <named list [3]>
#> 2 L002 Ben Okafor 12 <NA> <named list [3]>
#> 3 L003 Dev Patel NA 2025-02-03 <named list [3]>
# A missing element is NA (hoist() has no default argument). .ptype checks the
# type of a column, and .transform converts each component as it is extracted.
shelf |>
hoist(
record,
member_name = c("member", "name"),
card = list("member", "card"),
.transform = list(card = \(x) x * 100L),
.ptype = list(card = integer())
) |>
select(member_name, card)
#> # A tibble: 3 × 2
#> member_name card
#> <chr> <int>
#> 1 Hugo Silva 700
#> 2 Ben Okafor 1200
#> 3 Dev Patel NA
tryCatch(
hoist(shelf, record, card = list("member", "card"), .ptype = character()),
error = \(e) cat(conditionMessage(e), "\n")
)
#> Can't convert `..1` <integer> to <character>.
# Elements that are not named cannot become columns without names_sep.
tryCatch(
unnest_wider(tibble::tibble(x = list(1:2, 3:4)), x),
error = \(e) cat(conditionMessage(e), "\n")
)
#> ℹ In column: `x`.
#> ℹ In row: 1.
#> Caused by error:
#> ! Can't unnest elements with missing names.
#> ℹ Supply `names_sep` to generate automatic names.
unnest_wider(tibble::tibble(x = list(1:2, 3:4)), x, names_sep = "_")
#> # A tibble: 2 × 2
#> x_1 x_2
#> <int> <int>
#> 1 1 2
#> 2 3 4JSON read with the jsonlite package (see Serialization: RDS, RData and JSON; jsonlite is not run on this page) has the same shape as the lists above when it is read with simplifyVector = FALSE: an object is a named list and an array a list. With the default, simplifyVector = TRUE, jsonlite already rectangles the common cases, turning an array of objects into a data frame whose nested objects are data-frame columns. The rectangling article of tidyr works through real JSON examples, and the whole procedure is the one of the two blocks above: put the list in a column, unnest one level at a time, and decide at each level whether it is wider or longer.
Iterating with purrr
purrr is the tidyverse’s answer to the apply family. The idea is the same as vapply(): a function is applied to every element, and the suffix of the function name says what kind of vector you get back, so a result of the wrong shape is an error that names the element and not a surprise further down the script. The apply Family and Functionals has a full comparison table of purrr and base functions and its own runs of the error messages; this section covers what that page leaves out: the shapes the results come in, combining them, and turning errors into values. The data comes first and the function second in every purrr function, which is what a pipe wants. purrr needs R 4.1 or later (purrr 1.1.0 dropped support for older versions).
| purrr | Base R | Difference worth knowing |
|---|---|---|
|
|
|
|
|
one value per element, of exactly the declared type, with vctrs rules for conversion |
|
|
only length-1 inputs are recycled; a data frame is a list of columns for |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
return the error, a default or the captured messages as values |
map() and the type-specific variants
map(.x, .f, ..., .progress = FALSE) returns a list with the names of .x. Its siblings map_lgl(), map_int(), map_dbl() and map_chr() return an atomic vector and require every result to be a single value of that type. The conversions follow the vctrs rules: an integer can become a double, a double only an integer if no precision is lost, and a logical and the numbers 0 and 1 are interchangeable; a number is no longer converted to a string (deprecated in purrr 1.0.0, an error since purrr 1.2.0), so as.character() has to be explicit. map_vec() is the version for results that are not one of the four basic types (dates, factors, date-times): it keeps the class, where map_dbl() would have dropped it. modify() keeps the type of its input, so a data frame in is a data frame out.
.f can be written in four ways. A named function and an anonymous function (\(x) x + 1) are the ones to use. A formula (~ .x + 1) works, but the purrr documentation lists it as no longer recommended. A string, an integer or a list is a shortcut for pluck() on each element: "name" takes the element called name, 2L the second element, and list("tags", 1L) the first element of tags; a .default is returned where the element is missing or NULL. The documentation also advises against passing constant arguments through ... and in favour of a lambda that calls the function with them, which makes it clear which arguments belong to which function.
library(purrr)
source("scenario.R")
# map() always returns a list, with the names of its input.
pages <- list(short = c(88, 96), long = c(387, 432, 410))
map(pages, mean)
#> $short
#> [1] 92
#>
#> $long
#> [1] 409.6667
#>
# The suffix says the type of the result: exactly one value of that type each.
map_dbl(pages, mean)
#> short long
#> 92.0000 409.6667
map_int(pages, length)
#> short long
#> 2 3
map_chr(pages, \(p) paste(range(p), collapse = "-"))
#> short long
#> "88-96" "387-432"
map_lgl(pages, \(p) any(p > 400))
#> short long
#> FALSE TRUE
# A character, integer or list passed as .f extracts by name or position.
members <- list(
list(name = "Ana Ruiz", card = 7L, tags = c("staff", "poetry")),
list(name = "Ben Okafor", card = 12L, tags = "crime")
)
map_chr(members, "name")
#> [1] "Ana Ruiz" "Ben Okafor"
map_int(members, "card")
#> [1] 7 12
map_chr(members, list("tags", 1L))
#> [1] "staff" "crime"
map_chr(members, "email", .default = NA_character_)
#> [1] NA NA
# map_vec() keeps richer classes, where map_dbl() would drop them.
dates <- list(a = as.Date("2026-06-30"), b = as.Date("2026-07-01"))
map_vec(dates, \(d) d + 7)
#> a b
#> "2026-07-07" "2026-07-08"
map_dbl(dates, \(d) as.numeric(d + 7))
#> a b
#> 20641 20642
# Data frame in, one result per column.
map_dbl(books[c("year", "pages")], \(col) mean(col, na.rm = TRUE))
#> year pages
#> 1998.7333 268.2143
map_int(books, \(col) sum(is.na(col)))
#> title author isbn year genre pages
#> 0 0 0 1 0 2
# modify() keeps the type of the input: a data frame in, a data frame out.
books[1:3, c("title", "author")] |>
modify(\(col) toupper(trimws(col)))
#> title author
#> 1 THE DISPOSSESSED URSULA K. LE GUIN
#> 2 PRIDE AND PREJUDICE JANE AUSTEN
#> 3 THE QUIET STREETS MARA QUILLSource: purrr package documentation: map
map2(), pmap(), imap() and walk()
map2() walks two vectors in step and pmap() any number: its first argument is a list of the arguments, matched by name or position, and a data frame is such a list (one element per column), so pmap() is the row-wise map of a data frame. Since purrr 1.0.0 these functions follow the tidyverse recycling rules: only a length-1 input is recycled and the others must have the same length. imap() calls the function with each element and its name, or its position when the vector has no names. walk(), walk2(), pwalk() and iwalk() are for the side effect of the function (printing, writing a file); they return their first input invisibly so that they can sit in the middle of a pipeline.
library(purrr)
source("scenario.R")
# map2() walks two vectors in step; the lengths must match (or be 1).
title <- trimws(books$title[1:3])
author <- books$author[1:3]
map2_chr(title, author, \(t, a) paste0(t, " - ", a))
#> [1] "The Dispossessed - Ursula K. Le Guin"
#> [2] "Pride and Prejudice - Jane Austen"
#> [3] "the quiet streets - Mara Quill"
# pmap() takes a list (or a data frame) of the arguments, matched by name.
books[1:3, c("title", "year", "pages")] |>
pmap_chr(\(title, year, pages) {
sprintf("%s (%s): %s pages", trimws(title), year, pages)
})
#> [1] "The Dispossessed (1974): 387 pages"
#> [2] "Pride and Prejudice (1813): 432 pages"
#> [3] "the quiet streets (2019): 214 pages"
# imap() passes the name, or the position when there is no name.
imap_chr(c(a = "x", b = "y"), \(value, name) paste0(name, "=", value))
#> a b
#> "a=x" "b=y"
imap_chr(c("x", "y"), \(value, i) paste0(i, "=", value))
#> [1] "1=x" "2=y"
# walk() is for side effects: it returns its input invisibly, so it can sit in
# the middle of a pipeline.
loans <- loans[nzchar(loans$member), ]
loans_by_member <- split(loans$loan_id, trimws(tolower(loans$member)))
loans_by_member[1:3] |>
iwalk(\(ids, member) cat(member, ":", length(ids), "loans\n"))
#> ana ruiz : 14 loans
#> ben okafor : 5 loans
#> chloe martin : 9 loans
walk(c("first", "second"), \(s) cat("shelf:", s, "\n"))
#> shelf: first
#> shelf: second
# The result of walk() is its input, invisibly.
res <- walk(1:2, \(i) i)
identical(res, 1:2)
#> [1] TRUE
# map2() and pmap() recycle only length-1 inputs.
tryCatch(map2(1:2, 1:3, `+`), error = \(e) cat(conditionMessage(e), "\n"))
#> Can't recycle `.x` (size 2) to match `.y` (size 3).
map2_dbl(1:3, 10, `+`)
#> [1] 11 12 13Combining the results
A list of data frames is combined with list_rbind() and list_cbind(), and a list of vectors with list_c(), all added in purrr 1.0.0. They replace map_dfr() and map_dfc(), whose names suggested the strict single-value contract of map_dbl() but which accepted results of any size and relied on dplyr::bind_rows(). The purrr documentation marks map_dfr() and its relatives superseded, which means that they still work and receive only critical bug fixes: map(x, f) |> list_rbind() is the replacement. list_flatten() removes one level of nesting and replaces flatten(); its name_spec is a glue template for the names ("{outer}_{inner}" by default), and is_node (purrr 1.2.0) says which elements count as nodes. list_simplify() reduces a list of length-1 vectors to one vector (the simplify() replacement), and list_transpose() turns a list of records into a record of vectors (the transpose() replacement).
library(purrr)
source("scenario.R")
# Per-member loan counts as a list of one-row data frames ...
loans <- loans[nzchar(loans$member), ]
loans$member <- trimws(tolower(loans$member))
by_member <- split(loans, loans$member)
counts <- map(by_member, \(d) {
data.frame(n = nrow(d), first = min(d$loan_date, na.rm = TRUE))
})
# ... list_rbind() stacks them; names_to turns the list names into a column.
list_rbind(counts)
#> n first
#> 1 14 2025-03-27
#> 2 5 2025-01-16
#> 3 9 2025-04-23
#> 4 6 2025-01-21
#> 5 6 2025-01-26
#> 6 2 2025-12-11
#> 7 7 2025-03-11
#> 8 10 2025-01-14
list_rbind(counts, names_to = "member") |>
head(3)
#> member n first
#> 1 ana ruiz 14 2025-03-27
#> 2 ben okafor 5 2025-01-16
#> 3 chloe martin 9 2025-04-23
# list_cbind() binds side by side; list_c() concatenates vectors.
list_c(list(1:2, 3:4, 5L))
#> [1] 1 2 3 4 5
# list_flatten() removes one level of nesting; name_spec controls the names.
nested <- list(a = list(x = 1, y = 2), b = list(z = 3))
list_flatten(nested)
#> $a_x
#> [1] 1
#>
#> $a_y
#> [1] 2
#>
#> $b_z
#> [1] 3
#>
list_flatten(nested, name_spec = "{outer}/{inner}")
#> $`a/x`
#> [1] 1
#>
#> $`a/y`
#> [1] 2
#>
#> $`b/z`
#> [1] 3
#>
# list_simplify() reduces a list of length-1 vectors to a vector; with
# strict = FALSE it leaves a list that cannot be simplified.
list_simplify(list(1L, 2L, 3L))
#> [1] 1 2 3
list_simplify(list(1L, 2:3), strict = FALSE)
#> [[1]]
#> [1] 1
#>
#> [[2]]
#> [1] 2 3
#>Selecting and folding
keep() and discard() select the elements for which a predicate is TRUE and FALSE; keep_at() and discard_at() (purrr 1.0.0) select by name, and detect() and detect_index() return the first match; every(), some() and none() summarise a predicate over the whole input, and since purrr 1.2.0 the predicate must return a logical scalar. reduce(.x, .f, ..., .init, .dir) folds the list from the left with a two-argument function, and accumulate() keeps the intermediate values; with .dir = "backward" they fold from the right (the right = TRUE of Reduce(); the older reduce_right() was removed in purrr 1.2.0). reduce2() takes a second list for the third argument of the function.
library(purrr)
source("scenario.R")
# keep() and discard() select by a predicate (compare Filter()).
keep(books$pages, \(p) !is.na(p) && p > 400)
#> [1] 432 410
discard(books$pages, is.na) |> length()
#> [1] 14
# keep_at() and discard_at() select by name.
cols <- map(books, \(col) sum(is.na(col)))
keep_at(cols, c("year", "pages")) |> names()
#> [1] "year" "pages"
# detect(), detect_index(), every(), some() and none().
detect(books$pages, \(p) !is.na(p) && p > 400)
#> [1] 432
detect_index(books$pages, \(p) !is.na(p) && p > 400)
#> [1] 2
every(books$title, is.character)
#> [1] TRUE
some(books$pages, is.na)
#> [1] TRUE
none(books$pages, \(p) isTRUE(p > 1000))
#> [1] TRUE
# reduce() folds a list into one value; .init supplies the start value.
reduce(1:4, `+`)
#> [1] 10
reduce(list(1:3, 2:5, 3:6), intersect)
#> [1] 3
reduce(c("a", "b", "c"), paste, sep = "/")
#> [1] "a/b/c"
# accumulate() keeps the intermediate results.
accumulate(1:4, `+`)
#> [1] 1 3 6 10
accumulate(c("a", "b", "c"), paste0)
#> [1] "a" "ab" "abc"
# A fold over data frames: join one table to the next by loan_id.
parts <- list(
loans[1:3, c("loan_id", "member")],
loans[1:3, c("loan_id", "loan_date")],
loans[1:3, c("loan_id", "return_date")]
)
reduce(parts, merge, by = "loan_id")
#> loan_id member loan_date return_date
#> 1 L001 Hugo Silva 2025-01-14 2025-02-04
#> 2 L002 Ben Okafor 2025-01-16 2025-01-26
#> 3 L003 Dev Patel 2025-01-21 2025-01-22Turning errors into values
safely(), possibly() and quietly() take a function and return a function that does not stop. safely(.f) returns a list with the elements result and error, exactly one of them NULL, so a map over a whole vector gives a list of such pairs, and list_transpose() turns it into one list of results and one of errors. possibly(.f, otherwise) returns otherwise instead of the error; its default NULL (purrr 1.0.0) is replaced here by NA_real_ so the result can be a numeric vector. quietly(.f) captures the printed output, the messages and the warnings of a call, and returns them with the result. insistently() retries a function with a rate (rate_delay(), rate_backoff()), which is the right tool for a flaky web request and is not run here. All of them build on tryCatch() and withCallingHandlers() of Conditions and Error Handling, and their error is the condition object, so its message and call can be read.
library(purrr)
# safely() wraps a function so it returns list(result, error) and never stops.
safe_log <- safely(log)
safe_log(100)
#> $result
#> [1] 4.60517
#>
#> $error
#> NULL
#>
str(safe_log("a"))
#> List of 2
#> $ result: NULL
#> $ error :List of 2
#> ..$ message: chr "non-numeric argument to mathematical function"
#> ..$ call : language .f(...)
#> ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"
# Over a vector: transpose the list of pairs into results and errors.
res <- map(list(1, 10, "a", 100), safe_log) |>
list_transpose()
res$result
#> [[1]]
#> [1] 0
#>
#> [[2]]
#> [1] 2.302585
#>
#> [[3]]
#> NULL
#>
#> [[4]]
#> [1] 4.60517
#>
map_lgl(res$error, is.null)
#> [1] TRUE TRUE FALSE TRUE
# possibly() returns a default instead of the error. The default is NULL
# unless otherwise is given (purrr 1.0.0).
possible_log <- possibly(log, otherwise = NA_real_)
map_dbl(list(1, 10, "a", 100), possible_log)
#> [1] 0.000000 2.302585 NA 4.605170
# quietly() captures output, messages and warnings as well as the result.
loud <- \(x) {
cat("computing\n")
message("about to take the log")
warning("negative input")
log(abs(x))
}
q <- quietly(loud)(-1)
names(q)
#> [1] "result" "output" "warnings" "messages"
q$output
#> [1] "computing"
q$messages
#> [1] "about to take the log\n"
q$warnings
#> [1] "negative input"
q$result
#> [1] 0What purrr stops on
The strictness is the point. The block shows the text of the failures that the map family raises, each preceded by the iteration that failed. Since purrr 1.0.0 the error raised by the function inside map() is wrapped in an error that tracks the iteration, and since purrr 1.0.1 that wrapper has the class purrr_error_indexed, records the location (the position) and the name (for a named input) and keeps the original condition as parent. A progress bar is requested with .progress = TRUE.
library(purrr)
msg <- function(expr) {
res <- tryCatch(expr, error = identity, warning = identity)
cat(conditionMessage(res), "\n")
}
# A result of the wrong length or the wrong type (compare vapply()).
msg(map_dbl(1:3, \(x) c(x, x)))
#> ℹ In index: 1.
#> Caused by error:
#> ! Result must be length 1, not 2.
msg(map_chr(1:2, \(x) x))
#> ℹ In index: 1.
#> Caused by error:
#> ! Can't coerce from an integer to a string.
msg(map_int(1:2, \(x) x + 0.5))
#> ℹ In index: 1.
#> Caused by error:
#> ! Can't coerce from a number to an integer.
msg(map_dbl(c("a", "b"), \(x) x))
#> ℹ In index: 1.
#> Caused by error:
#> ! Can't coerce from a string to a double.
# The error is wrapped: it knows the index and, for named input, the name.
err <- tryCatch(map_dbl(list(a = 1, b = 1:2), \(x) x), error = identity)
class(err)
#> [1] "purrr_error_indexed" "rlang_error" "error"
#> [4] "condition"
err$location
#> [1] 2
err$name
#> [1] "b"
class(err$parent)
#> [1] "simpleError" "error" "condition"
conditionMessage(err$parent)
#> [1] "Result must be length 1, not 2."
# A function that fails inside map() reports the iteration too.
msg(map(list(1, "a"), \(x) x + 1))
#> ℹ In index: 2.
#> Caused by error in `x + 1`:
#> ! non-numeric argument to binary operator
# map2()/pmap() length rules and an NA in a predicate.
msg(map2(1:2, 1:3, `+`))
#> Can't recycle `.x` (size 2) to match `.y` (size 3).
msg(pmap(list(1:2, 1:3), `+`))
#> Can't recycle `.l[[1]]` (size 2) to match `.l[[2]]` (size 3).
msg(keep(c(1, NA, 3), \(x) x > 1))
#> ℹ In index: 2.
#> Caused by error:
#> ! `.p()` must return a single `TRUE` or `FALSE`, not `NA`.
msg(reduce(list(), `+`))
#> Must supply `.init` when `.x` is empty.
# An empty input gives an empty vector of the right type.
map_chr(list(), identity)
#> character(0)
map_dbl(NULL, identity)
#> numeric(0)Source: purrr package documentation: map
purrr can also run a map in parallel: wrapping .f in in_parallel() (purrr 1.1.0) sends it to daemons started with the mirai package, falling back to sequential evaluation when none are set. That is described on Parallel and Asynchronous Computing and is not run here.
Text with stringr
stringr wraps the stringi library, which uses the ICU (International Components for Unicode) library. Its functions all start with str_, take the vector of strings first and the pattern second, treat NA the same way everywhere (the result is NA, except where a function says otherwise) and, since stringr 1.5.0, follow the tidyverse recycling rules: only a length-1 pattern is recycled, and a longer one that does not match the length of the strings is an error. Since stringr 1.6.0 all the relevant functions preserve the names of their input. The base R functions they correspond to are taught on Strings and Regular Expressions, which also has a shorter version of the mapping; the table below follows the package’s own From base R article and lists the functions used on this page.
| Base R | stringr | Difference worth knowing |
|---|---|---|
|
|
pattern second; |
|
|
same result; the package gives each one a name |
|
|
backreferences are written |
|
|
one element per input, |
|
|
a list, as in base R |
|
|
a character matrix with the whole match and one column per group |
|
|
|
|
|
a matrix with |
|
|
|
|
|
negative positions count from the end; |
|
|
same values, including |
|
|
|
|
|
locale-aware, with a |
|
|
|
|
|
|
|
|
|
|
|
collation of the |
The first block runs the correspondences and checks several of them with identical(): str_detect() is grepl() with the arguments swapped, str_replace_all() is gsub(), and str_length() is nchar(). It shows the places where the two differ: str_extract() has one element per input where regmatches() drops the elements that did not match, and str_sub() with a negative position is the end of the string.
library(stringr)
source("scenario.R")
x <- c(" the quiet streets ", "THE LONG TABLE", NA)
# Detect: str_detect() is grepl() with the string first and the pattern second.
str_detect(books$genre, "iction")
#> [1] TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE
#> [13] TRUE TRUE TRUE FALSE
identical(str_detect(books$genre, "iction"), grepl("iction", books$genre))
#> [1] TRUE
str_starts(books$isbn, "978-0-0")
#> [1] FALSE FALSE FALSE TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE
#> [13] TRUE TRUE FALSE TRUE
# Replace: backreferences are \\1 in both; a named vector replaces several.
str_replace_all(books$isbn[1:2], "-", "")
#> [1] "9780306406157" "9780141439518"
identical(str_replace_all(books$isbn, "-", ""), gsub("-", "", books$isbn))
#> [1] TRUE
str_replace(books$author[8], "^(.*), (.*)$", "\\2 \\1")
#> [1] "Ursula K. Le Guin"
str_replace_all("a b c", c("a" = "A", "\\s+" = " "))
#> [1] "A b c"
# Extract: one element per input, NA when nothing matches.
str_extract(c("Loan L012", "none"), "[0-9]+")
#> [1] "012" NA
regmatches(c("Loan L012", "none"), regexpr("[0-9]+", c("Loan L012", "none")))
#> [1] "012"
str_extract("2026-06-30", "(\\d{4})-(\\d{2})", group = 2)
#> [1] "06"
str_match("2026-06-30", "(\\d{4})-(\\d{2})-(\\d{2})")
#> [,1] [,2] [,3] [,4]
#> [1,] "2026-06-30" "2026" "06" "30"
str_extract_all("L012 L034", "[0-9]+")
#> [[1]]
#> [1] "012" "034"
#>
# Split.
str_split("a,b,c", ",")
#> [[1]]
#> [1] "a" "b" "c"
#>
str_split_fixed("Le Guin, Ursula K.", ", ", 2)
#> [,1] [,2]
#> [1,] "Le Guin" "Ursula K."
str_split_i(c("a-b-c", "d-e"), "-", 2)
#> [1] "b" "e"
# Substrings: negative positions count from the end.
str_sub(books$isbn[1:3], -4)
#> [1] "15-7" "51-8" "0100"
substr(books$isbn[1:3], nchar(books$isbn[1:3]) - 3, nchar(books$isbn[1:3]))
#> [1] "15-7" "51-8" "0100"
str_sub("The Dispossessed", 5)
#> [1] "Dispossessed"
str_length(x)
#> [1] 20 14 NA
identical(str_length(x), nchar(x))
#> [1] TRUECleaning, joining and formatting
The shelf data is cleaned by a handful of calls. str_trim() removes leading and trailing white space and str_squish() also reduces inner runs of white space to one blank, which is the repair of "Iris Fenwick" and the double blanks of the titles. The case functions take a locale. str_c() and paste0() differ on NA: paste0("a", NA) is the text "aNA", str_c("a", NA) is NA. str_flatten() collapses a vector into one string, with a different separator for the last element (last =), and str_flatten_comma() is the comma-separated special case, which also gets the Oxford comma of a two-element list right. str_glue() wraps glue::glue(): the text between braces is an R expression evaluated in the calling environment, and a doubled brace is a literal brace. It is easier to read than sprintf() for text with several values, and sprintf() is better when a value needs a width or a number of decimals ("%5.2f"). str_interp(), an older interpolation function of the package (${expression}), is marked superseded in favour of str_glue(); it still works and gives no warning.
The last part of the block cleans four columns in one data.frame() call. The author column goes from ten distinct spellings to eight (the two orders of "Le Guin" collapse, and the two blanks of "Iris Fenwick" disappear), and the genre column from nine to six.
library(stringr)
source("scenario.R")
x <- c(" the quiet streets ", "THE LONG TABLE", NA)
# Trim, squish and change case.
str_trim(x)
#> [1] "the quiet streets" "THE LONG TABLE" NA
str_squish(" Iris Fenwick ")
#> [1] "Iris Fenwick"
identical(str_trim(x), trimws(x))
#> [1] TRUE
str_to_lower(x)
#> [1] " the quiet streets " "the long table" NA
str_to_title(str_to_lower(x))
#> [1] " The Quiet Streets " "The Long Table" NA
str_to_snake("Loans Per Genre")
#> [1] "loans_per_genre"
# Padding and truncating.
str_pad(c("7", "42"), width = 4, side = "left", pad = "0")
#> [1] "0007" "0042"
str_trunc("The Dispossessed", width = 10)
#> [1] "The Dis..."
# Joining: NA propagates in str_c() but is pasted as "NA" by paste0().
paste0("a", x)
#> [1] "a the quiet streets " "aTHE LONG TABLE" "aNA"
str_c("a", x)
#> [1] "a the quiet streets " "aTHE LONG TABLE" NA
str_c("loan-", sprintf("%03d", 1:3), collapse = "; ")
#> [1] "loan-001; loan-002; loan-003"
str_flatten(c("fiction", "poetry", "mystery"), collapse = ", ", last = " and ")
#> [1] "fiction, poetry and mystery"
# Interpolation: str_glue() and sprintf() side by side.
title <- "Ten Winters"
year <- 2022L
str_glue("{title} ({year})")
#> Ten Winters (2022)
sprintf("%s (%d)", title, year)
#> [1] "Ten Winters (2022)"
str_glue("{title} has {nchar(title)} characters")
#> Ten Winters has 11 characters
str_glue("{{literal}} braces and {year + 1L}")
#> {literal} braces and 2023
# The shelf cleaning: one pipeline for titles, authors, genres and ISBNs.
clean <- data.frame(
title = str_to_title(str_squish(books$title)),
author = str_squish(books$author) |>
str_replace("^(.*), (.*)$", "\\2 \\1"),
genre = str_to_lower(str_trim(books$genre)),
isbn = str_remove_all(books$isbn, "[^0-9X]")
)
clean[c(1, 3, 8, 10), ]
#> title author genre isbn
#> 1 The Dispossessed Ursula K. Le Guin science fiction 9780306406157
#> 3 The Quiet Streets Mara Quill fiction 9780001000100
#> 8 The Lamp At Noon Ursula K. Le Guin classic 9780006000624
#> 10 The Long Table Iris Fenwick fiction 9780008000851
n_unique <- \(x) length(unique(x))
c(authors = n_unique(books$author), cleaned = n_unique(clean$author))
#> authors cleaned
#> 10 8
c(genres = n_unique(books$genre), cleaned = n_unique(clean$genre))
#> genres cleaned
#> 9 6Pattern modifiers: regex(), fixed(), coll() and boundary()
A string used as a pattern is a regular expression. Wrapping it in a modifier changes what it means: regex() takes the flags ignore_case, multiline, comments and dotall; fixed() compares the bytes of a literal string and is what fixed = TRUE is in base R; coll() compares strings with the collation rules of a locale, which matters for case folding (the Turkish dotted and dotless i) and for characters that can be written in two ways; and boundary() matches the edges of characters, words, sentences or line breaks, so that str_split(x, boundary("word")) splits on words without a pattern. str_equal() compares two strings by Unicode equivalence, which == does not. str_escape() makes a literal string safe to paste into a regular expression, the stringr counterpart of the escaping routine of the base page.
The regular-expression dialect is the part that differs most from base R. Base R has three engines: TRE (the default, POSIX extended expressions), PCRE2 (perl = TRUE) and no regular expression at all (fixed = TRUE). stringr has one, ICU: it has lookaround, named groups, the \p{...} property classes and flags, and its \d and \w are Unicode-aware, so \d matches the Arabic-Indic digits that grepl() with the default engine did not match in the base page. It is not identical to PCRE2: the error messages are ICU’s own (they carry a constant such as U_REGEX_MISMATCHED_PAREN), and a few escapes differ, so a pattern copied from a perl = TRUE call is tested before it is trusted. The help for stringr’s modifiers and the package’s Regular expressions article describe the dialect.
library(stringr)
# A pattern is a regular expression unless it is wrapped in a modifier.
str_detect("a.b", ".")
#> [1] TRUE
str_detect("axb", fixed("."))
#> [1] FALSE
str_detect("a.b", fixed("."))
#> [1] TRUE
str_detect("FICTION", regex("fiction", ignore_case = TRUE))
#> [1] TRUE
str_detect("a\nb", regex("a.b", dotall = TRUE))
#> [1] TRUE
str_extract_all(
"a1 b22 c333",
regex("[a-z]\\d+ # a letter and digits", comments = TRUE)
)
#> [[1]]
#> [1] "a1" "b22" "c333"
#>
# coll() compares with the rules of a locale: case folding differs in Turkish.
str_detect("I", coll("i", ignore_case = TRUE))
#> [1] TRUE
str_detect("I", coll("i", ignore_case = TRUE, locale = "tr"))
#> [1] FALSE
str_equal("é", "é")
#> [1] TRUE
"é" == "é"
#> [1] FALSE
# boundary() splits on text boundaries instead of a pattern.
str_split("The Dispossessed is a novel.", boundary("word"))
#> [[1]]
#> [1] "The" "Dispossessed" "is" "a" "novel"
#>
str_count("The Dispossessed is a novel.", boundary("word"))
#> [1] 5
str_split("Hi there. Bye now.", boundary("sentence"))
#> [[1]]
#> [1] "Hi there. " "Bye now."
#>
# ICU regular expressions: digits are Unicode-aware, and lookaround and
# property classes such as \p{L} work. TRE (base R) does not match them.
str_detect(c("42", "x7", "١٢"), "\\d")
#> [1] TRUE TRUE TRUE
grepl("\\d", c("42", "x7", "١٢"))
#> [1] TRUE TRUE FALSE
str_extract_all("Zoë and Ürsula", "\\p{L}+")
#> [[1]]
#> [1] "Zoë" "and" "Ürsula"
#>
str_extract("loan L012 returned", "(?<=loan )L\\d+")
#> [1] "L012"
str_replace_all("camelCaseName", "(?<=[a-z])(?=[A-Z])", "_")
#> [1] "camel_Case_Name"
# A pattern from user input is escaped with str_escape() or used with fixed().
str_escape("a.b*c")
#> [1] "a\\.b\\*c"
str_detect("a.b*c", str_escape("a.b*c"))
#> [1] TRUEWhat stringr stops on
Most of the strictness was added in stringr 1.5.0 (2022-12-02): an empty pattern or a boundary() pattern in str_detect(), str_starts(), str_ends() and str_subset() is an error (str_detect(x, "") used to be TRUE for every non-empty string), an NA pattern is an error since stringr 1.6.0, and arguments of the wrong type are errors where they used to be warnings. The block prints the messages. Deprecated functions and arguments give a lifecycle warning of class lifecycle_warning_deprecated, once per session. An invalid pattern raises an error from ICU, which has no preceding warning, unlike the TRE and PCRE2 errors of base R. str_view() is the debugging tool: it prints the strings with the matches highlighted (since 1.5.0 with ANSI colour in the console and no HTML widget); here the colour is replaced by angle brackets.
library(stringr)
msg <- function(expr) {
res <- tryCatch(expr, error = identity, warning = identity)
cat(class(res)[[1L]], ":", conditionMessage(res), "\n")
}
# An invalid pattern is an ICU error, not a TRE one.
msg(str_detect("a", "("))
#> simpleError : Incorrectly nested parentheses in regex pattern. (U_REGEX_MISMATCHED_PAREN, context=`(`)
msg(str_detect("a", "[z-a]"))
#> simpleError : In a character range [x-y], x is greater than y. (U_REGEX_INVALID_RANGE, context=`[z-a]`)
# Since stringr 1.5.0 only length-1 inputs are recycled ...
msg(str_detect(letters, c("x", "y")))
#> vctrs_error_incompatible_size : Can't recycle `string` (size 26) to match `pattern` (size 2).
msg(str_c("a", 1:2, c("x", "y", "z")))
#> vctrs_error_incompatible_size : Can't recycle `..2` (size 2) to match `..3` (size 3).
# ... and patterns that make no sense are errors, not silent TRUE values.
msg(str_detect("abc", ""))
#> rlang_error : `pattern` can't be the empty string (`""`).
msg(str_detect("abc", boundary("word")))
#> rlang_error : `pattern` can't be a boundary.
msg(str_detect("abc", NA_character_))
#> rlang_error : `pattern` can not contain NAs.
msg(str_replace_all("abc", "b", 1))
#> rlang_error : `replacement` must be a character vector, not the number 1.
# Deprecated arguments and functions warn once per session.
msg(str_like("a", "A", ignore_case = TRUE))
#> lifecycle_warning_deprecated : The `ignore_case` argument of `str_like()` is deprecated as of stringr 1.6.0.
#> ℹ `str_like()` is always case sensitive.
#> ℹ Use `str_ilike()` for case insensitive string matching.
msg(str_view_all("abc", "b"))
#> lifecycle_warning_deprecated : `str_view_all()` was deprecated in stringr 1.5.0.
#> ℹ Please use `str_view()` instead.
# str_view() shows the matches in the console; the angle brackets stand for
# the highlighting that a colour terminal would use.
str_view(c("Midnight Ledger", "Ten Winters"), "[aeiou]")
#> [1] │ M<i>dn<i>ght L<e>dg<e>r
#> [2] │ T<e>n W<i>nt<e>rsFactor levels with forcats
forcats is the factor counterpart of stringr: every function starts with fct_, takes the factor first, validates its input (a non-factor, non-character input is an error since forcats 1.0.0) and returns a factor. The underlying type is the factor of Factors, an integer vector with a levels attribute, and forcats leaves the codes alone and rewrites the levels. The highlights are fct_relevel() (move some levels to the front, or after = another position), fct_infreq() (order by frequency), fct_lump_*() (collapse the rare levels into "Other"), fct_reorder() (order by a summary of another variable, the standard preparation of a bar chart for ggplot2), fct_recode() and fct_collapse() (rename and merge) and fct_count() (a tibble of counts, where table() returns a table). The factors page has a shorter forcats paragraph with the base R equivalents; the block below uses other functions of the package.
| forcats | Base R | Difference worth knowing |
|---|---|---|
|
|
several levels at once, and |
|
|
|
|
|
|
|
|
warns about missing values in |
|
|
a level that does not exist is a warning |
|
|
the last one turns |
|
|
an error when a value is not in |
The block works on the shelf genres, trimmed and lower-cased, which is the six-level factor of the factors page. fct_count() replaces table() when the counts are going to be used as data. The level order that fct_infreq() chooses is the order in which a bar chart or a table() will list them. fct_lump_n() has a ties.method because several levels can tie for the last place that is kept: the default "min" keeps all of them, so n = 2 can keep three levels.
library(forcats)
source("scenario.R")
genre <- factor(tolower(trimws(books$genre)))
levels(genre)
#> [1] "classic" "fiction" "mystery" "non-fiction"
#> [5] "poetry" "science fiction"
# fct_relevel(): move levels by name; fct_inorder(), fct_infreq(), fct_inseq().
levels(fct_relevel(genre, "poetry", "mystery"))
#> [1] "poetry" "mystery" "classic" "fiction"
#> [5] "non-fiction" "science fiction"
levels(fct_relevel(genre, "classic", after = Inf))
#> [1] "fiction" "mystery" "non-fiction" "poetry"
#> [5] "science fiction" "classic"
levels(fct_infreq(genre))
#> [1] "fiction" "non-fiction" "science fiction" "classic"
#> [5] "mystery" "poetry"
fct_count(genre, sort = TRUE)
#> # A tibble: 6 × 2
#> f n
#> <fct> <int>
#> 1 fiction 4
#> 2 non-fiction 3
#> 3 science fiction 3
#> 4 classic 2
#> 5 mystery 2
#> 6 poetry 2
# Lumping the rare levels: by count, by proportion or by a minimum count.
table(fct_lump_n(genre, n = 2, ties.method = "first"))
#>
#> fiction non-fiction Other
#> 4 3 9
table(fct_lump_prop(genre, prop = 0.2))
#>
#> fiction Other
#> 4 12
table(fct_lump_min(genre, min = 3, other_level = "rare"))
#>
#> fiction non-fiction science fiction rare
#> 4 3 3 6
# fct_recode() renames; fct_collapse() merges several levels into one.
fct_recode(genre, "sci-fi" = "science fiction")[1:3]
#> [1] sci-fi classic fiction
#> Levels: classic fiction mystery non-fiction poetry sci-fi
levels(fct_collapse(genre, factual = c("non-fiction", "classic")))
#> [1] "factual" "fiction" "mystery" "poetry"
#> [5] "science fiction"
levels(fct_other(genre, keep = c("fiction", "poetry")))
#> [1] "fiction" "poetry" "Other"
# fct_reorder(): sort the levels by a summary of another variable.
levels(fct_reorder(genre, books$pages, .fun = median, .na_rm = TRUE))
#> [1] "poetry" "fiction" "non-fiction" "mystery"
#> [5] "classic" "science fiction"
levels(
fct_reorder(genre, books$pages, .fun = median, .na_rm = TRUE, .desc = TRUE)
)
#> [1] "science fiction" "classic" "mystery" "non-fiction"
#> [5] "fiction" "poetry"
# fct_expand(), fct_drop() and fct_na_value_to_level() change the set of levels.
levels(fct_expand(genre, "drama"))
#> [1] "classic" "fiction" "mystery" "non-fiction"
#> [5] "poetry" "science fiction" "drama"
levels(fct_drop(genre[genre != "poetry"]))
#> [1] "classic" "fiction" "mystery" "non-fiction"
#> [5] "science fiction"
with_na <- factor(c("fiction", NA, "poetry"))
levels(fct_na_value_to_level(with_na, level = "unknown"))
#> [1] "fiction" "poetry" "unknown"The warnings and errors of the package are the stringr kind: they name the argument and what was wrong. fct_reorder() removes the missing values of the variable it summarises, and since forcats 1.0.0 it warns (.na_rm = TRUE silences the warning and .na_rm = FALSE keeps the NA values); fct_recode() and fct_relevel() warn about levels that do not exist, because a typo in a level name is the most common mistake; and fct() is factor() without the silent NA.
library(forcats)
source("scenario.R")
msg <- function(expr) {
res <- tryCatch(expr, error = identity, warning = identity)
if (inherits(res, "condition")) {
cat(class(res)[[1L]], ":", conditionMessage(res), "\n")
} else {
print(res)
}
}
genre <- factor(tolower(trimws(books$genre)))
# fct_recode() warns about a level that does not exist.
msg(fct_recode(genre, "sci-fi" = "scifi"))
#> simpleWarning : Unknown levels in `f`: scifi
# fct_reorder() removes the missing values of the summarised variable, with a
# warning, unless .na_rm says otherwise.
msg(fct_reorder(genre, books$pages))
#> rlang_warning : `fct_reorder()` removing 2 missing values.
#> ℹ Use `.na_rm = TRUE` to silence this message.
#> ℹ Use `.na_rm = FALSE` to preserve NAs.
# Input that is not a factor or character vector is an error, and so is
# moving a level that does not exist.
msg(fct_infreq(1:3))
#> rlang_error : `f` must be a factor or character vector, not an integer vector.
msg(fct_relevel(genre, "drama"))
#> rlang_warning : 1 unknown level in `f`: drama
# fct() is factor() that refuses values missing from the levels.
msg(fct(c("fiction", "drama"), levels = c("fiction", "poetry")))
#> rlang_error : All values of `x` must appear in `levels` or `na`
#> ℹ Missing level: "drama"
factor(c("fiction", "drama"), levels = c("fiction", "poetry"))
#> [1] fiction <NA>
#> Levels: fiction poetryDates and times with lubridate
lubridate does not replace the classes of Dates and Times: its parsers return ordinary Date and POSIXct objects, so the result works with every base function. What it adds is a shorter way to parse, to read and set the parts of a date, and three classes for spans of time (Interval, Period and Duration) with arithmetic that follows the calendar. Since lubridate 1.9.0 (2022-11-06) the computations on time zones and on updating dates are done by the timechange package. The first block compares the common tasks with base R:
| Task | Base R | lubridate |
|---|---|---|
parse an ISO date or date-time |
|
|
parse another order of the parts |
|
|
read a part |
|
|
month or weekday name |
|
|
first day of the month |
|
|
difference in days |
|
|
add one calendar month |
|
|
a date-time in another zone |
|
|
Parsing and reading the parts
The parsers are named after the order of the parts in the text and guess the separators: ymd() accepts 2026-06-30, 2026/06/30, 20260630 and 2026 Jun 30, dmy() and mdy() the other orders, and ymd_hms(), ymd_hm() and ymd_h() add a time. The date-time parsers take tz (the default is "UTC"). parse_date_time(x, orders = ) accepts a vector of orders for text that mixes formats. A text that cannot be parsed is NA with a warning. The accessors read a part (year(), month(), day(), wday(), yday(), quarter(), week()) and most of them can also assign one, as in month(d) <- 7. month(label = TRUE) and wday(label = TRUE) return ordered factors, named in the language of the LC_TIME locale (here English), and abbr = FALSE gives the full names. wday() counts from Sunday unless week_start = 1 or the option lubridate.week.start says otherwise. Assigning a part gives NA if the result does not exist: month(d) <- 2 on 2026-06-30 would be 30 February, which is why the example assigns July.
library(lubridate, warn.conflicts = FALSE)
source("scenario.R")
# The parsers are named for the order of the parts and guess the separators.
ymd("2026-06-30")
#> [1] "2026-06-30"
ymd(c("2026-06-30", "2026/07/01", "20260702", "2026 Jul 3"))
#> [1] "2026-06-30" "2026-07-01" "2026-07-02" "2026-07-03"
dmy("30/06/2026")
#> [1] "2026-06-30"
mdy("June 30, 2026")
#> [1] "2026-06-30"
ymd_hms("2026-06-30 09:30:00", tz = "UTC")
#> [1] "2026-06-30 09:30:00 UTC"
ymd_hm("2026-06-30 09:30", tz = "Europe/Madrid")
#> [1] "2026-06-30 09:30:00 CEST"
class(ymd("2026-06-30"))
#> [1] "Date"
class(ymd_hms("2026-06-30 09:30:00"))
#> [1] "POSIXct" "POSIXt"
# Several formats in one vector: parse_date_time() with orders.
parse_date_time(
c("30/06/2026", "2026-07-01", "Jul 2 2026"),
orders = c("dmy", "ymd", "mdy")
)
#> [1] "2026-06-30 UTC" "2026-07-01 UTC" "2026-07-02 UTC"
# A failure is NA with a warning that says how many.
tryCatch(ymd(c("2026-02-30", "2026-03-01")), warning = \(w) conditionMessage(w))
#> [1] " 1 failed to parse."
suppressWarnings(ymd(c("2026-02-30", "2026-03-01")))
#> [1] NA "2026-03-01"
# The accessors read a part, and many can also assign one.
d <- ymd("2026-06-30")
c(year(d), month(d), day(d), wday(d), yday(d), quarter(d), week(d))
#> [1] 2026 6 30 3 181 2 26
month(d, label = TRUE)
#> [1] Jun
#> 12 Levels: Jan < Feb < Mar < Apr < May < Jun < Jul < Aug < Sep < ... < Dec
month(d, label = TRUE, abbr = FALSE)
#> [1] June
#> 12 Levels: January < February < March < April < May < June < ... < December
wday(d, label = TRUE)
#> [1] Tue
#> Levels: Sun < Mon < Tue < Wed < Thu < Fri < Sat
wday(d, label = TRUE, week_start = 1)
#> [1] Tue
#> Levels: Mon < Tue < Wed < Thu < Fri < Sat < Sun
month(d) <- 7
d
#> [1] "2026-07-30"
update(ymd("2026-06-30"), year = 2027, month = 1, day = 15)
#> [1] "2027-01-15"
# The loan dates of the shelf data: the impossible 2025-02-30 is NA, and the
# calendar months of both years are counted together.
raw <- read.csv("loans.csv", colClasses = "character")
parsed <- suppressWarnings(ymd(raw$loan_date))
is.na(parsed[20])
#> [1] TRUE
sum(is.na(parsed))
#> [1] 1
table(month(parsed, label = TRUE))
#>
#> Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
#> 8 5 6 7 5 3 7 1 4 7 3 3Intervals, comparisons and rounding
An Interval is the span between two specific instants, built with interval(start, end) or with the operator %--%. It knows its start and end (int_start(), int_end()), its length in seconds (int_length()) and can be converted to a period or measured in any unit with time_length(), which is the right way to say "days between two dates" when months and years are involved. %within% asks if a date, or an interval, lies inside an interval, and int_overlaps() whether two intervals share any time (lubridate 1.9.0 fixed %within% for intervals that run backwards). The loans of the shelf data are intervals from the loan date to the return date, and the 25 overdue returned loans are the ones longer than loan_days. floor_date(), ceiling_date() and round_date() round to a unit ("day", "week", "month", "quarter", "year" and the smaller ones), with week_start for the first day of the week, which is how the loans per quarter are counted.
library(lubridate, warn.conflicts = FALSE)
source("scenario.R")
# An interval has a start and an end: interval() or the %--% operator.
loan <- ymd("2026-06-09") %--% ymd("2026-06-30")
loan
#> [1] 2026-06-09 UTC--2026-06-30 UTC
class(loan)
#> [1] "Interval"
#> attr(,"package")
#> [1] "lubridate"
int_start(loan)
#> [1] "2026-06-09 UTC"
int_end(loan)
#> [1] "2026-06-30 UTC"
int_length(loan)
#> [1] 1814400
as.period(loan)
#> [1] "21d 0H 0M 0S"
time_length(loan, "day")
#> [1] 21
time_length(loan, "week")
#> [1] 3
# %within% asks whether a date (or an interval) is inside another interval.
ymd("2026-06-15") %within% loan
#> [1] TRUE
ymd("2026-07-15") %within% loan
#> [1] FALSE
june <- interval(ymd("2026-06-01"), ymd("2026-06-30"))
loan %within% june
#> [1] TRUE
int_overlaps(loan, interval(ymd("2026-06-25"), ymd("2026-07-10")))
#> [1] TRUE
# Intervals of the shelf loans: loan date to return date, 21 days allowed.
returned <- loans[!is.na(loans$loan_date) & !is.na(loans$return_date), ]
returned <- returned[returned$return_date >= returned$loan_date, ]
spans <- interval(returned$loan_date, returned$return_date)
length(spans)
#> [1] 51
summary(time_length(spans, "day"))
#> Min. 1st Qu. Median Mean 3rd Qu. Max.
#> 1.00 15.00 21.00 20.33 26.50 37.00
sum(time_length(spans, "day") > loan_days)
#> [1] 25
sum(returned$return_date - returned$loan_date > loan_days)
#> [1] 25
# Rounding dates to a unit.
d <- ymd("2026-06-30")
floor_date(d, "month")
#> [1] "2026-06-01"
ceiling_date(d, "month")
#> [1] "2026-07-01"
round_date(d, "quarter")
#> [1] "2026-07-01"
floor_date(d, "week", week_start = 1)
#> [1] "2026-06-29"
floor_date(ymd_hms("2026-06-30 14:45:10", tz = "UTC"), "hour")
#> [1] "2026-06-30 14:00:00 UTC"
# Returned loans per quarter, with the quarters as dates.
table(floor_date(returned$loan_date, "quarter"))
#>
#> 2025-01-01 2025-04-01 2025-07-01 2025-10-01 2026-01-01
#> 9 7 12 13 10Periods and durations
A period is a span in calendar units, a duration a span in exact seconds, and the two differ whenever the calendar does not have a fixed length. days(1) is "one day on the clock" and ddays(1) is 86400 seconds. Across the change to daylight saving time the first keeps the clock reading (noon is still noon the next day) and the second adds 24 hours of elapsed time, so the two results are an hour apart. Months and years are written as periods (months(), years()) because a month has no fixed length; the duration dmonths() converts a month into a fixed number of seconds. Adding a period that lands on a day that does not exist gives NA; %m+% and %m-% are the operators that roll back to the last valid day, and they are what to use for "same day next month". The Europe/Madrid change of 2026-03-29 and the leap day of 2024-02-29 are the examples.
| Class | Made by | Is |
|---|---|---|
|
|
the span between two instants, with a start and an end |
|
|
a number of calendar units; the length in seconds depends on where it is added |
|
|
a number of seconds; always the same length |
Time zones have two operations that look alike. with_tz() keeps the instant and changes how it is displayed, which is what format(x, tz = ) does in base R; force_tz() keeps the clock reading and changes the instant, which is a re-interpretation of the same text in another zone and is the repair for a date-time that was read with the wrong zone. The block ends with the check that decides which one an operation is: the first is == to the original, the second is not.
library(lubridate, warn.conflicts = FALSE)
# A period is a calendar span ("one day"), a duration an exact number of
# seconds ("86400 seconds"). They agree until a clock change intervenes.
days(1)
#> [1] "1d 0H 0M 0S"
ddays(1)
#> [1] "86400s (~1 days)"
is.period(days(1))
#> [1] TRUE
is.duration(ddays(1))
#> [1] TRUE
as.numeric(ddays(1))
#> [1] 86400
# Daylight saving time: Europe/Madrid moved the clocks on 2026-03-29.
before <- ymd_hms("2026-03-28 12:00:00", tz = "Europe/Madrid")
before + days(1)
#> [1] "2026-03-29 12:00:00 CEST"
before + ddays(1)
#> [1] "2026-03-29 13:00:00 CEST"
# Months and years have no fixed length: periods follow the calendar. Adding
# a month to January 31 gives the impossible February 31, which is NA.
jan31 <- ymd("2026-01-31")
jan31 + months(1)
#> [1] NA
jan31 %m+% months(1)
#> [1] "2026-02-28"
jan31 %m-% months(2)
#> [1] "2025-11-30"
ymd("2024-02-29") + years(1)
#> [1] NA
ymd("2024-02-29") %m+% years(1)
#> [1] "2025-02-28"
ymd("2026-06-30") + weeks(3)
#> [1] "2026-07-21"
# Arithmetic with periods and durations.
days(2) + hours(3)
#> [1] "2d 3H 0M 0S"
dhours(3) + dminutes(30)
#> [1] "12600s (~3.5 hours)"
as.duration(days(2))
#> [1] "172800s (~2 days)"
as.period(ddays(2) + dhours(3))
#> [1] "2d 3H 0M 0S"
# seq(by = "month") overflows the short months; %m+% rolls back instead.
seq(ymd("2026-01-31"), by = "month", length.out = 3)
#> [1] "2026-01-31" "2026-03-03" "2026-03-31"
ymd("2026-01-31") %m+% months(0:2)
#> [1] "2026-01-31" "2026-02-28" "2026-03-31"
# Time zones: with_tz() keeps the instant, force_tz() keeps the clock.
x <- ymd_hms("2026-06-30 09:30:00", tz = "UTC")
with_tz(x, "Europe/Madrid")
#> [1] "2026-06-30 11:30:00 CEST"
force_tz(x, "Europe/Madrid")
#> [1] "2026-06-30 09:30:00 CEST"
with_tz(x, "America/New_York")
#> [1] "2026-06-30 05:30:00 EDT"
tz(x)
#> [1] "UTC"
x == with_tz(x, "Asia/Tokyo")
#> [1] TRUE
x == force_tz(x, "Asia/Tokyo")
#> [1] FALSEFailures and impossible values
A parse failure is a warning whose text begins with a blank and the count of failed elements (the text is 1 failed to parse.), or All formats failed to parse. No formats found. when none of them fits; the value is NA. A date-time that does not exist because the clocks skipped it is a failure of the same kind. A period has whole-number fields, and an unknown unit is an error that names the unit. The block prints the messages, and the base R counterparts are in Dates and Times.
library(lubridate, warn.conflicts = FALSE)
msg <- function(expr) {
res <- tryCatch(expr, error = identity, warning = identity)
if (inherits(res, "condition")) {
cat(class(res)[[1L]], ":", conditionMessage(res), "\n")
} else {
print(res)
}
}
# A parse failure is a warning, and the value is NA. The text begins with a
# blank, and the count is the number of failed elements.
msg(ymd("not a date"))
#> simpleWarning : All formats failed to parse. No formats found.
msg(ymd(c("2026-06-30", "oops", NA, "2026-13-01")))
#> simpleWarning : 2 failed to parse.
suppressWarnings(ymd(c("2026-06-30", "oops", NA, "2026-13-01")))
#> [1] "2026-06-30" NA NA NA
# A time that does not exist is a failure too: the clocks of Europe/Madrid
# jumped from 02:00 to 03:00 on 2026-03-29.
msg(ymd_hms("2026-03-29 02:30:00", tz = "Europe/Madrid"))
#> simpleWarning : 1 failed to parse.
ymd_hms("2026-03-29 03:30:00", tz = "Europe/Madrid")
#> [1] "2026-03-29 03:30:00 CEST"
# Periods have whole-number fields; units and rounding units are checked.
msg(days(1.5))
#> simpleError : invalid class “Period” object: periods must have integer values
msg(period(1, "fortnight"))
#> simpleError : Invalid unit name: fortnight
msg(floor_date(ymd("2026-06-30"), "fortnight"))
#> simpleError : Invalid unit specification 'fortnight'
#>
days(1) + hours(36)
#> [1] "1d 36H 0M 0S"Putting it together: the shelf loans
The last script uses all six packages on the running scenario, with the pipeline the earlier sections took apart. stringr cleans the text columns of books, forcats orders the genre factor by frequency, lubridate turns each loan into an interval and measures it in days (a loan that is still out ends on shelf_today), tidyr nests the loans of each book into a list-column, purrr maps three summaries over it, and tidyr pivots the final table. The ISBN 978030640615 (one digit short) matches no book, so those loans disappear at the inner join; the loan returned before it was lent has a negative days_out and is filtered out before the summaries. The overdue count and the rate per genre can be compared with the base R analyses of Data Frames.
library(dplyr, warn.conflicts = FALSE)
library(tidyr)
library(purrr)
library(stringr)
library(forcats)
library(lubridate, warn.conflicts = FALSE)
source("scenario.R")
# Clean the text with stringr and order the genre factor by frequency.
shelf_books <- books |>
mutate(
title = str_to_title(str_squish(title)),
author = str_squish(author) |> str_replace("^(.*), (.*)$", "\\2 \\1"),
genre = str_to_lower(str_trim(genre)) |> fct_infreq(),
isbn = str_remove_all(isbn, "[^0-9X]")
)
levels(shelf_books$genre)
#> [1] "fiction" "non-fiction" "science fiction" "classic"
#> [5] "mystery" "poetry"
# Days out: an interval from the loan to the return, or to shelf_today when
# the book is still out. Negative values are the impossible return dates.
shelf_loans <- loans |>
filter(!is.na(loan_date)) |>
mutate(
member = str_to_title(str_squish(member)),
end = coalesce(return_date, shelf_today),
days_out = time_length(interval(loan_date, end), "day")
)
range(shelf_loans$days_out)
#> [1] -3 84
# Nest the loans of each book and map a summary over the list-column.
per_book <- shelf_loans |>
filter(days_out >= 0) |>
nest(.by = isbn) |>
mutate(
n_loans = map_int(data, nrow),
mean_days = map_dbl(data, \(d) mean(d$days_out)),
overdue = map_int(data, \(d) sum(d$days_out > loan_days))
) |>
select(!data) |>
inner_join(shelf_books[c("isbn", "title", "genre")], by = "isbn")
per_book |>
arrange(desc(overdue)) |>
head(4)
#> # A tibble: 4 × 6
#> isbn n_loans mean_days overdue title genre
#> <chr> <int> <dbl> <int> <chr> <fct>
#> 1 9780001000100 8 25.4 5 The Quiet Streets fiction
#> 2 9780013001331 6 34.2 4 Ten Winters fiction
#> 3 9780306406157 7 27.6 3 The Dispossessed science fiction
#> 4 9780010001044 4 31 3 Notes On Gravity science fiction
# Back to genres: the factor order from fct_infreq() orders the summary.
per_book |>
summarise(
books = n(), loans = sum(n_loans), overdue = sum(overdue),
.by = genre
) |>
arrange(genre) |>
mutate(overdue_rate = round(overdue / loans, 2))
#> # A tibble: 6 × 5
#> genre books loans overdue overdue_rate
#> <fct> <int> <int> <int> <dbl>
#> 1 fiction 4 22 12 0.55
#> 2 non-fiction 2 5 3 0.6
#> 3 science fiction 3 15 7 0.47
#> 4 classic 2 5 3 0.6
#> 5 mystery 2 4 2 0.5
#> 6 poetry 2 6 4 0.67
# The wide report: loans per year and genre, with a total column.
shelf_loans |>
inner_join(shelf_books[c("isbn", "genre")], by = "isbn") |>
count(year = year(loan_date), genre) |>
pivot_wider(
names_from = year, values_from = n, values_fill = 0L, names_prefix = "y"
) |>
arrange(genre) |>
mutate(total = y2025 + y2026)
#> # A tibble: 6 × 4
#> genre y2025 y2026 total
#> <fct> <int> <int> <int>
#> 1 fiction 16 7 23
#> 2 non-fiction 4 1 5
#> 3 science fiction 8 7 15
#> 4 classic 4 1 5
#> 5 mystery 4 0 4
#> 6 poetry 5 1 6Pitfalls
| Pitfall | What happens | What to do |
|---|---|---|
|
a warning and list-columns |
aggregate first, or pass |
|
an error that counts the values that were too short or too long |
use |
|
the records vanish from the result |
|
|
|
|
|
a list that is carried through the pipeline |
use |
|
they work but are superseded |
|
constant arguments after |
they work, and the purrr documentation recommends against them |
a lambda: |
|
it returns its input, not the value of the function |
use |
|
a wrong answer, or a recycling error when the lengths differ |
the string is the first argument, the pattern the second |
|
|
decide which one is wanted; |
|
an error since stringr 1.5.0 |
use |
|
|
|
|
|
|
|
the time zone is |
pass |
|
the names are in the language of |
set the locale (see Encodings, Locales and Internationalization) or pass |
|
whitespace is trimmed, empty text fields are |
check the result with |
Version notes
The R entries come from the R NEWS (R NEWS). The package entries come from the changelog (NEWS) page of each package, and only what those pages say is repeated here. The dates are the CRAN release dates printed on those pages; the readr 2.2.0 page has no date.
| R release | Change that matters here |
|---|---|
R 4.1.0 |
The native pipe operator and the function shorthand |
R 4.5.0 |
|
| tidyr | Change |
|---|---|
1.3.2 (2025-12-19) |
|
1.3.1 (2024-01-24) |
The |
1.3.0 (2023-01-24) |
New |
1.2.0 (2022-02-01) |
|
1.0.0 (2019-09-11) |
New |
| purrr | Change |
|---|---|
1.2.2 (2026-04-10) |
Fixes for CRAN checks. |
1.2.1 (2026-01-09) |
Compatibility with vctrs 0.7.0. |
1.2.0 (2025-11-04) |
Functions and arguments deprecated in purrr 0.3.0 are removed ( |
1.1.0 (2025-07-10) |
R 4.1 or later is required; |
1.0.1 (2023-01-10) |
The error that wraps a failure in |
1.0.0 (2022-12-20) |
New |
| stringr | Change |
|---|---|
1.6.0 (2025-11-04) |
All relevant functions preserve names; |
1.5.2 (2025-09-08) |
R CMD check fixes. |
1.5.0 (2022-12-02) |
Tidyverse recycling rules; |
1.4.0 (2019-02-10) |
|
1.3.0 (2018-02-19) |
|
| forcats | Change |
|---|---|
1.0.1 (2025-09-25) |
|
1.0.0 (2023-01-29) |
|
0.5.2 (2022-08-19) |
|
0.5.0 (2020-03-01) |
|
| lubridate | Change |
|---|---|
1.9.5 (2026-02-04) |
lubridate is MIT licensed; a fix to the orders of time stamps. |
1.9.4 (2024-12-08) |
|
1.9.3 (2023-09-27) |
|
1.9.0 (2022-11-06) |
Updating and time-zone computations are built on the timechange package; the |
| readr | Change |
|---|---|
2.2.0 |
|
2.1.0 (2021-11-11) |
Files are read eagerly by default again; |
2.0.0 (2021-07-20) |
The second edition parsing and writing engine, implemented by the vroom package; |
The tidyverse packages label their functions with the four lifecycle stages of the lifecycle package: experimental, stable, deprecated and superseded. A superseded function stays in the package and gets only critical bug fixes, and it gives no warning, which is why so many old tutorials still run (each superseded form named in the next section ran without a warning on the versions of this page). A deprecated function has a better alternative and is scheduled for removal, and calling it gives a warning once per session; the reduce_right() of purrr, removed in purrr 1.2.0, is an example of a function that went all the way.
Where older tutorials differ
Most of the code in older tutorials still runs, because the tidyverse supersedes before it deprecates. The table lists what current code does instead and why. Nothing in the first column is recommended.
| Older tutorial writes | Current practice | Why | Since |
|---|---|---|---|
|
|
the reference pages say development is complete and recommend the pivots, which handle several value columns and have clearer argument names |
tidyr 1.0.0 |
|
|
superseded: they remain with critical fixes only; the new functions make a mismatched number of pieces an error with a debug mode |
tidyr 1.3.0 |
|
|
the 1.0.0 syntax; the lazyeval functions ending in an underscore are defunct or deprecated |
tidyr 1.0.0 |
|
|
superseded: the |
purrr 1.0.0 |
|
|
the purrr documentation lists the formula lambda as no longer recommended, and the base shorthand needs no package |
R 4.1.0 |
|
an explicit |
the typed maps follow the strict vctrs conversions |
purrr 1.0.0, error in 1.2.0 |
|
|
they are deprecated in purrr 1.0.0 and give a lifecycle warning, for example |
purrr 1.0.0 |
|
|
superseded |
purrr 1.0.0 |
|
|
superseded; |
stringr 1.3.0 (added |
|
|
deprecated |
stringr 1.5.0, 1.6.0 |
|
a real pattern, and a vector of patterns of length one or of the same length |
the first returned |
stringr 1.5.0 |
|
|
|
forcats 1.0.0, 0.5.0 |
|
|
literal data without |
readr 2.2.0 |
|
|
the first is |
always |
|
attach the packages that are used, and the native pipe |
the pipe is in base R since 4.1.0, and the tidyverse documentation uses it |
R 4.1.0 |
|
|
|
always |
For the full list of changes by release see What’s New and Migrating Older Code.
Where to go next
-
The tidyverse and dplyr for the data-frame verbs (
filter(),mutate(),summarise(), joins) that these packages are used with, and tibbles. -
data.table for
melt()anddcast()as the data.table reshaping functions, andfread()as the fast reader. -
Data Frames for the base R
reshape(),merge(),aggregate()andsplit()that the tidyr verbs correspond to. -
The apply Family and Functionals and Vectorisation and Iteration for
lapply(),vapply(),Map()andReduce(). -
Strings and Regular Expressions for the base regex engines (TRE, PCRE2, fixed) and the base string functions.
-
Factors and Dates and Times for the base classes forcats and lubridate work on.
-
Serialization: RDS, RData and JSON for jsonlite and reading JSON before rectangling it.
-
Reading and Writing Text Data for
read.csv(), encodings and the readr counterparts. -
Conditions and Error Handling for
tryCatch()and the conditions behindsafely()andpossibly(). -
ggplot2 and the Grammar of Graphics for the plots that long tables and reordered factors are prepared for.
-
Parallel and Asynchronous Computing for
in_parallel()and the mirai package.
References
-
tidyr package documentation: pivot_longer, pivot_wider, separate_wider_delim, separate_longer_delim, complete, nest, hoist, separate, Pivoting, Rectangling, Tidy data, Changelog
-
purrr package documentation: map, map2, list_c, reduce, safely, purrr and base R, Changelog
-
stringr package documentation: str_detect, str_glue, modifiers, From base R, Regular expressions, Changelog
-
forcats package documentation: fct_lump, fct_recode, Changelog
-
lubridate package documentation: ymd, interval, period, mplus, Changelog
-
readr package documentation: read_delim, problems, Changelog
-
lifecycle package documentation: Lifecycle stages and deprecate_soft
-
R NEWS: Changes in R 4.1.0 and 4.5.0