Encodings, Locales and Internationalization

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

Checked against R 4.6.1 (2026-06-24) on 2026-10-20; R 4.6.2 is scheduled for 2026-10-29. Everything on this page is base R (the base, utils and tools packages) except one short example that uses withr 3.0.3 to change a locale for a single expression; the lintr 3.4.0 default linters were used only to check the scripts.

Every example below is a script that was run with Rscript on R 4.6.1 with warnings printed as they happen, and the lines starting with #> are what it printed. The scripts pass the default lintr linters. Errors and warnings that a script shows on purpose are caught with tryCatch() or withCallingHandlers() and printed as their conditionMessage(), so the text is exactly what R says. The scripts that use the library scenario of the section (books, loans, loan_days) start with source("scenario.R"), as described on the section index. The examples about text use a few made-up donated titles and author names, because the shelf data itself is plain ASCII. In the scripts a non-ASCII character is written as a Unicode escape (\u00e9 is the e with an acute accent), which keeps the script independent of the encoding of the file that holds it.

Two things about the machine that produced the output matter for this page. The test session ran in a UTF-8 English (US) locale; its name is printed as en_US.UTF-8, which is only an example of such a name, and on your machine the same call prints whatever your system calls its locale. And the container had the translation catalogues of R installed, so the examples of translated messages show real Spanish, German and French text; on a build without them the same calls return the English text, as the section on message translation explains.

Text looks like the simplest data type until the first accented letter arrives from a file written on another computer. Then three separate systems have an opinion: the encoding of the bytes (how bytes become characters), the locale of the session (how characters are classified, sorted, dated and numbered) and the language of the messages (which translation of an error R shows). R keeps them apart on purpose, and almost every confusing text problem in R comes from mixing two of them up. This page takes them in that order: the encoding first, with the functions that declare, convert, read and write it; then what goes wrong when the bytes do not match the encoding; then UTF-8 as the native encoding of recent Windows; then the locale categories and collation; and finally translated messages, including how to make your own messages translatable. Strings themselves (paste(), substr(), regular expressions) are on Strings and Regular Expressions, and reading data files is on Reading and Writing Text Data.

Encodings in one page

A computer stores bytes; an encoding is the agreement about which characters a sequence of bytes stands for. ASCII uses the values 0 to 127 for 128 characters and is the common core of nearly every encoding in use. Latin-1 (ISO 8859-1) uses one byte for each of 256 characters, so é is the single byte e9 (hexadecimal). UTF-8 encodes the whole of Unicode, using one byte for ASCII characters and two to four bytes for the others, so é is the two bytes c3 a9, a Japanese character takes three bytes and an emoji four. Because the same byte can mean different characters in different encodings, a file or a string without a known encoding cannot be read reliably: that is the whole difficulty.

R stores every element of a character vector as a sequence of bytes with a small flag next to it. The R Internals manual describes the design: the encoding of a string can be flagged as Latin-1 or as UTF-8 with two spare bits, and a string with neither flag is in the native encoding, the one of the current locale (R Internals: Encodings for CHARSXPs). The rationale given there is that UTF-8 is the nearest thing to a universal format and that Latin-1 was the old assumption of R. Two consequences follow, and the rest of the page depends on them:

  • A character vector can hold elements in different encodings. Comparison, unique() and match() take the declared encodings into account, so "caf\u00e9" declared UTF-8 and the same word declared Latin-1 are equal.

  • A string that is ASCII never carries a mark, because its bytes mean the same in every supported encoding. Only strings with non-ASCII bytes have something to declare.

The four values that Encoding() reports are these.

Encoding() Meaning Where the mark comes from

"unknown"

The string is ASCII, or its bytes are in the native encoding of the session and nothing was declared.

Most input: readLines(), read.csv(), scan() and rawToChar() give unmarked strings unless told otherwise.

"UTF-8"

The bytes are UTF-8, whatever the session locale.

A \uxxxx or \Uxxxxxxxx escape in the parser, intToUtf8(), iconv(to = "UTF-8"), enc2utf8(), encoding = "UTF-8" arguments, and console input in a UTF-8 session.

"latin1"

The bytes are Latin-1 (on Windows, the Windows-1252 superset).

iconv(to = "latin1"), enc2native() in a Latin-1 session, encoding = "latin1" arguments, Encoding<-.

"bytes"

The bytes are not text: R must not translate or count characters in them.

Encoding<-, readLines(encoding = "bytes"), and the results of string functions applied to such strings.

Source: R documentation: Encoding

The help page of Encoding() lists the other ways a string gets its mark: the encoding argument of scan(), read.table(), readLines() and parse() declares the encoding of what they read, iconv() declares the encoding from its to argument, and the parser marks strings that contain \u or \U escapes (R documentation: Encoding). Most character manipulation functions set the mark on their output when it was declared on their input, but they do not preserve it; substr() does, and paste() and sprintf() mark their result as UTF-8 if any input is UTF-8 and as bytes if any input is bytes.

Characters, bytes and widths

The first script builds a vector of four donated titles with an ASCII one, one with an accented letter, one in Japanese and one with an emoji, and measures each in the three ways nchar() offers: type = "chars" counts characters (code points), type = "bytes" counts the bytes stored, and type = "width" counts the columns a monospaced terminal font uses, where a Japanese character takes two columns (R documentation: nchar). It also shows the escapes, charToRaw(), which exposes the bytes of a string, and utf8ToInt() and intToUtf8(), which convert between a string and its Unicode code points (R documentation: utf8Conversion).

# Titles donated to the shelf: ASCII, accented, Japanese and one with an emoji
titles <- c(
  "Midnight Ledger", "Caf\u00e9 des Sables", "\u6771\u4eac\u306e\u672c",
  "Happy \U0001F600"
)
titles
#> [1] "Midnight Ledger" "Café des Sables" "東京の本"        "Happy 😀"
Encoding(titles)
#> [1] "unknown" "UTF-8"   "UTF-8"   "UTF-8"

# Size in bytes, characters and terminal columns
nchar(titles, type = "bytes")
#> [1] 15 16 12 10
nchar(titles, type = "chars")
#> [1] 15 15  4  7
nchar(titles, type = "width")
#> [1] 15 15  8  8

# The bytes behind the second title: the e-acute takes two
charToRaw("caf\u00e9")
#> [1] 63 61 66 c3 a9
utf8ToInt("\u00e9")
#> [1] 233
intToUtf8(c(72, 233, 8364))
#> [1] "Hé€"
intToUtf8(c(72, 233, 8364), multiple = TRUE)
#> [1] "H" "é" "€"

# One character, written in five ways; the last one is typed directly
identical("\u00e9", "\U000000e9")
#> [1] TRUE
identical("\u00e9", intToUtf8(233))
#> [1] TRUE
identical("\u00e9", iconv("\xe9", "latin1", "UTF-8"))
#> [1] TRUE
identical("\u00e9", "é")
#> [1] TRUE

# Is the string valid UTF-8? NA is
validUTF8(c("caf\u00e9", "caf\xe9", NA))
#> [1]  TRUE FALSE  TRUE

Source: R documentation: Encoding

Things to notice in the output:

  • "Midnight Ledger" is "unknown" because it is ASCII; the three others are "UTF-8" because they were written with \u and \U escapes, which mark the string as UTF-8 in any locale. The é typed directly in the last comparison is also marked UTF-8, because Rscript parses the file in a UTF-8 session, which is why it equals the escaped one; strings read from data files are not marked (see Reading and writing files).

  • The same title is 16 bytes, 15 characters and 15 columns for "Café des Sables", and the Japanese title is 12 bytes, 4 characters and 8 columns. Nothing is wrong: three different questions have three different answers. Use nchar(x, "chars") for the length a person would count, "bytes" for storage and buffer limits, and "width" for aligning text on a terminal.

  • An é is c3 a9 in UTF-8, and the Unicode code point 233 (U+00E9) is the same character. The code points of an emoji are above 65535, so the escape needs the long form \U0001F600 with eight hexadecimal digits.

  • validUTF8() tests whether each element is valid UTF-8 (NA counts as valid). "caf\xe9" is not: a lone e9 byte is a Latin-1 letter, and in UTF-8 it is the start of a three-byte sequence that never completes. Section When the bytes do not match shows what R does with such strings.

Writing non-ASCII characters in source code as escapes is the portable choice. R Extensions calls Unicode \uxxxx escapes the portable way to put arbitrary text into strings in package code, and asks for non-ASCII characters to be kept out of comments (Writing R Extensions: Encoding issues). R CMD check reports non-ASCII characters in code, and tools::showNonASCIIfile() finds them.

Declaring, converting and the bytes mark

Encoding(x) reads the declared encodings and Encoding(x) <- value sets them. Setting the encoding is a declaration: it tells R how to read the bytes that are already there, and it changes no byte. enc2utf8() and enc2native() are the converting functions: they re-encode each element to UTF-8 or to the native encoding, taking the declared encoding into account. They are primitives designed to do minimal copying, so they cost almost nothing on strings that are already in the target encoding (R documentation: Encoding).

# Four bytes that spell "caf" and an e-acute in Latin-1: 63 61 66 e9
raw_bytes <- as.raw(c(0x63, 0x61, 0x66, 0xe9))
x <- rawToChar(raw_bytes)
Encoding(x)
#> [1] "unknown"
x
#> [1] "caf\xe9"
nchar(x, type = "bytes")
#> [1] 4

# Declaring the encoding changes how R reads the bytes, not the bytes
Encoding(x) <- "latin1"
x
#> [1] "café"
Encoding(x)
#> [1] "latin1"
charToRaw(x)
#> [1] 63 61 66 e9

# Converting changes the bytes
y <- enc2utf8(x)
y
#> [1] "café"
Encoding(y)
#> [1] "UTF-8"
charToRaw(y)
#> [1] 63 61 66 c3 a9
identical(x, y)
#> [1] TRUE
x == y
#> [1] TRUE

# Declaring the wrong encoding is not an error, just wrong text
z <- y
Encoding(z) <- "latin1"
z
#> [1] "café"

# bytes: R refuses to treat the string as text
b <- y
Encoding(b) <- "bytes"
b
#> [1] "caf\\xc3\\xa9"
Encoding(b)
#> [1] "bytes"
tryCatch(nchar(b), error = conditionMessage)
#> [1] "number of characters is not computable in \"bytes\" encoding, element 1"
nchar(b, type = "bytes")
#> [1] 5
tryCatch(toupper(b), error = conditionMessage)
#> [1] "translating strings with \"bytes\" encoding is not allowed"

# enc2native() changes nothing in a UTF-8 session
identical(enc2native(y), y)
#> [1] TRUE

Source: R documentation: Encoding

The script starts from four raw bytes that spell "caf" and a Latin-1 e-acute. rawToChar() gives an unmarked string, which R tries to read as UTF-8 and prints with the byte shown as \xe9. After Encoding(x) <- "latin1" the same bytes (charToRaw() still shows e9) print as café. enc2utf8() produces a different string at byte level (c3 a9) that R still considers equal to the Latin-1 one, because comparison translates both to a common encoding first: identical(x, y) and x == y are both TRUE.

Declaring the wrong encoding is not an error. The variable z holds the UTF-8 bytes c3 a9 declared as Latin-1, and R faithfully shows the two Latin-1 letters é: the classic mojibake that results from a wrong declaration or a missing conversion. The cure for mojibake is never Encoding<- on text that is already correct; it is to find where the encoding was lost and convert there.

The "bytes" mark is for the rare case that a string must be handled as bytes. A bytes string prints its bytes in hexadecimal (with the backslashes escaped when printed), cannot be counted in characters (nchar(b) stops with number of characters is not computable in "bytes" encoding, element 1) and cannot be translated (toupper(b) stops with translating strings with "bytes" encoding is not allowed). nchar(type = "bytes") still works. Since R 4.3.0 gsub(), sub() and strsplit() mark their output as bytes when any input is marked bytes, and readLines(encoding = "bytes") was added to process special text files byte by byte without creating invalid strings (R NEWS: Changes in R 4.3.0). The last line of the script confirms that enc2native() returns the same string in a UTF-8 session, because UTF-8 is the native encoding there.

Converting with iconv()

iconv(x, from, to, sub, mark, toRaw) converts a character vector between encodings using the facilities of the operating system. from and to are encoding names such as "UTF-8", "latin1" or "ASCII"; "" means the native encoding. Names are matched without regard to case and "utf8" is fixed up to "UTF-8" (from R 4.3.3), but the names available are platform-dependent, and iconvlist() lists the ones the system knows (R documentation: iconv). If from = "", the declared encoding of each element is used in preference to the native encoding (R 4.3.0).

authors <- c(
  "Jos\u00e9 \u00c1lvarez", "Zo\u00eb Hart", "Bj\u00f6rn Sj\u00f6berg",
  "Mara Quill"
)

# Latin-1 can hold all of these, so converting to it works and is marked
latin <- iconv(authors, from = "UTF-8", to = "latin1")
Encoding(latin)
#> [1] "latin1"  "latin1"  "latin1"  "unknown"
nchar(latin, type = "bytes")
#> [1] 12  8 13 10

# ASCII cannot, so the elements that do not fit become NA ...
iconv(authors, from = "UTF-8", to = "ASCII")
#> [1] NA           NA           NA           "Mara Quill"

# ... unless sub says what to put in place of each unconvertible byte
iconv(authors, "UTF-8", "ASCII", sub = "?")
#> [1] "Jos?? ??lvarez"  "Zo?? Hart"       "Bj??rn Sj??berg" "Mara Quill"
iconv(authors, "UTF-8", "ASCII", sub = "")
#> [1] "Jos lvarez"  "Zo Hart"     "Bjrn Sjberg" "Mara Quill"
iconv(authors, "UTF-8", "ASCII", sub = "byte")
#> [1] "Jos<c3><a9> <c3><81>lvarez"  "Zo<c3><ab> Hart"
#> [3] "Bj<c3><b6>rn Sj<c3><b6>berg" "Mara Quill"
iconv(authors, "UTF-8", "ASCII", sub = "Unicode")
#> [1] "Jos<U+00E9> <U+00C1>lvarez"  "Zo<U+00EB> Hart"
#> [3] "Bj<U+00F6>rn Sj<U+00F6>berg" "Mara Quill"
iconv(authors, "UTF-8", "ASCII", sub = "c99")
#> [1] "Jos\\u00e9 \\u00c1lvarez"  "Zo\\u00eb Hart"
#> [3] "Bj\\u00f6rn Sj\\u00f6berg" "Mara Quill"

# Transliteration: the nearest ASCII look-alike (it depends on the OS iconv)
iconv(authors, "UTF-8", "ASCII//TRANSLIT")
#> [1] "Jose Alvarez"  "Zoe Hart"      "Bjorn Sjoberg" "Mara Quill"

# Characters that Latin-1 does not have
iconv("Happy \U0001F600 \u20ac5", "UTF-8", "latin1")
#> [1] NA
iconv("Happy \U0001F600 \u20ac5", "UTF-8", "latin1", sub = "Unicode")
#> [1] "Happy <U+1F600> <U+20AC>5"

# toRaw = TRUE returns the bytes of the target encoding
iconv("Zo\u00eb", "UTF-8", "UTF-16LE", toRaw = TRUE)
#> [[1]]
#> [1] 5a 00 6f 00 eb 00
#>
iconv("Zo\u00eb", "UTF-8", "latin1", toRaw = TRUE)
#> [[1]]
#> [1] 5a 6f eb
#>

# NA stays NA, and encoding names are matched without regard to case
iconv(c("Zo\u00eb", NA), "utf8", "LATIN1")
#> [1] "Zoë" NA

# Latin-1 bytes read as UTF-8 are invalid, so that conversion fails
bad <- rawToChar(as.raw(c(0x5a, 0x6f, 0xeb)))
iconv(bad, "UTF-8", "UTF-8")
#> [1] NA
iconv(bad, "latin1", "UTF-8")
#> [1] "Zoë"

# An unknown encoding name is an error
tryCatch(iconv("a", "no-such-encoding", "UTF-8"), error = conditionMessage)
#> [1] "unsupported conversion from 'no-such-encoding' to 'UTF-8'"

# The number of known encodings differs between systems
is.character(iconvlist())
#> [1] TRUE
length(iconvlist()) > 100
#> [1] TRUE

Source: R documentation: iconv

The behaviours worth remembering:

  • An element that cannot be converted becomes NA (or NULL with toRaw = TRUE). Converting names with accents to "ASCII" gives NA for three of the four authors. Always check anyNA() on the result of a conversion.

  • sub says what replaces each non-convertible byte: a string such as "?" or "", "byte" for <c3><a9>-style hexadecimal, and "Unicode" or "c99" for <U+00E9> and \u00e9 forms (they apply when converting from UTF-8; for other inputs they act like "byte"). Because sub works on bytes, a single accented letter replaced with "?" becomes ??, which is why the help page recommends "Unicode" or "c99". Since R 4.4.0 sub = "Unicode" always zero-pads to four hexadecimal digits.

  • "ASCII//TRANSLIT" asks for the nearest ASCII look-alike. It is the portable-looking way to strip accents, and it is not portable: glibc gave Jose Alvarez, macOS and musl give other results, and the help page lists the three outcomes for the same input. R 4.3.2 emulates transliteration with substitution where the OS lacks it (Alpine Linux), so the result is at least human-readable rather than NA. Do not use it for keys or file names that must be the same on every machine.

  • mark = TRUE (the default) declares the output as Latin-1 or UTF-8 when to is one of them; the first block shows "latin1" for the three converted names and "unknown" for the ASCII one.

  • toRaw = TRUE returns the converted bytes as raw vectors, which is how to build a file in a given encoding: the UTF-16LE bytes of Zoë are 5a 00 6f 00 eb 00. The next section uses it to write a Latin-1 file.

  • Implementations differ in how strictly they validate their input. Converting the invalid string from "UTF-8" to "UTF-8" gives NA, while converting the same bytes from "latin1" gives the correct Zoë. The help page warns that iconv implementations typically do not do much validity checking and will often mis-convert invalid input.

  • An unsupported name is an error at the call: unsupported conversion from 'no-such-encoding' to 'UTF-8'. The number of encodings in iconvlist() is a property of the system (over a thousand names in the test container); the script only checks the shape.

Neither iconv() nor the Encoding() functions understand a file; for that the encoding belongs to the connection, which is the next section.

Reading and writing files in other encodings

There are two different arguments called "encoding" for reading text, and they do different things. The R Data Import/Export manual states the problem: any file that is not pure ASCII needs its encoding to be known, and an 8-bit encoding cannot be detected with certainty (R Data Import/Export: Encodings). Once you know it, you choose between re-encoding on input and declaring the encoding of what comes in:

Where What it does Use it when

file(path, encoding = "latin1") and fileEncoding = "latin1" in read.table() and its relatives

Re-encodes the file to the native encoding as it is read. In readLines() and scan() the connection gives UTF-8 strings that are marked as such.

The file is in a known non-native encoding that you want converted: the usual case.

readLines(encoding = "latin1"), read.csv(encoding = "latin1"), scan(encoding =)

Marks the strings as Latin-1 or UTF-8 (or bytes), and does not re-encode.

The input is already known to be Latin-1 or UTF-8, and you want R to handle it in that encoding even though the session is not.

options(encoding = "latin1")

The default encoding of every connection that does not give one.

Rarely, and with care: it affects file(), readLines(), source() and everything built on them.

Source: R documentation: connections

The help pages say the same in their own words: readLines(encoding =) "is not used to re-encode the input", to do that the encoding must be part of the connection or options(encoding =); and read.table() documents fileEncoding as re-encoding through a connection and encoding as only declaring (R documentation: readLines, R documentation: read.table). The connection help page adds that encoding names "" and "native.enc" both mean the native encoding, that re-encoding applies only to text-mode connections, and that "utf8", "mac" and "macroman" are not portable names ("UTF-8" is, and "macintosh" is the official name of Mac Roman).

The next script builds a small Latin-1 CSV of donated books from UTF-8 text with iconv(toRaw = TRUE) and writeBin(), so it needs no file from outside, and then reads it in six ways.

# A donations list as a Latin-1 CSV, built from UTF-8 text
donated <- c(
  "title,author,year",
  "Caf\u00e9 des Sables,Zo\u00eb Hart,2019",
  "Garten der M\u00e4rchen,Bj\u00f6rn Sj\u00f6berg,2011",
  "Midnight Ledger,Iris Fenwick,2016"
)
path <- tempfile(fileext = ".csv")
latin1_bytes <- iconv(
  paste0(donated, "\n", collapse = ""), "UTF-8", "latin1",
  toRaw = TRUE
)[[1L]]
writeBin(latin1_bytes, path)

# One byte per accented letter: five bytes fewer than the UTF-8 text
file.size(path)
#> [1] 120
sum(nchar(donated, type = "bytes") + 1L) - file.size(path)
#> [1] 5

# Default: the bytes are taken as UTF-8, which they are not
raw_lines <- readLines(path)
raw_lines[2L]
#> [1] "Caf\xe9 des Sables,Zo\xeb Hart,2019"
Encoding(raw_lines[2L])
#> [1] "unknown"
validUTF8(raw_lines)
#> [1]  TRUE FALSE FALSE  TRUE

# readLines(encoding =) only declares the encoding, it does not convert
declared <- readLines(path, encoding = "latin1")
declared[2L]
#> [1] "Café des Sables,Zoë Hart,2019"
Encoding(declared)
#> [1] "unknown" "latin1"  "latin1"  "unknown"
charToRaw(declared[2L])
#>  [1] 43 61 66 e9 20 64 65 73 20 53 61 62 6c 65 73 2c 5a 6f eb 20 48 61 72 74 2c
#> [26] 32 30 31 39

# A connection with encoding = converts on the way in, to UTF-8
con <- file(path, encoding = "latin1")
converted <- readLines(con)
close(con)
converted[2L]
#> [1] "Café des Sables,Zoë Hart,2019"
Encoding(converted)
#> [1] "unknown" "UTF-8"   "UTF-8"   "unknown"
charToRaw(converted[2L])
#>  [1] 43 61 66 c3 a9 20 64 65 73 20 53 61 62 6c 65 73 2c 5a 6f c3 ab 20 48 61 72
#> [26] 74 2c 32 30 31 39

# read.csv() has both: fileEncoding converts, encoding only declares
d1 <- read.csv(path, fileEncoding = "latin1")
d1
#>                title        author year
#> 1    Café des Sables      Zoë Hart 2019
#> 2 Garten der Märchen Björn Sjöberg 2011
#> 3    Midnight Ledger  Iris Fenwick 2016
Encoding(d1$title)
#> [1] "unknown" "unknown" "unknown"
d2 <- read.csv(path, encoding = "latin1")
d2$title
#> [1] "Café des Sables"    "Garten der Märchen" "Midnight Ledger"
Encoding(d2$title)
#> [1] "latin1"  "latin1"  "unknown"
identical(d1$title, d2$title)
#> [1] TRUE

# With neither, the strings are not valid UTF-8
d3 <- read.csv(path)
d3$title
#> [1] "Caf\xe9 des Sables"    "Garten der M\xe4rchen" "Midnight Ledger"
validUTF8(d3$title)
#> [1] FALSE FALSE  TRUE

# options(encoding =) sets the default for every file() connection
old <- options(encoding = "latin1")
default_conn <- readLines(path)
options(old)
default_conn[2L]
#> [1] "Café des Sables,Zoë Hart,2019"

# readLines(encoding = "bytes") marks every non-ASCII line as bytes
bytes_lines <- readLines(path, encoding = "bytes")
Encoding(bytes_lines)
#> [1] "unknown" "bytes"   "bytes"   "unknown"

unlink(path)

Source: R documentation: connections

What each reading does:

  • readLines(path) returns strings whose é and ä are single Latin-1 bytes. They are not valid UTF-8 (validUTF8() is FALSE for the two lines that have accents), they print as Caf\xe9 des Sables, and the ASCII lines are unaffected.

  • readLines(path, encoding = "latin1") shows the same bytes (e9 in charToRaw()) marked Latin-1, so R prints Café des Sables. Nothing was converted; if you now write the strings with writeLines() to a UTF-8 file, R converts them then.

  • file(path, encoding = "latin1") converts on the way in: the bytes of the result are c3 a9, and the strings are marked UTF-8.

  • read.csv(path, fileEncoding = "latin1") converts through such a connection; the result has clean native strings (Encoding() says "unknown" because the native encoding is UTF-8 here). read.csv(path, encoding = "latin1") only marks them, and the two results are identical() for comparison purposes.

  • read.csv(path) with neither argument gives strings that are not valid UTF-8, which print as Caf\xe9 des Sables. This is the output of a script run on a laptop that works and on a server that does not, when the laptop is a Windows machine that treated the file as Windows-1252.

  • options(encoding = "latin1") made the plain readLines(path) convert. Restore the option, as the script does with options(old); it is global state that changes how every later file is read.

  • readLines(encoding = "bytes") (R 4.3.0) marks the non-ASCII lines as "bytes", which keeps them intact for byte-level processing.

Writing is the mirror image. Output connections always assume that the text handed to them is in the native encoding and re-encode it to the encoding of the connection; write.csv(fileEncoding = "latin1") does the same through a connection it opens. writeLines(useBytes = TRUE) skips the conversion and writes the bytes of each string exactly as stored, which is correct only when you know what the bytes are (R documentation: writeLines). A character that the target encoding does not have cannot be written to a re-encoding connection.

donors <- data.frame(
  title = c("Caf\u00e9 des Sables", "Garten der M\u00e4rchen"),
  year = c(2019L, 2011L)
)
path <- tempfile(fileext = ".csv")
first_bytes <- \(n) head(readBin(path, "raw", 200L), n)

# write.csv() writes in the native encoding (UTF-8 here)
write.csv(donors, path, row.names = FALSE)
first_bytes(24L)
#>  [1] 22 74 69 74 6c 65 22 2c 22 79 65 61 72 22 0a 22 43 61 66 c3 a9 20 64 65
file.size(path)
#> [1] 66

# fileEncoding converts on the way out
write.csv(donors, path, row.names = FALSE, fileEncoding = "latin1")
file.size(path)
#> [1] 64
readLines(path, encoding = "latin1")[2L]
#> [1] "\"Café des Sables\",2019"

# The same with a connection, where encoding converts text written to it
con <- file(path, "w", encoding = "latin1")
writeLines(donors$title, con)
close(con)
file.size(path)
#> [1] 35

# A character that Latin-1 lacks: R warns and stops writing at that character
con <- file(path, "w", encoding = "latin1")
withCallingHandlers(
  writeLines("Happy \U0001F600 end", con),
  warning = \(w) {
    cat("warning:", conditionMessage(w), "\n")
    invokeRestart("muffleWarning")
  }
)
#> warning: invalid char string in output conversion
close(con)
readLines(path, warn = FALSE)
#> [1] "Happy "

# writeLines(useBytes = TRUE) writes the bytes of the string as they are
latin <- iconv("Caf\u00e9", "UTF-8", "latin1")
writeLines(latin, path, useBytes = TRUE)
file.size(path)
#> [1] 5
writeLines(latin, path)
file.size(path)
#> [1] 6

# A UTF-8 file with a byte order mark: a UTF-8 session drops the mark
bom <- as.raw(c(0xef, 0xbb, 0xbf))
writeBin(c(bom, charToRaw("title,year\nMidnight Ledger,2016\n")), path)
head(readBin(path, "raw", 6L), 6L)
#> [1] ef bb bf 74 69 74
charToRaw(readLines(path)[[1L]])
#>  [1] 74 69 74 6c 65 2c 79 65 61 72
names(read.csv(path))
#> [1] "title" "year"

# A UTF-16 file, as some Windows programs write it
text <- "title,year\nMidnight Ledger,2016\n"
utf16 <- iconv(text, "UTF-8", "UTF-16LE", toRaw = TRUE)
writeBin(c(as.raw(c(0xff, 0xfe)), utf16[[1L]]), path)
tryCatch(read.csv(path), error = conditionMessage, warning = conditionMessage)
#> [1] "line 1 appears to contain embedded nulls"
read.csv(path, fileEncoding = "UTF-16LE")
#>             title year
#> 1 Midnight Ledger 2016
unlink(path)

Source: R documentation: writeLines

In the output the UTF-8 CSV is 66 bytes and the Latin-1 one 64, two fewer for two accented letters; readLines(path, encoding = "latin1") shows the written file read back correctly. Writing the emoji to a Latin-1 connection gives the warning invalid char string in output conversion; with the warning muffled, the file holds Happy and a blank, the text up to the unconvertible character, and nothing after it. The useBytes = TRUE call wrote five bytes (the four letters, Café in Latin-1, and the newline) and the plain call wrote six (the UTF-8 é is two bytes).

A byte order mark (BOM) is the character U+FEFF at the start of a file. It is common in files from Microsoft applications and rare elsewhere. R 3.0.0 made readLines() and scan() (and hence read.table()) discard a UTF-8 BOM in a UTF-8 locale, and added the connection encoding name "UTF-8-BOM" to discard it explicitly in any locale (R NEWS: Changes in R 3.0.0). That is why the script’s first column is called title, not something with three strange characters in front. A UTF-16 file is different: read as bytes it contains NUL bytes, and R reports line 1 appears to contain embedded nulls; the fix is fileEncoding = "UTF-16LE", which also strips the BOM. The connection help page notes the two encoding names "UCS-2LE" and "UTF-16LE" are treated specially because they are what Windows calls "Unicode" text files.

The R Data Import/Export manual suggests finding out the encoding with the command-line tool file (on Windows it comes with Rtools), and with od or a hex editor when that fails; it also names the practical guesses: a Unix-like system or macOS produces UTF-8, Windows may produce UTF-16, and otherwise an 8-bit encoding unless the file comes from a Chinese, Japanese or Korean locale. Community packages can guess an encoding from the bytes (readr::guess_encoding() and stringi::stri_enc_detect()), but a guess is a guess: the manual’s advice is to ask whoever made the file.

When the bytes do not match the encoding

A string whose bytes are not valid in the encoding R believes it has is the source of the most common text errors. The next script makes one such string, the four bytes of café in Latin-1 held as an unmarked string in a UTF-8 session, and applies nineteen calls to it, printing for each either the value or the text of the error or the warning that R raised. A message can quote the offending bytes, so the script prints them as <xx> with iconv(sub = "byte"); without that the invalid byte would appear in the page.

# Latin-1 bytes in a string that R believes is UTF-8
bad <- rawToChar(as.raw(c(0x63, 0x61, 0x66, 0xe9)))
bad
#> [1] "caf\xe9"
validUTF8(bad)
#> [1] FALSE
Encoding(bad)
#> [1] "unknown"

# A message can quote the bad bytes, so show them as <xx> when printing it
show_bytes <- \(text) iconv(text, "UTF-8", "UTF-8", sub = "byte")

# What does a call do with it? Report the condition text, or the value
outcome <- function(expr) {
  tryCatch(
    paste("value", deparse1(expr)),
    error = \(e) paste("error:", show_bytes(conditionMessage(e))),
    warning = \(w) paste("warning:", show_bytes(conditionMessage(w)))
  )
}
calls <- alist(
  nchar(bad),
  nchar(bad, allowNA = TRUE),
  nchar(bad, type = "bytes"),
  toupper(bad),
  toupper(enc2utf8(bad)),
  chartr("a", "b", bad),
  substr(bad, 1L, 2L),
  substr(bad, 4L, 4L),
  substring(bad, 1:4, 1:4),
  strsplit(bad, ""),
  grepl("a", bad),
  sub("a", "A", bad),
  gsub("a", "A", bad, perl = TRUE),
  regexpr("f", bad, fixed = TRUE),
  trimws(bad),
  utf8ToInt(bad),
  iconv(bad, "UTF-8", "latin1"),
  paste(bad, "ok"),
  bad == "cafe"
)
for (call in calls) {
  cat(deparse1(call), "->", outcome(eval(call)), "\n")
}
#> nchar(bad) -> error: invalid multibyte string, element 1
#> nchar(bad, allowNA = TRUE) -> value NA_integer_
#> nchar(bad, type = "bytes") -> value 4L
#> toupper(bad) -> error: invalid multibyte string 1
#> toupper(enc2utf8(bad)) -> error: invalid input 'caf<e9>' in 'utf8towcs'
#> chartr("a", "b", bad) -> error: invalid input multibyte string 1
#> substr(bad, 1L, 2L) -> value "ca"
#> substr(bad, 4L, 4L) -> error: invalid multibyte string at '<e9>'
#> substring(bad, 1:4, 1:4) -> error: invalid multibyte string at '<e9>'
#> strsplit(bad, "") -> warning: input string 1 is invalid UTF-8
#> grepl("a", bad) -> warning: unable to translate 'caf<e9>' to a wide string
#> sub("a", "A", bad) -> warning: unable to translate 'caf<e9>' to a wide string
#> gsub("a", "A", bad, perl = TRUE) -> error: input string 1 is invalid UTF-8
#> regexpr("f", bad, fixed = TRUE) -> warning: input string 1 is invalid in this locale
#> trimws(bad) -> error: input string 1 is invalid UTF-8
#> utf8ToInt(bad) -> value NA_integer_
#> iconv(bad, "UTF-8", "latin1") -> value NA_character_
#> paste(bad, "ok") -> value "caf\xe9 ok"
#> bad == "cafe" -> value FALSE

# The parser reports the line, not the byte
outcome(parse(text = "x <- 'caf\xe9'", keep.source = FALSE))
#> [1] "error: invalid multibyte character in parser (<input>:2:0)"

# A string cannot contain a NUL
outcome(rawToChar(as.raw(c(0x61, 0x00, 0x62))))
#> [1] "error: embedded nul in string: 'a\\0b'"
outcome(parse(text = "'a\\0b'"))
#> [1] "error: nul character not allowed (<input>:1:4)"

Source: R documentation: nchar

The calls fall into four groups:

  • Functions that need characters, and stop. nchar() (invalid multibyte string, element 1), toupper() and tolower() (invalid multibyte string 1), chartr() (invalid input multibyte string 1), substr() once it reaches the bad byte (invalid multibyte string at '<e9>', the message that older tutorials and questions quote), gsub(perl = TRUE) and trimws() (input string 1 is invalid UTF-8). nchar(allowNA = TRUE) returns NA instead of stopping, and !is.na(nchar(x, "chars", allowNA = TRUE)) is a test of validity. Converting to UTF-8 first does not help if the string was mislabelled: toupper(enc2utf8(bad)) fails with invalid input 'caf<e9>' in 'utf8towcs'.

  • Regular-expression functions, which warn. grepl() and sub() warn unable to translate 'caf<e9>' to a wide string, strsplit() and the PCRE engine say input string 1 is invalid UTF-8, and fixed-string matching says input string 1 is invalid in this locale. The result after the warning is usually NA or FALSE, which is how the rows silently disappear from a filter.

  • Functions that work on the bytes, and succeed. nchar(type = "bytes"), paste(), ==, substr() of the first characters, and utf8ToInt() (which returns NA). They carry the invalid string along, and it shows up later, usually when the data is written to a UTF-8 file, plotted, or sent to a database.

  • The parser. Source text with an invalid byte stops with invalid multibyte character in parser (<input>:2:0), which reports the position and not the byte. The remedy is to read the file with the right encoding (source(file, encoding = "latin1") or parse(file, encoding = ...)), or to convert the file. A string cannot contain a NUL byte: rawToChar() stops with embedded nul in string: 'a\0b' and the parser with nul character not allowed (<input>:1:4).

The cures are the same as for mojibake: find the step where a declaration or a conversion was missed, and fix it there. validUTF8() locates the bad elements, iconv(x, "latin1", "UTF-8") (or Encoding(x) <- "latin1" followed by enc2utf8()) fixes the ones that really are Latin-1, and iconv(x, "UTF-8", "UTF-8", sub = "byte") makes the bad bytes visible without converting anything else.

Width, padding and case

Because characters, bytes and columns differ, so do the functions that pad and cut. The sprintf() help page states that field widths and precisions of %s conversions are interpreted as bytes, not characters, as the C standard says; format() pads to display width, and formatC() documents its width as characters while its result in this session was eight columns wide (R documentation: sprintf, R documentation: format). Table-making code that mixes sprintf() with accented or Japanese names produces ragged columns.

# Three words that are four, four and two characters long
words <- c("Cafe", "Caf\u00e9", "\u6771\u4eac")
data.frame(
  chars = nchar(words, type = "chars"),
  bytes = nchar(words, type = "bytes"),
  width = nchar(words, type = "width")
)
#>   chars bytes width
#> 1     4     4     4
#> 2     4     5     4
#> 3     2     6     4

# Padding to eight: sprintf() counts bytes, the other two count columns
padded <- list(
  sprintf = sprintf("%-8s", words),
  formatC = formatC(words, width = -8L),
  format = format(words, width = 8L)
)
paste0(padded$sprintf, "|")
#> [1] "Cafe    |" "Café   |"  "東京  |"
paste0(padded$formatC, "|")
#> [1] "Cafe    |" "Café    |" "東京    |"
paste0(padded$format, "|")
#> [1] "Cafe    |" "Café    |" "東京    |"
vapply(padded, \(p) nchar(p, type = "width"), integer(3L))
#>      sprintf formatC format
#> [1,]       8       8      8
#> [2,]       7       8      8
#> [3,]       6       8      8
vapply(padded, \(p) nchar(p, type = "bytes"), integer(3L))
#>      sprintf formatC format
#> [1,]       8       8      8
#> [2,]       8       9      9
#> [3,]       8      10     10

# Truncating: strtrim() counts display columns, substr() characters
strtrim(words, 3L)
#> [1] "Caf" "Caf" "東"
substr(words, 1L, 3L)
#> [1] "Caf"  "Caf"  "東京"

# Case conversion works character by character, so one letter stays one letter
toupper("stra\u00dfe")
#> [1] "STRAßE"

Source: R documentation: sprintf

The first table is the nchar() triple for the three words. In the second, sprintf("%-8s") produced strings that are eight bytes long, so Café got three spaces and the Japanese word two, and the third and fourth tables show the display widths (8 for formatC() and format(), 6 and 7 for sprintf()). strtrim() cuts to a number of display columns (the Japanese word does not fit three columns, so only its first character remains), and substr() counts characters. toupper() maps character by character, so ß stays ß and Straße becomes STRAßE, not STRASSE: case conversion is not a full Unicode operation in base R, and its result depends on the character classification of the locale (LC_CTYPE). The stringr and stringi packages offer full Unicode case mapping and padding by width; they are described on Strings and Regular Expressions.

UTF-8 on Windows

Until R 4.1 the Windows port of R ran in a locale that used a single-byte or double-byte code page: Windows-1252 on a Western European machine, 932 on a Japanese one. Text outside the code page could not be represented in the native encoding, so Windows users needed encoding, fileEncoding and iconv() far more than users of Linux and macOS did, and the same script printed different results on the two systems.

R 4.2.0 changed that. The NEWS entry says that R uses UTF-8 as the native encoding on recent Windows systems (at least Windows 10 version 1903, Windows Server 2022 or Windows Server 1903), and that as part of this change R uses UCRT as the C runtime; UCRT should be installed manually on systems older than Windows 10 or Windows Server 2016 before installing R (R NEWS: Changes in R 4.2.0). The same release added .ucrt variants of Makevars, Makefile, configure and cleanup for packages, and a field crt in R.version (only on Windows) whose value is "ucrt". This is a property of Windows builds; the page cannot run it in the Linux container, so what follows is taken from the manuals, not from a run.

What changed for a user:

  • The native encoding is UTF-8, so Encoding() of ordinary typed or read text is "unknown" and means UTF-8, as on the other systems. A script that was written for Windows-1252 and saved as such will now contain invalid UTF-8, and has to be converted or read with encoding or fileEncoding.

  • l10n_info() on Windows has two additional components, codepage (the code page R is using) and system.codepage (the one Windows is using; added in R 4.1.0). The help page lists 65001 as the UTF-8 code page "where supported" (on a recent Windows both components are 65001), and says R does not allow the C locale on Windows and uses 1252 as the default code page (R documentation: l10n_info). Elsewhere l10n_info() has codeset instead, the encoding name as reported by the OS (added in R 4.1.0).

  • Locale names are the UCRT ones. The Sys.setlocale() help page says that from R 4.2 on a recent Windows the system code page is 65001 and one should always use locale names ending in .UTF-8 (except "C" and ""), because otherwise Windows may add a different character set; names such as "German" or "English" from older tutorials may therefore bring a different character set (R documentation: Sys.setlocale).

  • On Windows Sys.setlocale("LC_MESSAGES", ...) is not supported; the language of messages is chosen with the LANGUAGE environment variable or Sys.setLanguage().

  • The internal case-changing functions are used by default on Windows, which circumvents problems of the UCRT runtime (for example with the e-acute).

  • Rgui, the Windows GUI, gained its own fixes in R 4.2.1: text injection through SendInput, the performance of txtProgressBar() in a multibyte locale, and the script editor, whose scripts with non-ASCII characters must be converted to UTF-8 before reading them in R 4.2.1 or newer (R NEWS: Changes in R 4.2.1).

Writing R Extensions agrees with this from the package side: the encoding issues of a package "have been much alleviated" by the change in R 4.2.0, though users of earlier R on earlier Windows may still hit them (Writing R Extensions: Encoding issues). In R 4.5.0 R CMD check stopped reporting marked UTF-8 or Latin-1 strings in package datasets by default, for the same reason: almost all supported R versions now use UTF-8.

Files are a separate matter. A Windows tool can still write Windows-1252, UTF-16 or UTF-8 with a BOM, and the file’s encoding is whatever the tool chose. The functions of the earlier sections (fileEncoding, encoding, iconv()) are the answer, whatever the platform of the session.

Locales

A locale is the description of the user’s environment: the preferred language, the encoding of characters, the currency and its conventions. The R Installation and Administration manual says it consists of a human language (a lower-case ISO 639 code such as en), a territory (an upper-case ISO 3166 code such as GB, used mainly for the currency), a charset encoding and an optional modifier such as @euro or @latin; R is principally concerned with the language, for translations, and with the charset (R Installation and Administration: Locales). The names are operating-system specific:

A locale is made of categories, and Sys.getlocale() and Sys.setlocale() work on one at a time (R documentation: Sys.setlocale). .LC.categories lists the nine R knows; the effects of the six that always exist are these.

Category What it controls

LC_CTYPE

Which bytes are characters and which characters are letters, digits or punctuation, and the multibyte encoding itself; it decides toupper(), [[:alpha:]], nchar() and whether é can be printed.

LC_COLLATE

The order of strings in sort(), order(), < and > on strings, and in ICU’s rules.

LC_TIME

Month and weekday names in format(), months(), weekdays() and strptime(); date() is not affected.

LC_NUMERIC

The decimal separator of the C library. R keeps this at "C" and gives a warning when it is changed.

LC_MONETARY

The currency conventions that Sys.localeconv() reports.

LC_MESSAGES

The language of messages (not on Windows); LANGUAGE takes precedence over it.

LC_ALL

All of the above: but setting it sets only LC_COLLATE, LC_CTYPE, LC_MONETARY and LC_TIME.

Source: R documentation: Sys.setlocale

The next script prints the categories, asks for the facts about the encoding, and exercises the failure cases of Sys.setlocale().

# The nine categories R knows about (the last three are not on Windows)
.LC.categories
#> [1] "LC_ALL"         "LC_COLLATE"     "LC_CTYPE"       "LC_MONETARY"
#> [5] "LC_NUMERIC"     "LC_TIME"        "LC_MESSAGES"    "LC_PAPER"
#> [9] "LC_MEASUREMENT"

# This session's locale (an example name: the one on your machine will differ)
Sys.getlocale("LC_CTYPE")
#> [1] "en_US.UTF-8"
Sys.getlocale("LC_COLLATE")
#> [1] "en_US.UTF-8"
Sys.getlocale("LC_NUMERIC")
#> [1] "C"

# The facts about the encoding come from l10n_info(), not from the name
l10n_info()[c("MBCS", "UTF-8", "Latin-1")]
#> $MBCS
#> [1] TRUE
#>
#> $`UTF-8`
#> [1] TRUE
#>
#> $`Latin-1`
#> [1] FALSE
#>

# localeToCharset() guesses the charset from a locale name (just a string)
localeToCharset("en_GB")
#> [1] "ISO8859-1"
localeToCharset("de_DE.UTF-8")
#> [1] "UTF-8"     "ISO8859-1"
localeToCharset("ja_JP.eucJP")
#> [1] "EUC-JP"
localeToCharset("C")
#> [1] "ASCII"

# Asking for a locale the system does not have: a warning and an empty result
req <- withCallingHandlers(
  Sys.setlocale("LC_TIME", "xx_XX.UTF-8"),
  warning = \(w) {
    cat("warning:", conditionMessage(w), "\n")
    invokeRestart("muffleWarning")
  }
)
#> warning: OS reports request to set locale to "xx_XX.UTF-8" cannot be honored
req
#> [1] ""
Sys.getlocale("LC_TIME")
#> [1] "en_US.UTF-8"

# An unknown category is an error
tryCatch(Sys.setlocale("LC_FOO", "C"), error = conditionMessage)
#> [1] "invalid 'category' argument"

# LC_ALL sets only four categories, so the message language keeps its value
before <- Sys.getlocale("LC_MESSAGES")
saved <- c(
  ctype = Sys.getlocale("LC_CTYPE"), collate = Sys.getlocale("LC_COLLATE"),
  monetary = Sys.getlocale("LC_MONETARY"), time = Sys.getlocale("LC_TIME")
)
invisible(Sys.setlocale("LC_ALL", "C"))
Sys.getlocale("LC_TIME")
#> [1] "C"
identical(Sys.getlocale("LC_MESSAGES"), before)
#> [1] TRUE
invisible(Sys.setlocale("LC_CTYPE", saved[["ctype"]]))
invisible(Sys.setlocale("LC_COLLATE", saved[["collate"]]))
invisible(Sys.setlocale("LC_MONETARY", saved[["monetary"]]))
invisible(Sys.setlocale("LC_TIME", saved[["time"]]))
Sys.getlocale("LC_TIME")
#> [1] "en_US.UTF-8"

# Dates: month and weekday names come from LC_TIME
d <- as.Date("2026-06-30")
format(d, "%A %d %B %Y")
#> [1] "Tuesday 30 June 2026"
months(d)
#> [1] "June"
weekdays(d, abbreviate = TRUE)
#> [1] "Tue"

# Numbers: LC_NUMERIC must stay "C"; changing it warns
withCallingHandlers(
  invisible(Sys.setlocale("LC_NUMERIC", saved[["ctype"]])),
  warning = \(w) {
    cat("warning:", conditionMessage(w), "\n")
    invokeRestart("muffleWarning")
  }
)
#> warning: setting 'LC_NUMERIC' may cause R to function strangely
Sys.getlocale("LC_NUMERIC")
#> [1] "en_US.UTF-8"
invisible(Sys.setlocale("LC_NUMERIC", "C"))
Sys.getlocale("LC_NUMERIC")
#> [1] "C"

# The conventions of the locale: decimal point, grouping and currency
conv <- Sys.localeconv()
conv[c("decimal_point", "thousands_sep", "currency_symbol")]
#>   decimal_point   thousands_sep currency_symbol
#>             "."              ""             "$"

# options(OutDec) is the safe way to print a decimal comma
print(1234.5)
#> [1] 1234.5
old <- options(OutDec = ",")
print(1234.5)
#> [1] 1234,5
format(1234.5, big.mark = ".")
#> [1] "1.234,5"
options(old)

Source: R documentation: Sys.setlocale

The points to take from it:

  • The locale name is not the encoding. The Sys.getlocale() help page warns against using the LC_CTYPE value to find the character set: UTF-8 locales end in .UTF-8, .utf8 or nothing (macOS), and Latin-9 locales have several spellings. l10n_info() has three logical components, MBCS, UTF-8 and Latin-1, plus codeset on Unix-alikes (R documentation: l10n_info). localeToCharset() in utils guesses a character set from a locale name string, and its result for en_GB or de_DE.UTF-8 is a guess based on the name only (R documentation: localeToCharset).

  • A locale the system does not have is not an error, only a warning that depends on the OS: here OS reports request to set locale to "xx_XX.UTF-8" cannot be honored, with the empty string as the result and the old value kept. Always check the value returned by Sys.setlocale(). An unknown category, on the other hand, is an error (invalid 'category' argument).

  • LC_ALL is not everything. After Sys.setlocale("LC_ALL", "C") the LC_TIME category is C, while LC_MESSAGES kept its old value. Sys.getlocale() with no argument returns one long string of all categories whose format depends on the OS (;-separated on Linux and Windows, / on macOS), and the help page says it should not be fed back to Sys.setlocale(); query categories individually, as the script does to restore them.

  • LC_NUMERIC warns. Setting it to anything other than "C" gives setting 'LC_NUMERIC' may cause R to function strangely. The help page explains: input conversions in R itself are unaffected, but reading and writing ASCII save files will be, as may packages that do their own input and output; and that to produce graphical or text output with a decimal comma options(OutDec) is often preferable. Setting it back to "C" gives no warning (since R 3.0.3). The script shows print(1234.5) giving 1234.5 and, with options(OutDec = ","), 1234,5.

  • Sys.localeconv() returns the conventions of the current locale as a named character vector (decimal point, thousands separator, currency symbol and many more); in this session LC_NUMERIC is C, so the decimal point is . and the thousands separator is empty (R documentation: Sys.localeconv).

  • The month and weekday names come from LC_TIME. The container has only English, C and POSIX locales, so the script can show only the failed request and the English names. With a German locale installed, the same format(d, "%A %d %B %Y") would print the German weekday and month names; that is the point of the Sys.setlocale() help page example, which is marked as not run, and it was not run here. Dates and Times has the formats.

Locales also come from the environment before R starts. R takes the initial values from the usual environment variables: LC_ALL overrides the individual LC_* variables, which override LANG. A job started by cron, a minimal container or a CI runner may have none of them, and R then runs in the C locale, where characters above 127 cannot be classified; the macOS section of the manual says the same of remote logins and batch jobs. The next script shows what a C LC_CTYPE does to a string that is declared UTF-8.

word <- "caf\u00e9"
nchar(word)
#> [1] 4
l10n_info()[["UTF-8"]]
#> [1] TRUE

# In a C locale R has no multibyte encoding, so non-ASCII text shows as escapes
saved <- Sys.getlocale("LC_CTYPE")
invisible(Sys.setlocale("LC_CTYPE", "C"))
l10n_info()[c("MBCS", "UTF-8")]
#> $MBCS
#> [1] FALSE
#>
#> $`UTF-8`
#> [1] FALSE
#>
nchar(word)
#> [1] 4
word
#> [1] "caf<U+00E9>"
toupper(word)
#> [1] "CAF<U+00E9>"
Encoding(word)
#> [1] "UTF-8"
invisible(Sys.setlocale("LC_CTYPE", saved))

# Back in the UTF-8 session the same string is fine again
word
#> [1] "café"
toupper(word)
#> [1] "CAFÉ"

# A new process that starts in the C locale: an escape is marked UTF-8, but
# an accent typed in the file is just bytes
child <- tempfile(fileext = ".R")
writeLines(c(
  "escaped <- \"caf\\u00e9\"",
  "typed <- \"caf\u00e9\"",
  "print(Sys.getlocale(\"LC_CTYPE\"))",
  "print(Encoding(c(escaped, typed)))",
  "print(nchar(c(escaped, typed)))",
  "print(typed)"
), child)
out <- system2("Rscript", child, stdout = TRUE, env = "LC_ALL=C")
cat(out, sep = "\n")
#> [1] "C"
#> [1] "UTF-8"   "unknown"
#> [1] 4 5
#> [1] "caf\303\251"
unlink(child)

Source: R documentation: l10n_info

In the C locale l10n_info() says there is no multibyte character set, so the declared-UTF-8 string is printed with the code point escaped as <U+00E9> (the nchar() help page notes that for marked-UTF-8 strings the counts are still computed in UTF-8, so nchar() is still 4). The last part of the script starts a new Rscript process with LC_ALL=C: there the escaped string is still marked UTF-8, but an accent typed in the file is only bytes, which print as octal escapes (caf\303\251) and count as 5; these two forms are what a log file from a misconfigured server looks like. Rscript and R CMD BATCH in a C locale are the usual place for this; setting LANG=en_US.UTF-8 (or the name of a UTF-8 locale that the machine has, for example C.UTF-8 where it exists) in the job’s environment fixes it, and the rest of the code is unchanged. See Scripts and the Command Line and Sessions, Workspaces and Startup for how environment variables reach R.

Collation: why sort() depends on the machine

Sorting strings depends on LC_COLLATE. R does not sort strings by bytes unless asked to; it asks the operating system (or ICU, see below) for the order of the current locale, in which accented letters sort with their base letters and, in most English-language locales, apple sorts before Apple before banana. The R FAQ lists this as a difference between R and S: in S the C locale is used, whereas in R the current operating system locale decides which characters are alphanumeric and how they are sorted, which affects ordering in sorts and comparisons such as whether "aA" < "Bb" is true or false (R FAQ 3.3.3: Others).

words <- c("banana", "Apple", "cherry", "apple", "Banana")

# The session's collation order: dictionary order, lower case first
saved <- Sys.getlocale("LC_COLLATE")
sort(words)
#> [1] "apple"  "Apple"  "banana" "Banana" "cherry"

# The C locale compares code points: every capital sorts before every lower case
invisible(Sys.setlocale("LC_COLLATE", "C"))
Sys.getlocale("LC_COLLATE")
#> [1] "C"
sort(words)
#> [1] "Apple"  "Banana" "apple"  "banana" "cherry"
"a" < "B"
#> [1] FALSE
invisible(Sys.setlocale("LC_COLLATE", saved))
"a" < "B"
#> [1] TRUE

# method = "radix" always uses the C-locale order, whatever the session locale
sort(words, method = "radix")
#> [1] "Apple"  "Banana" "apple"  "banana" "cherry"

# Accents: the order is the locale's, not the code points'
accented <- c("zebra", "\u00e9clair", "eclair", "Zulu", "\u00e0 la carte")
sort(accented)
#> [1] "à la carte" "eclair"     "éclair"     "zebra"      "Zulu"
sort(accented, method = "radix")
#> [1] "Zulu"       "eclair"     "zebra"      "à la carte" "éclair"
order(accented)
#> [1] 5 3 2 1 4

# ICU is the collation engine when R was built with it
capabilities("ICU")
#>  ICU
#> TRUE
icuGetCollate()
#> [1] "root"

# icuSetCollate() changes the rules for the session; "es" sorts n-tilde
# after n and "da" sorts aa after z
spanish <- c("\u00f1u", "nz", "nu", "oso")
sort(spanish)
#> [1] "nu"  "ñu"  "nz"  "oso"
icuSetCollate(locale = "es")
icuGetCollate()
#> [1] "es"
sort(spanish)
#> [1] "nu"  "nz"  "ñu"  "oso"
danish <- c("aa", "z", "\u00e5", "b")
icuSetCollate(locale = "da")
sort(danish)
#> [1] "b"  "z"  "å"  "aa"

# Back to the session locale, then one attribute at a time
icuSetCollate(locale = "default")
sort(danish)
#> [1] "å"  "aa" "b"  "z"
mixed <- c("banana", "Apple", "apple", "Banana")
sort(mixed)
#> [1] "apple"  "Apple"  "banana" "Banana"
icuSetCollate(case_first = "upper")
sort(mixed)
#> [1] "Apple"  "apple"  "Banana" "banana"
icuSetCollate(locale = "default")
sort(mixed)
#> [1] "apple"  "Apple"  "banana" "Banana"

Source: R documentation: icuSetCollate

What the output shows:

  • In the session locale sort(words) puts apple before Apple and ignores case first. After Sys.setlocale("LC_COLLATE", "C") it gives Apple Banana apple banana cherry: all capitals before all lower-case letters, by code point. The comparison "a" < "B" follows: TRUE in the session locale, FALSE in the C locale. The Sys.setlocale() help page says the C setting turns off locale-specific sorting "usually (but not on all platforms)".

  • sort(method = "radix") is always in the C order, whatever the locale, and is also the fast method. It is the right choice for reproducible output (keys, snapshots, test expectations) when the order itself does not matter to a human. For character vectors the default method uses the locale.

  • The accented words show why: à la carte sorts before eclair, and éclair after it, in the session locale, while the radix order puts all of the accented words after the ASCII ones. order() returns the same permutation as sort() does.

  • ICU. capabilities("ICU") says whether R was built with ICU (International Components for Unicode), the library that implements the Unicode collation rules. The Installation manual says that ICU gives much more precise control over collation than the OS and is used by default where available (R Installation and Administration: Useful libraries and programs); the icuSetCollate() help page adds that on Windows it is used only if R_ICU_LOCALE is set at start-up or icuSetCollate() is called. icuGetCollate() reports the locale ICU is using; for English locales it reports "root", the default rules, because "most English locales fall back to root" (they contain no special collation rules).

  • icuSetCollate(locale = "es") changes the rules for the whole session: ñu now sorts after nz, as in the Spanish alphabet where ñ is a letter after n; with locale = "da" the Danish å and aa sort after z. icuSetCollate(locale = "default") goes back to the OS locale, and passing a locale resets all earlier customizations. Other arguments tune one attribute: case_first = "upper" flips the case order, the help page gives case_level = "on", strength = "primary" as the way to ignore accent differences, and alternate_handling = "shifted" as the way to ignore space and punctuation. Sys.setlocale("LC_COLLATE", ...) also resets ICU, and since R 3.6.0 a C setting through LC_ALL, LC_COLLATE or Sys.setlocale() takes precedence over R_ICU_LOCALE.

R itself uses the C order wherever reproducibility matters. The default collation of a package’s R files is the C locale unless the Collate field of DESCRIPTION says otherwise, and R CMD check runs the tests with collation set to C (and the examples and tests with LANGUAGE=en) to minimise differences between platforms (Writing R Extensions: Checking packages). Test your own code the same way, rather than relying on the order that your laptop happens to give.

Changing a locale for one expression

Changing a locale and forgetting to restore it is a frequent source of "works in the console, fails in the script" bugs. The withr package (3.0.3 here) has with_locale(), which sets categories for the duration of one expression, and local_locale(), which does so until the end of the calling function; both restore the old values on exit, even after an error (withr package documentation: with_locale). Without it, save the value of Sys.getlocale(category) and restore it in on.exit(), as the earlier scripts did by hand.

words <- c("banana", "Apple", "cherry", "apple", "Banana")
packageVersion("withr") >= "3.0.0"
#> [1] TRUE

# with_locale() sets categories for one expression and restores them after
withr::with_locale(c(LC_COLLATE = "C"), sort(words))
#> [1] "Apple"  "Banana" "apple"  "banana" "cherry"
sort(words)
#> [1] "apple"  "Apple"  "banana" "Banana" "cherry"

# local_locale() does the same until the end of the function that calls it
by_code_point <- function(x) {
  withr::local_locale(c(LC_COLLATE = "C"))
  sort(x)
}
by_code_point(words)
#> [1] "Apple"  "Banana" "apple"  "banana" "cherry"
sort(words)
#> [1] "apple"  "Apple"  "banana" "Banana" "cherry"

# The categories are the same ones as Sys.setlocale()
Sys.getlocale("LC_COLLATE")
#> [1] "en_US.UTF-8"

Source: withr package documentation: with_locale

The language of messages

The language of R’s messages is a third setting, separate from the encoding and from the locale categories above. The Installation manual defines how it is chosen: the preferred language is by default taken from the locale, and this can be overridden first by the environment variable LANGUAGE and then by LC_ALL, LC_MESSAGES and LANG; LANGUAGE is used only to select the language for messages (R Installation and Administration: Localization of messages). The same section says more:

  • LANGUAGE can hold a colon-separated list in decreasing order of preference, such as se:de or es_MX:es; the special value en@quot gives American messages with the pairs of single quotes translated to Unicode directional quotes.

  • Messages are divided into domains: R for the C-level messages of the interpreter, R-pkg for the R-level stop(), warning() and message() texts of each package (R-base for the base package), pkg for the C-level messages of each package, and RGui for the menus of the Windows GUI. This is what makes the system extensible: when a package is loaded, its catalogues can be loaded too.

  • Translations are looked up as specifically as possible, so an Austrian de_AT catalogue is used in preference to a generic German one, and a catalogue that exists but lacks a translation falls back to the less specific ones. R has catalogues for es (Spanish as written in Spain, used by default in Latin America too) and pt_BR (used for Brazil but not Portugal), and for en_GB, which translates the Americanisms of the standard messages (gray) into British English.

  • If no catalogue is found, or a message is not translated, English is used. A translation in the right language but the wrong charset is re-encoded on the fly.

  • Translated messages may be cached, so changing the language of an error that has already been shown in another language may not work. On Windows Sys.setlocale("LC_MESSAGES", ...) is not available.

R 4.2.0 added Sys.setLanguage(lang, unset = "en"), described in its NEWS entry as a partly experimental utility, solving the main problem of PR#18055 (R NEWS: Changes in R 4.2.0). It sets LANGUAGE, flushes the cache with bindtextdomain(NULL) and returns the previous setting invisibly, with an attribute ok that tells whether it worked. R 4.5.0 made it warn about some failures to change the language. The documentation of gettext() covers it with the translation functions (R documentation: gettext).

# Where the language of messages comes from (an example value: yours differs)
Sys.getenv("LANGUAGE")
#> [1] ""
Sys.getenv("LANG")
#> [1] "en_US.UTF-8"

# Is the translation machinery built in?
capabilities("NLS")
#>  NLS
#> TRUE
is.character(bindtextdomain("R"))
#> [1] TRUE

# The English text of warnings that R raises itself
first_warning <- function(expr) tryCatch(expr, warning = conditionMessage)
first_warning(log(-1))
#> [1] "NaNs produced"
first_warning(as.integer("twelve"))
#> [1] "NAs introduced by coercion"
first_warning(1:3 + 1:2)
#> [1] "longer object length is not a multiple of shorter object length"

# Sys.setLanguage() sets LANGUAGE and returns the previous value
previous <- Sys.setLanguage("es")
previous
#> [1] "en"
#> attr(,"ok")
#> [1] TRUE
first_warning(log(-1))
#> [1] "Se han producido NaNs"
first_warning(as.integer("twelve"))
#> [1] "NAs introducidos por coerción"
first_warning(1:3 + 1:2)
#> [1] "longitud de objeto mayor no es múltiplo de la longitud de uno menor"

# gettext() finds the same translations when given the domain "R"
gettext("NaNs produced", domain = "R")
#> [1] "Se han producido NaNs"
gettext("NaNs produced", domain = NA)
#> [1] "NaNs produced"
gettext("NaNs produced")
#> [1] "NaNs produced"

# Other languages in the same session
invisible(Sys.setLanguage("de"))
first_warning(log(-1))
#> [1] "NaNs wurden erzeugt"
invisible(Sys.setLanguage("fr"))
first_warning(log(-1))
#> [1] "Production de NaN"

# Back to English, with the value Sys.setLanguage() returned
invisible(Sys.setLanguage(previous))
first_warning(log(-1))
#> [1] "NaNs produced"

# A language without a catalogue is accepted: no message, no translation
res <- Sys.setLanguage("tlh")
attr(res, "ok")
#> [1] TRUE
first_warning(log(-1))
#> [1] "NaNs produced"
invisible(Sys.setLanguage("en"))

Source: R documentation: Sys.setLanguage

The test container has the Spanish, German and French catalogues of R installed, so the three languages give real translations: Se han producido NaNs, NaNs wurden erzeugt and Production de NaN for the same NaNs produced warning, and NAs introducidos por coerción and longitud de objeto mayor no es múltiplo de la longitud de uno menor for two others. If the build does not have the catalogues, the same calls return the English text and ok can be FALSE with the warning no natural language support or missing translations. capabilities("NLS") and bindtextdomain("R") (non-NULL only when translation is available) tell which case you are in. A language for which no catalogue exists is accepted silently: Sys.setLanguage("tlh") returned ok TRUE and the message stayed English, which is documented behaviour ("no translation will happen, without any message").

The language is read when a message is looked up, but a message that was already looked up in one language stays in it. The next script sets LANGUAGE by hand with Sys.setenv(), and then flushes the cache.

first_warning <- function(expr) tryCatch(expr, warning = conditionMessage)

# Setting LANGUAGE by hand: a message already looked up keeps its language
first_warning(log(-1))
#> [1] "NaNs produced"
Sys.setenv(LANGUAGE = "es")
first_warning(log(-1))
#> [1] "Se han producido NaNs"
first_warning(as.integer("twelve"))
#> [1] "NAs introducidos por coerción"
Sys.setenv(LANGUAGE = "de")
first_warning(log(-1))
#> [1] "Se han producido NaNs"
first_warning(as.integer("twelve"))
#> [1] "NAs introducidos por coerción"

# bindtextdomain(NULL) empties the cache; that is what Sys.setLanguage() adds
bindtextdomain(NULL)
#> [1] TRUE
first_warning(log(-1))
#> [1] "NaNs wurden erzeugt"
invisible(Sys.setLanguage("en"))
first_warning(log(-1))
#> [1] "NaNs produced"

Source: R documentation: gettext

After Sys.setenv(LANGUAGE = "es") the log(-1) warning, which had already been shown in English, is now Spanish, but after Sys.setenv(LANGUAGE = "de") it stays Spanish while the other warning, as.integer("twelve"), which was Spanish too, also stays Spanish, because both had been looked up. bindtextdomain(NULL) empties the cache, after which the next lookup is German; that is the whole of what Sys.setLanguage() adds to Sys.setenv(LANGUAGE = ...). In a script, call it before the first message you want translated, or use the variable LANGUAGE in the environment of the process.

Two more things about the language. It follows the OS default and the locale when LANGUAGE is not set, so a Spanish laptop shows Spanish errors; code that parses messages (grepl("not found", conditionMessage(e))) breaks there, and the condition class or the arguments of the condition are the robust thing to test (see Conditions and Error Handling). And messages written in a bug report should be in English: set LANGUAGE=en (or Sys.setLanguage("en")) before reproducing, which is what R CMD check does for tests and examples.

Translating your own messages

gettext(), ngettext() and gettextf() are the building blocks that stop(), warning() and message() use. gettext(..., domain = NULL, trim = TRUE) looks up each string in the message catalogue of a domain and returns it unchanged when there is none. If domain is NULL, the domain is inferred: a call made from a function in the namespace of package pkg (including via stop(), warning() or message()) uses the domain R-pkg; at the top level there is no default domain and nothing is translated. domain = NA suppresses translation (as in stop("...", domain = NA)), and domain = "" is the same. With trim = TRUE white space around the string is ignored when looking for the translation; trim = FALSE (new in R 4.2.0) is for C-level messages such as "Execution halted\n". ngettext(n, msg1, msg2) chooses between the singular and plural forms of a message by the integer n; in English msg1 is returned for n == 1 and msg2 for 0, 2, 3 and so on, but the plural rules of other languages differ and are the translators' business, which is why you give the whole sentence for each form. gettextf(fmt, ...) is sprintf() with a translated format (R documentation: gettext, R documentation: sprintf).

source("scenario.R")

# gettext(): look the text up in a message catalogue
# With no catalogue, or domain = NA, the same text comes back
gettext("Return date")
#> [1] "Return date"
gettext("Return date", domain = NA)
#> [1] "Return date"

# ngettext(n, singular, plural): English picks the first for n == 1
for (n in 0:2) {
  print(ngettext(n, "book", "books"))
}
#> [1] "books"
#> [1] "book"
#> [1] "books"

# Translate whole sentences, not words, because plural rules differ
returned <- loans[!is.na(loans$return_date), ]
days_out <- as.integer(returned$return_date - returned$loan_date)
overdue <- sum(days_out > loan_days, na.rm = TRUE)
one <- "%d returned loan was overdue\n"
many <- "%d returned loans were overdue\n"
cat(sprintf(ngettext(overdue, one, many), overdue))
#> 25 returned loans were overdue
cat(sprintf(ngettext(1L, one, many), 1L))
#> 1 returned loan was overdue

# gettextf() is sprintf() applied to a translated format
with_isbn <- sum(nzchar(books$isbn))
gettextf("%d books on the shelf, %d with an ISBN", nrow(books), with_isbn)
#> [1] "16 books on the shelf, 15 with an ISBN"

# Sys.setLanguage() refuses to change the language in a C locale
old <- Sys.getlocale("LC_CTYPE")
invisible(Sys.setlocale("LC_CTYPE", "C"))
withCallingHandlers(
  invisible(Sys.setLanguage("es")),
  warning = \(w) {
    cat("warning:", conditionMessage(w), "\n")
    invokeRestart("muffleWarning")
  }
)
#> warning: in a C locale: cannot set language
invisible(Sys.setlocale("LC_CTYPE", old))
invisible(Sys.setLanguage("en"))

# and it checks its argument
tryCatch(Sys.setLanguage("Spanish"), error = conditionMessage)
#> [1] "lang == \"C\" || grepl(\"^[a-z][a-z]\", lang) is not TRUE"
tryCatch(Sys.setLanguage(c("es", "fr")), error = conditionMessage)
#> [1] "length(lang) == 1L is not TRUE"

Source: R documentation: gettext

The script counts how many of the 60 loans of the shelf scenario were returned late, 25, and builds the sentence with sprintf(ngettext(n, one, many), n). Note the pattern: the format contains the number as %d, both forms are complete sentences, and the number is substituted afterwards. Gluing words, as in paste(n, ngettext(n, "book", "books")), hands the translator a fragment with no context and no way to reorder it. The second half of the script shows the two guards of Sys.setLanguage(): it warns in a C locale: cannot set language when the LC_CTYPE category is C (the language of the message cannot be output in that character set) and returns ok FALSE, and it checks its argument with stopifnot() (Spanish is not a language code; the argument must be a single string starting with two lower-case letters, or "C").

The package side

A package that wants its messages translated needs no code beyond using gettext-family functions: translation of the strings inside stop(), warning() and message() calls is automatic, and so are the messages enclosed in gettext() and gettextf() (domain = NA turns it off). What is needed is the catalogue, and Writing R Extensions describes the process in one section (Writing R Extensions: Internationalization). The catalogue files live in the po/ directory of the source package: a template R-pkg.pot (pkg.pot for the C-level messages), and one R-ll.po (or ll.po) per language code ll, which msgfmt compiles to the binary .mo files that are installed. tools::xgettext2pot() builds the R-level template from all the strings in gettext, gettextf, stop, warning and message calls, and tools::xgettext() returns the strings it found, which is more useful when tidying up error messages; both are pure R, while msgfmt, msgmerge and xgettext come from GNU gettext, which must be installed. tools::update_pkg_po() automates the maintenance: with no translations it creates po/, the template and the en@quot pseudo-language (which turns quotes into directional quotes in UTF-8 locales), and on later runs it merges updated templates into the .po files, marks changed entries as fuzzy (fuzzy translations are not used, so the coverage report tells you which need a human), checks C-style formats with tools::checkPofile() and installs the result (Writing R Extensions: Preparing translations). Corrections to the translations of R itself go to the translation teams listed at developer.r-project.org. The community package potools offers a friendlier workflow around the same files; it is named here as community material and was not run. The last script of the page writes a tiny source tree with four translatable messages and lets xgettext2pot() produce the template, which is the only step of the pipeline that does not need the GNU tools; the header of the file, with a creation date, is cut from the output.

# A tiny package source tree with translatable messages
pkg <- file.path(tempdir(), "shelfpkg")
dir.create(file.path(pkg, "R"), recursive = TRUE)
writeLines(c(
  "report_overdue <- function(n) {",
  "  if (n == 0L) message(\"No overdue loans\")",
  "  fmt <- ngettext(n, \"%d loan is overdue\", \"%d loans are overdue\")",
  "  message(sprintf(fmt, n))",
  "  if (n > 100L) warning(\"that is a lot of overdue loans\")",
  "  stop(gettextf(\"unknown member %s\", sQuote(\"x\")), call. = FALSE)",
  "}"
), file.path(pkg, "R", "report.R"))

# xgettext() lists the strings it finds, xgettext2pot() writes the template
found <- tools::xgettext(pkg)
length(found)
#> [1] 1
lengths(found, use.names = FALSE)
#> [1] 4
pot <- file.path(tempdir(), "R-shelfpkg.pot")
tools::xgettext2pot(pkg, pot)
lines <- readLines(pot)

# Print the entries, without the header that holds a creation date
cat(lines[-seq_len(which(lines == "")[[1L]])], sep = "\n")
#> msgid "No overdue loans"
#> msgstr ""
#>
#> msgid "that is a lot of overdue loans"
#> msgstr ""
#>
#> msgid "unknown member %s"
#> msgstr ""
#>
#> msgid        "%d loan is overdue"
#> msgid_plural "%d loans are overdue"
#> msgstr[0]    ""
#> msgstr[1]    ""
unlink(pkg, recursive = TRUE)

Source: R documentation: xgettext

xgettext() found four strings in the single file (the plural pair counts as one message), and the template has one msgid for each. Note how the plural message is a msgid/msgid_plural pair with a msgstr[0] and a msgstr[1]; translators for languages with more plural forms add msgstr[2] and so on and state the rule in the Plural-Forms header. A .po file filled in for Spanish, compiled and installed, makes every message(sprintf(fmt, n)) of the package speak Spanish to users whose language is es. See Creating Packages for the rest of the package layout. Writing R Extensions also notes that it is safest to give domain = "R-pkg" explicitly in ngettext() calls, and that it is necessary in earlier versions of R unless the call is made directly from a function in the package.

Common errors and warnings

The table collects the messages captured above. All were produced on R 4.6.1 in a UTF-8 session; those marked "Windows" or "build" depend on the system and are described, not run.

Message Where it comes from What to do

invalid multibyte string, element 1

nchar() on a string whose bytes are not valid in the native encoding.

Find where the encoding was lost; convert with iconv() or read with fileEncoding. nchar(allowNA = TRUE) gives NA.

invalid multibyte string 1, invalid input multibyte string 1

toupper(), tolower(), chartr().

The same.

invalid multibyte string at '<e9>'

substr() or substring() reaching the invalid byte.

The same.

input string 1 is invalid UTF-8 (warning or error)

strsplit(), gsub(perl = TRUE), trimws(), regexpr(perl = TRUE).

The same; the PCRE engine is stricter than the default one.

unable to translate 'caf<e9>' to a wide string

grepl(), sub(), regexec() and others.

The same. The result is usually NA or FALSE, so the rows vanish silently.

input string 1 is invalid in this locale

regexpr(fixed = TRUE) and the other fixed-string matchers.

The same.

invalid input 'caf<e9>' in 'utf8towcs'

toupper() on a string that is declared UTF-8 but is not.

Do not declare an encoding that the bytes do not have.

invalid multibyte character in parser (<input>:2:0)

parse(), source(), eval(parse(...)) on text with an invalid byte.

Read with encoding or convert the file; keep source files in UTF-8 or ASCII.

embedded nul in string: 'a\0b', nul character not allowed (<input>:1:4)

rawToChar() and the parser with a NUL byte.

Use skipNul = TRUE in readLines() and scan(), or work with raw vectors.

number of characters is not computable in "bytes" encoding, element 1

nchar() on a string marked "bytes".

Use nchar(type = "bytes"), or convert the string with iconv() first.

translating strings with "bytes" encoding is not allowed

toupper() and other functions that must re-encode a "bytes" string.

The same.

unsupported conversion from 'no-such-encoding' to 'UTF-8'

iconv() with an encoding name the system does not know.

Look the name up in iconvlist(), or use "UTF-8", "latin1", "ASCII".

invalid char string in output conversion

Writing text that the connection’s encoding cannot represent.

Choose an encoding that has the character (usually UTF-8) or replace it first with iconv(sub = ...).

line 1 appears to contain embedded nulls

Reading a UTF-16 or UTF-32 file as if it were a byte-oriented text file.

fileEncoding = "UTF-16LE" (or the encoding of the file).

OS reports request to set locale to "xx_XX.UTF-8" cannot be honored

Sys.setlocale() with a locale the system lacks (a warning on Linux; the value returned is "").

locale -a (or the Windows list) shows the names; install the locale.

invalid 'category' argument

Sys.setlocale() with a category that is not one of .LC.categories.

Use a category from .LC.categories.

setting 'LC_NUMERIC' may cause R to function strangely

Sys.setlocale("LC_NUMERIC", ...) to anything other than "C".

Use options(OutDec = ",") or format(decimal.mark = ",").

in a C locale: cannot set language

Sys.setLanguage() when LC_CTYPE is C.

Run R in a UTF-8 locale (LANG).

no natural language support or missing translations (build)

Sys.setLanguage() on a build of R without translations.

Use a build with NLS enabled; messages stay English.

Source: R documentation: gettext

Version notes

The rows for R 3.x come from the NEWS of the 3.x series (R NEWS: Changes in R 3.0.0, 3.0.3 and 3.6.0); the others from the NEWS of the 4.x series.

Version Change

R 3.0.0

readLines() and scan() (hence read.table()) in a UTF-8 locale discard a UTF-8 byte order mark; the connection encoding name "UTF-8-BOM" discards it in any locale.

R 3.5.0

Undefined or control positions of ISO 8859-1 are interpreted as Windows code page 1252 when converting, for example to UTF-8, where the system allows it (from the Encoding help page).

R 3.6.0

Setting the C locale for collation through LC_ALL, LC_COLLATE or Sys.setlocale() takes precedence over the environment variable R_ICU_LOCALE.

R 4.1.0

l10n_info() gains codeset (non-Windows) and system.codepage (Windows).

R 4.2.0

UTF-8 is the native encoding on recent Windows, with UCRT as the C runtime; Sys.setLanguage() is added; gettext() and gettextf() get trim.

R 4.2.1

Rgui regressions of the UTF-8 change fixed; the script editor needs UTF-8 scripts.

R 4.3.0

readLines(encoding = "bytes"); iconv(from = "") takes declared encodings into account; gsub(), sub() and strsplit() mark output as "bytes" when an input is.

R 4.3.2

iconv(to = "ASCII//TRANSLIT") is emulated by substitution where the OS lacks it.

R 4.3.3

iconv() fixes up variant names such as "utf8" case-insensitively.

R 4.4.0

iconv(sub = "Unicode") always zero-pads to four hexadecimal digits.

R 4.5.0

Sys.setLanguage() warns about some failures to change the language; R CMD check no longer reports marked UTF-8 or Latin-1 strings in datasets by default.

Source: R NEWS: Changes in R 4.1.0 to 4.5.0

Where older tutorials differ

  • "Fix garbled text with Encoding(x) <- "UTF-8"." That declares, it does not convert. It repairs text only when the bytes already are UTF-8 and the mark was missing; on Latin-1 bytes it makes things worse. Use iconv(), enc2utf8() or a fileEncoding.

  • "Windows cannot do UTF-8, convert everything to Latin-1 or cp1252." True before R 4.2.0. Current Windows builds use UTF-8 as the native encoding and UCRT locale names ending in .UTF-8; code written around code pages (Sys.setlocale("LC_ALL", "English"), Sys.setlocale("LC_COLLATE", "German")) should be revisited.

  • options(encoding = "UTF-8") at the top of every script, or read.csv(..., encoding = "UTF-8") to convert. The option is global state, and the encoding argument only declares; fileEncoding is the one that converts.

  • fileEncoding = "UTF-8-BOM" on every spreadsheet CSV. Needed only where the session is not UTF-8; since R 3.0.0 a UTF-8 session discards the mark by itself.

  • iconv(x, to = "ASCII//TRANSLIT") to make accent-free keys. Its result differs between glibc, macOS and musl. Use a fixed mapping (chartr()) or stringi::stri_trans_general() when the result must be the same everywhere.

  • Sys.setenv(LANGUAGE = "en") to get English messages mid-session. A message already looked up keeps its language; Sys.setLanguage("en") (R 4.2.0) also empties the cache.

  • "R sorts like S, in C order." R uses the locale of the session, so sort() gives different orders on different machines; the FAQ lists it as a difference from S. Use method = "radix" or LC_COLLATE = "C" for reproducible order and icuSetCollate() when you need a specific language’s rules.

  • Non-ASCII characters in package R code or comments, with Encoding: latin1 in DESCRIPTION. Use \uxxxx escapes in strings and keep comments ASCII; declare Encoding: UTF-8 if the package has any non-ASCII file.

  • Messages hard-coded in French or German in package code. Writing R Extensions asks for the internationalization facilities instead.

The wider list of differences between old and current R is on What’s New and Migrating Older Code.

Where to go next

References