Serialization: JSON, YAML, CSV and Marshal

JSON.parse(source, **options) turns a string into Ruby objects. Keys of objects are strings unless symbolize_names: true is given. Symbolised keys are convenient with dig, with keyword arguments and with hash patterns, and they are safe: unlike the interned symbols of old Rubies, symbols are garbage collected, so a hostile document cannot exhaust memory through key names. JSON.parse accepts any JSON value at the top level, not only an object or an array. A failure raises JSON::ParserError whose message carries the line and column; nil is not a string, so JSON.parse(nil) is a TypeError that you must prevent yourself (a missing HTTP body or an unset environment variable is the usual source).

Source: Ruby 4.0 documentation: JSON.parse and JSON::ParserError

The error message format changed between json versions (Ruby 3.3 says unexpected token at '{"a": }'), so match on the class JSON::ParserError, never on the text. JSON.load_file(path, **options) reads and parses a file, and JSON.parse! / JSON.load_file! are variants with the dangerous defaults (unlimited nesting, allow_nan) that exist for trusted input only.

JSON.generate(object) and object.to_json produce a compact document, and JSON.pretty_generate(object) an indented one (two spaces by default; indent:, space:, object_nl: and array_nl: change the layout). to_json is defined for Hash, Array, String, Integer, Float, true, false and nil. Every other object inherits the generic Object#to_json, which is to_s.to_json: a string with the object’s description in it. That is the most common serialization surprise, and it is the reason Data, Struct, Range, Set and Time all come out as quoted strings.

Rules worth knowing: hash keys are converted with to_s (so nil becomes the empty string, 1 becomes "1" and two keys such as 1 and "1" collide); symbols become strings and cannot be told apart on the way back; NaN and the infinities raise JSON::GeneratorError because JSON has no representation for them; strings must be valid in their encoding (a byte sequence that is not valid UTF-8 raises JSON::GeneratorError, and so does a BINARY string with high bytes, whose message is "\xFF" from ASCII-8BIT to UTF-8; a BINARY string that is pure ASCII is accepted); big integers are written in full (10**30) and are exact in Ruby but may be rounded by a JavaScript reader, which is why identifiers above 2**53 are often sent as strings. strict: true makes the generator reject anything that is not one of the six JSON kinds instead of falling back to to_s, which turns the silent surprise above into an error you can see. script_safe: true escapes / and the line separators U+2028 and U+2029 so the output can be placed inside an HTML <script> element, and ascii_only: true escapes all non-ASCII characters.

Source: Ruby 4.0 documentation: JSON.generate, JSON.pretty_generate and JSON::GeneratorError

Times are the other recurring question. Time#to_json gives "1970-01-01 00:00:00 UTC", which is not the ISO 8601 form most APIs expect. Write the format you mean: require "time" and call time.utc.iso8601(3) (or Time.iso8601 to read it back), and keep dates as ISO 8601 strings in the document. DateTime is deprecated and not needed for this (see Time and Dates). Money and other exact decimals should travel as strings or integer cents, not floats: JSON.parse(text, decimal_class: BigDecimal) parses non-integer numbers directly into BigDecimal (the bigdecimal gem is a bundled gem since 3.4, so it needs a Gemfile entry under Bundler).

There is no as_json in plain Ruby (that is a Rails convention). The idiom is two small methods on the class: a to_h that builds the plain hash, and a to_json(*) that forwards to it. Writing to_json(*) rather than to_json matters: JSON.pretty_generate and nested generation pass a state object to every to_json they reach, and an override that drops the argument loses the pretty-printing and the depth tracking. Reading is the mirror image, a class method such as from_h that uses fetch so a missing key is a KeyError with the key’s name instead of a nil that fails far away. Because from_h calls the constructor, the Shelf::Isbn validation runs on the way in: untrusted JSON cannot produce an invalid ISBN.

JSON::Coder (new in the json shipped with Ruby 4.0; Ruby 3.4 does not have it) offers a single place to say what happens to objects JSON does not know: it takes a block that is called for each such object and returns something JSON can represent. The same coder dumps and loads, and its load has none of the legacy additions.

Source: Ruby 4.0 documentation: JSON::Coder and Data

JSON.parse is deliberately strict. The options below loosen it for specific, known input.

Generation has matching options: max_nesting, allow_nan, strict, script_safe, ascii_only, indent, space, object_nl and array_nl. JSON.dump(object) and JSON.load(source) are the older, more lenient pair: dump tolerates NaN and writes to an IO when given one; load returns nil for nil or an empty string and accepts a callable that sees every parsed value. Prefer parse/generate in new code; use load only when you need those leniencies.

The trap is the json/add/* family (json/add/range, json/add/time, json/add/set, json/add/struct, json/add/bigdecimal, json/add/core and others). They teach classes to write a "json_class" key and to rebuild themselves from it: with create_additions: true, a document that says "json_class": "Range" is turned into a Range, and any class that defines json_create can be named by the attacker the same way. That is object injection, the JSON counterpart of loading a YAML or Marshal stream. JSON.parse leaves it off, whereas JSON.load switches it on implicitly, which json 2.18 still does while printing a deprecation warning (JSON.load implicit support for create_additions: true is deprecated and will be removed in 3.0, use JSON.unsafe_load or explicitly pass create_additions: true). The rule: never require "json/add/..." in code that parses outside data, and never pass create_additions: true or call JSON.unsafe_load on it. A "json_class" key without the opt-in is plain data.

Source: Ruby 4.0 documentation: JSON.load, JSON.unsafe_load and JSON::NestingError

Large inputs: JSON.parse reads the whole document into memory. For a stream of records use JSON Lines (one document per line, not part of the JSON standard) and parse each line separately with File.foreach(path) { |line| JSON.parse(line) }; there is no streaming parser in the standard library.

The loaders are where YAML gets dangerous: a YAML document may carry a tag such as !ruby/object:Foo, and the loader will allocate a Foo, set its instance variables and even call methods defined on it. Psych 4.0 (Ruby 3.1) closed this by default: YAML.load now behaves like safe_load with Symbol permitted, and the old behaviour has to be asked for by name, YAML.unsafe_load. Older books and blog posts that call YAML.load on untrusted data as a normal practice describe the unsafe Psych 3 behaviour (Ruby 3.0 and before). The loaders compare as follows.

The extra arguments of safe_load (and of load, which accepts the same keywords) are:

The rule of thumb: use safe_load with an explicit permitted_classes for everything and unsafe_load for nothing. If you need an object of your own class back, convert it from a hash after loading (Book.from_h(hash)), exactly as with JSON, rather than letting the document name classes.

Source: Ruby 4.0 documentation: Psych.safe_load, Psych.load, Psych.unsafe_load and Psych::DisallowedClass

The exceptions form a small family: Psych::Exception (a RuntimeError) is the root, Psych::SyntaxError reports malformed text and carries file, line, column, problem and context, Psych::DisallowedClass reports a refused class or symbol and Psych::BadAlias / Psych::AliasesNotEnabled report an alias that is undefined or not permitted. A missing file raises the usual Errno::ENOENT from YAML.load_file and YAML.safe_load_file. A duplicate key in a mapping is not an error in Psych: the last one wins silently, which hides copy-and-paste mistakes in hand-edited configuration.

Because YAML.load accepts Symbol, a file written with to_yaml from a hash with symbol keys (:name: Ico, with a leading colon) reads back with symbol keys. That is a Ruby-only dialect: a Python or JavaScript consumer sees the colon as part of the key. Dump string keys (config.transform_keys(&:to_s).to_yaml) for files other tools read.

An anchor (&defaults) names a node and an alias (*defaults) reuses it, which removes repetition in configuration files such as CI definitions and Rails database.yml. The merge key <<: *defaults copies the anchored mapping into the current one, and keys written after it override the copied ones. Psych resolves the merge key; plain YAML 1.2 does not define it. The same machinery is why aliases are off by default in safe_load: a document of nested aliases can expand to an enormous structure (the "billion laughs" attack), so aliases: true belongs to files you trust. When dumping, Psych writes an anchor wherever the same object appears twice, and reading it with unsafe_load or aliases: true restores the sharing, as the last lines of the example show.

Source: Ruby 4.0 documentation: Psych.load_stream, Psych::AliasesNotEnabled and Psych::BadAlias

An unquoted YAML scalar is typed by its appearance, and Psych follows YAML 1.1: yes, no, on, off, y and n in several capitalisations are booleans, ~ and an empty value are nil, 010 is octal 8, 0x1f is hexadecimal, underscores are digit separators, 1:30 is a base-60 number (5400) and 2024-01-05 is a Date. The classic casualty is the country code NO, which loads as false (the "Norway problem"). Version numbers written as 1.10 load as the float 1.1. The defence is to quote anything that is text ("1.10", "NO", "0123") and to check types after loading; Psych does the reverse when dumping and quotes any string that would be misread, which is why YAML.dump("code" => "0123") writes '0123'. A single-letter or boolean-looking key such as n is even quoted as a key.

Objects dump through their instance variables by default; a class can control the document by defining encode_with(coder) (write entries into the coder hash-like object) and init_with(coder) (read them back), which is how to keep caches and handles out of a file. Struct dumps as !ruby/struct:Name. Data dumps as !ruby/data:Name with the Psych that ships with Ruby 4.0; Ruby 3.4’s Psych 5.2.2 wrote an empty !ruby/object:Name {} and lost the members, so a YAML file with Data instances is not portable to older Rubies. Set dumps as !ruby/object:Set with a hash: mapping. None of these tags can be loaded by safe_load unless the class is permitted, and for a Struct or Data the member names are symbols, so Symbol has to be permitted too.

Source: Ruby 4.0 documentation: Psych.dump, Psych::Handler and YAML 1.1 specification

Two more Psych facilities are worth knowing. YAML.safe_dump (with permitted_classes, permitted_symbols and aliases) refuses to write what safe_load could not read back, a cheap guard against leaking an object graph into a configuration file. YAML.load_stream reads several --- documents, and YAML.parse gives the syntax tree (Psych::Nodes) when you need comments or line numbers, which load discards: Psych does not preserve comments in a load and dump round trip.

The csv library was a default gem until Ruby 3.3 and became a bundled gem in Ruby 3.4: it still installs with Ruby and require "csv" works in a plain script, but under Bundler it must be listed in the Gemfile, or the require fails. Ruby 3.3 already printed a warning about the coming change, and Ruby 4.0 behaves as shown below. A gemspec for a gem that uses CSV needs add_dependency "csv" as well. The same move applied to base64, bigdecimal and observer in 3.4, and to logger, benchmark, ostruct, pstore and others in 4.0; see Standard Library Tour and RubyGems and Bundler for the full table. The experiment below uses a Gemfile with no entry for csv, then one with it. With no entry, Bundler hides the gem and Ruby explains the problem:

After adding one line to the Gemfile:

Source: Ruby 4.0 documentation: Standard library and csv gem documentation

CSV.parse(string) returns an array of arrays of strings, and CSV.parse_line parses one line. CSV.read(path) and CSV.foreach(path) do the same for files, foreach streaming rows without loading the file (it returns an Enumerator without a block). With headers: true, the first row names the columns and each row becomes a CSV::Row, an ordered hash-like object (row["title"], row.to_h, row.fields, row.headers) held in a CSV::Table; a table can be indexed by row (table[0]), by column name (table["year"] returns the column) and iterated like an array. CSV.table(path) is shorthand for read with headers, converters: :numeric and symbol headers.

Everything is a String or nil until you convert it. converters: runs a conversion on each field (:integer, :float, :numeric, :date, :date_time, :time, :all, or any lambda taking the field and, optionally, a CSV::FieldInfo); header_converters: normalises header names (:downcase, :symbol, :symbol_raw). Converters are guesses: :numeric turns the ISBN 9780132350884 into an Integer, which drops leading zeros, so convert by column in your own code when the digits are an identifier. A custom converter should return its input unchanged when it does not apply.

The dialect options are col_sep: (for example ";" or "\t"), row_sep: (default autodetected), quote_char:, skip_blanks:, skip_lines: (a regexp or string), strip:, nil_value: and empty_value: (what an empty and a quoted-empty field become), liberal_parsing: (tolerate stray quotes) and encoding:. For a file starting with a byte order mark, as Excel writes, use encoding: "bom|utf-8". Errors are CSV::MalformedCSVError (with line_number), reported for an unclosed quote or illegal quoting, and CSV::InvalidEncodingError when the bytes do not match the encoding (the usual symptom of reading a Latin-1 export as UTF-8). A repeated header keeps both columns in the row but to_h keeps only the first value.

Source: csv gem documentation: CSV.parse, CSV.foreach and Recipes: parsing

CSV.generate { |csv| csv << row } builds a string, CSV.generate_line(array) one line, and CSV.open(path, "w") { |csv| ... } writes a file. A row is an array or, when headers are set, a hash or a CSV::Row. write_headers: true with headers: writes the header line first. Quoting is automatic: a field is quoted when it contains the separator, a quote or a line break, quotes are doubled, and nil becomes an empty field while "" becomes a quoted empty one (force_quotes: true quotes everything). Pass col_sep: and row_sep: to match the consumer, and open files for writing in the encoding the consumer expects.

Formula injection. A spreadsheet treats a cell that begins with =, `, `-` or `@` as a formula, so data typed by a user into the catalogue (a title such as `=HYPERLINK(…​)+) can run when somebody opens the export. CSV quoting does not prevent it. Neutralise untrusted text when exporting for a spreadsheet by prefixing the dangerous cell with an apostrophe, as the helper in the example does (the apostrophe is the convention that tells spreadsheet programs the cell is text). Do not apply it to data meant for programs, where it would corrupt the value.

Source: csv gem documentation: CSV.generate, CSV.open and RFC 4180

Marshal.dump(object, io = nil, limit = -1) writes to a string, or to an IO when given one, in which case several dumps can be written one after another and read back in order. Marshal.load(source, proc = nil, freeze: false) reads from a string or an IO. The proc form calls the proc for each object as it is rebuilt and uses its return value in place of the object, which is how the deep-copy or filtering tricks work. freeze: true (Ruby 3.1) returns a deeply frozen graph with deduplicated strings; Marshal.restore is an alias of load. The result of dump has the BINARY encoding, so write it with File.binwrite and read it with File.binread.

The best-known use is the deep copy, Marshal.load(Marshal.dump(object)), which is why RuboCop exempts exactly this form from its Security/MarshalLoad cop. It is short but crude: it only works on graphs of dumpable objects, it loses singleton methods, and it is slow compared with writing dup for your own classes (see Object Lifecycle, Freezing and Copying).

Several kinds of object cannot be dumped at all, and raise TypeError: Proc and Method objects, IO objects, Thread, Mutex, Binding, Dir, a Hash with a default proc, an object with singleton methods, and instances of anonymous classes or modules. Attempting to dump a graph that contains one of them fails as a whole, which is the practical reason to keep caches and handles out of the objects you serialize (or define marshal_dump, below).

Source: Ruby 4.0 documentation: Marshal.dump, Marshal.load and Marshal format documentation

A class takes over its serialization with marshal_dump (return any dumpable object that holds the state) and marshal_load(state) (the instance is allocated without calling initialize, and receives the state). That is the way to leave a cache or a handle out and rebuild it lazily. The older pair, an instance method _dump(limit) returning a string and a class method _load(string) returning the object, still works and is what some C extensions use. Both protocols are the reason loading is dangerous: they are code that runs because the data asked for the class. Subclasses of core types and classes with initialize_copy-style needs usually do better with marshal_dump than with the default instance-variable dump.

Source: Ruby 4.0 documentation: Marshal format documentation

Source: Ruby 4.0 documentation: Marshal

The Ruby documentation says it plainly: do not use Marshal.load on data from an untrusted source. The loader resolves class names from the stream, allocates instances of any class that is loaded in the process, restores their instance variables without running initialize, and calls marshal_load, _load, and, while rebuilding hashes and sets, hash and eql? on the restored objects. An attacker who controls the bytes chooses which of those methods run and on what; chaining methods that already exist in the application and its gems (so-called gadget chains) has led to remote code execution in many Ruby applications and gems, and there is no allow-list option in Marshal.load to make it safe. The same holds for YAML.unsafe_load, for JSON.unsafe_load with additions, and for any cookie, cache entry, queue message or file that an outsider can write to, because Marshal data in a shared cache is only as trusted as everyone who can write to the cache. The remedies are to use JSON (or YAML.safe_load) for anything that crosses a trust boundary, to sign data with a secret (OpenSSL::HMAC) and verify the signature before loading if Marshal is unavoidable, and to restrict write access to the storage. The examples below are all harmless; the lesson is that load does not validate.

The second danger is that initialize does not run. The Shelf::Isbn value object checks its digits in initialize, so Isbn.new rejects a bad number, but Marshal.load of tampered bytes builds an Isbn holding it, with no error. Invariants enforced in constructors are not invariants of loaded objects. The third is versioning: if the class changed (a member renamed, a class moved or removed), the stream still names the old shape, and loading raises ArgumentError for a class that no longer exists and silently yields objects with missing or stale instance variables for one that does. A Marshal cache therefore needs a version in its key, and a failed load must be treated as a cache miss.

Source: Ruby 4.0 documentation: Marshal and Ruby 4.0 documentation: Security