Email, Feeds & Web Scraping

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

This page is written against Node.js 26 (26.10.0, released 2026-09-21, the Current line that becomes LTS on 2026-10-28), with the differences of Node.js 24 "Krypton" (24.21.0, Active LTS) and Node.js 22 "Jod" (22.23.3, Maintenance) called out; versions and stability levels were checked against the official documentation on 2026-10-04, and every example below was run on Node.js 26, 24 and 22 against a local catch-all SMTP server and a local test website, with the exceptions listed in What was run, and what was not. None of this needs a new runtime feature: email, feeds and scraping are ecosystem work built on three things Node.js does provide, global fetch (see Fetch & HTTP Clients), the node:crypto primitives for tokens (see Crypto, Passwords & Secrets) and timers for pacing (see Timers). What matters here is the surrounding discipline: a mail you send carries your domain’s reputation, a feed you poll is someone’s server, and a page you scrape belongs to someone who may have said no. The running scenario is the Bookshelf application described in Introduction & Architecture: a newsletter, a news reader and a price watcher.

The stability and version map

Table 1. APIs and packages covered on this page
API or package Stability or version (resolved 2026-10-03) Version notes

Global fetch()

No stability banner; no longer experimental since v21.0.0

Unflagged in v18.0.0. Present in all three lines, so the feed and scraping examples run unchanged on 22, 24 and 26.

AbortSignal.timeout()

Stable

Added in v17.3.0 and v16.14.0. Used as the request deadline in every client example below.

nodemailer

10.0.14 (npm latest)

The Nodemailer documentation states that version 10 requires Node.js 20 or later and ships both an ES module and a CommonJS build. Staying on Node.js 22 is therefore fine; Node.js 18 and older need the 9.x line.

smtp-server, mailparser

3.19.17 and 3.9.34

Maintained by the Nodemailer author. Used here only to build the local catch-all server and to read the messages it receives. They are test tools, not part of a sending application.

cheerio

1.2.0

jQuery-like API over a parsed document; it parses markup and never runs scripts. Its package.json declares engines.node of 20.18.1 or later.

puppeteer

25.12.0

Controls Chrome or Firefox. npm i puppeteer also downloads a browser; puppeteer-core does not. Its package.json declares engines.node of 22.12.0 or later, so it runs on the 22, 24 and 26 lines but not on early 22.x releases.

playwright

1.63.0

Browser automation for Chromium, Firefox and WebKit. Its browsers are installed by a separate command. Declares engines.node of 20 or later.

Everything on the page except fetch and node:crypto is a third-party package, so its version moves independently of the Node.js release lines; treat the numbers above as the state on the date shown and check npm view <package> version before you copy an install line.

Sending email with Nodemailer

Email delivery is a chain of independent systems: your application hands a message to a mail server over SMTP, that server relays it, and the recipient’s provider decides whether to deliver, junk or reject it. Nodemailer is the de facto library for the first hop. It builds the MIME message (headers, text and HTML alternatives, attachments, encodings) and speaks SMTP, so your code deals with objects instead of RFC 5322 text.

Transports

nodemailer.createTransport(options) returns a transporter, and the options select the transport, the mechanism that actually takes the message away.

Table 2. Transports bundled with Nodemailer
Transport Selected by Use it for

SMTP (the default)

host, port, secure, auth; or service for a well-known provider

Talking to any SMTP server or relay. The default port is 587, or 465 when secure is true.

Sendmail

sendmail: true

Piping the message to a local sendmail-compatible binary. Rare on containers, which usually ship none.

Stream and JSON

streamTransport: true, jsonTransport: true

Not delivering at all: unit tests and previews that receive the generated message as a buffer, stream or JSON object.

Amazon SES and community transports

A transport object passed to createTransport()

Posting to an HTTPS API instead of SMTP; the Nodemailer documentation lists the bundled and community options.

Two TLS options on the SMTP transport matter more than the rest. With secure: false (the default) the connection starts in plain text and is upgraded with STARTTLS if the server offers it; with secure: true it is TLS from the first byte, which is what port 465 expects. requireTLS: true makes sending fail instead of silently continuing in plain text when the server does not offer STARTTLS, and ignoreTLS: true disables STARTTLS altogether. Production code should use secure: true on 465 or requireTLS: true on 587 and never ignoreTLS; the examples below set ignoreTLS only because the local test server has no certificate. Credentials and the DKIM key belong in process.env, loaded with --env-file (see CLI Options, Environment & Config File).

For many messages, set pool: true. A pooled transporter keeps maxConnections connections open (default 5) and reuses them for maxMessages messages each, instead of paying a TCP and TLS handshake per mail, and rateDelta with rateLimit caps how many messages may leave per time window, which is how you stay under a provider’s sending limit. Call transporter.close() when you are done, or the pooled connections keep the process alive.

Testing against a local SMTP server

Never develop against a real mailbox. A catch-all SMTP server that accepts every message and delivers nothing is a few lines with the smtp-server package (a stand-alone tool such as Mailpit does the same job with a web inbox, and Nodemailer’s own guide also describes the hosted Ethereal service). Listening on port 0 lets the operating system pick a free port, and the addresses use the reserved .test top-level domain, so nothing can reach a real person.

Install the packages used on this page (scratch project, never in a repository you commit)
npm i nodemailer smtp-server mailparser cheerio puppeteer

Source: Nodemailer: About and Nodemailer: Testing with Ethereal

HTML and text parts, templating and the unsubscribe headers

A message should carry both a text and an html body: Nodemailer sends them as a multipart/alternative pair, and clients that cannot or will not render HTML (and spam filters that distrust HTML-only mail) use the text part. Nodemailer has no template engine of its own; it offers a compile plugin stage (registered with transporter.use()) for template plugins, and for a handful of mails a function that returns { text, html } is enough. The one rule that is not optional is escaping: every value interpolated into HTML is data from somebody, and an unescaped name or URL is an injection into a message that arrives under your domain.

The example sends one confirmation mail with both parts, a List-Unsubscribe header created from the list option, the RFC 8058 List-Unsubscribe-Post header, and a DKIM signature, then reads the message back from the local server.

Generate a throwaway DKIM key pair in the scratch directory and pass the private key through the environment
openssl genrsa -out dkim-private.pem 2048
openssl rsa -in dkim-private.pem -pubout -out dkim-public.pem
DKIM_PRIVATE_KEY="$(cat dkim-private.pem)" node send.mjs
send.mjs: HTML + text, escaping, List-Unsubscribe and DKIM against a local catch-all server
import { once } from 'node:events';
import { SMTPServer } from 'smtp-server';
import { simpleParser } from 'mailparser';
import nodemailer from 'nodemailer';

// Local catch-all SMTP server: accepts everything, delivers nothing.
const inbox = [];
const server = new SMTPServer({
  authOptional: true,
  disabledCommands: ['STARTTLS'],
  onData(stream, session, callback) {
    simpleParser(stream).then((mail) => {
      inbox.push({ envelope: session.envelope, mail });
      callback();
    }, callback);
  },
});
server.listen(0, '127.0.0.1');
await once(server.server, 'listening');
const { port } = server.server.address();

const transporter = nodemailer.createTransport({
  host: '127.0.0.1',
  port,
  secure: false,
  ignoreTLS: true,
});
await transporter.verify();

const esc = (s) => String(s).replace(/[&<>"']/g, (c) => `&#${c.charCodeAt(0)};`);
const render = ({ name, url }) => ({
  text: `Hello ${name},\n\nConfirm your subscription: ${url}\n`,
  html: `<p>Hello ${esc(name)},</p><p><a href="${esc(url)}">Confirm your subscription</a></p>`,
});

const info = await transporter.sendMail({
  from: '"Bookshelf" <news@bookshelf.example.test>',
  to: 'reader@example.test',
  subject: 'Confirm your subscription',
  ...render({ name: 'Ada <b>', url: 'https://bookshelf.example.test/confirm?t=abc' }),
  list: { unsubscribe: { url: 'https://bookshelf.example.test/u/abc', comment: 'Unsubscribe' } },
  headers: { 'List-Unsubscribe-Post': 'List-Unsubscribe=One-Click' },
  dkim: {
    domainName: 'bookshelf.example.test',
    keySelector: 'k1',
    privateKey: process.env.DKIM_PRIVATE_KEY,
    // RFC 8058: the one-click header must be signed too, and it is not in the default list
    headerFieldNames: 'from:to:subject:date:message-id:mime-version:content-type:list-unsubscribe:list-unsubscribe-post',
  },
});
console.log(info.accepted, info.rejected, info.response);
const { envelope, mail } = inbox[0];
console.log(envelope.mailFrom.address, '->', envelope.rcptTo.map((r) => r.address));
console.log(mail.subject, '|', JSON.stringify(mail.text), '|', mail.html);
for (const { key, line } of mail.headerLines) {
  if (['list-unsubscribe', 'list-unsubscribe-post', 'dkim-signature'].includes(key)) {
    const flat = line.replace(/\s+/g, ' ');
    console.log(key === 'dkim-signature' ? flat.replace(/ b=.*/, ' b=…') : flat);
  }
}
transporter.close();
await new Promise((r) => server.close(r));
Output (the signature value and body hash are truncated)
[ 'reader@example.test' ] [] 250 OK: message queued
news@bookshelf.example.test -> [ 'reader@example.test' ]
Confirm your subscription | "Hello Ada <b>,\n\nConfirm your subscription: https://bookshelf.example.test/confirm?t=abc\n" | <p>Hello Ada &#60;b&#62;,</p><p><a href="https://bookshelf.example.test/confirm?t=abc">Confirm your subscription</a></p>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bookshelf.example.test; q=dns/txt; s=k1; bh=…; h=from:to:subject:date:message-id:mime-version:content-type:list-unsubscribe:list-unsubscribe-post; b=…
List-Unsubscribe-Post: List-Unsubscribe=One-Click
List-Unsubscribe: <https://bookshelf.example.test/u/abc> (Unsubscribe)

Source: Nodemailer: Message configuration, Nodemailer: List headers, Nodemailer: DKIM, Nodemailer: SMTP transport and Nodemailer: Plugins

Three observations from running it. The HTML escaping turned the hostile name Ada <b> into text in the HTML part and left it alone in the text part, as it should. transporter.verify() is a cheap start-up check that the server is reachable and accepts the connection. And by default Nodemailer signs a fixed list of headers that includes List-Unsubscribe but not List-Unsubscribe-Post: the h= tag of the signature shows exactly which headers are covered, and the example adds the second one through headerFieldNames because RFC 8058 requires both headers to be covered by the signature. That default was observed with Nodemailer 10.0.14, so print the h= tag for your own version before relying on it.

SPF, DKIM, DMARC and List-Unsubscribe basics

The receiving provider cannot trust the From: header, so it checks three DNS-published policies. Each answers a different question, and mail sent from your own domain should pass all three.

Table 3. The authentication records and what each proves
Mechanism What it checks What you publish

SPF (RFC 7208)

Whether the connecting server’s IP address is allowed to send for the envelope sender’s domain.

A TXT record on the sending domain listing the IP ranges or providers (for example an include: for your mail provider).

DKIM (RFC 6376)

Whether the message was signed by a key that belongs to the signing domain, and whether the signed headers and body are unchanged in transit.

A TXT record at <selector>._domainkey.<domain> holding the public key. The private key signs; Nodemailer’s dkim option does it with domainName, keySelector and privateKey.

DMARC (RFC 7489)

Whether SPF or DKIM passed and the passing domain aligns with the visible From: domain, and what to do when it does not.

A TXT record at _dmarc.<domain>, starting with p=none to collect reports and tightening to quarantine or reject once legitimate senders are aligned.

If you send through a provider’s SMTP relay, the provider usually signs for you once you add its DNS records, and the DKIM option in Nodemailer is for the case where you sign yourself. Keep the private key out of the repository and the image: read it from an environment variable or a secret store, rotate it by publishing a new selector, and remember that the public key in DNS is the only part that is public. The DNS side can be checked without sending anything, for example with dig TXT <selector>._domainkey.<domain>; DNS lookups from Node.js itself are covered in TCP, UDP & DNS.

The unsubscribe headers are the part that is easy to forget. List-Unsubscribe (RFC 2369) tells a mail client where an unsubscribe request goes, so it can show its own "Unsubscribe" button. RFC 8058 adds List-Unsubscribe-Post: List-Unsubscribe=One-Click: the client then sends an HTTPS POST to the URL by itself, without the person visiting a page, and the endpoint must unsubscribe immediately with no login or confirmation. Mailbox providers have tied these to delivery. Google’s sender guidelines require senders of more than 5,000 messages a day to Gmail accounts to authenticate with SPF, DKIM and DMARC, support one-click unsubscribe and keep user-reported spam rates low (under 0.3%), and the Yahoo sender guidance likewise asks for DMARC and an easy, one-click unsubscribe; read the current pages, because thresholds change.

Source: RFC 7208: SPF, RFC 6376: DKIM, RFC 7489: DMARC, RFC 2369: List-Unsubscribe, RFC 8058: One-Click Unsubscribe and Gmail: Email sender guidelines

Sending mail to people is regulated and policed, and the technical design should make the right behaviour the easy one. Email only people who asked, record when and how they did, and make leaving as easy as joining. The usual pattern is double opt-in: the sign-up form only stores a pending subscription and sends a confirmation message, and the address becomes active when the person follows the link in it. That proves the address belongs to the person (so nobody can subscribe a stranger) and gives you a record of consent. Two details are easy to get wrong. The token in the link must be unguessable and single use, so generate it from crypto.randomBytes(), store only its hash with an expiry time, and look the pending subscription up by that hash; an email address in a GET link is not a token, and the address does not belong in the link at all. And the confirmation and unsubscribe endpoints change state, so they need the protections described in Security Best Practices; the one-click unsubscribe endpoint in particular must accept the unauthenticated POST that mail providers send. Legal requirements depend on where you and your recipients are, so ask someone qualified rather than inferring them from a library’s documentation; tracking pixels, which the books' mailer examples use, add a consent question of their own and are better avoided.

The example below combines the token handling with a pooled transporter that is rate limited to two messages per second, and runs against its own local server.

consent.mjs: double opt-in tokens and a rate-limited pool
import { once } from 'node:events';
import { createHash, randomBytes } from 'node:crypto';
import { SMTPServer } from 'smtp-server';
import nodemailer from 'nodemailer';

// Double opt-in: the link carries only a random token; the store is keyed by the token's hash.
const TOKEN_TTL_MS = 48 * 60 * 60 * 1000;
const sha256 = (s) => createHash('sha256').update(s).digest('hex');
const pending = new Map(); // sha256(token) -> { email, expiresAt } (a table with a unique index in a real service)
function startSubscription(email, now = Date.now()) {
  const token = randomBytes(32).toString('base64url');
  pending.set(sha256(token), { email, expiresAt: now + TOKEN_TTL_MS });
  return { email, url: `https://bookshelf.example.test/confirm?token=${token}` }; // no address in the URL
}
function confirm(token, now = Date.now()) {
  const key = sha256(String(token)); // a lookup by digest of a 256-bit random token: nothing useful leaks
  const record = pending.get(key);
  if (!record) return null;
  pending.delete(key); // single use, and an expired token is removed too
  return record.expiresAt > now ? record.email : null;
}

// Pooled transport with a rate limit, against a local catch-all SMTP server that records recipients.
const recipients = [];
const server = new SMTPServer({ authOptional: true, disabledCommands: ['STARTTLS'],
  onData(stream, session, cb) {
    stream.resume();
    stream.on('end', () => { recipients.push(...session.envelope.rcptTo.map((r) => r.address)); cb(); });
  } });
server.listen(0, '127.0.0.1');
await once(server.server, 'listening');
const transporter = nodemailer.createTransport({
  host: '127.0.0.1', port: server.server.address().port, ignoreTLS: true,
  pool: true, maxConnections: 2, rateDelta: 1000, rateLimit: 2, // at most 2 messages per second
});

const subscriptions = ['a@example.test', 'b@example.test', 'c@example.test'].map((e) => startSubscription(e));
const started = performance.now();
await Promise.all(subscriptions.map(({ email, url }) => transporter.sendMail({
  from: 'news@bookshelf.example.test', to: email, subject: 'Confirm your subscription', text: url, // to the address being confirmed
})));
console.log('sent', recipients.length, 'in', Math.round((performance.now() - started) / 100) / 10, 's to', recipients.sort());
console.log(subscriptions[0].url.replace(/token=.{6}.*/, 'token=…'));

const tokenOf = (i) => new URL(subscriptions[i].url).searchParams.get('token');
console.log('wrong token:', confirm('wrong-token'), '| right token:', confirm(tokenOf(0)), '| reused:', confirm(tokenOf(0)));
console.log('after 49 hours:', confirm(tokenOf(1), Date.now() + 49 * 60 * 60 * 1000));
transporter.close();
await new Promise((r) => server.close(r));
Output (the token is truncated)
sent 3 in 1 s to [ 'a@example.test', 'b@example.test', 'c@example.test' ]
https://bookshelf.example.test/confirm?token=…
wrong token: null | right token: a@example.test | reused: null
after 49 hours: null

The last two lines show a wrong token rejected, the right one accepted once (returning the address it confirms), the same link refused the second time, and a link used after its 48-hour lifetime refused. Each confirmation goes to the address it confirms, and the link carries only the token, so the address does not end up in proxy logs, browser history or Referer headers. Sending to a list is background work: the request that stores a subscription should enqueue the mail and return, which is the subject of Background Jobs, Scheduling & Queues. A NestJS view of the same task is in i18n, Mail & File Storage.

Source: Nodemailer: Pooled SMTP, Node.js API: crypto.randomBytes() and Node.js API: crypto.createHash()

Consuming RSS and Atom feeds with fetch

A feed is an XML document that a site publishes so programs can follow it: RSS 2.0 (<rss><channel><item>) and Atom (RFC 4287, <feed><entry>) are the two formats you meet. They differ in the details a parser must absorb: RSS puts the link in the text of <link> and the date in <pubDate> (an RFC 822 date), while Atom puts the link in an href attribute and the date in <updated> (an RFC 3339 timestamp). Node.js ships no XML parser, so the choice is a dedicated feed-parsing package or an XML-aware library you already have; the example uses Cheerio in xml: true mode, which is enough for well-formed feeds and shows both shapes in one function. For anything that must survive malformed real-world feeds, prefer a maintained feed parser.

A feed reader is a polling client, so the manners matter more than the parsing. Send an identifying User-Agent, set a deadline with AbortSignal.timeout(), treat non-2xx answers as errors (fetch only rejects on network failure), and use conditional requests: remember the ETag (or Last-Modified) from the last answer and send it back as If-None-Match (or If-Modified-Since), so an unchanged feed costs the server a 304 with no body. Poll every few minutes at the fastest, honour any Retry-After, and spread many feeds out instead of firing them together at the top of the hour.

feeds.mjs: read RSS and Atom with fetch, an honest user agent and a conditional request
import * as cheerio from 'cheerio';
import { startSite } from './site.mjs';

const site = await startSite();
const UA = 'BookshelfFeedReader/1.0 (+https://bookshelf.example.test/bot; bot@example.test)';
const cache = new Map(); // url -> { etag, items }

async function readFeed(url) {
  const known = cache.get(url);
  const res = await fetch(url, {
    headers: { 'user-agent': UA, accept: 'application/rss+xml, application/atom+xml, application/xml;q=0.9',
      ...(known && { 'if-none-match': known.etag }) },
    signal: AbortSignal.timeout(5_000),
  });
  if (res.status === 304) return { items: known.items, cached: true };
  if (!res.ok) throw new Error(`Feed ${url} answered ${res.status}`);
  const $ = cheerio.load(await res.text(), { xml: true });
  const items = $('item, entry').map((_, el) => {
    const $el = $(el);
    return {
      title: $el.children('title').text(),
      link: $el.children('link').attr('href') ?? $el.children('link').text(),
      date: new Date($el.children('pubDate, updated').text()),
    };
  }).get();
  if (res.headers.get('etag')) cache.set(url, { etag: res.headers.get('etag'), items });
  return { items, cached: false };
}

const urls = [`${site.origin}/feed.xml`, `${site.origin}/atom.xml`];
const results = await Promise.all(urls.map(readFeed));
console.table(results.flatMap((r) => r.items).map((i) => ({ ...i, date: i.date.toISOString() })));
const again = await readFeed(urls[0]);
console.log('second request cached:', again.cached, '| items:', again.items.length);
console.log(site.hits.join('\n'));
site.close();
The local test website the examples use (site.mjs)
import { createServer } from 'node:http';
import { once } from 'node:events';

const rss = `<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"><channel><title>Bookshelf news</title><link>http://site.example.test/</link>
<item><title>Node.js 26 &amp; you</title><link>http://site.example.test/a</link><guid>a</guid><pubDate>Mon, 21 Sep 2026 10:00:00 GMT</pubDate><description><![CDATA[<p>First <b>item</b></p>]]></description></item>
<item><title>Second post</title><link>http://site.example.test/b</link><guid>b</guid><pubDate>Tue, 22 Sep 2026 10:00:00 GMT</pubDate><description>Plain text</description></item>
</channel></rss>`;
const atom = `<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>Bookshelf Atom</title>
<entry><title>Atom entry</title><id>urn:uuid:1</id><link rel="alternate" href="http://site.example.test/c"/><updated>2026-09-23T10:00:00Z</updated><summary>Atom summary</summary></entry>
</feed>`;
const books = `<!doctype html><html><head><title>Catalogue</title></head><body>
<ul id="books"><li class="book" data-id="1"><a href="/books/1">Efficient &amp; Fast</a> <span class="price">12.50</span></li>
<li class="book" data-id="2"><a href="/books/2">Streams in Depth</a> <span class="price">9.99</span></li></ul></body></html>`;
const spa = `<!doctype html><html><body><div id="app">loading…</div>
<script>setTimeout(()=>{document.querySelector('#app').innerHTML='<h1 class="title">Rendered by JS</h1>'},50)</script></body></html>`;

export async function startSite() {
  const hits = [];
  const server = createServer((req, res) => {
    hits.push(`${req.method} ${req.url} UA=${req.headers['user-agent']}`);
    const send = (type, body, extra = {}) => {
      res.writeHead(200, { 'content-type': type, ...extra });
      res.end(body);
    };
    switch (req.url) {
      case '/robots.txt':
        return send('text/plain', 'User-agent: *\nDisallow: /private/\nCrawl-delay: 1\n');
      case '/feed.xml': {
        const etag = '"v1"';
        if (req.headers['if-none-match'] === etag) { res.writeHead(304, { etag }); return res.end(); }
        return send('application/rss+xml; charset=utf-8', rss, { etag });
      }
      case '/atom.xml': return send('application/atom+xml', atom);
      case '/books': return send('text/html; charset=utf-8', books);
      case '/spa': return send('text/html; charset=utf-8', spa);
      default: res.writeHead(404); return res.end('not found');
    }
  });
  server.listen(0, '127.0.0.1');
  await once(server, 'listening');
  return { origin: `http://127.0.0.1:${server.address().port}`, hits, close: () => server.close() };
}
Output of feeds.mjs (the local port is replaced by the test site’s fixed .test links)
┌─────────┬────────────────────┬──────────────────────────────┬────────────────────────────┐
│ (index) │ title              │ link                         │ date                       │
├─────────┼────────────────────┼──────────────────────────────┼────────────────────────────┤
│ 0       │ 'Node.js 26 & you' │ 'http://site.example.test/a' │ '2026-09-21T10:00:00.000Z' │
│ 1       │ 'Second post'      │ 'http://site.example.test/b' │ '2026-09-22T10:00:00.000Z' │
│ 2       │ 'Atom entry'       │ 'http://site.example.test/c' │ '2026-09-23T10:00:00.000Z' │
└─────────┴────────────────────┴──────────────────────────────┴────────────────────────────┘
second request cached: true | items: 2
GET /feed.xml UA=BookshelfFeedReader/1.0 (+https://bookshelf.example.test/bot; bot@example.test)
GET /atom.xml UA=BookshelfFeedReader/1.0 (+https://bookshelf.example.test/bot; bot@example.test)
GET /feed.xml UA=BookshelfFeedReader/1.0 (+https://bookshelf.example.test/bot; bot@example.test)

The request log on the server side shows what the example claims: three requests, each carrying the user agent, and the third one answered with 304 from the cached ETag. The Promise.all() fan-out is fine for two feeds; for hundreds, bound the concurrency and add per-host delays as described in the next section. The feed’s <description> can contain HTML, as the CDATA item in the test feed does; treat it as untrusted markup and sanitise it before showing it in a page.

Source: Node.js Learn: Fetch, MDN: Fetch API, MDN: Conditional requests, RSS 2.0 Specification, RFC 4287: The Atom Syndication Format and Cheerio: Configuring Cheerio

Scraping responsibly

Scraping means fetching pages meant for people and extracting data from them. It is sometimes the only way to get data, and it is also how well-meaning programs get blocked, sued or blamed for an outage. Before writing a selector, work through this list, in order.

Table 4. Checklist before scraping a site
Question What to do

Is there an API, a feed or a data export?

Use it. It is more stable than markup and it is what the site expects automated clients to use.

What do the terms of service say?

Read them. Many sites forbid automated access or reuse of their content, and no technical measure overrides that. If they forbid it, ask for permission or stop.

What does robots.txt say?

Fetch /robots.txt before anything else and obey it. RFC 9309 defines the format and states that its rules are not a form of access authorization, so a path that is not disallowed is not thereby permitted by the terms of service. Per the RFC, a 4xx answer means no published rules, while an unreachable file (5xx) means you should assume the whole site is off limits; cache the file for at most a day.

How fast may you go?

Slowly. Crawl-delay is a widespread but non-standard directive (RFC 9309 does not define it), and a polite default of one request per second per host, sequential, is a better start than concurrency. Back off on 429 and 503, and honour Retry-After.

Who are you?

Send a truthful User-Agent with a name, a version and a contact URL or address, so the operator can reach you instead of banning an IP range. Do not spoof a browser’s user agent to slip past a block; a block is an answer.

What are you collecting?

Personal data has its own legal regime. Collect only what the feature needs and do not republish whole sites.

The example below applies the checklist to the local test site: it reads robots.txt (only the group for *, which is enough here; use a maintained parser in production), skips a disallowed path, sends an identifying user agent, and waits between requests. It then parses the catalogue page with Cheerio.

Cheerio for static HTML

Cheerio parses markup into a tree and offers a jQuery-like API ($('li.book'), .find(), .attr(), .text(), .map()) to query it. It never runs scripts, loads sub-resources or builds a layout, which makes it fast and safe, and means it sees exactly what the server sent. cheerio.load(html) returns the query function, and { xml: true } switches to an XML parser, as the feed example does. Because the site’s own markup is the only contract, prefer stable hooks such as data-* attributes, ids and semantic classes over positional selectors, resolve relative links with new URL(href, base), and treat every extracted string as untrusted input.

scrape.mjs: robots.txt, an honest user agent, a delay and Cheerio
import { setTimeout as sleep } from 'node:timers/promises';
import * as cheerio from 'cheerio';
import { startSite } from './site.mjs';

const site = await startSite();
const UA = 'BookshelfBot/1.0 (+https://bookshelf.example.test/bot; bot@example.test)';
const headers = { 'user-agent': UA };

// Minimal robots.txt reader: the group for "*" only (use a maintained parser in production).
async function loadRules(origin) {
  const res = await fetch(`${origin}/robots.txt`, { headers });
  if (res.status >= 500) throw new Error('robots.txt unreachable: treat the site as disallowed');
  if (!res.ok) return { disallow: [], delayMs: 1000 }; // 4xx: no rules published
  const disallow = [];
  let delayMs = 1000;
  let applies = false;
  for (const raw of (await res.text()).split('\n')) {
    const [field, ...rest] = raw.split('#')[0].split(':');
    const value = rest.join(':').trim();
    switch (field.trim().toLowerCase()) {
      case 'user-agent': applies = value === '*'; break;
      case 'disallow': if (applies && value) disallow.push(value); break;
      case 'crawl-delay': if (applies) delayMs = Number(value) * 1000; break;
    }
  }
  return { disallow, delayMs };
}

const rules = await loadRules(site.origin);
const allowed = (path) => !rules.disallow.some((prefix) => path.startsWith(prefix));
console.log(rules, allowed('/books'), allowed('/private/x'));

for (const path of ['/books', '/private/x']) {
  if (!allowed(path)) { console.log('skipped by robots.txt:', path); continue; }
  const res = await fetch(site.origin + path, { headers, signal: AbortSignal.timeout(5_000) });
  const $ = cheerio.load(await res.text());
  const books = $('li.book').map((_, el) => ({
    id: Number($(el).attr('data-id')),
    title: $(el).find('a').text(),
    href: new URL($(el).find('a').attr('href'), site.origin).href,
    price: Number($(el).find('.price').text()),
  })).get();
  console.log($('title').text(), books);
  await sleep(rules.delayMs); // honour Crawl-delay (non-standard, but a polite default)
}
console.log(site.hits.join('\n'));
site.close();
Output (the port differs on each run)
{ disallow: [ '/private/' ], delayMs: 1000 } true false
Catalogue [
  { id: 1, title: 'Efficient & Fast', href: 'http://127.0.0.1:<port>/books/1', price: 12.5 },
  { id: 2, title: 'Streams in Depth', href: 'http://127.0.0.1:<port>/books/2', price: 9.99 }
]
skipped by robots.txt: /private/x
GET /robots.txt UA=BookshelfBot/1.0 (+https://bookshelf.example.test/bot; bot@example.test)
GET /books UA=BookshelfBot/1.0 (+https://bookshelf.example.test/bot; bot@example.test)

The server log confirms the order: robots.txt first, then the one allowed page, and no request for /private/x. Cheerio decoded the & entity in the title, and the prices were converted with Number() because the DOM only ever gives strings.

Source: Cheerio: Introduction, Cheerio: Loading, Cheerio: Selecting, Cheerio: Extracting data, RFC 9309: Robots Exclusion Protocol and Node.js API: timersPromises.setTimeout()

Puppeteer and Playwright for rendered pages

Many sites today ship an almost empty HTML shell and build the page in the browser with JavaScript, so fetch plus Cheerio sees the shell and nothing else. The test site’s /spa page demonstrates it: its <div id="app"> says loading… until a script replaces it. For such pages (and for anything behind a login flow you are entitled to automate, or a real user-interaction test) you need a real browser engine. Puppeteer drives Chrome or Firefox and Playwright drives Chromium, Firefox and WebKit, both through a high-level API: launch a browser, open a page, navigate, wait for something to appear, read it.

The cost is large compared with Cheerio: a whole browser process per run, a few hundred megabytes on disk (the Chrome for Testing build Puppeteer downloaded here is about 370 MB), seconds of start-up, and far more memory per page. So decide in this order: an API or feed; then fetch plus Cheerio, possibly against the JSON endpoint the page itself calls (visible in the browser’s network tab, and still subject to the checklist above); and a headless browser only when the data truly exists only after rendering. Whatever the tool, the checklist applies unchanged: browsers are automated clients too, so set an honest user agent and keep the delay.

render.mjs: the same page seen by fetch plus Cheerio and by Puppeteer
import * as cheerio from 'cheerio';
import puppeteer from 'puppeteer';
import { startSite } from './site.mjs';

const site = await startSite();
const staticHtml = await (await fetch(`${site.origin}/spa`)).text();
console.log('fetch + Cheerio sees:', cheerio.load(staticHtml)('#app').text());

const browser = await puppeteer.launch(); // headless by default
try {
  const page = await browser.newPage();
  await page.setUserAgent('BookshelfBot/1.0 (+https://bookshelf.example.test/bot; bot@example.test)');
  await page.goto(`${site.origin}/spa`, { waitUntil: 'domcontentloaded' });
  const title = await page.waitForSelector('#app .title');
  console.log('Puppeteer sees:', await title.evaluate((el) => el.textContent));
  const html = await page.content();
  console.log('Cheerio on rendered HTML:', cheerio.load(html)('h1').text());
} finally {
  await browser.close();
  site.close();
}
Output
fetch + Cheerio sees: loading…
Puppeteer sees: Rendered by JS
Cheerio on rendered HTML: Rendered by JS

Three points about the example. puppeteer.launch() is headless by default and npm i puppeteer downloaded a Chrome for Testing build for it (observed here; the cached browser is outside node_modules); puppeteer-core skips the download and needs an executablePath or channel. waitForSelector() waits for the element to exist instead of sleeping for a guessed time, which is the habit that separates a reliable scraper from a flaky one. And the finally block closes the browser even when navigation throws, because an orphaned browser process is the commonest way these scripts leak memory. page.content() returns the rendered HTML, so you can hand it to Cheerio and keep the extraction code you already have.

Playwright’s equivalent differs mainly in naming: browser.newContext() isolates a session and takes the userAgent option, and page.locator(selector) returns a lazily resolved, auto-waiting handle.

pw.mjs: the same rendered page with Playwright
import { chromium } from 'playwright';
import { startSite } from './site.mjs';
const site = await startSite();
const browser = await chromium.launch({ executablePath: process.env.CHROME_PATH });
try {
  const context = await browser.newContext({ userAgent: 'BookshelfBot/1.0 (+https://bookshelf.example.test/bot; bot@example.test)' });
  const page = await context.newPage();
  await page.goto(`${site.origin}/spa`);
  console.log(await page.locator('#app .title').textContent());
} finally { await browser.close(); site.close(); }
Output
Rendered by JS

Playwright installs its browsers with a separate command (npx playwright install chromium), a download of its own that was not run for this page. The example ran by pointing executablePath at the Chrome for Testing build that Puppeteer had already downloaded, which is a supported option of launch(), and it worked on Node.js 26 only; in your project, install the browser Playwright manages and drop the executablePath line. Check each library’s declared engines.node when you pin a version, as the version map shows.

Source: Puppeteer: What is Puppeteer?, Puppeteer: Installation, Puppeteer: Page.waitForSelector(), Playwright: Installation and Playwright: Browsers

What was run, and what was not

Table 5. Verification status of the examples
Example Node.js versions run Environment

send.mjs, consent.mjs

26.10.0, 24.21.0, 22.23.3

Local smtp-server on a random loopback port; addresses on the reserved .test domain; a throwaway DKIM key kept outside the repository. No mail left the machine.

feeds.mjs, scrape.mjs

26.10.0, 24.21.0, 22.23.3

Local test website on a random loopback port (site.mjs). No third-party site was contacted.

render.mjs (Puppeteer 25.12.0)

26.10.0, 24.21.0, 22.23.3

Headless Chrome for Testing against the local test website only.

pw.mjs (Playwright 1.63.0)

26.10.0 only

Against the local test website, using Puppeteer’s browser through executablePath. Playwright’s own browser download was not run.

The server in site.mjs is deliberately tiny. Real sites send redirects, compressed bodies, cookies, consent banners and rate-limit headers, and the examples do not model them.

What changed recently

Table 6. Changes since the older material, with versions
Change Detail

Global fetch everywhere

Unflagged in Node.js 18 and no longer experimental since 21, so node-fetch and axios are no longer needed for feed polling or simple scraping; the Node.js project documents a codemod from axios to fetch.

Nodemailer 10

Requires Node.js 20 or later and ships ES module and CommonJS builds. The createTransport() and sendMail() calls used here are unchanged from earlier majors, but read the release notes when upgrading across majors, since options are occasionally removed.

Puppeteer 25

The current line is 25.x. In its TypeScript declarations headless is boolean | 'shell' (default true, the new headless mode) and Page has no waitForTimeout(), so older listings that pass headless: 'new' or sleep with page.waitForTimeout() do not match the current API; use the default and waitForSelector() or the other wait methods.

Mailbox-provider requirements

Large senders to Gmail must authenticate with SPF, DKIM and DMARC and support one-click unsubscribe, which makes RFC 8058 headers a delivery requirement and no longer a courtesy.

Install scripts in npm

npm 11.19 printed a warning that the puppeteer package has a postinstall script that is not covered by an allowlist, yet the download still happened in this run (observed). Read the warning for your npm version, because script policies are tightening across npm.

Where the books are outdated

This section compares the page with the two O’Reilly books consulted while preparing this guide: Node.js Projects by Jonathan Wexler (2025), whose mailer, feed-reader and scraper chapters overlap this page, and Efficient Node.js by Samer Buna (2025). Both are recent; the points below concern habits worth updating and safer forms.

  • The feed-reader project polls on a fixed setInterval() timer. Production polling should await each round before scheduling the next, send conditional requests with the ETag or Last-Modified validators, set a deadline with AbortSignal.timeout() and check the response status, so that a slow response cannot stack overlapping requests and an unchanged feed costs a 304.

  • The mailer project installs the 7.x line of Nodemailer; the current major is 10 and requires Node.js 20 or later. Its unsubscribe feature is a GET route keyed by the email address; the safe form is an unguessable per-subscriber token, the List-Unsubscribe header and the RFC 8058 one-click POST, because a GET can be triggered by link prefetchers and a guessable address lets anyone unsubscribe someone else. Mail authentication (SPF, DKIM, DMARC) and consent get little room in a tutorial but decide whether the mail is delivered. Open-tracking pixels deserve a deliberate consent decision.

  • The scraper project sets a user-agent string chosen to get the same response a browser receives and does not discuss robots.txt or terms of service. The checklist above is the safe form: an identifying user agent, robots rules, a delay and a permission check come before any selector. The chapter’s use of a headless browser for pages that return only a shell is still the right idea; current Puppeteer defaults and waiting methods differ from older listings.

  • Neither book covers AbortSignal.timeout() for deadlines on every outbound request, nor the built-in proxy support in recent Node.js lines that scraping behind a corporate proxy now depends on (see Fetch & HTTP Clients).

References