Performance & the storage/caching model

Index through the _bulk API, not single-document requests: one round trip carries many actions and amortises coordination cost. There is no universally correct batch size — start around 5—​15 MB of raw data per request, measure indexing throughput, and increase until it plateaus or you see es_rejected_execution_exception from a full write queue. Send bulk requests from several client threads in parallel to keep every data node busy.

Omitting _id (the empty index action line above) lets Elasticsearch assign a UID it knows is new, so it skips the "does this id already exist?" lookup that every explicit-id index or upsert must do against existing segments. That check gets more expensive as the shard grows, so auto-generated ids keep bulk-load speed flat. Use an explicit id only when the source system’s key is the natural identity and you need idempotent re-indexing.

A refresh turns the in-memory indexing buffer into a new searchable segment. It happens automatically about once per second on an index that is being searched, which is what makes Elasticsearch near real-time rather than real-time — see Near real-time search. Each refresh has a fixed cost, so during a large one-off load, lengthen the interval (or disable it) and force one refresh at the end.

Indexing into events with one primary and zero replicas means each document is indexed once instead of twice; adding the replica afterwards copies finished segments over the network, which is cheaper than concurrent indexing on both copies. Keep at least one replica for anything you cannot afford to rebuild.

A refresh makes writes visible; it does not make them durable. Durability comes from the translog, an append-only log each shard writes every operation to. By default (index.translog.durability: request) the translog is fsync-ed before every indexing request is acknowledged, so an acknowledged write survives a node crash. Setting it to async fsyncs only every index.translog.sync_interval (default 5s), which raises indexing throughput at the cost of losing the last few seconds of acknowledged writes on a crash — acceptable for data you can replay from an upstream source.

A flush writes the current in-memory segments to disk via a Lucene commit and then trims the translog. It runs automatically when the translog grows past index.translog.flush_threshold_size (default 512mb); you rarely trigger it by hand.

Because segments are immutable, every refresh creates another one, a delete only marks a document as gone in a .liv file, and an update is a delete plus a re-index. Left alone, a busy shard would accumulate thousands of small segments, each an extra file to open and search. Lucene continuously merges smaller segments into fewer larger ones in the background, physically dropping deleted documents as it goes. Merging is I/O- and CPU-heavy; indices.merge.scheduler.max_thread_count bounds it per shard.

Once a time-based index (yesterday’s logs, last month’s orders) will receive no more writes, merge it down to one segment. This removes all deleted-document overhead and gives the fastest possible searches, but it is expensive and must run only on indices that are truly read-only — a force-merged segment is never picked for automatic merging again, so if writes resume you can be left with one huge segment full of deletes.

Index lifecycle & scaling automates the rollover-then-forcemerge pattern with ILM.

A clause in filter context answers yes/no, computes no _score, and its result bitset is cached in the node query cache for reuse across queries on the same shard. A clause in query context computes relevance and is not cached. Move every exact condition — terms, ranges, dates, status flags — into filter (or must_not), and keep only the clauses whose ranking you actually use in must / should.

Search API & pagination covers the two contexts in more detail.

Two independent caches sit in front of shard-level work:

By default Elasticsearch spreads successive searches across shard copies (adaptive replica selection). Passing a stable preference string — a user id, a session id — routes that user’s repeated searches to the same copies, so they hit warm filesystem cache, node query cache, and (for size: 0) shard request cache.

from / size paging makes every shard build from + size hits and is capped at index.max_result_window (10000). For anything beyond the first few pages use search_after with a point-in-time, which costs the same per page no matter how deep you are. See Paginate search results and the worked example in Search API & pagination.

Mapping & field types is the full reference for these parameters.

index.codec: best_compression switches stored fields and _source from LZ4 to DEFLATE, typically cutting on-disk size 15—​25% in exchange for slightly slower fetches and merges. Set it at creation time; it applies to segments written afterwards, so pair it with _forcemerge on existing read-only indices.

_source is the stored original JSON. Disabling it saves disk, but you lose everything that reads the document back: reindex, update and update-by-query, the highlight feature, and the ability to reindex into a new mapping at all. Prefer _source filtering on the way out, or synthetic _source, over disabling it. If you still must:

Synthetic _source stops storing the raw JSON and instead reconstructs it on demand from doc_values and the inverted index. You keep reindex, update, and highlight while paying roughly the storage of the fields themselves and no more. The reconstructed document is normalised (fields sorted, values de-duplicated, some formatting lost), which is fine for most uses.