Administration, monitoring & snapshots

This section documents the current Elasticsearch 9.x line (with 8.19 as the final 8.x release) as published at the Elasticsearch documentation, which is the reference these pages are written and verified against. No specific patch version is pinned. Some capabilities (Kibana-only UIs, the ML/NLP model-management workflow, cross-cluster replication, and parts of the paid / serverless-only surface) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, as Elasticsearch iterates quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Running a cluster is mostly three activities: reading its current state through the _cat and stats APIs, watching a short list of health signals, and keeping restorable backups with snapshot & restore. This page covers the operational APIs, what to alert on, and how snapshots and Snapshot Lifecycle Management work. For the shard-allocation model these tools report on, see Cluster, nodes & shards; for the write-path knobs you tune in response, see Performance tuning.

The _cat APIs

The _cat (compact and aligned text) APIs render cluster state as columns meant for a terminal, not for parsing. Add ?v for a header row, ?s=<col> to sort, ?h=<col1,col2> to pick columns, and ?format=json (or yaml, cbor, smile) when a script needs the data — though the real JSON APIs below are the supported machine interface. See Compact and aligned text (CAT) APIs.

# Overall status: green / yellow / red, node and shard counts, pending tasks.
GET /_cat/health?v

# Nodes: heap %, CPU, load, disk, and which one is master (*).
GET /_cat/nodes?v&h=name,node.role,master,heap.percent,ram.percent,cpu,load_1m,disk.used_percent

# Indices: health, docs, store size -- largest first.
GET /_cat/indices?v&s=store.size:desc

# Shards: every shard, its state (STARTED / RELOCATING / INITIALIZING / UNASSIGNED),
# the node it sits on, and -- for unassigned ones -- why.
GET /_cat/shards/my-index?v&h=index,shard,prirep,state,docs,store,node,unassigned.reason

# Allocation: how many shards and how much disk per node.
GET /_cat/allocation?v

# Thread pools: active / queue / rejected, for the pools that matter (search, write).
GET /_cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected,completed

?format=json turns any of them into an array of objects:

GET /_cat/indices?format=json&s=store.size:desc

Cluster and node stats & info APIs

These return structured JSON and are what monitoring should scrape.

# Cluster health, optionally waiting for a status or drilling into one index.
GET /_cluster/health
GET /_cluster/health/my-index?level=shards&wait_for_status=yellow&timeout=30s

# Cluster-wide rollups: node count, total shards, JVM versions, OS, mappings,
# field-type usage across every index.
GET /_cluster/stats

# Per-node counters -- the heavy one. Scope it to the groups you need.
GET /_nodes/stats/jvm,os,fs,thread_pool,indices

# Static per-node info: roles, JVM args, installed plugins, settings, OS/CPU.
GET /_nodes
GET /_nodes/plugins

See Cluster APIs for the full set, Nodes stats API, and Nodes info API.

Why a shard is unassigned: _cluster/allocation/explain

When _cat/health is yellow or red, this API explains — in prose — why the allocator will not place a specific shard: a disk watermark, an allocation filter, max_retries exceeded after repeated failures, no copy available, and so on.

# Explain the first unassigned shard the cluster finds.
GET /_cluster/allocation/explain

# Or point at an exact shard.
POST /_cluster/allocation/explain
{ "index": "my-index", "shard": 0, "primary": true }

# After fixing the cause, clear a "max_retries exceeded" state with a forced retry.
POST /_cluster/reroute?retry_failed=true

See Cluster allocation explain API.

Where CPU is going: _nodes/hot_threads

_nodes/hot_threads samples the busiest JVM threads on each node and prints their stack traces — the first thing to pull when a node is CPU-bound.

GET /_nodes/hot_threads
GET /_nodes/<node-id>/hot_threads?threads=5&type=cpu&interval=500ms

See Nodes hot threads API.

Running and queued work: _tasks and pending tasks

The Task Management API lists in-flight actions (searches, bulks, reindex, snapshots) and can cancel the cancellable ones. _cluster/pending_tasks shows cluster-state updates queued on the master — a persistently non-empty queue means the master is a bottleneck.

GET /_tasks?actions=*search&detailed
GET /_tasks?actions=*reindex,*snapshot&detailed

# Cancel a long-running task by its id (node:taskNumber).
POST /_tasks/oTUltX4IQMOUUVeiohTt8A:12345/_cancel

GET /_cluster/pending_tasks

See Task management API and Pending cluster tasks API.

What to watch

The overview at Monitor a cluster frames these; the signals below are the ones worth alerting on.

JVM heap pressure and GC

Read jvm.mem.heap_used_percent from _nodes/stats/jvm. Sustained above ~85% means the node is close to circuit-breaker trips and long stop-the-world GC pauses; frequent old-generation collections (jvm.gc.collectors.old.collection_count climbing) confirm it. The fix is usually fewer/smaller shards, lighter aggregations, or more nodes — not a bigger heap (keep it at half of RAM and under ~31 GB so compressed object pointers stay on).

GET /_nodes/stats/jvm?filter_path=nodes.*.name,nodes.*.jvm.mem.heap_used_percent,nodes.*.jvm.gc
# Real-memory circuit breaker trips show up here:
GET /_nodes/stats/breaker

Thread-pool rejections

Each pool (search, write, snapshot, …) has a bounded queue; once it is full, further tasks are rejected and the client sees 429 Too Many Requests / es_rejected_execution_exception. A non-zero, growing rejected column in _cat/thread_pool means the cluster cannot keep up — back off and retry on the client, then add capacity. Rejections are a healthy back-pressure signal, not a bug to configure away.

GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected&s=rejected:desc

See Thread pools.

Disk watermarks and the read-only block

The allocator enforces three disk watermarks: at low (default 85%) it stops putting new shards on a node; at high (90%) it moves shards off; at flood_stage (95%) it applies index.blocks.read_only_allow_delete: true to every index with a shard on that node, so writes fail with a cluster_block_exception until disk is freed. Elasticsearch releases the block automatically once usage drops below high, but you can clear it explicitly after making room.

GET /_cat/allocation?v&h=node,disk.percent,disk.used,disk.avail

# After freeing disk, lift the block if it is still set.
PUT /*/_settings
{ "index.blocks.read_only_allow_delete": null }

See Disk-based shard allocation. Deleting data does not help while an index is read-only-allow-delete only for deletes; free space at the OS level instead.

Search and indexing latency

_nodes/stats/indices exposes cumulative time and counts; latency is the delta of time_in_millis over the delta of the operation count between two samples.

GET /_nodes/stats/indices/search,indexing,merges,refresh,flush?filter_path=nodes.*.name,nodes.*.indices
# search.query_time_in_millis / search.query_total  -> avg query latency
# indexing.index_time_in_millis / indexing.index_total -> avg index latency

Unassigned shards

Any UNASSIGNED primary makes the cluster red and its data unavailable; an unassigned replica makes it yellow. List them, then ask why:

GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state
GET /_cluster/allocation/explain

Slow logs

Per-index thresholds write slow queries, fetches, and indexing operations to dedicated log files. Nothing is logged until you set a threshold; -1 disables a level.

PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "2s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "5s",
  "index.indexing.slowlog.source": "1000"
}

See Slow log.

Stack Monitoring

For dashboards, retention, and alerting on all of the above, ship metrics to a separate monitoring cluster with the Elastic Agent elasticsearch integration (or Metricbeat) and view them in Kibana’s Stack Monitoring app rather than polling these APIs by hand. Legacy self-monitoring (xpack.monitoring.collection.enabled) still exists but Agent-based collection is the current path. See Monitor a cluster.

Snapshot & restore

A snapshot is a copy of index and/or cluster state written to a repository — a shared filesystem or object store. Snapshots are incremental at the segment-file level: each one stores only files not already present in the repository from an earlier snapshot, so after the first, snapshots are small and cheap. See Snapshot and restore.

Register a repository

# Shared filesystem -- path.repo must be set in elasticsearch.yml on every node.
PUT /_snapshot/backups
{ "type": "fs", "settings": { "location": "/mnt/es-backups", "compress": true } }

# S3 (needs the repository-s3 plugin / preinstalled on Elastic Cloud).
PUT /_snapshot/s3-backups
{ "type": "s3", "settings": { "bucket": "my-es-snapshots", "base_path": "prod", "region": "eu-west-1" } }

# GCS and Azure are analogous: "type": "gcs" (bucket, base_path) with repository-gcs,
# "type": "azure" (container, base_path) with repository-azure.

# Verify every node can reach it.
POST /_snapshot/s3-backups/_verify
GET  /_snapshot/_all

See Register a snapshot repository.

Take a snapshot

PUT /_snapshot/s3-backups/snap-2026-09-06?wait_for_completion=false
{
  "indices": "logs-*,metrics-*",
  "ignore_unavailable": true,
  "include_global_state": true,
  "metadata": { "taken_by": "cron", "reason": "nightly" }
}

GET /_snapshot/s3-backups/snap-2026-09-06/_status
GET /_cat/snapshots/s3-backups?v&s=end_epoch:desc

include_global_state: true also captures cluster settings, index templates, ingest pipelines, and stored scripts. Only one snapshot per repository runs at a time; concurrent index writes are fine — the snapshot reflects the point it started.

sequenceDiagram participant C as Client participant M as Master participant D as Data nodes participant R as Repository C->>M: PUT /_snapshot/s3-backups/snap-2 (indices: logs-*) M->>D: snapshot each primary shard D->>D: flush, list this shard's Lucene segment files D->>R: read repo index for logs-* Note over D,R: segment files already in R from snap-1 are skipped D->>R: upload only new segment files D-->>M: shard done (n new files, m reused) M->>R: write snap-2 metadata (points at all files: new + reused) M-->>C: snapshot SUCCESS

Restore

You cannot restore over an open index — close it first, or restore under new names with a rename pattern.

# Whole snapshot.
POST /_snapshot/s3-backups/snap-2026-09-06/_restore
{ "indices": "logs-*", "include_global_state": false }

# Partial + rename: restore one index as a copy, tweak settings on the way in.
POST /_snapshot/s3-backups/snap-2026-09-06/_restore
{
  "indices": "logs-2026.09.05",
  "rename_pattern": "logs-(.+)",
  "rename_replacement": "restored-logs-$1",
  "index_settings": { "index.number_of_replicas": 0 },
  "include_aliases": false
}

GET /_cat/recovery/restored-*?v&active_only=true

See Restore a snapshot. Restoring include_global_state: true overwrites cluster settings and templates — usually left false except in a full-cluster recovery.

Snapshot Lifecycle Management (SLM)

SLM runs snapshots on a schedule and deletes old ones by a retention rule, so you do not cron the calls yourself.

PUT /_slm/policy/nightly
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "s3-backups",
  "config": { "indices": "*", "include_global_state": true },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}

POST /_slm/policy/nightly/_execute
GET  /_slm/policy/nightly
GET  /_slm/stats

See Snapshot lifecycle management. SLM needs the manage_slm cluster privilege and a periodic POST /_slm/_execute_retention (scheduled automatically by slm.retention_schedule).

Beyond backup

The same repository mechanism powers searchable snapshots — mounting a snapshot as a read-only, partially-cached index so the cold and frozen ILM tiers keep data queryable at object-store cost. For keeping a live second copy of an index in another cluster (disaster recovery, geo-locality), use cross-cluster replication rather than frequent snapshots. Both are linked from Snapshot and restore.

Contrast with MongoDB

The document-database equivalents cover the same ground with different tools: see MongoDB Administration & Monitoring for its server-status and profiler counterparts to the stats APIs and slow log, and MongoDB Backup for mongodump / filesystem snapshots against Elasticsearch’s repository-based, incremental model. For access control on every API on this page, see Security.