Instrumenting applications

This section documents the current Prometheus 3.x line (latest 3.14.0) as published at the Prometheus documentation and the current Grafana 13.x line (latest 13.2.1) as published at the Grafana documentation, which are the references these pages are written and verified against. No specific patch version is pinned. Some capabilities (Grafana Enterprise-only features, Grafana Cloud-hosted services, and long-term-storage systems like Thanos/Cortex/Mimir) are linked, not documented in depth.

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production, since both projects iterate quickly.

This section’s bibliography lists the reference material consulted while preparing these pages.

Scraping only works once something exposes a /metrics endpoint in a format Prometheus understands. This page covers the official client libraries used to instrument application code, the text-based exposition format those libraries emit (and its standardization as OpenMetrics), how a scraper and a target negotiate which of the two to speak, and the naming conventions that keep metrics and labels usable as an application grows.

Client libraries

Prometheus maintains official client libraries for Go, Java/Scala, Python, Ruby, and Rust, plus a longer list of community libraries for other languages. Each implements the same core model — registries, the four metric types, and an HTTP handler that serves the current state in the exposition format — with idioms native to its language. The full list, with install instructions per language, is Client libraries; a library not yet covered there, or one you write for an unsupported language, should follow Writing client libraries so it stays compatible with the format and conventions below.

Go (`client_golang`)

client_golang is the reference implementation and the library Prometheus’s own server and exporters use internally. A Counter is registered once, at startup, and incremented from request-handling code; promhttp.Handler() wires the default registry to an HTTP endpoint:

package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var httpRequestsTotal = promauto.NewCounterVec(
	prometheus.CounterOpts{
		Name: "http_requests_total",
		Help: "Total number of HTTP requests processed.",
	},
	[]string{"method", "code"},
)

func handler(w http.ResponseWriter, r *http.Request) {
	httpRequestsTotal.WithLabelValues(r.Method, "200").Inc()
	w.Write([]byte("ok"))
}

func main() {
	http.HandleFunc("/", handler)
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":8080", nil)
}
// https://prometheus.io/docs/instrumenting/clientlibs/

promauto registers the metric with the default registry as it is created, so there is no separate Register call to forget. WithLabelValues must be called with label values in the same order the label names were declared.

Java/Scala (`simpleclient` / `client_java`)

The Java ecosystem has two generations of official client: the older simpleclient modules (io.prometheus:simpleclient) and the newer client_java (io.prometheus:prometheus-metrics-core), which follows the OpenMetrics data model natively. Both register metrics against a CollectorRegistry and expose it over HTTP with a small embedded server or a servlet filter:

import io.prometheus.client.Counter;
import io.prometheus.client.exporter.HTTPServer;

public class App {
  static final Counter httpRequestsTotal = Counter.build()
      .name("http_requests_total")
      .help("Total number of HTTP requests processed.")
      .labelNames("method", "code")
      .register();

  public static void main(String[] args) throws Exception {
    HTTPServer server = new HTTPServer.Builder()
        .withPort(8080)
        .build();

    httpRequestsTotal.labels("GET", "200").inc();
  }
}
// https://prometheus.io/docs/instrumenting/clientlibs/

HTTPServer serves the default registry’s content on /metrics on the given port. Spring Boot applications typically go through Micrometer’s Prometheus registry instead of calling this library directly; that integration is out of scope for this page.

Python, Ruby, and Rust

prometheus_client (Python) and prometheus-client (Ruby) follow the same registry-and-decorator shape as the libraries above — a Counter/Gauge/Histogram/Summary object is created once and updated inline with request handling, then served via a small built-in HTTP server or WSGI/Rack middleware. The Rust client (prometheus) mirrors the same four types with an explicit Registry and a TextEncoder that a web framework’s own handler calls to produce the response body. For an axum service built with the metrics facade and metrics-exporter-prometheus instead, see Axum: Metrics & SLOs. Installation and a minimal example for each are on the client libraries page linked above; the patterns — instrument business logic directly, expose /metrics for scraping, avoid per-request metric creation — are the same across all five official languages, and are spelled out in more depth in Instrumentation. What each metric type actually means and when to reach for a Histogram versus a Summary is covered in Data Model & Metric Types and Histograms and summaries.

The exposition format and OpenMetrics

Whatever library produces it, /metrics returns plain text: one line per sample, plus # HELP and # TYPE comment lines that document each metric once. This is what a scraper actually parses:

curl -s http://localhost:8080/metrics
# HELP http_requests_total Total number of HTTP requests processed.
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1027
http_requests_total{method="POST",code="500"} 3

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.5346432e+07
# https://prometheus.io/docs/instrumenting/exposition_formats/

This line-oriented layout is Prometheus’s own text-based exposition format, defined at Exposition formats. It was later submitted to, and standardized by, the CNCF as OpenMetrics — a vendor-neutral spec for the same wire format, with a stricter grammar (for example, every counter’s name must end in _total, and an explicit # EOF line closes the payload) so that other systems can produce or consume it without depending on Prometheus itself. Every current official client library can emit either format from the same registry; which one a given scrape actually gets is decided per request, not per client, as described next.

Content negotiation

A scrape target does not hard-code which format it returns. The Prometheus server sends an Accept header on every scrape request listing the formats it is willing to receive, in preference order, and the client library’s HTTP handler picks the best match:

# Ask explicitly for the OpenMetrics text format instead of the Prometheus legacy format.
curl -s -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' \
  http://localhost:8080/metrics

If no Accept header is sent, or none of the offered types are recognized, a client library falls back to its own default (the legacy Prometheus text format, for most current libraries). Because both formats describe the same underlying samples, this negotiation is invisible to application code — instrumentation is written once against the client library’s API, and the wire format served for any given scrape is a detail between the Prometheus server and that library’s HTTP handler. The exact media types, parameters, and fallback rules are defined in Content negotiation.

Metric and label naming

Names are part of a metric’s public API: they get baked into dashboards, alerting rules, and other teams' PromQL queries, so getting them right at instrumentation time avoids a breaking rename later. The full guidance is Metric and label naming; the rules that matter most day to day:

Namespace with an underscore-separated prefix that identifies the owning system or application, e.g. http_requests_total, process_cpu_seconds_total — not a dotted or camelCase name.
Suffix the unit, in base units. Use _seconds (not milliseconds), _bytes (not kilobytes or megabytes), _total for a monotonically increasing count. Prometheus and PromQL assume base units throughout (rate(), histogram bucket boundaries, and so on), so a metric in non-base units silently breaks any query written against the convention.
Every counter ends in _total. This is required by OpenMetrics and conventional even when only emitting the legacy format; it is also how a reader distinguishes a counter (meant to be wrapped in rate()/increase()) from a gauge at a glance.
Put varying dimensions in labels, not in the metric name. http_requests_total{method="GET"} is one metric with a method label, not a family of http_requests_get_total, http_requests_post_total metrics — the label form is what makes aggregation across dimensions (sum by (method) (…)) possible at all.
Keep label cardinality bounded. Each distinct combination of label values creates a new time series, stored and indexed independently. A label drawn from a small, closed set (method, status_code, region) is safe; a label drawn from an effectively unbounded set (a raw user ID, a full request path with path parameters, a UUID, a timestamp) is not — it causes cardinality explosion, inflating memory and query latency on the server for every application that mislabels this way. Bucket or normalize high-cardinality values (a route template like /users/:id instead of /users/1042) before attaching them as a label.

These conventions apply equally to metrics defined by hand and to ones a exporter derives from some other system; they also set the vocabulary that PromQL selects against and that recording and alerting rules build on.

Continue with Exporters & the Pushgateway for instrumenting systems you cannot modify directly, or Data Model & Metric Types for the underlying Counter/Gauge/Histogram/Summary model these libraries implement.