DNS and Service Discovery

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production; see the section bibliography for the reference material consulted while preparing these pages.

Services gives every Service a stable virtual IP; this page covers the other half of "stable" — a stable name a Pod can look up to find that IP, without ever hard-coding it. Kubernetes solves this with a cluster-internal DNS service, CoreDNS, which every Pod is configured to use by default. This page is written against Kubernetes 1.37 and CoreDNS as shipped with it; see the section index for the full version baseline. The Bookshelf web Deployment resolving catalog-api and reviews-api by name, and catalog-api resolving postgres (the headless Service from StatefulSets), are the running examples below.

CoreDNS

CoreDNS is a general-purpose, plugin-chained DNS server that Kubernetes deploys as the cluster’s DNS addon: an ordinary Deployment (typically two replicas for availability) in kube-system, fronted by an ordinary Service, both reconciled exactly like any application workload. Its configuration is a Corefile ConfigMap, plugin-based — the kubernetes plugin is what makes it watch the API server for Services and Pods and answer queries about them; other plugins add caching, health, metrics, forwarding to upstream resolvers, and logging.

kubelet injects the cluster DNS Service’s ClusterIP as nameserver in every Pod’s /etc/resolv.conf (unless dnsPolicy says otherwise, below), so applications need no client library or SDK to do service discovery — they just resolve a hostname the way they always do.

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get svc -n kube-system kube-dns
kubectl -n kube-system get configmap coredns -o yaml
NAME                       READY   STATUS    RESTARTS   AGE
coredns-668d6bf9bc-4z2wq   1/1     Running   0          3d
coredns-668d6bf9bc-9k7jp   1/1     Running   0          3d

NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   3d

The Service is still named kube-dns for backward compatibility even though CoreDNS has been the default implementation since Kubernetes 1.13 — an old habit worth knowing so -l k8s-app=kube-dns and svc/kube-dns don’t look like a leftover kube-dns installation.

Source: DNS for Services and Pods

A, AAAA and SRV records for Services and Pods

CoreDNS’s kubernetes plugin answers three kinds of query, all under the cluster domain (cluster.local unless the cluster was configured otherwise):

Record Resolves to

A/AAAA for a normal Service

<svc>.<ns>.svc.cluster.local → the Service’s ClusterIP

A/AAAA for a headless Service (clusterIP: None)

<svc>.<ns>.svc.cluster.local → the set of every backing Pod’s own IP (no load-balancing, the client picks)

A/AAAA for a headless Service’s per-pod record

<pod-hostname>.<svc>.<ns>.svc.cluster.local → that one Pod’s IP — this is exactly how StatefulSets give each ordinal a resolvable name, e.g. postgres-0.postgres.bookshelf.svc.cluster.local

SRV for a named port

<port-name>.<protocol>.<svc>.<ns>.svc.cluster.local → the port number and the Service’s (or, for a headless Service, each Pod’s) domain name

A/AAAA for a Pod, by IP

<pod-ipv4-with-dashes>.<ns>.pod.cluster.local, e.g. 172-17-0-3.bookshelf.pod.cluster.local — rarely used directly; it exists mainly so a Pod’s own IP is resolvable when something needs to reverse a connection

The one form worth memorizing is <svc>.<ns>.svc.cluster.local, and it is worth being precise about the segment order and separators: it is service, then namespace, then the literal svc, each dot-separated, ending in the cluster domain. This matters because Kubernetes: Up and Running's storage chapter writes external-service DNS names such as mysql.svc.default.cluster — neither the segment order (svc before the namespace, instead of after it) nor the domain suffix (cluster instead of cluster.local) matches any name CoreDNS has ever produced; treat it as an editorial slip and use mysql.default.svc.cluster.local (or, cross-namespace, <svc>.<namespace>.svc.cluster.local) instead. In practice a Pod rarely needs to type the full name at all, because of the search domains below.

Source: DNS for Services and Pods

sequenceDiagram participant W as Pod web participant R as resolv.conf resolver participant C as CoreDNS participant A as kube-apiserver W->>R: resolve "catalog-api" R->>C: catalog-api.bookshelf.svc.cluster.local (ndots search expansion) C->>A: watch cache: Service catalog-api in bookshelf C-->>R: A record: ClusterIP 10.96.12.4 R-->>W: 10.96.12.4 W->>W: connect to 10.96.12.4:8080 (kube-proxy load-balances to a Pod)

Search domains and ndots

kubelet writes a Pod’s /etc/resolv.conf with a search list and an ndots option, which is why web can resolve plain catalog-api instead of the full catalog-api.bookshelf.svc.cluster.local:

nameserver 10.96.0.10
search bookshelf.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 means: if the name being looked up has fewer than 5 dots, the resolver tries it with each search suffix appended, in order, before trying it as an absolute name; a name with 5 or more dots is tried as absolute first. catalog-api (0 dots) therefore first tries catalog-api.bookshelf.svc.cluster.local — a same-namespace Service resolves on the very first query. A cross-namespace lookup such as postgres.other-ns (1 dot) still resolves, just one query later, by matching the svc.cluster.local suffix. The pitfall this creates: any external hostname with fewer than 5 dots (an internal company host, or even some public domains) pays for up to 4 failed internal lookups before the 5th, absolute query finally succeeds — extra latency and extra load on CoreDNS, multiplied across every Pod. dnsConfig (below) is how to opt individual Pods out of that cost for known-external names.

Source: DNS for Services and Pods

dnsPolicy and dnsConfig

spec.dnsPolicy on a Pod chooses where its resolv.conf comes from in the first place:

dnsPolicy Effect

ClusterFirst (default)

Use the values above; queries that don’t match the cluster domain suffix are forwarded to the node’s upstream resolvers as configured in CoreDNS’s forward plugin

ClusterFirstWithHostNet

The same, but for a Pod that also sets hostNetwork: true — otherwise such a Pod would inherit the node’s own resolv.conf and never see cluster DNS

Default

Inherit the node’s own resolv.conf (misleadingly named — it is not the cluster default)

None

Ignore both of the above; dnsConfig alone supplies everything

spec.dnsConfig layers extra nameservers, searches and options on top of whichever policy is in effect (or supplies them outright under None):

apiVersion: v1
kind: Pod
metadata:
  name: notifier-worker
  namespace: bookshelf
spec:
  dnsPolicy: "ClusterFirst"
  dnsConfig:
    options:
      - name: ndots
        value: "2"
  containers:
    - name: notifier
      image: ghcr.io/example/bookshelf-notifier:1.0.0

Lowering ndots on a Pod that talks mostly to external hosts (a third-party payment API, a managed database outside the cluster) trades away same-namespace short names for fewer wasted internal lookups; it is a per-Pod decision, not a cluster-wide one.

Source: DNS for Services and Pods

Environment-variable discovery versus DNS

Besides DNS, kubelet injects one set of environment variables per Service that existed before a Pod was created, following the Docker "legacy links" convention: <SVCNAME>_SERVICE_HOST and <SVCNAME>_SERVICE_PORT, plus a handful of <SVCNAME>_PORT_<port>_TCP_* variables. Book 1’s chapter 3 describes this env-var mechanism as the way a Pod finds a Service, alongside DNS, and it is still real — but its ordering caveat is the part worth carrying forward carefully: a Pod only receives environment variables for Services that already existed when that Pod was scheduled. A Service created after a consuming Pod is running gives that Pod no new variables at all — there is no re-injection, no update, nothing to poll. Create catalog-api-svc after web is already running, and web’s container never sees `CATALOG_API_SVC_SERVICE_HOST, even though a kubectl exec nslookup in the very same Pod resolves it correctly, because DNS is looked up live on every query while environment variables are a one-time snapshot taken at Pod start. In a cluster where Services and consuming Deployments are usually applied together (as every Bookshelf manifest does), this rarely bites — but a namespace where Services get added incrementally, or Pods restart in an unpredictable order, is exactly where it does. DNS has no such ordering dependency, which is why it is the recommended mechanism and the env vars exist mainly for compatibility with software written before in-cluster DNS was assumed.

Source: Services — Environment variables

Debugging with a dnsutils Pod

The standard first step when name resolution misbehaves is a disposable Pod carrying nslookup and friends:

apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: bookshelf
spec:
  containers:
    - name: dnsutils
      image: registry.k8s.io/e2e-test-images/agnhost:2.40
      imagePullPolicy: IfNotPresent
  restartPolicy: Always
kubectl apply -f dnsutils.yaml
kubectl exec -i -t dnsutils -n bookshelf -- nslookup catalog-api
kubectl exec -i -t dnsutils -n bookshelf -- cat /etc/resolv.conf
kubectl exec -i -t dnsutils -n bookshelf -- nslookup postgres-0.postgres.bookshelf.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10:53

Name:   catalog-api.bookshelf.svc.cluster.local
Address: 10.96.12.4

When that comes back empty or times out, work through the pieces in order: is a CoreDNS Pod actually Running (kubectl get pods -n kube-system -l k8s-app=kube-dns); does the kube-dns Service have any endpoints (kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns); does the target Service itself have endpoints (a Service with a selector matching zero Pods resolves fine but every connection then fails, a frequent red herring); and, only then, CoreDNS’s own logs (kubectl logs -n kube-system -l k8s-app=kube-dns, after adding the log plugin to the Corefile if nothing shows up). Almost every "DNS is broken" report turns out to be one of the first three, not CoreDNS itself.

Source: Debugging DNS Resolution

NodeLocal DNSCache

At any real query volume, every Pod’s DNS traffic still has to leave its node, cross kube-proxy’s DNAT rules and reach whichever node a CoreDNS replica happens to be scheduled on — and UDP DNS queries through that DNAT path are a known source of dropped packets and 5-to-30-second client-side timeouts under load, because of conntrack races and a shared UDP conntrack table filling up. NodeLocal DNSCache (stable since Kubernetes 1.18) addresses this by running an additional node-local-dns DaemonSet — one caching CoreDNS instance per node, listening on a link-local address (and optionally the kube-dns ClusterIP too, in iptables mode) — so that most queries never leave the node at all, and the ones that do go from the local cache to the real CoreDNS Service over TCP instead of UDP, which closes cleanly instead of waiting out a UDP conntrack timeout. It is deployed alongside CoreDNS, not instead of it: CoreDNS remains the source of truth for the cluster, node-local-dns is purely a cache in front of it, and enabling it is a cluster-admin operation (the DaemonSet manifest plus, in IPVS clusters, pointing kubelet’s --cluster-dns at the local address) rather than anything an application manifest needs to know about.

Source: Using NodeLocal DNSCache in Kubernetes clusters

SkyDNS and kube2sky (history)

Book 1, written against the Kubernetes 1.0 era, has no concept of CoreDNS at all — cluster DNS back then was SkyDNS, a general-purpose etcd-backed DNS server, paired with kube2sky, a sidecar that watched the API server and wrote Service records into SkyDNS’s etcd store. That two-process, etcd-mediated design was replaced by kube-dns (a single Pod with dnsmasq caching in front of a Go-based kubedns component reading the API directly) and later, from Kubernetes 1.13, by CoreDNS — one process, no separate etcd, a plugin chain instead of two cooperating daemons. None of SkyDNS, kube2sky or kube-dns has any part in a current cluster; the kube-dns name survives only as the Service’s name, for compatibility with anything still hard-coding it.

References