Kubernetes Pods Explained: Why Connecting to a Database Pod Is Hard

This article examines the architecture of Kubernetes pods and explains why directly connecting an external client to a database running inside a pod is more involved than the equivalent task with a standalone Docker container. The discussion is grounded in the networking model that Kubernetes uses and in the Service abstraction that the model requires. A common experience for engineers new to Kubernetes is that kubectl exec into a Postgres pod followed by psql -h localhost works as expected, while a parallel attempt from a developer laptop, using the pod IP reported by kubectl get pods -o wide, times out with no error. The credentials, the database, and the apparent network are the same, yet the second connection never completes. This outcome is not the result of a defect; it is a direct consequence of how the platform is designed, and understanding the design is the first step toward connecting to in-cluster databases in a reliable manner.

Why Kubernetes Exists in the First Place

Docker addressed a genuine packaging problem by allowing an application and its dependencies to be shipped as a single reproducible image that behaves consistently on a developer laptop, a continuous integration runner, and a production virtual machine. Readers who have not yet worked through the container model will find the Docker containers, from dev to production guide a useful prerequisite for the material that follows. Once an organization operates several dozen containers distributed across several dozen servers, however, Docker alone becomes insufficient. Several questions arise that the single-host model does not answer: which host should run a given container, what should happen if that host fails outside business hours, how can a new version be rolled out without dropping in-flight requests, how do containers on one host discover containers on another, and how should compute and memory budgets be enforced.

Kubernetes is the answer that became the industry consensus. It originated inside Google as a re-implementation of the Borg system and was open-sourced in 2014. Kubernetes is best understood as a cluster operating system: the operator declares the desired state of the workload, and a chain of controllers continuously reconciles the actual state of the cluster with that declaration. The unit that Kubernetes manages is not a container directly but a pod, a wrapper around one or more tightly coupled containers that share a network identity and storage. All higher-level objects, including Deployments, Services, StatefulSets, Jobs, and CronJobs, are abstractions that ultimately specify which pods should exist, where they should run, and how they should be exposed.

The control plane performs the coordination function, while worker nodes execute the workload. Each worker runs three components in its base layer. The kubelet is the agent that receives instructions from the API server and applies them to the local node. The container runtime, which is now almost always containerd or CRI-O, executes the containers themselves; Docker as a runtime was deprecated in version 1.20 and removed in version 1.24. The kube-proxy process programs the kernel iptables or IPVS rules that route Service IPs to actual pod endpoints. Pods are scheduled on top of these three components.

The Pod: Smaller Than a VM, Bigger Than a Container

A pod is the smallest deployable unit in Kubernetes. The simplest pod runs a single container, but the abstraction exists precisely because the smallest unit that an engineer sometimes needs to ship is more than one container. A common configuration places a primary application container alongside a sidecar container that handles logging, TLS termination, metric scraping, or database proxying. All containers in the same pod share a single network namespace, which means they can communicate over localhost, and they can share filesystem volumes. They are always scheduled together onto the same node, and their lifecycles are linked: they are created together, restarted together, and terminated together.

The manifest below shows the minimum specification for a pod. Three details are worth noting. The apiVersion: v1 field indicates that pods are part of the core Kubernetes API. The specification contains a single container running an Nginx image. There is no top-level restart policy, which reflects the fact that bare pods are not self-healing: when a node fails, the pod fails with it. Bare pods are therefore rarely used in production. They are useful primarily for one-off tests and as a teaching device.

# pod.yaml — the simplest possible pod
apiVersion: v1
kind: Pod
metadata:
  name: hello-pod
  labels:
    app: hello
spec:
  containers:
  - name: web
    image: nginx:1.27
    ports:
    - containerPort: 80
    resources:
      requests:
        cpu: "50m"
        memory: "64Mi"
      limits:
        cpu: "200m"
        memory: "128Mi"

The object that an operator actually deploys is a Deployment, a controller that maintains a desired number of identical pods, handles rolling updates, and recreates pods when nodes fail. The Deployment owns a ReplicaSet, which in turn owns the pods. Operators rarely reference pods directly. Instead, they reference the Deployment, and Kubernetes manages the underlying pods.

# deployment.yaml — a real workload
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: ghcr.io/acme/api:1.4.2
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet: { path: /healthz, port: 8000 }
          initialDelaySeconds: 5
        livenessProbe:
          httpGet: { path: /livez, port: 8000 }
          initialDelaySeconds: 15
        env:
        - name: DB_HOST
          value: "postgres.db.svc.cluster.local"
        - name: DB_PORT
          value: "5432"

The Flat Networking Model and Its Implications

The Kubernetes networking model rests on four rules that appear simple on the surface but carry substantial implications for how traffic flows inside a cluster:

1. Every pod gets its own IP address, drawn from a cluster-wide CIDR range that does not overlap with the node IPs.
2. Pods on the same node can communicate without NAT.
3. Pods on different nodes can communicate without NAT.
4. The IP a pod sees as its own is the same IP that other pods see when they talk to it.

The fourth point carries the most weight. In a typical single-host Docker configuration, a container holds a private IP on a bridge network, and outbound traffic is translated through the host using NAT. Kubernetes deliberately avoids this arrangement. Every pod is a first-class participant in a single flat network, regardless of the physical machine that hosts it. The component that implements this property is a CNI plugin (Container Network Interface), of which several mature implementations exist, including Calico, Cilium, Flannel, Weave, the AWS VPC CNI, and the native plugin used by GKE. These plugins implement the same contract but differ in their mechanisms, which range from overlays based on VXLAN tunnels, to BGP route advertisement, to native cloud routing, to eBPF-based data planes.

This flat addressing scheme is convenient for application code, which simply connects to an IP address and proceeds, but it is demanding for operators who must reason about traffic flows. Every pod is mutually addressable inside the cluster, which means that in the absence of explicit policies, every pod is able to reach every database, every cache, and every internal API. This property becomes important in a later section, which explains why directly addressing a database pod is more fragile than it appears.

Services: How Pods Actually Find Each Other

Because pod IPs change on every restart, they cannot appear in a connection string. The Kubernetes solution to this problem is a Service, a stable virtual IP and DNS name that load-balances traffic to a set of pods identified by labels. Individual pods come and go, but the Service remains. Inside the cluster, every Service automatically receives a DNS name of the form service-name.namespace.svc.cluster.local, which is resolved by CoreDNS, the built-in DNS resolver of the cluster.

The manifest below defines a ClusterIP Service for a Postgres pod. The selector field is worth attention, because the Service matches by labels rather than by name. Any pod that carries the label app: postgres in the same namespace automatically becomes a backend.

# service.yaml — ClusterIP for Postgres
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: db
spec:
  type: ClusterIP            # default, can omit
  selector:
    app: postgres
  ports:
  - name: pg
    port: 5432              # the Service port
    targetPort: 5432        # the container port
    protocol: TCP

From any other pod inside the cluster, the command psql -h postgres.db.svc.cluster.local -p 5432 will succeed. The same command issued from a developer laptop will hang indefinitely. The next section examines the reasons for this gap in detail.

Why Direct Connections to a Database Pod Fail

The assumption that breaks down for engineers coming from a plain Docker workflow is the idea that a container can be reached as long as one knows its IP address and port. In Kubernetes, almost every element of that assumption is incorrect. The pod IP is real but private, ephemeral, and exists only on a network shared between the nodes of a single cluster. The container port is open inside the container but is not automatically exposed at any higher layer. There is no host-level port-publishing equivalent to docker run -p 5432:5432; the field hostPort exists but is discouraged for production use. The following paragraphs examine each failure mode in turn.

Pod IPs are ephemeral. The moment a pod restarts, for any reason ranging from a node reboot, to a failed liveness probe, to a manual kubectl rollout restart, to an eviction by the scheduler, the new pod receives a new IP address from the address pool that the CNI manages. Any client that retains a reference to the previous IP is now communicating with nothing, or in the worst case, with whatever pod has been allocated the recycled address. This is the reason pod IPs should never be written into a configuration file. The correct address to record is a Service DNS name, which CoreDNS resolves at lookup time.

The ClusterIP is not visible from outside the cluster. The Service IP that kubectl get svc reports, such as 10.96.42.7 in the earlier example, is a virtual IP. It does not belong to any physical or virtual network interface and exists only as an entry in the iptables tables that kube-proxy maintains on each node. A laptop outside the cluster has no route to the Service CIDR 10.96.0.0/12, and even a statically added route would not help, because no kernel outside the cluster contains the rules required to translate that virtual address.

Pods do not use the host network by default. Setting hostNetwork: true on a pod causes the container to share the network namespace of the node, with the consequence that the container port maps directly to a port on the node. This configuration is used by CNI agents, node-exporter, and similar infrastructure components. Applying it to a database, however, is poor practice: IP isolation is lost, port collisions become possible, and any node failure takes the database with it, since the address is tied to a specific host and cannot be moved.

NetworkPolicies can explicitly deny traffic. When the cluster runs a CNI that supports NetworkPolicy, which most modern plugins do, operators can write rules such as “only pods labeled role: api in the app namespace may connect to pods labeled app: postgres in the db namespace on port 5432.” When a default-deny baseline is in place and no allow rule has been written, all traffic is dropped. When no policies are present at all, all traffic is permitted, which presents its own security concerns.

# networkpolicy.yaml — only the api can talk to postgres
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-allow-api
  namespace: db
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: app
      podSelector:
        matchLabels:
          role: api
    ports:
    - protocol: TCP
      port: 5432

The container port is not automatically exposed at the node level. Docker users are accustomed to -p 5432:5432, which binds a host port to a container port. Kubernetes provides no equivalent automatic mapping. The containerPort field in a pod specification is documentation: it informs operators and tooling that the container intends to listen on the indicated port, but it does not open a path through any higher layer. External reachability requires a Service of the appropriate type and, in cloud environments, a security group rule that permits traffic to whichever node port the cloud load balancer or NodePort uses.

Databases are stateful, and stateful pods require stateful controllers. A plain Deployment treats its pods as interchangeable replicas. The Deployment will reschedule postgres-0 from node 2 to node 5 when node 2 becomes unhealthy, mounting whichever PersistentVolume is available, or no volume at all if the PersistentVolumeClaim has been deleted. A database instead requires a StatefulSet, which assigns each pod a stable identity such as postgres-0 or postgres-1, a stable per-pod DNS name served by a headless Service, and a stable PersistentVolumeClaim that remains attached to the same ordinal across reschedules. A misconfiguration in this area is a common cause of data loss for teams new to running databases on Kubernetes.

The request path is long, and any single weak link breaks it. When an external client reaches a pod, the request typically traverses the following sequence: client, public DNS, cloud load balancer, node IP, iptables DNAT, pod IP, container port, Postgres listener, pg_hba.conf check, and finally authentication. A misconfiguration at any stage, such as an incorrect TLS certificate, a security group blocking the load balancer health check, a pg_hba.conf rule that denies the source CIDR, or a Postgres listener bound to 127.0.0.1 inside the container rather than 0.0.0.0, produces a connection failure that appears identical to a network problem from the perspective of the client.

Three Reliable Connection Patterns

Three legitimate patterns exist for connecting a client to a database that runs in a pod, and the appropriate choice depends primarily on the location of the client. Selecting among them is largely a question of which client requires the connection and for how long.

Pattern A: In-cluster application to in-cluster database

This pattern is the default and the most reliable choice. The application pod sets DB_HOST=postgres.db.svc.cluster.local as an environment variable and opens a connection. CoreDNS resolves the name, kube-proxy translates the virtual IP into the address of a real pod through DNAT, and the connection succeeds. Pod restarts on either side remain transparent because every endpoint is named rather than pinned to a specific IP. This is also the pattern that Airflow workloads adopt when they run with the KubernetesExecutor described in the Apache Airflow data pipeline orchestration guide, in which each task is launched as a pod that reaches the database through a Service. The same pattern applies to dbt jobs running on Kubernetes and to Kafka consumer workloads running in pods.

Pattern B: Local developer to in-cluster database

The command kubectl port-forward opens a tunnel from a local port on a developer machine, through the Kubernetes API server, to a port on a pod. It is intended for development and one-off administrative tasks. The example below uses it against the headless Service that the next subsection defines:

# forward localhost:5432 to the postgres-0 pod's port 5432
kubectl port-forward -n db pod/postgres-0 5432:5432

# Or forward through the headless Service to whichever endpoint is selected
kubectl port-forward -n db svc/postgres 5432:5432

# Now from another terminal, on your laptop:
psql -h localhost -p 5432 -U app -d production

The Python client below connects through the forwarded port. The connection string specifies localhost, which is correct on the developer laptop. Inside the cluster, the same code would instead specify postgres.db.svc.cluster.local.

# dev_query.py — assumes "kubectl port-forward" is running
import os
import psycopg2
from psycopg2.extras import RealDictCursor

# Local dev: connect through kubectl port-forward
# In production (in-cluster), DB_HOST would be postgres.db.svc.cluster.local
DB_HOST = os.environ.get("DB_HOST", "localhost")
DB_PORT = int(os.environ.get("DB_PORT", "5432"))
DB_NAME = os.environ.get("DB_NAME", "production")
DB_USER = os.environ.get("DB_USER", "app")
DB_PASS = os.environ["DB_PASS"]  # required, no default

def fetch_recent_orders(limit: int = 50):
    """Read the most recent orders — example dev-time query."""
    with psycopg2.connect(
        host=DB_HOST,
        port=DB_PORT,
        dbname=DB_NAME,
        user=DB_USER,
        password=DB_PASS,
        connect_timeout=5,
        sslmode="require",   # still enforce TLS even on port-forward
    ) as conn:
        with conn.cursor(cursor_factory=RealDictCursor) as cur:
            cur.execute(
                "SELECT id, customer_id, total_cents, created_at "
                "FROM orders ORDER BY created_at DESC LIMIT %s",
                (limit,),
            )
            return cur.fetchall()

if __name__ == "__main__":
    rows = fetch_recent_orders()
    for row in rows:
        print(row)

Pattern C: External application to in-cluster database

This is the pattern about which most teams should hesitate. When an application outside the cluster needs to read from or write to the database, the preferred architecture is almost always to expose an API over HTTP or gRPC through an Ingress with TLS and authentication, and to let the API mediate access to the database. Legitimate cases for direct external access nevertheless exist, including analytics tools, business intelligence dashboards, and replication to external systems. In those cases the pattern takes the following shape: a Service of type LoadBalancer backed by the database pods, fronted by a cloud network load balancer, with the security group restricted to specific client CIDRs, mandatory TLS, and a credential rotation policy. When a managed database such as Amazon RDS, Google Cloud SQL, or Aurora can be substituted, that option is usually preferable. Operating Postgres inside Kubernetes is technically feasible, but it represents a significant operational commitment.

The StatefulSet plus headless Service pattern

A headless Service is the object produced when clusterIP: None is set in the specification. Rather than allocating a virtual IP, this configuration produces DNS A records, with one record per pod backend. When combined with a StatefulSet, the result is a set of stable per-pod hostnames, such as postgres-0.postgres.db.svc.cluster.local and postgres-1.postgres.db.svc.cluster.local. This naming arrangement is precisely what a primary-replica database deployment requires. The application directs writes to the hostname of the primary and reads to the hostname of any replica.

# headless service + statefulset for postgres
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: db
  labels:
    app: postgres
spec:
  clusterIP: None          # headless — no virtual IP
  selector:
    app: postgres
  ports:
  - name: pg
    port: 5432
    targetPort: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: db
spec:
  serviceName: postgres    # MUST match the headless Service name
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: postgres
        image: postgres:16.3
        ports:
        - containerPort: 5432
          name: pg
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3-ssd
      resources:
        requests:
          storage: 200Gi

Production databases almost always benefit from a purpose-built operator layered on top of this scaffolding, such as CloudNativePG, the postgres-operator developed by Zalando, or Crunchy PGO. These operators handle primary election, streaming replication, backups, point-in-time recovery, and rolling minor-version upgrades. Selecting an appropriate database backend is a separate concern; the database comparison for preprocessed time-series data serves as a useful companion reference for that decision.

Kubernetes 1.36 and What Changed in 2026

Kubernetes 1.36 is the most recent minor release as of this writing in May 2026, and it continues the project’s emphasis on stronger security defaults and on first-class support for AI workloads. According to the official release page (Source: kubernetes.io/releases, as of 2026-05-20), the project actively maintains release branches for the three most recent minor versions, currently 1.34, 1.35, and 1.36. Version 1.33 entered maintenance on 2026-04-28 and reaches end of life on 2026-06-28. The release cadence is rapid enough that operators running anything older than 1.33 are already outside the supported window.

The promotion of User Namespaces to general availability is the most prominent security change in 1.36. When user namespaces are enabled, the root user inside a container is mapped to an unprivileged user on the host. This arrangement substantially reduces the impact of a container escape: even when an attacker compromises a container running as UID 0, they emerge on the host as a high-numbered unprivileged user, such as UID 100000, with no special privileges. For database pods specifically, a compromised Postgres container no longer translates directly into root access on the node. In combination with seccomp and AppArmor profiles, this change closes one of the long-standing gaps between Kubernetes security and traditional virtual machine isolation.

Mutating Admission Policies, also promoted to general availability, bring declarative mutations expressed in the Common Expression Language (CEL) to the admission chain, replacing many uses of webhook-based mutating admission controllers. Operators can now write policies that, for example, automatically inject sidecar containers, attach labels, set default resource requests, or enforce image-registry rules, without operating a separate webhook server. The result is less infrastructure to maintain and fewer failure modes when a webhook becomes unavailable.

Fine-Grained Kubelet API Authorization, now generally available, allows the kubelet to enforce per-verb RBAC on its own API rather than treating all operations uniformly. This change matters for hardening: tools that require nodes/proxy can be restricted to read-only operations, and the kubelet can refuse risky combinations that previously required cluster-admin privileges in order to be fully restricted.

Beyond security, version 1.36 continues to invest in AI workload support. It introduces refinements to Dynamic Resource Allocation (DRA) for GPU scheduling, adds support for accelerator partitioning, and improves the ability of the scheduler to handle long-running training jobs alongside short-lived inference pods. The trajectory is clear: the pattern of Kubernetes as an AI platform, which grew rapidly in 2024 and 2025 as model-serving workloads migrated off bespoke infrastructure, has been a first-class concern for two consecutive release cycles. For language and runtime choices when developing operators or controllers around these new APIs, the Python and Rust comparison provides a useful framing. The controller-runtime ecosystem in Go remains dominant, but Rust-based operators are gaining ground for performance-sensitive components.

Frequently Asked Questions

Can a pod have more than one container?

Yes, and it is a common design pattern. The most frequent reason is the sidecar — a helper container that does logging, TLS termination, service-mesh proxying (Envoy in Istio or Linkerd), or connection pooling. All containers in a pod share a network namespace and can share volumes, but they remain separate processes with separate filesystems. Use multiple containers when their lifecycles are genuinely coupled. If the answer to “can these scale independently?” is yes, they belong in separate pods.

Why not just expose every database pod with a NodePort and connect directly?

NodePort opens the same port on every node in the cluster, in the 30000–32767 range, and routes it to whichever pod backs the Service. Three problems: the port numbers are non-standard so client tooling fights you, every node becomes an attack surface for the database, and you still need a cloud security group or firewall rule to control who can hit those ports. NodePort is fine for on-prem clusters without a cloud LB or for very specific debug scenarios. It is not a substitute for proper Service architecture.

Is kubectl port-forward safe to use in production?

It is safe to use, but it should not be how production traffic flows. The tunnel runs through the API server and consumes API-server resources. It bypasses NetworkPolicy — if you can port-forward, you can connect, regardless of how strict your in-cluster policies are. RBAC controls who can use it, and you should treat pods/portforward on a database namespace as a sensitive verb subject to audit. For production traffic, use a real Service.

What is the difference between a StatefulSet and a Deployment?

A Deployment treats pods as interchangeable. It will scale up by spinning up new pods with random suffix names, scale down by killing any of them, and roll updates in parallel. A StatefulSet maintains ordered, named pods (name-0, name-1, name-2) that always come up in order, always shut down in reverse order, and each get their own stable PersistentVolumeClaim. Use Deployment for stateless apps. Use StatefulSet for anything that has identity — databases, message brokers, ZooKeeper, distributed coordination services. Kafka brokers running in Kubernetes are a textbook StatefulSet workload.

Should I actually run my database in Kubernetes, or use a managed service?

For most teams below the scale of needing a database engineer on the org chart, managed (RDS, Cloud SQL, Aurora, AlloyDB, Spanner) is the right answer. Operating a stateful workload well — backups, point-in-time recovery, minor-version upgrades, failover, performance tuning, observability — is a continuous engineering investment that managed services amortize across thousands of customers. Run databases in your cluster when you have a real reason: cost at scale, regulatory data residency, latency requirements that make a separate database tier unworkable, or a database that managed offerings do not provide. The operator ecosystem (CloudNativePG and friends) makes this much more tractable than it was five years ago, but it is still real work.

Related Reading

References

• Kubernetes releases — official release and support timeline
• Kubernetes cluster networking concepts
• Kubernetes 1.36 sneak peek (official blog, March 2026)
• Connecting applications with Services (official tutorial)
• InfoQ: Kubernetes 1.36 released (May 2026)

Conclusion

Connecting to a database that runs in a Kubernetes pod feels harder than it should because Kubernetes is solving a different problem than many engineers initially assume. It is not an elaborate replacement for docker run. It is a cluster operating system whose entire networking model is designed around the principle that pods communicate with other pods through stable abstractions, and external clients reach applications through carefully chosen entry points. The pod IP revealed by kubectl get pods -o wide is a debugging convenience rather than an address suitable for client traffic. The ClusterIP shown by kubectl get svc is a virtual construct held together by iptables rules. The correct address for production traffic originating inside the cluster is a DNS name served by CoreDNS and backed by a Service whose membership the controllers maintain. The correct address from outside the cluster is whatever the LoadBalancer, Ingress, or bastion-host configuration specifies, and it is never a pod IP.

Three points are worth retaining from this discussion. First, kubectl port-forward is well suited to development workflows and unsuited to production traffic. Second, stateful workloads require a StatefulSet, a PersistentVolumeClaim, and a headless Service in combination, or data loss is likely. Third, in Kubernetes 1.36 and beyond, security defaults are tightening, with User Namespaces reaching general availability as the most consequential change, which benefits anyone running databases in pods. Even with these improvements, however, the number of ways in which a connection between an external client and an in-cluster database can fail remains large enough that exposing Postgres directly to the public internet is almost always inferior to placing an API tier in front of the database. The recommended approach is to build the conservative, layered version first, and to reserve more aggressive shortcuts for cases that genuinely warrant them.