This clustering strategy discovers node hostnames from SRV records for a headless Service. The example below uses a StatefulSet to give Pods stable hostnames.

This strategy requires per-Pod DNS identities exposed by a headless Service. A StatefulSet provides the stable hostnames used in this example. If you want to avoid that, you could use Cluster.Strategy.Kubernetes.DNS.

It assumes that all Erlang nodes are using longnames - <basename>@<domain>:

• all nodes are using the same <basename>
• all nodes are using unique <domain>

In <basename>@<domain>:

• <basename> would be the value configured by :application_name option.
• <domain> would be the value which is controlled by following options:
  • :service
  • :namespace
  • :resolver

Getting <basename>

As said above, the basename is configured by :application_name option.

The basename of each peer's running node must match :application_name. The Mix release name is only the default basename in the examples below; an explicit RELEASE_NODE can override it. See Kubernetes node naming for runtime release configuration and troubleshooting long names, peer identities and connectivity.

Getting <domain>

For more information, see the kubernetes stateful-application documentation

Setup

Getting this strategy to work requires:

1. giving each Pod a hostname matching its SRV target (a StatefulSet supplies stable identities)
2. exposing above StatefulSet by a headless service (otherwise, the SRV query won't work as expected)
3. setting the name of Erlang node according to hostname of pods

First, deploying pods as a StatefulSet which is exposed by a headless service. And here is an example of a corresponding Kubernetes definition:

apiVersion: v1
kind: Service
metadata:
  name: "myapp-headless"
  labels:
    app: myapp
spec:
  ports:
  - port: 4000
    name: web
  clusterIP: None
  selector:
    app: myapp
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myapp
spec:
  serviceName: "myapp-headless"
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:v1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 4000
          name: http
          protocol: TCP

Then, set the name of Erlang node by using the hostname of pod. If you use mix releases, you can configure the required options in rel/env.sh.eex:

# rel/env.sh.eex
export RELEASE_DISTRIBUTION=name
export RELEASE_NODE="<%= @release.name %>@$(hostname -f)"

For DNS publication, readiness and persistent database startup, see Kubernetes startup and readiness.

Polling Interval

The default interval to sync topologies is 5000 (5 seconds). You can configure it with :polling_interval option.

An example configuration

config :libcluster,
  topologies: [
    erlang_nodes_in_k8s: [
      strategy: Elixir.Cluster.Strategy.Kubernetes.DNSSRV,
      config: [
        service: "_web._tcp.myapp-headless",
        application_name: "myapp",
        namespace: "default",
        polling_interval: 10_000
      ]
    ]
  ]

Select the SRV query

For the Service's named web TCP port, set service: "_web._tcp.myapp-headless". The strategy appends the namespace and svc.<cluster-domain>; with namespace default and the default domain, the query is _web._tcp.myapp-headless.default.svc.cluster.local.. Do not pass the full FQDN as :service. Set CLUSTER_DOMAIN if your cluster uses another domain. StatefulSet.spec.serviceName remains myapp-headless, without the port/protocol prefix.

Kubernetes specifies SRV names as _port-name._protocol.service.namespace.svc.cluster-domain. A headless Service returns per-Pod targets; a normal ClusterIP Service returns its Service name. See Kubernetes DNS records. Bare Service SRV queries may work with some DNS implementations; existing configurations remain accepted, but named-port queries avoid relying on that extension. This is not a CoreDNS-only strategy.

Only the SRV target contributes to application_name@target. The SRV port, priority and weight do not select the Erlang distribution listener; the default distribution uses EPMD for that. An HTTP port in the discovery Service therefore does not make distribution use HTTP.

Check the SRV targets

From an application Pod, inspect the selected Service and the exact query:

kubectl get service myapp-headless -n default -o yaml
kubectl get statefulset myapp -n default -o yaml
dig +short SRV _web._tcp.myapp-headless.default.svc.cluster.local.

Expected targets include myapp-0.myapp-headless.default.svc.cluster.local. and myapp-1.myapp-headless.default.svc.cluster.local., not just the Service FQDN. The final dot in dig output denotes an absolute DNS name. Compare the actual Erlang resolver result and libcluster's logged target with the peer's Node.self(); do not append dots to Erlang node names as a workaround. Check the Service selector, clusterIP: None, named port/protocol, StatefulSet serviceName, namespace, cluster domain and published endpoint readiness.

For example, myapp@myapp-headless.default.svc.cluster.local is not the same node identity as myapp@myapp-0.myapp-headless.default.svc.cluster.local, even if both hostnames reach a Pod. Use the node naming checks and attempt Node.connect/1 with exactly the target libcluster logs, from the running application node.

Once identities match, check EPMD and the distribution listener from each peer. To bound the listener to TCP 9100 in a Mix release, add this to rel/env.sh.eex:

case "$RELEASE_COMMAND" in
  start*|daemon*)
    export ELIXIR_ERL_OPTIONS="${ELIXIR_ERL_OPTIONS:-} -kernel inet_dist_listen_min 9100 inet_dist_listen_max 9100"
    ;;
esac

Apply the bound only to application startup, so rpc and remote-console helper VMs can use their own ports. Permit TCP 4369 (default EPMD) and 9100 between Pods in network policies and firewalls. In a mesh, check interception, authorization and TLS policy for both connections; Erlang distribution is TCP, not HTTP. Configure the actual distribution ports as opaque TCP according to Istio protocol selection. If DNS, identity and direct connectivity agree, reproduce in a disposable deployment with and without sidecars before attributing the remaining failure to Istio. No mesh bypass or global TLS-policy change is required by this strategy.