Kubernetes clustered deployment with replicas

[brettinternet]

I've been playing around with Livebook's cluster mode on a bare-metal Kubernetes cluster. I initially had some trouble getting the replicas to discover each other, but now that it's working, I wanted to share the deployment setup.

You can see my full helm deployment here.

For storage, I use a RWX persistent longhorn volume to allow me to scale the deployment replicas.

I have a headless service with ports targeted for Livebook's container. The headless service load balances DNS queries for the replicated pods which is what Livebook expects for clustering. Side note: This is similar to what libcluster recommends for its Kubernetes DNS strategy. However, Livebook does not use libcluster.

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: livebook
  labels:
    app.kubernetes.io/instance: livebook
    app.kubernetes.io/name: livebook
  name: livebook
  namespace: default
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/instance: livebook
    app.kubernetes.io/name: livebook

Here are the pod's environment variables:

TZ: "${TIMEZONE}"
LIVEBOOK_PORT: 8080
LIVEBOOK_APP_SERVICE_URL: "https://livebook.${MY_DOMAIN}"
LIVEBOOK_CLUSTER: dns:livebook.default.svc.cluster.local
LIVEBOOK_DISTRIBUTION: name
LIVEBOOK_SECRET_KEY_BASE: secret
LIVEBOOK_COOKIE: secret
POD_IP:
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: status.podIP

I pass these environment variables in a configMap to /app/user/app.sh:

#!/usr/bin/env bash

export LIVEBOOK_IP="${POD_IP:-$(hostname -i)}"
export LIVEBOOK_NODE="livebook@$LIVEBOOK_IP"

The LIVEBOOK_CLUSTER: dns:livebook.default.svc.cluster.local environment value is what DNSCluster lib expects, which Livebook uses for node discovery.

Then, if I dig the headless service in the Kubernetes cluster I should see both pod replicas:

$ dig livebook.default.svc.cluster.local

; <<>> DiG 9.18.25 <<>> livebook.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28602
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: c6d710dd9880727b (echoed)
;; QUESTION SECTION:
;livebook.default.svc.cluster.local. IN	A

;; ANSWER SECTION:
livebook.default.svc.cluster.local. 5 IN A	10.69.1.55
livebook.default.svc.cluster.local. 5 IN A	10.69.0.38

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10) (UDP)
;; WHEN: Fri Jun 07 19:09:01 UTC 2024
;; MSG SIZE  rcvd: 175

So now with the query livebook.default.svc.cluster.local should find all the Elixir nodes. 🙂

[brettinternet]

My friend pointed out that it's a better idea to separate the headless service from the web service so that the web service can properly load balance the pods with incoming requests.

So I created a separate headless service:

---
apiVersion: v1
kind: Service
metadata:
  name: livebook-headless
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app.kubernetes.io/name: livebook
    app.kubernetes.io/instance: livebook

And update the cluster query var to point to that service instead: LIVEBOOK_CLUSTER: dns:livebook-headless.default.svc.cluster.local.