NSQ cluster in Kubernetes - go

Question

I'm trying to set up an NSQ cluster in Kubernetes and having issues.
Basically, I want to scale out NSQ and NSQ Lookup. I have a stateful set(2 nodes) definition for both of them. To not post the whole YAML file, I'll post only part of it for NSQ
NSQ container template
command:
- /nsqd
- -data-path
- /data
- -lookupd-tcp-address
- nsqlookupd.default.svc.cluster.local:4160
here nsqlookupd.default.svc.cluster.local is a K8s headless service, by doing this I'm expecting the NSQ instance to open a connection with all of the NSQ Lookup instances, which in fact is not happening. It just opens a connection with a random one. However, if I explicitly list all of the NSQ Lookup hosts like this, it works.
command:
- /nsqd
- -data-path
- /data
- -lookupd-tcp-address
- nsqlookupd-0.nsqlookupd:4160
- -lookupd-tcp-address
- nsqlookupd-1.nsqlookupd:4160
I also wanted to use the headless service DNS name in --broadcast-address for both NSQ and NSQ Lookup but that doesn't work as well.
I'm using nsqio go library for Publishing and Consuming messages, and it looks like I can't use headless service there as well and should explicitly list NSQ/NSQ Lookup pod names when initializing a consumer or publisher.
Am I using this in the wrong way?
I mean I want to have horizontally scaled NSQ and NSQLookup instances and not hardcode the addresses.

Answer 1

you could use statfulset and headless service to achieve this goal

Answer 2

Try using the below config yaml to deploy it into the K8s cluster. Or else you should checkout the Official Helm : https://github.com/nsqio/helm-chart
Using headless service you can discover all pod IPs.
apiVersion: v1
kind: Service
metadata:
name: nsqlookupd
labels:
app: nsq
spec:
ports:
- port: 4160
targetPort: 4160
name: tcp
- port: 4161
targetPort: 4161
name: http
publishNotReadyAddresses: true
clusterIP: None
selector:
app: nsq
component: nsqlookupd
---
apiVersion: v1
kind: Service
metadata:
name: nsqd
labels:
app: nsq
spec:
ports:
- port: 4150
targetPort: 4150
name: tcp
- port: 4151
targetPort: 4151
name: http
clusterIP: None
selector:
app: nsq
component: nsqd
---
apiVersion: v1
kind: Service
metadata:
name: nsqadmin
labels:
app: nsq
spec:
ports:
- port: 4170
targetPort: 4170
name: tcp
- port: 4171
targetPort: 4171
name: http
selector:
app: nsq
component: nsqadmin
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nsqlookupd
spec:
serviceName: "nsqlookupd"
replicas: 3
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nsq
component: nsqlookupd
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nsq
- key: component
operator: In
values:
- nsqlookupd
topologyKey: "kubernetes.io/hostname"
containers:
- name: nsqlookupd
image: nsqio/nsq:v1.1.0
imagePullPolicy: Always
resources:
requests:
cpu: 30m
memory: 64Mi
ports:
- containerPort: 4160
name: tcp
- containerPort: 4161
name: http
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
readinessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 2
command:
- /nsqlookupd
terminationGracePeriodSeconds: 5
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nsqd
spec:
serviceName: "nsqd"
replicas: 3
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: nsq
component: nsqd
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- nsq
- key: component
operator: In
values:
- nsqd
topologyKey: "kubernetes.io/hostname"
containers:
- name: nsqd
image: nsqio/nsq:v1.1.0
imagePullPolicy: Always
resources:
requests:
cpu: 30m
memory: 64Mi
ports:
- containerPort: 4150
name: tcp
- containerPort: 4151
name: http
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
readinessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 2
volumeMounts:
- name: datadir
mountPath: /data
command:
- /nsqd
- -data-path
- /data
- -lookupd-tcp-address
- nsqlookupd-0.nsqlookupd:4160
- -lookupd-tcp-address
- nsqlookupd-1.nsqlookupd:4160
- -lookupd-tcp-address
- nsqlookupd-2.nsqlookupd:4160
- -broadcast-address
- $(HOSTNAME).nsqd
env:
- name: HOSTNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
terminationGracePeriodSeconds: 5
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes:
- "ReadWriteOnce"
storageClassName: ssd
resources:
requests:
storage: 1Gi
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nsqadmin
spec:
replicas: 1
template:
metadata:
labels:
app: nsq
component: nsqadmin
spec:
containers:
- name: nsqadmin
image: nsqio/nsq:v1.1.0
imagePullPolicy: Always
resources:
requests:
cpu: 30m
memory: 64Mi
ports:
- containerPort: 4170
name: tcp
- containerPort: 4171
name: http
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 10
readinessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
command:
- /nsqadmin
- -lookupd-http-address
- nsqlookupd-0.nsqlookupd:4161
- -lookupd-http-address
- nsqlookupd-1.nsqlookupd:4161
- -lookupd-http-address
- nsqlookupd-2.nsqlookupd:4161
terminationGracePeriodSeconds: 5
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nsq
spec:
rules:
- host: nsq.example.com
http:
paths:
- path: /
backend:
serviceName: nsqadmin
servicePort: 4171