The LSD People Blog

GKE mixed TCP/UDP loadbalancer

GKE MIXED UPD/TCP LOADBALANCER

Recently I’ve been experimenting with Google Kubernetes Engine (GKE) and in particular the ingress from external to the GKE cluster. There are essentially 3 network options, fairly well described.

  1. ClusterIP (service exposed internally to cluster)
  2. NodePort (port opened and mapped on the node)
  3. LoadBalancer (external IP created and forwarded to application. This can both be at a TCP/UDP level or at a layer 7 http/s level)

LoadBalancer is the probably the type you want if you want if you want external ingress into your application

LoadBalancer is the probably the type you want if you want if you want external ingress into your application.

I wanted to run a Go Ethereum (geth) client on GKE. For peer discovery and communication you want to an external IP that will send TCP 30303 and UDP 30303 to your application.

No worries, i’ve made a bunch of LoadBalancer types before. Something like this (adapted from my helm3 chart)

apiVersion: v1
kind: Service
metadata:
  name: lit-lb
  labels:
    helm.sh/chart: lnd-lit-0.1.0
    app.kubernetes.io/name: lnd-lit
    app.kubernetes.io/instance: lit
    app.kubernetes.io/version: "v0.3.1-alpha"
    app.kubernetes.io/managed-by: Helm
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: lnd-lit
    app.kubernetes.io/instance: lit
  ports:
    – protocol: TCP
       port: 443
    loadBalancerSourceRanges:
  – 129.205.33.85/32

You can have the last one for free (loadBalancerSourceRanges) – That actually does the GKE firewall which can limit which IP can access your ingress. (This can be super useful for an app which you want to access, but its not ideal for the rest of the world to access)

The above code speaks to GKE and created a Google Loadbalancer (you can see it in your console) and connect it to the service running on the node. What you will also probably notice is that this is only for TCP. I need both UDP and TCP. No worries, i’ll just add that in via another – protocol: UDP section under the ports. Right?

The issue

What happens is, you run into this “cannot create an external load balancer  with mix protocols” – essentially saying you cant (well, GKE) cant create a mixed type TCP/UDP loadbalancer. This upstream issue references it.

This causes a problem as geth expects both the TCP and UDP to arrive to it on 30303. It also seemed that you couldn’t change this to be different.

How we fixed it

Highlights

  1. Switch to using NodePort
  2. Create a static IP on Google Network services
  3. Use kubectl to create a 2x LoadBalancer’s and pass in the loadBalancerIP: “StaticIPhere” (one for UDP and one for TCP in my application)
  4. Monitor in Google console and kubectl that the services are created and bound

 

In Detail

  1. Switch to using NodePort

The initial thinking was to switch from LoadBalancer to NodePort. This would listen on the node to any incoming traffic on 30303 TCP and UDP.

apiVersion: v1
kind: Service
metadata:
  name: geth-np
  labels:
    helm.sh/chart: geth-0.1.0
    app.kubernetes.io/name: geth
    app.kubernetes.io/instance: geth
    app.kubernetes.io/version: "v1.9.24"
    app.kubernetes.io/managed-by: Helm
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: geth
    app.kubernetes.io/instance: geth
  ports:
  - name: geth-tcp
    port: 30303
    targetPort: 30303
    protocol: TCP
    nodePort: 30303
  - name: geth-udp
    port: 30303
    targetPort: 30303
    protocol: UDP
    nodePort: 30303

This should be simple enough. This will open a node port from TCP 30303 to 30303 on the node. Confirm you have this with <kubectl get svc>

 

2. Create a static IP on Google Network services

Create a static IP. Put it in the same region as your GKE servers and attached to none


 

3. Use kubectl to create two load balancers.

This is an important step as we couldnt get it working doing it via the gui. Kubectl does some further magic I haven’t quiet worked out yet. Be sure to replace your staticIP with one you reserved

---
apiVersion: v1
kind: Service
metadata:
  name: geth-lb-tcp
spec:
  selector:
    app.kubernetes.io/instance: geth
    app.kubernetes.io/name: geth
  ports:
    - port: 30303
      targetPort: 30303
      protocol: TCP
  type: LoadBalancer
  loadBalancerIP: "StaticIPHere"
---
apiVersion: v1
kind: Service
metadata:
  name: geth-lb-udp
spec:
  selector:
    app.kubernetes.io/instance: geth
    app.kubernetes.io/name: geth
  ports:
    - port: 30303
      targetPort: 30303
      protocol: UDP
  type: LoadBalancer
  loadBalancerIP: "StaticIPHere"

4. Monitor that its working

geth-lb-tcp          LoadBalancer   192.168.65.47     StaticHere              30303:31232/TCP                   3d9h
geth-lb-udp          LoadBalancer   192.168.95.91    StaticHere              30303:31571/UDP                   3d10h

 

That should hopefully be it

You should have a mixed TCP/UDP loadbalancer on with the same external IP sending packets to your application.

Post a Comment

Your email address will not be published. Required fields are marked *