Configuring Split-Horizon Ingress with Istio and MongoDB in Kubernetes

Prerequisites and Assumptions

MongoDB Enterprise Operator and Ops Manager infrastructure is already deployed.
Istio is not injecting mesh network sidecars on the deployment namespace. We will need to ensure TLS connections terminate at the mongod directly.
You have an accessible Load Balancer and created DNS records for your external horizon addresses to point to this public address.
You have cert-manager deployed and an Issuer (Certificate Authority) configured

TLS Certificates from cert-manager

The following certificate request includes the Subject Alternative Names list of both the external connection addresses as well as the internal pods’ fully qualified domain name (FQDN). The certificate usages including both server auth and client auth is required for MongoDB, regardless of split-horizon situations. The spec.secretName includes an important “prefix” followed by the MongoDB metadata.name and a cert suffix, e.g. mdb-{name}-cert. This is required for the Operator to hook onto the kubernetes secret that cert-manager generates. The Operator is then expected to create a new secret (type opaque) that takes the name format of mdb-{name}-cert-pem (the operator concatenated the tls secret’s key and cert into a single data field). MongoDB configuration typically requires a single PEM file that contains both the private key and certificate. The Operator mounts this secret to the MongoDB statefulset’s pods.

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-scram-enabled-replica-set-cert
  namespace: mongodb
spec:
  # Secret names are always required.
  secretName: mdb-my-scram-enabled-replica-set-cert
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
      - jalder
  # The use of the common name field has been deprecated since 2000 and is
  # discouraged from being used.
  commonName: my-tls-enabled-rs-0
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - server auth
    - client auth
  # At least one of a DNS Name, URI, or IP address is required.
  dnsNames:
    - my-scram-enabled-replica-set-0.my-scram-enabled-replica-set-svc.mongodb.svc.cluster.local
    - my-scram-enabled-replica-set-1.my-scram-enabled-replica-set-svc.mongodb.svc.cluster.local
    - my-scram-enabled-replica-set-2.my-scram-enabled-replica-set-svc.mongodb.svc.cluster.local
    - mdb0.k3s.jalder.tech
    - mdb1.k3s.jalder.tech
    - mdb2.k3s.jalder.tech
  # Issuer references are always required.
  issuerRef:
    name: ca-issuer
    kind: Issuer
    group: cert-manager.io

MongoDB manifest

The following MongoDB manifest enables both TLS and SCRAM-SHA-256 authentication mechanism. It’s presumed your Ops Manager project connection details are in a configmap named my-project and your Ops Manager API keys are in kubernetes secret my-credentials. Note the certsSecretPrefix is set to mdb which was mentioned as a requirement in the previous step. This informs the Operator which TLS secret maps to this MongoDB deployment. The cert-manager Issuer’s Certificate Authority chain is saved in a k8s configmap named custom-ca with a data field named ca-pem (e.g. kubectl create cm custom-ca --from-file=ca-pem). The replicaSetHorizons has been populated with an ordered array of the external addresses or hostnames that you will reach the service at. Note that IP Addresses are not allowed in MongoDB split-horizon as the TLS SNI RFC states that SNI does not accommodate IP Addresses (some libraries will automatically look up the rDNS of an IP when negotiating TLS anyways).

---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: my-scram-enabled-replica-set
  namespace: mongodb
spec:
  credentials: my-credentials
  members: 3
  opsManager:
    configMapRef:
      name: my-project
  security:
    certsSecretPrefix: mdb
    tls:
      enabled: true
      ca: custom-ca
    authentication:
      agents:
        mode: SCRAM
      enabled: true
      modes:
      - SCRAM
  type: ReplicaSet
  version: 4.4.12-ent
  connectivity:
    replicaSetHorizons:
    - "istio-gw": "mdb0.k3s.jalder.tech:27017"
    - "istio-gw": "mdb1.k3s.jalder.tech:27017"
    - "istio-gw": "mdb2.k3s.jalder.tech:27017"

MongoDB Pod Services

Three services are created, one for each pod. The selector targets each pod individually, this is important for the external MongoDB drivers as they need to contact each replica set member directly and individually in order to route queries correctly.

---
apiVersion: v1
kind: Service
metadata:
  name: my-scram-enabled-replica-set-0
  namespace: mongodb
spec:
  selector:
    statefulset.kubernetes.io/pod-name: my-scram-enabled-replica-set-0
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: my-scram-enabled-replica-set-1
  namespace: mongodb
spec:
  selector:
    statefulset.kubernetes.io/pod-name: my-scram-enabled-replica-set-1
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: my-scram-enabled-replica-set-2
  namespace: mongodb
spec:
  selector:
    statefulset.kubernetes.io/pod-name: my-scram-enabled-replica-set-2
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017

MongoDB VirtualService Routing

An istio VirtualService is created to bind the internal pod services with the gateway service. Note the sniHosts route individually to their respective replica set members. We’re catching all wildcard subdomains on the external address, adjust this as appropriate for your environment. The route destinations point to the pod service’s metadata.name. We will be routing port 27017 1:1 throughout this project, but you may change ports at the horizon if desired (update the replicaSetHorizons with appropriate ports).

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-scram-enabled-replica-set
  namespace: mongodb
spec:
  hosts:
  - "*.k3s.jalder.tech" # external DNS hostnames pointing at the LoadBalancer
  gateways:
  - mongodb/mdb-gateway # can omit the namespace if gateway is in same namespace as virtual service.
  tls:
  - match:
    - port: 27017
      sniHosts:
      - mdb0.k3s.jalder.tech
    route:
    - destination:
        host: my-scram-enabled-replica-set-0
  - match:
    - port: 27017
      sniHosts:
      - mdb1.k3s.jalder.tech
    route:
    - destination:
        host: my-scram-enabled-replica-set-1
  - match:
    - port: 27017
      sniHosts:
      - mdb2.k3s.jalder.tech
    route:
    - destination:
        host: my-scram-enabled-replica-set-2

Istio Gateway

This is a minimal Istio Gateway bound to the ingressgateway. Your plans may deviate from here, this example is meant to simplify the project, merge the appropriate servers entry into your existing Istio gateway if appropriate.

The important portions are the tls.mode set to PASSTHROUGH and the port.protocol set to TLS. MongoDB split-horizon functionality requires that the original TLS Client Hello message (where the desired SNI is requested), reaches the mongod process directly, no early termination of TLS can be accommodated (aside from edge cases like relaying the original SNI).

---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: mdb-gateway
  namespace: mongodb
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 27017
      name: mongo
      protocol: TLS
    tls:
      mode: PASSTHROUGH
    hosts:
    - "*"

Connect and Validate Horizon

If everything went as planned, you should now be able to connect to the deployment at the external horizon addresses. MongoDB split-horizon configuration means that a client driver’s request for SDAM (server discovery and monitoring) queries returns the horizon address rather than the internal addresses. Connecting to the deployment and calling db.hello() (formerly db.isMaster()) from mongo shell will return internal hosts or horizon hosts when configured correctly. Note the external hostnames are returned as expected.

Connect:

mongo mongodb://mdb0.k3s.jalder.tech/?replicaSet=my-scram-enabled-replica-set --tls --tlsCAFile ca.crt

Confirm Horizons

MongoDB Enterprise my-scram-enabled-replica-set:PRIMARY> db.hello()
{
        "topologyVersion" : {
                "processId" : ObjectId("62571946dc7cc56286694ab1"),
                "counter" : NumberLong(6)
        },
        "hosts" : [
                "mdb0.k3s.jalder.tech:27017",
                "mdb1.k3s.jalder.tech:27017",
                "mdb2.k3s.jalder.tech:27017"
        ],
        "setName" : "my-scram-enabled-replica-set",
        "setVersion" : 35,
        "isWritablePrimary" : true,
        "secondary" : false,
        "primary" : "mdb0.k3s.jalder.tech:27017",
        "me" : "mdb0.k3s.jalder.tech:27017",
        "electionId" : ObjectId("7fffffff000000000000000b"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1649893435, 1),
                        "t" : NumberLong(11)
                },
                "lastWriteDate" : ISODate("2022-04-13T23:43:55Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1649893435, 1),
                        "t" : NumberLong(11)
                },
                "majorityWriteDate" : ISODate("2022-04-13T23:43:55Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2022-04-13T23:43:56.222Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "connectionId" : 5865,
        "minWireVersion" : 0,
        "maxWireVersion" : 9,
        "readOnly" : false,
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1649893435, 1),
                "signature" : {
                        "hash" : BinData(0,"B5f/lRzd/KGVnSze5Y6stVRCd/E="),
                        "keyId" : NumberLong("7085396157957931012")
                }
        },
        "operationTime" : Timestamp(1649893435, 1)
}

Jack Alder

MongoDB Istio Split-Horizon Replica Set