Multi-Cluster Ops Manager Without a Service Mesh

The Ops Manager is responsible for facilitating workloads such as backing up data, monitoring database performance and more. To make your multi-cluster Ops Manager and the Application Database deployment resilient to entire data center or zone failures, deploy the Ops Manager Application and the Application Database on multiple Kubernetes clusters.

Prerequisites

Before you begin the following procedure, perform the following actions:

Install kubectl.
Complete the GKE Clusters procedure or the equivalent.
Complete the TLS Certificates procedure or the equivalent.
Complete the ExternalDNS procedure or the equivalent.
Complete the Deploy the MongoDB Operator procedure.
Set the required environment variables as follows:

# This script builds on top of the environment configured in the setup guides.# It depends (uses) the following env variables defined there to work correctly.# If you don't use the setup guide to bootstrap the environment, then define them here.#  ${K8S_CLUSTER_0_CONTEXT_NAME}#  ${K8S_CLUSTER_1_CONTEXT_NAME}#  ${K8S_CLUSTER_2_CONTEXT_NAME}#  ${OM_NAMESPACE}#  ${CUSTOM_DOMAIN}#  ${DNS_ZONE}
# Defaults for the RustFS setup module and the MongoDBOpsManager backup stores.# If you use your own S3 storage - override any of these.export S3_OPLOG_BUCKET_NAME="${S3_OPLOG_BUCKET_NAME:-s3-oplog-store}"export S3_SNAPSHOT_BUCKET_NAME="${S3_SNAPSHOT_BUCKET_NAME:-s3-snapshot-store}"export S3_ENDPOINT="${S3_ENDPOINT:-rustfs.rustfs.svc.cluster.local}"export S3_ACCESS_KEY="${S3_ACCESS_KEY:-rustfsadmin}"export S3_SECRET_KEY="${S3_SECRET_KEY:-rustfsadmin123}"
export OPS_MANAGER_VERSION="8.0.5"export APPDB_VERSION="8.0.5-ent"
export OPS_MANAGER_EXTERNAL_DOMAIN="opsmanager.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_0_EXTERNAL_DOMAIN="${K8S_CLUSTER_0}.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_1_EXTERNAL_DOMAIN="${K8S_CLUSTER_1}.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_2_EXTERNAL_DOMAIN="${K8S_CLUSTER_2}.${CUSTOM_DOMAIN}"
# Run-scoped names for project-global GCP load balancer resources (KUBE-268).# Multiple CI runs share the same GCP project; with fixed global names, one run's# pre-clean/teardown deletes a concurrent run's load balancer mid-flight.# Derive the run-specific suffix from ${DNS_ZONE} (set in the ExternalDNS setup# guide). For docs users there is no run suffix and the names stay unchanged.lb_suffix="${DNS_ZONE#"mongodb"}"export OM_FIREWALL_RULE_NAME="fw-ops-manager-hc${lb_suffix}"export OM_HEALTHCHECK_NAME="om-healthcheck${lb_suffix}"export OM_BACKEND_SERVICE_NAME="om-backend-service${lb_suffix}"export OM_URL_MAP_NAME="om-url-map${lb_suffix}"export OM_LB_PROXY_NAME="om-lb-proxy${lb_suffix}"export OM_CERTIFICATE_NAME="om-certificate${lb_suffix}"export OM_FORWARDING_RULE_NAME="om-forwarding-rule${lb_suffix}"
# Retry wrapper for gcloud commands to handle transient GCP API errors# (ConnectionError, RemoteDisconnected, etc.). Functions are inherited by# subshells, so this is available inside snippet functions run by sample_test_runner.gcloud_retry() {  for attempt in 1 2 3; do    if gcloud "$@"; then return 0; fi    if (( attempt < 3 )); then      echo "gcloud failed (attempt ${attempt}/3), retrying in $((attempt * 5))s..." >&2      sleep $((attempt * 5))    fi  done  return 1}

Source Code

You can find all included source code in the MongoDB Kubernetes Operator repository.

Procedure

Generate TLS certificates.

kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" apply -f - <<EOFapiVersion: cert-manager.io/v1kind: Certificatemetadata:  name: om-certspec:  dnsNames:  - ${OPS_MANAGER_EXTERNAL_DOMAIN}  duration: 240h0m0s  issuerRef:    name: my-ca-issuer    kind: ClusterIssuer  renewBefore: 120h0m0s  secretName: cert-prefix-om-cert  usages:  - server auth  - client auth---apiVersion: cert-manager.io/v1kind: Certificatemetadata:  name: om-db-certspec:  dnsNames:  - "*.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}"  - "*.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}"  - "*.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}"  duration: 240h0m0s  issuerRef:    name: my-ca-issuer    kind: ClusterIssuer  renewBefore: 120h0m0s  secretName: cert-prefix-om-db-cert  usages:  - server auth  - client authEOF

Add TLS certificate to GCP.

mkdir -p certs
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get secret cert-prefix-om-cert -o jsonpath="{.data['tls\.crt']}" | base64 --decode > certs/tls.crtkubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get secret cert-prefix-om-cert -o jsonpath="{.data['tls\.key']}" | base64 --decode > certs/tls.key
gcloud_retry compute ssl-certificates create "${OM_CERTIFICATE_NAME}" --certificate=certs/tls.crt --private-key=certs/tls.key

Create the required Kubernetes components for a load balancer.

This load balancer distributes traffic between all the replicas of Ops Manager across all 3 clusters.

gcloud_retry compute firewall-rules create "${OM_FIREWALL_RULE_NAME}" \   --action=allow \   --direction=ingress \   --target-tags=mongodb \   --source-ranges=130.211.0.0/22,35.191.0.0/16 \   --rules=tcp:8443
gcloud_retry compute health-checks create https "${OM_HEALTHCHECK_NAME}" \    --use-serving-port \    --request-path=/monitor/health
gcloud_retry compute backend-services create "${OM_BACKEND_SERVICE_NAME}" \    --protocol HTTPS \    --health-checks "${OM_HEALTHCHECK_NAME}" \    --global
gcloud_retry compute url-maps create "${OM_URL_MAP_NAME}" \    --default-service "${OM_BACKEND_SERVICE_NAME}"
gcloud_retry compute target-https-proxies create "${OM_LB_PROXY_NAME}" \    --url-map "${OM_URL_MAP_NAME}" \    --ssl-certificates="${OM_CERTIFICATE_NAME}"
gcloud_retry compute forwarding-rules create "${OM_FORWARDING_RULE_NAME}" \    --global \    --target-https-proxy="${OM_LB_PROXY_NAME}" \    --ports=443

NAME                                  NETWORK  DIRECTION  PRIORITY  ALLOW     DENY  DISABLEDfw-ops-manager-hc-4f4a5aab3498f278b0  default  INGRESS    1000      tcp:8443        FalseNAME                               PROTOCOLom-healthcheck-4f4a5aab3498f278b0  HTTPSNAME                                   BACKENDS  PROTOCOLom-backend-service-4f4a5aab3498f278b0            HTTPSNAME                           DEFAULT_SERVICEom-url-map-4f4a5aab3498f278b0  backendServices/om-backend-service-4f4a5aab3498f278b0NAME                            SSL_CERTIFICATES                   URL_MAP                        REGION  CERTIFICATE_MAPom-lb-proxy-4f4a5aab3498f278b0  om-certificate-4f4a5aab3498f278b0  om-url-map-4f4a5aab3498f278b0

Add an "A" record to your DNS zone with your external domain.

ip_address=$(gcloud_retry compute forwarding-rules describe "${OM_FORWARDING_RULE_NAME}" --global --format="get(IPAddress)")
gcloud_retry dns record-sets create "${OPS_MANAGER_EXTERNAL_DOMAIN}" --zone="${DNS_ZONE}" --type="A" --ttl="300" --rrdatas="${ip_address}"

Create credentials for the Ops Manager admin user.

kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${OM_NAMESPACE}" create secret generic om-admin-user-credentials \  --from-literal=Username="admin" \  --from-literal=Password="Passw0rd@" \  --from-literal=FirstName="Jane" \  --from-literal=LastName="Doe"

Deploy Ops Manager.

kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata:  name: omspec:  topology: MultiCluster  version: "${OPS_MANAGER_VERSION}"  adminCredentials: om-admin-user-credentials  externalConnectivity:    type: ClusterIP    annotations:      cloud.google.com/neg: '{"exposed_ports": {"8443":{}}}'  opsManagerURL: "https://${OPS_MANAGER_EXTERNAL_DOMAIN}"  security:    certsSecretPrefix: cert-prefix    tls:      ca: ca-issuer  clusterSpecList:    - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"      members: 1    - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"      members: 2  applicationDatabase:    version: "${APPDB_VERSION}"    topology: MultiCluster    security:      certsSecretPrefix: cert-prefix      tls:        ca: ca-issuer    clusterSpecList:      - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"        members: 2        externalAccess:          externalDomain: "${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}"          externalService:            annotations:               external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}"      - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"        members: 2        externalAccess:          externalDomain: "${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}"          externalService:            annotations:               external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}"      - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}"        members: 1        externalAccess:          externalDomain: "${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}"          externalService:            annotations:               external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}"  backup:    enabled: falseEOF

Wait for the Kubernetes Operator to enter a pending state.

Wait for both the Application Database and Ops Manager deployments to complete.

echo "Waiting for Application Database to reach Pending phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s
echo "Waiting for Ops Manager to reach Pending phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Pending opsmanager/om --timeout=600s

Waiting for Application Database to reach Pending phase...mongodbopsmanager.mongodb.com/om condition metWaiting for Ops Manager to reach Pending phase...mongodbopsmanager.mongodb.com/om condition met

Set up load balancer services.

svcneg0=$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg -o=jsonpath='{.items[0].metadata.name}')
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" \  wait svcneg "${svcneg0}" --for=condition=Initialized --timeout=300s \  || { kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg "${svcneg0}" -o yaml; exit 1; }
gcloud_retry compute backend-services add-backend "${OM_BACKEND_SERVICE_NAME}" \    --global \    --network-endpoint-group="${svcneg0}" \    --network-endpoint-group-zone="${K8S_CLUSTER_0_ZONE}" \    --balancing-mode RATE --max-rate-per-endpoint 5

svcneg1=$(kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg -o=jsonpath='{.items[0].metadata.name}')
kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" \  wait svcneg "${svcneg1}" --for=condition=Initialized --timeout=300s \  || { kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg "${svcneg1}" -o yaml; exit 1; }
gcloud_retry compute backend-services add-backend "${OM_BACKEND_SERVICE_NAME}" \    --global \    --network-endpoint-group="${svcneg1}" \    --network-endpoint-group-zone="${K8S_CLUSTER_1_ZONE}" \    --balancing-mode RATE --max-rate-per-endpoint 5

Wait for Ops Manager to enter a running state.

echo "Waiting for Application Database to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=1200secho; echo "Waiting for Ops Manager to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=1200secho; echo "MongoDBOpsManager resource"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/omecho; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods

Waiting for Application Database to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
Waiting for Ops Manager to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
MongoDBOpsManager resourceNAME   REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGSom                8.0.5     Running              Running         Disabled         18m   
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-4f4a5aab3498f278b0NAME        READY   STATUS    RESTARTS   AGEom-0-0      1/1     Running   0          16mom-db-0-0   2/2     Running   0          4m57som-db-0-1   2/2     Running   0          5m48s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-4f4a5aab3498f278b0NAME        READY   STATUS    RESTARTS   AGEom-1-0      1/1     Running   0          16mom-1-1      1/1     Running   0          9m42som-db-1-0   2/2     Running   0          3m14som-db-1-1   2/2     Running   0          4m5s

Optional. Deploy S3-compatible storage for testing.

This step provides a script that deploys a simple RustFS instance for testing purposes. You can skip this step if you have AWS S3 or other S3-compatible buckets available.

Note

RustFS is only for testing and isn't suitable for production. For production, use your own S3 storage.

kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${RUSTFS_NAMESPACE}" --dry-run=client -o yaml | \  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${RUSTFS_NAMESPACE}" delete job rustfs-create-buckets --ignore-not-found=true || true
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${RUSTFS_NAMESPACE}" apply -f - <<EOFapiVersion: cert-manager.io/v1kind: Certificatemetadata:  name: rustfs-certspec:  dnsNames:  - rustfs.${RUSTFS_NAMESPACE}.svc.cluster.local  duration: 240h0m0s  issuerRef:    name: my-ca-issuer    kind: ClusterIssuer  renewBefore: 120h0m0s  secretName: rustfs-tls  usages:  - server auth---apiVersion: v1kind: Servicemetadata:  name: rustfs  labels:    app: rustfsspec:  selector:    app: rustfs  ports:    - name: s3-https      port: 443      targetPort: 9000    - name: s3      port: 9000      targetPort: 9000---apiVersion: apps/v1kind: Deploymentmetadata:  name: rustfs  labels:    app: rustfsspec:  replicas: 1  selector:    matchLabels:      app: rustfs  template:    metadata:      labels:        app: rustfs      annotations:        # RustFS needs no mesh sidecar; keep behavior identical with and        # without Istio.        sidecar.istio.io/inject: "false"    spec:      securityContext:        runAsNonRoot: true        runAsUser: 10001        runAsGroup: 10001        fsGroup: 10001        seccompProfile:          type: RuntimeDefault      containers:        - name: rustfs          image: quay.io/rustfs/rustfs:1.0.0          securityContext:            allowPrivilegeEscalation: false            capabilities:              drop: ["ALL"]            runAsNonRoot: true          env:            - name: RUSTFS_ACCESS_KEY              value: "${S3_ACCESS_KEY}"            - name: RUSTFS_SECRET_KEY              value: "${S3_SECRET_KEY}"            - name: RUSTFS_VOLUMES              value: /data            - name: RUSTFS_ADDRESS              value: 0.0.0.0:9000            - name: RUSTFS_CONSOLE_ENABLE              value: "false"            - name: RUSTFS_TLS_PATH              value: /opt/tls          ports:            - name: s3              containerPort: 9000          readinessProbe:            httpGet:              scheme: HTTPS              path: /health              port: 9000            initialDelaySeconds: 5            periodSeconds: 3          volumeMounts:            - name: data              mountPath: /data            - name: tls              mountPath: /opt/tls              readOnly: true      volumes:        - name: data          emptyDir: {}        - name: tls          secret:            secretName: rustfs-tls            items:              - key: tls.crt                path: rustfs_cert.pem              - key: tls.key                path: rustfs_key.pem---apiVersion: batch/v1kind: Jobmetadata:  name: rustfs-create-bucketsspec:  backoffLimit: 1  template:    metadata:      annotations:        # istio-proxy keeps running after the aws-cli container exits, so an        # injected Job never reaches Complete.        sidecar.istio.io/inject: "false"    spec:      restartPolicy: Never      containers:        - name: aws-cli          image: public.ecr.aws/aws-cli/aws-cli:latest          env:            - name: AWS_ACCESS_KEY_ID              value: "${S3_ACCESS_KEY}"            - name: AWS_SECRET_ACCESS_KEY              value: "${S3_SECRET_KEY}"            - name: AWS_DEFAULT_REGION              value: us-east-1            - name: S3_ENDPOINT              value: "https://${S3_ENDPOINT}"            - name: S3_OPLOG_BUCKET_NAME              value: "${S3_OPLOG_BUCKET_NAME}"            - name: S3_SNAPSHOT_BUCKET_NAME              value: "${S3_SNAPSHOT_BUCKET_NAME}"          command: ["sh", "-ec"]          args:            - |              aws_opts="--endpoint-url \$S3_ENDPOINT --no-verify-ssl --cli-connect-timeout 5 --cli-read-timeout 10"              attempt=0              until aws \$aws_opts s3api list-buckets; do                attempt=\$((attempt + 1))                if [ "\$attempt" -ge 24 ]; then                  echo "RustFS endpoint \$S3_ENDPOINT not reachable after \$attempt attempts" >&2                  exit 1                fi                sleep 5              done              aws \$aws_opts s3api create-bucket --bucket "\$S3_OPLOG_BUCKET_NAME"              aws \$aws_opts s3api create-bucket --bucket "\$S3_SNAPSHOT_BUCKET_NAME"EOF
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${RUSTFS_NAMESPACE}" wait --for=condition=available deployment/rustfs --timeout=300skubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${RUSTFS_NAMESPACE}" wait --for=condition=complete job/rustfs-create-buckets --timeout=240s

Configure Kubernetes Secrets for Ops Manager backups.

kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-access-secret \  --from-literal=accessKey="${S3_ACCESS_KEY}" \  --from-literal=secretKey="${S3_SECRET_KEY}"
# RustFS serves a cert-manager certificate; OM must trust the CA that signed it.kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-ca-cert \  --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${RUSTFS_NAMESPACE}" get secret rustfs-tls -o jsonpath="{.data['ca\.crt']}" | base64 --decode)"

Enable S3 backups in Ops Manager.

kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata:  name: omspec:  topology: MultiCluster  version: "${OPS_MANAGER_VERSION}"  adminCredentials: om-admin-user-credentials  externalConnectivity:    type: ClusterIP    annotations:      cloud.google.com/neg: '{"exposed_ports": {"8443":{}}}'  opsManagerURL: "https://${OPS_MANAGER_EXTERNAL_DOMAIN}"  security:    certsSecretPrefix: cert-prefix    tls:      ca: ca-issuer  clusterSpecList:    - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"      members: 1      backup:        members: 0    - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"      members: 2      backup:        members: 0    - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}"      members: 0      backup:        members: 1  applicationDatabase:    version: "${APPDB_VERSION}"    topology: MultiCluster    security:      certsSecretPrefix: cert-prefix      tls:        ca: ca-issuer    clusterSpecList:      - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"        members: 2        externalAccess:          externalDomain: "${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}"          externalService:            annotations:               external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}"      - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"        members: 2        externalAccess:          externalDomain: "${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}"          externalService:            annotations:               external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}"      - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}"        members: 1        externalAccess:          externalDomain: "${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}"          externalService:            annotations:               external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}"  backup:    enabled: true    s3Stores:      - name: my-s3-block-store        s3SecretRef:          name: "s3-access-secret"        pathStyleAccessEnabled: true        s3BucketEndpoint: "${S3_ENDPOINT}"        s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}"        customCertificateSecretRefs:          - name: s3-ca-cert            key: ca.crt    s3OpLogStores:      - name: my-s3-oplog-store        s3SecretRef:          name: "s3-access-secret"        s3BucketEndpoint: "${S3_ENDPOINT}"        s3BucketName: "${S3_OPLOG_BUCKET_NAME}"        pathStyleAccessEnabled: true        customCertificateSecretRefs:          - name: s3-ca-cert            key: ca.crtEOF

Wait for Ops Manager to enter a running state.

echo; echo "Waiting for Backup to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200secho "Waiting for Application Database to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=1200secho; echo "Waiting for Ops Manager to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=1200secho; echo "MongoDBOpsManager resource"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/omecho; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods

Waiting for Backup to reach Running phase...mongodbopsmanager.mongodb.com/om condition metWaiting for Application Database to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
Waiting for Ops Manager to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
MongoDBOpsManager resourceNAME   REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGSom                8.0.5     Running              Running         Running          21m   
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-4f4a5aab3498f278b0NAME        READY   STATUS    RESTARTS   AGEom-0-0      1/1     Running   0          18mom-db-0-0   2/2     Running   0          7m34som-db-0-1   2/2     Running   0          8m25s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-4f4a5aab3498f278b0NAME        READY   STATUS    RESTARTS   AGEom-1-0      1/1     Running   0          18mom-1-1      1/1     Running   0          12mom-db-1-0   2/2     Running   0          5m51som-db-1-1   2/2     Running   0          6m42s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2-4f4a5aab3498f278b0NAME                   READY   STATUS    RESTARTS   AGEom-2-backup-daemon-0   1/1     Running   0          2m27som-db-2-0              2/2     Running   0          4m57s

Create MongoDB Organization and get credentials.

To configure credentials, you must create an Ops Manager organization, generate programmatic API keys in the Ops Manager UI, and create a secret with your Load Balancer IP. See Create Credentials for the Kubernetes Operator to learn more.

Back

TLS Certificates

Multi-Cluster Replica Sets

Multi-Cluster Ops Manager Without a Service Mesh

Prerequisites

Source Code

Procedure

Generate TLS certificates.

Add TLS certificate to GCP.

Create the required Kubernetes components for a load balancer.

Add an "A" record to your DNS zone with your external domain.

Create credentials for the Ops Manager admin user.

Deploy Ops Manager.

Wait for the Kubernetes Operator to enter a pending state.

Set up load balancer services.

Wait for Ops Manager to enter a running state.

Optional. Deploy S3-compatible storage for testing.

Deploy RustFS

Configure Kubernetes Secrets for Ops Manager backups.

Enable S3 backups in Ops Manager.

Wait for Ops Manager to enter a running state.

Create MongoDB Organization and get credentials.