4 Deployment Scenarios #

The most reasonable way to achieve fault tolerant exposure of the API servers is a load balancer. The load balancer will point to all the control plane nodes. The clients and Kubernetes components will talk to the load balancer: which will perform health checks against all the control planes and maintain an active pool of backends.

If only one load balancer is deployed this creates a single point of failure. For a complete HA solution, more than one load balancer is required.

Important

If your environment only contains one load balancer it cannot be considered highly available or fault tolerant, since the load balancer becomes the single point of failure.

Please refer to the SUSE CaaS Platform Administration Guide for more information on the rotation of certificates.

The CA certificate is downloaded from the cluster (present in the cluster-info secret inside the kube-public namespace). Since this information is public, there’s no restriction to download the CA certificate.

This certificate is saved under /etc/kubernetes/pki/ca.crt.

Certificates are stored under /var/lib/kubelet/pki on the worker nodes.

Please refer to the SUSE CaaS Platform Administration Guide for more information on the rotation of certificates.

Creating a cluster definition is the first step to initialize your cluster. You can execute this task as follows:

~/clusters$ skuba cluster init --control-plane 10.86.3.149 <CLUSTER_NAME>
[init] configuration files written to /home/my-user/clusters/<CLUSTER_NAME>

This operation happens strictly offline and will generate a folder structure like the following:

~/clusters > tree <CLUSTER_NAME>/
<CLUSTER_NAME>/
├── addons
│   ├── cilium
│   │   ├── base
│   │   │   └── cilium.yaml
│   │   └── patches
│   ├── cri
│   │   └── default_flags
│   ├── dex
│   │   ├── base
│   │   │   └── dex.yaml
│   │   └── patches
│   ├── gangway
│   │   ├── base
│   │   │   └── gangway.yaml
│   │   └── patches
│   ├── kured
│   │   ├── base
│   │   │   └── kured.yaml
│   │   └── patches
│   └── psp
│       ├── base
│       │   └── psp.yaml
│       └── patches
├── kubeadm-init.conf
└── kubeadm-join.conf.d
    ├── master.conf.template
    └── worker.conf.template

18 directories, 9 files

At this point, you can inspect all generated files, and if desired you can experiment by providing your custom settings using with declarative management of Kubernetes objects using Kustomize.

To provide custom settings of the form of strategic merge patch or a JSON 6902 patch go to addons patches directory for example addons/dex/pathes and create custom settings file for example addons/dex/pathes/custom.yaml

Read https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#patchstrategicmerge and https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#patchjson6902 to get more information.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: oidc-dex
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 5

From within your cluster definition folder, you can bootstrap your first master machine:

~/clusters/<CLUSTER_NAME>$ skuba node bootstrap --user sles --sudo --target <IP_ADDRESS/FQDN> <NODENAME>

This operation will read the kubeadm-init.conf file inside your cluster definition, will forcefully set certain settings to the ones required by SUSE CaaS Platform and will bootstrap the node remotely.

Prior to bootstrap it’s possible for you to tweak the configuration that will be used to create the cluster. You can:

After this operation has completed several modifications will have happened on your cluster definition folder:

Adding new master nodes to the cluster can be achieved by executing the following skuba command:

~/clusters/<CLUSTER_NAME>$ skuba node join --role master --user sles --sudo --target <IP_ADDRESS/FQDN> <NODENAME>

This operation will try to read the kubeadm-join.conf.d/<IP_ADDRESS/FQDN>.conf file if it exists. This allows you to set specific settings for this new node prior to joining it (a similar procedure to how kubeadm-init.conf file behaves when bootstrapping). If this file does not exist, skuba will read the kubeadm-join.conf.d/master.conf.template instead and will create this file automatically when skuba node join is called.

This operation will increase the etcd member count by one, so it’s recommended to always keep an odd number of master nodes because as described in previous sections an even number of nodes does not improve fault tolerance.

Adding new worker nodes to the cluster can be achieved by executing the following skuba command:

~/clusters/<CLUSTER_NAME>$ skuba node join --role worker --user sles --sudo --target <IP_ADDRESS/FQDN> <NODENAME>

This operation will try to read the kubeadm-join.conf.d/<IP_ADDRESS/FQDN>.conf file if it exists. This allows you to set specific settings for this new node prior to joining it (a similar procedure to how kubeadm-init.conf file behaves when bootstrapping). If this file does not exist, skuba will read the kubeadm-join.conf.d/worker.conf.template instead and will create this file automatically when skuba node join is called.

Removing nodes from the cluster requires you to execute skuba from a folder containing an admin.conf file, because this operation is performed exclusively using Kubernetes, and no access to the node being removed or other nodes for that matter is required.

For removing a node, the following command has to be executed:

~/clusters/<CLUSTER_NAME>$ skuba node remove <NODENAME>

If the node to be removed is a master, specific actions will be automatically executed, like removing the etcd member from the cluster. Note that this node cannot be added back to the cluster or any other skuba-initiated kubernetes cluster without reinstalling first.