1 Air gapped deployment #

For an air gapped scenario we assume a network separation into three logical parts.

The following instructions will use these three terms to refer to parts of the infrastructure. For example: "internal mirror" refers to the mirroring server on the air gapped network. The terms air gapped and internal will be used interchangeably.

In order to disconnect SUSE CaaS Platform from the external network, we provide ways for the components to retrieve data from alternative sources inside the internal (air gapped) network.

You will need to create a mirror server inside the internal network; which acts as a replacement for the default sources.

The three main sources that must be replaced are:

You will provide replacements for these resources on a dedicated server inside your internal (air gapped) network.

The internal mirror must be updated with data retrieved from the original upstream sources; in a trusted and secure fashion. To achieve this, you will need an additional mirroring server outside of the air gapped network which acts as a first stage mirror and allows retrieving data from the internet.

Updating of mirrors happens in three stages.

Once the replacement sources are in place, the key components are reconfigured to use the mirrors as their main sources.

Mirroring of the RPM repositories is handled by the Repository Mirroring Tool for SUSE Linux Enterprise Server 15. The tool provides functionality that mirrors the upstream SUSE package repositories on the local network. This is intended to minimize reliance on SUSE infrastructure for updating large volumes of machines. The air gapped deployment uses the same technology to provide the packages locally for the air gapped environment.

SUSE Linux Enterprise Server bundles software packages into so called modules. You must enable the SUSE CaaS Platform, SUSE Linux Enterprise Server and Containers Module modules in addition to the modules enabled by default. All enabled modules need to be mirrored inside the air gapped network in order to provide the necessary software for other parts of this scenario.

Repository Mirroring Tool (RMT) will provide a repository server that holds the packages and related metadata for SUSE Linux Enterprise Server; to install them like from the upstream repository. Data is synchronized once a day to the external mirror automatically or can be forced via the CLI.

You can copy this data to your trusted storage at any point and update the internal mirror.

SUSE CaaS Platform uses Helm as one method to install additional software on the cluster. The logic behind this relies on Charts, which are configuration files that tell Kubernetes how to deploy software and its dependencies. The actual software installed using this method is delivered as container images. The download location of the container image is stored inside the Helm chart.

Container images are provided by SUSE and others on so called registries. The SUSE container registry is used to update the SUSE CaaS Platform components.

To mirror container images inside the air gapped environment, you will run two container image registry services that are used to pull and in turn serve these images. The registry service is shipped as a container image itself.

Helm charts are provided independently from container images and can be developed by any number of sources. Please make sure that you trust the origin of container images referenced in the helm charts.

We provide helm-mirror to allow downloading all charts present in a chart repository in bulk and moreover to extract all container image URLs from the charts. skopeo is used to download all the images referred to in the Helm charts from their respective registry.

Helm charts will be provided to the internal network by a webserver and refer to the container images hosted on the internal registry mirror.

Once mirroring is configured, you will not have to modify Dockerfile(s) or Kubernetes manifests to use the mirrors. The requests are passed through the container engine which forwards them to the configured mirrors. For example: All images with a prefix registry.suse.com/ will be automatically pulled from the configured (internal) mirror instead.

For further information on registry mirror configuration, refer to https://documentation.suse.com/suse-caasp/4.5/html/caasp-admin/_miscellaneous.html#_configuring_container_registries_for_cri_o.

Note: Shared Mirror Server

If you have multiple SUSE CaaS Platform clusters or a very large number of nodes accessing the mirrors, you should increase the sizing of CPU/RAM.

Storage sizing depends on your intended update frequency and data retention model. If you want to keep snapshots or images of repository states at various points, you must increase storage size accordingly.

You will need to provide and maintain at least two machines in addition to your SUSE CaaS Platform cluster. These mirror servers will reside on the external part of your network and the internal (air gapped) network respectively.

For more information on the requirements of a SUSE Linux Enterprise 15 server, refer to: Installation Preparation.

Important: Adjust Number Of Mirror Servers

This scenario description does not contain any fallback contingencies for the mirror servers. Add additional mirror servers (behind a load balancer) if you require additional reliability/availability.

Transferring data from the external network mirror to the internal mirror can be performed in many ways. The most common way is portable storage (USB keys or external hard drives).

Sizing of the storage is dependent on the number of data sources that need to be stored. Container images can easily measure several Gigabytes per item; although they are generally smaller for Kubernetes related applications. The overall size of any given RPM repository is at least tens of Gigabytes. For example: At the time of writing, the package repository for SUSE Linux Enterprise Server contains approximately 36 GB of data.

The storage must be formatted to a file system type supporting files larger than 4 GB.

We recommend external storage with at least 128 GB.

Note: Mount Point For Storage In Examples

In the following procedures, we will assume the storage (when connected) is mounted on /mnt/storage . Please make sure to adjust the mountpoint in the respective command to where the device is actually available.

Note: Handling Of Trusted Storage

Data integrity checks, duplication, backup, and secure handling procedures of trusted storage are beyond the scope of this document.

Note: Deploy The Mirror Before SUSE CaaS PlatformCluster Deployment

The mirror on the air gapped network must be running and populated before

Now you need to perform the update procedure to do an initial sync of data between the upstream sources and the external mirror and the external and internal mirrors. Refer to: Section 1.5, “Updating RPM Repository Mirror”.

Follow these instructions to configure all SUSE CaaS Platform nodes to use the package repository mirror server in the air gapped network.

The container image registry is provided as a container image itself. You must download the registry container from SUSE and run it on the respective server.

Note: Which images to Mirror

CaaS Platform requires a base set of images to be mirrored, as they contain the core services needed to run the cluster.

This list of base images can be found under the following link: https://documentation.suse.com/external-tree/en-us/suse-caasp/4/skuba-cluster-images.txt

Alternatively, the list can be obtained from skuba - just run this command on the machine you have skuba installed on:

skuba cluster images

This will print out a list of the images skuba is expecting to use on the cluster to be bootstrapped.

Mirror those and setup the crio-registries to point to the location they are mirrored at.

Important

These images need to be available in the external and internal mirrors at the time you try to deploy SUSE CaaS Platform. Image tags will vary depending on the version of kubernetes you install.

CATEGORY	IMAGE	URL
KUBEADM	hyperkube	registry.suse.com/caasp/v4.5/hyperkube
KUBEADM	pause	registry.suse.com/caasp/v4.5/pause
KUBEADM	coredns	registry.suse.com/caasp/v4.5/coredns
KUBEADM	etcd	registry.suse.com/caasp/v4.5/etcd
ADDONS	cilium	registry.suse.com/caasp/v4.5/cilium-init
ADDONS	cilium	registry.suse.com/caasp/v4.5/cilium-operator
ADDONS	cilium	registry.suse.com/caasp/v4.5/cilium
ADDONS	dex	registry.suse.com/caasp/v4.5/caasp-dex
ADDONS	gangway	registry.suse.com/caasp/v4.5/gangway
ADDONS	kured	registry.suse.com/caasp/v4.5/kured
MISC	skuba-tooling	registry.suse.com/caasp/v4.5/skuba-tooling

Note: Internal Registry Mirror Is Read Only

For security reasons, the internal registry mirror is configured in read-only mode. Therefore, pushing container images to this mirror will not be possible. It can only serve images that were previously pulled and cached by the external mirror and then uploaded to the internal mirror.

You can modify and store your own container images on the external registry and transfer them with the other container images using the same process. If you need to be able to modify and store container images on the internal network, we recommend creating a new registry that will hold these images. The steps needed to run your own full container image registry are not part of this document.

For more information you can refer to: SLES15 - Docker Open Source Engine Guide: What is Docker Registry?.

We will re-use the nginx webserver that is running as part of Repository Mirroring Tool (RMT) to act as a reverse proxy for the container image registry service and to serve the chart repository files. This step is not necessary for the external host.

Now, you should have the registries set up and listening on port 5000 on their respective servers.

Important

The example provided with the installation is in the old v1 format of the CRI-O registries syntax. You must replace/remove all content from the example file and build a new file based on the v2 syntax.

The example below is written in the correct v2 syntax. registries.conf is written using TOML.

Configure /etc/containers/registries.conf to setup the mirroring from registry.suse.com to the internal mirror. This needs to be done on all cluster nodes. Make sure to adjust all the correct domain name for your local registry:

[[registry]]
prefix = "registry.suse.com"
location = "registry01.mydomain.local:5000/registry.suse.com"
[[registry]]
prefix = "docker.io"
location = "registry01.mydomain.local:5000/docker.io"
[[registry]]
prefix = "docker.io/library"
location = "registry01.mydomain.local:5000/docker.io"
[[registry]]
prefix = "quay.io"
location = "registry01.mydomain.local:5000/quay.io"
[[registry]]
prefix = "k8s.gcr.io"
location = "registry01.mydomain.local:5000/k8s.gcr.io"
[[registry]]
prefix = "gcr.io"
location = "registry01.mydomain.local:5000/gcr.io"

For detailed information about the configuration format see html/caasp-admin/_miscellaneous.html#_configuring_container_registries_for_cri_o.

Update the Helm chart repository by following the shared update procedure Section 1.8, “Updating Registry Mirror For Helm Charts”.

Add the webserver as a repo to helm.

This step needs to be performed on a machine where Helm is installed and configured in the SUSE CaaS Platform cluster. For steps to install Helm, refer to https://documentation.suse.com/suse-caasp/4.5/html/caasp-admin/_software_management.html#helm-install.

To initialize Helm, use the command:

helm init --client-only --skip-refresh

<SUSE_MIRROR> will be the user-defined name for this repository listed by Helm. The name of the repository must adhere to Helm Chart naming conventions.

helm repo add <SUSE_MIRROR> https://charts.<MYMIRROR.LOCAL>

From YaST register the node against the Repository Mirroring Tool (RMT) server. This will ensure the node zypper repositories are pointed against Repository Mirroring Tool (RMT). Moreover, all the available updates are going to be installed and there is no need to manually install updates right after the installation.

Ensure the admin node is registered against Repository Mirroring Tool (RMT), that will ensure the nodes that are provisioned by AutoYaST are registered against Repository Mirroring Tool (RMT) to have all the updates applied.

If you are using a self-signed certificate for the registry you can use the --dest-cert-dir /path/to/the/cert parameter to provide the certificate.

Refer to: Section 1.4.2, “Client Configuration”.

When registry mirror is using virtual repository URL. You may need to manually modify the Helm chart index.yaml and point the correct HTTPS base URL.