This is a draft document that was built and uploaded automatically. It may document beta software and be incomplete or even incorrect. Use this document at your own risk.
cf-plugin-backup backs up and restores your Cloud
Controller Database (CCDB), using the Cloud Foundry command line interface (cf CLI).
(See Section 26.1, “Using the cf CLI with SUSE Cloud Application Platform”.)
cf-plugin-backup is not a general-purpose backup and
restore plugin. It is designed to save the state of a KubeCF instance before
making changes to it. If the changes cause problems, use
cf-plugin-backup to restore the instance from scratch. Do
not use it to restore to a non-pristine KubeCF instance. Some of the
limitations for applying the backup to a non-pristine KubeCF instance are:
Application configuration is not restored to running applications, as the plugin does not have the ability to determine which applications should be restarted to load the restored configurations.
User information is managed by the User Account and Authentication (uaa) Server,
not the Cloud Controller (CC). As the plugin talks only to the CC it cannot
save full user information, nor restore users. Saving and restoring users
must be performed separately, and user restoration must be performed before
the backup plugin is invoked.
The set of available stacks is part of the KubeCF instance setup, and is not part of the CC configuration. Trying to restore applications using stacks not available on the target KubeCF instance will fail. Setting up the necessary stacks must be performed separately before the backup plugin is invoked.
Buildpacks are not saved. Applications using custom buildpacks not available on the target KubeCF instance will not be restored. Custom buildpacks must be managed separately, and relevant buildpacks must be in place before the affected applications are restored.
Download the plugin from https://github.com/SUSE/cf-plugin-backup/releases.
Then install it with cf, using the name of the plugin
binary that you downloaded:
tux > cf install-plugin cf-plugin-backup-1.0.8.0.g9e8438e.linux-amd64
Attention: Plugins are binaries written by potentially untrusted authors.
Install and use plugins at your own risk.
Do you want to install the plugin
backup-plugin/cf-plugin-backup-1.0.8.0.g9e8438e.linux-amd64? [yN]: y
Installing plugin backup...
OK
Plugin backup 1.0.8 successfully installed.Verify installation by listing installed plugins:
tux > cf plugins
Listing installed plugins...
plugin version command name command help
backup 1.0.8 backup-info Show information about the current snapshot
backup 1.0.8 backup-restore Restore the CloudFoundry state from a
backup created with the snapshot command
backup 1.0.8 backup-snapshot Create a new CloudFoundry backup snapshot
to a local file
Use 'cf repo-plugins' to list plugins in registered repos available to install.The plugin has three commands:
backup-info
backup-snapshot
backup-restore
View the online help for any command, like this example:
tux > cf backup-info --help
NAME:
backup-info - Show information about the current snapshot
USAGE:
cf backup-infoCreate a backup of your SUSE Cloud Application Platform data and applications. The command outputs progress messages until it is completed:
tux > cf backup-snapshot
2018/08/18 12:48:27 Retrieving resource /v2/quota_definitions
2018/08/18 12:48:30 org quota definitions done
2018/08/18 12:48:30 Retrieving resource /v2/space_quota_definitions
2018/08/18 12:48:32 space quota definitions done
2018/08/18 12:48:32 Retrieving resource /v2/organizations
[...]
Your Cloud Application Platform data is saved in the current directory in
cf-backup.json, and application data in the
app-bits/ directory.
View the current backup:
tux > cf backup-info
- Org systemRestore from backup:
tux > cf backup-restore
There are two additional restore options:
--include-security-groups and
--include-quota-definitions.
The following table lists the scope of the
cf-plugin-backup backup. Organization and space users
are backed up at the SUSE Cloud Application Platform level. The user account in
uaa/LDAP, the service instances and their application
bindings, and buildpacks are not backed up. The sections following the
table goes into more detail.
| Scope | Restore |
|---|---|
| Orgs | Yes |
| Org auditors | Yes |
| Org billing-manager | Yes |
| Quota definitions | Optional |
| Spaces | Yes |
| Space developers | Yes |
| Space auditors | Yes |
| Space managers | Yes |
| Apps | Yes |
| App binaries | Yes |
| Routes | Yes |
| Route mappings | Yes |
| Domains | Yes |
| Private domains | Yes |
| Stacks | not available |
| Feature flags | Yes |
| Security groups | Optional |
| Custom buildpacks | No |
cf backup-info reads the
cf-backup.json snapshot file found in the current
working directory, and reports summary statistics on the content.
cf backup-snapshot extracts and saves the following
information from the CC into a cf-backup.json snapshot
file. Note that it does not save user information, but only the references
needed for the roles. The full user information is handled by the
uaa server, and the plugin talks only to the CC. The
following list provides a summary of what each plugin command does.
Org Quota Definitions
Space Quota Definitions
Shared Domains
Security Groups
Feature Flags
Application droplets (zip files holding the staged app)
Orgs
Spaces
Applications
Users' references (role in the space)
cf backup-restore reads the
cf-backup.json snapshot file found in the current
working directory, and then talks to the targeted KubeCF instance to upload
the following information, in the specified order:
Shared domains
Feature flags
Quota Definitions (iff --include-quota-definitions)
Orgs
Space Quotas (iff --include-quota-definitions)
UserRoles
(private) Domains
Spaces
UserRoles
Applications (+ droplet)
Bound Routes
Security Groups (iff --include-security-groups)
The following list provides more details of each action.
Attempts to create domains from the backup. Existing domains are retained, and not overwritten.
Attempts to update flags from the backup.
Existing quotas are overwritten from the backup (deleted, re-created).
Attempts to create orgs from the backup. Attempts to update existing orgs from the backup.
Existing quotas are overwritten from the backup (deleted, re-created).
Expect the referenced user to exist. Will fail when the user is already associated with the space, in the given role.
Attempts to create domains from the backup. Existing domains are retained, and not overwritten.
Attempts to create spaces from the backup. Attempts to update existing spaces from the backup.
Expect the referenced user to exist. Will fail when the user is already associated with the space, in the given role.
Attempts to create apps from the backup. Attempts to update existing apps from the backup (memory, instances, buildpack, state, ...)
Existing groups are overwritten from the backup
An existing SUSE Cloud Application Platform deployment's data can be migrated to a new
SUSE Cloud Application Platform deployment through a backup and restore of its raw data. The
process involves performing a backup and restore of the
kubecf components respectively. This procedure is agnostic of
the underlying Kubernetes infrastructure and can be included as part of your
disaster recovery solution.
In order to complete a raw data backup and restore, the following are required:
Access to a running deployment of kubecf to create backups
with
Access to a new deployment of kubecf (deployed with a
kubecf-config-values.yaml configured according to
Step 1 of
Section 21.2.4, “Performing a Raw Data Restore”) to perform the
restore to
The following lists the data that is included as part of the backup (and restore) procedure:
The Cloud Controller Database (CCDB). In addition to what is encompassed by the CCDB listed in Section 21.1.3, “Scope of Backup”, this will include service binding data as well.
The Cloud Controller blobstore, which includes the types of binary large object (blob) files listed below. (See https://docs.cloudfoundry.org/concepts/architecture/cloud-controller.html#blob-store to learn more about each blob type.)
App Packages
Buildpacks
Resource Cache
Buildpack Cache
Droplets
User data
This process is intended for backing up and restoring to a target deployment with the same version as the source deployment. For example, data from a backup of SUSE Cloud Application Platform 2.1.1 should be restored to a SUSE Cloud Application Platform 2.1.1 deployment.
Perform the following steps to create a backup of your source SUSE Cloud Application Platform deployment.
Export the blobstore into a file.
tux > kubectl exec --namespace kubecf singleton-blobstore-0 --
tar cfz - --exclude=/var/vcap/store/shared/tmp /var/vcap/store/shared >
blob.tgzThe current UAA database configuration does not allow exporting of a mysqldump, so need to be more permissive.
tux > cat <<EOF | kubectl exec --stdin database-0 --namespace kubecf
-- mysql
SET GLOBAL pxc_strict_mode=PERMISSIVE;
SET GLOBAL
sql_mode='STRICT_ALL_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
set GLOBAL innodb_strict_mode='OFF';
EOFExport the UAA database into a file.
tux > kubectl exec --stdin database-0 --namespace kubecf --
mysqldump uaa > uaadb-src.sqlExport the Cloud Controller Database (CCDB) into a file.
tux > kubectl exec --stdin database-0 --namespace kubecf --
mysqldump cloud_controller > ccdb-src.sql
Save the CCDB encryption key(s). Adjust the A flag as
needed to include all keys.
tux > kubectl exec --stdin --tty --namespace kubecf api-0 -- bash
-c "cat /var/vcap/jobs/cloud_controller_ng/config/cloud_controller_ng.yml | grep
-A 10 db_encryption" > enc_key
Working with multiple Kubernetes clusters simultaneously can be confusing.
Ensure you are communicating with the desired cluster
by
setting $KUBECONFIG correctly.
Perform the following steps to restore your backed up data to the target SUSE Cloud Application Platform deployment.
Deploy the target SUSE Cloud Application Platform cluster following the steps for your platform.
For SUSE® CaaS Platform, see Chapter 4, Deploying SUSE Cloud Application Platform on SUSE CaaS Platform.
For Microsoft Azure Kubernetes Service, see Chapter 5, Deploying SUSE Cloud Application Platform on Microsoft Azure Kubernetes Service (AKS).
For Amazon Elastic Kubernetes Service, see Chapter 6, Deploying SUSE Cloud Application Platform on Amazon Elastic Kubernetes Service (EKS).
For Google Kubernetes Engine, see Chapter 7, Deploying SUSE Cloud Application Platform on Google Kubernetes Engine (GKE).
The current UAA database configuration does not allow importing of a mysqldump, so needs to be made more permissive.
tux > cat <<EOF | kubectl exec --stdin database-0 --namespace kubecf
-- mysql
SET GLOBAL pxc_strict_mode=PERMISSIVE;
SET GLOBAL
sql_mode='STRICT_ALL_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
set GLOBAL innodb_strict_mode='OFF';
EOFImport the UAA database.
tux > kubectl exec --stdin database-0 --namespace kubecf -- mysql
uaa < uaadb-src.sqlVerify the import is successful. The output should list the users from the deployment the backup was taken from.
tux > echo "select username from uaa.users;" | kubectl exec -i
database-0 --namespace kubecf -- mysqlImport the blobstore and restart the pod for changes to take affect.
tux >kubectl exec --stdin singleton-blobstore-0 --namespace kubecf -- tar xfz - -C / < blob.tgztux >kubectl delete pod --namespace kubecf singleton-blobstore-0
Drop the current CCDB and create a new instance.
tux > echo "drop database cloud_controller; create database
cloud_controller;" | \
kubectl exec -i database-0 --namespace kubecf -- mysqlImport the CCDB.
tux > kubectl exec --stdin database-0 --namespace kubecf -- mysql
cloud_controller < ccdb-src.sqlUpdate the encryption key.
Create a YAML configuration file containing the encryption key
information. The file structure should look similar to the following
example, called enc_key_values.yaml. Replace
the example values using the values from the enc_key
file generated earlier. Depending on the state of the cluster the
encryption keys were retrieved from, the key labels may differ and not
be encryption_key_0.
ccdb:
encryption:
rotation:
key_labels:
- encryption_key_0
current_key_label: encryption_key_0
credentials:
cc_db_encryption_key:
elqdi7TARO6NYELa9cUr6WwMYIvqaG4U0nMyfL1loDYi02C1Rrneov6fxxfd64je
ccdb_key_label_encryption_key_0:
tPhZZbMNYVWKs0II8e8pMxsJMokeReUrJAnQNdLaXEheTZVv5OpMe7vdyThhrkEP
In the above, the key
credentials.ccdb_key_label_encryption_key_0 is
based on the generic form
credentials.ccdb_key_label_XYZ. The
XYZ should be replaced with the value of the
current_key_label.
For example, if the current_key_label is
new_key, then
credentials.ccdb_key_label_new_key should be used.
Perform a helm upgrade for the changes to take
affect.
tux > helm upgrade kubecf suse/kubecf
\
--namespace kubecf \
--values kubecf-config-values.yaml \
--values enc_key_values.yaml \
--version 2.7.13
When all pods are fully running, verify the restore is successful. Example
commands to run include cf apps,
cf marketplace, or cf services.