Backup and Recovery

Backup and Recovery
Prev	Chapter 3. Operation and Maintenance	Next

Typical tasks of the Monitoring Service operator are to make regular backups, particularly of the data created during operation.

At regular intervals, you should make a backup of all:

Databases.
Configuration files of the individual agents and services.
Monitoring and log dashboards you have created and saved.

SUSE OpenStack Cloud Monitoring does not offer integrated backup and recovery mechanisms. Instead, use the mechanisms and procedures of the individual components.

Databases

You need to create regular backups of the following databases on the host where the Monitoring Service is installed:

Elasticsearch database for historic log data.
InfluxDB database for historic metrics data.
MariaDB database for historic configuration information.

It is recommended that backup and restore operations for databases are carried out by experienced operators only.

Preparations

Before backing up and restoring a database, we recommend stopping the Monitoring API and the Log API on the monasca-server node, and check that all data is processed. This ensures that no data is written to a database during a backup and restore operation. After backing up and restoring a database, restart the APIs.

To stop the Monitoring API and the Log API, use the following command:

systemctl stop apache2

To check that all Kafka queues are empty, list the existing consumer groups and check the LAG column for each group. It should be 0. For example:

kafka-consumer-groups.sh --zookeeper 192.168.56.81:2181 --list
kafka-consumer-groups.sh --zookeeper 192.168.56.81:2181 --describe \
 --group 1_metrics | column -t -s ','
kafka-consumer-groups.sh --zookeeper 192.168.56.81:2181 --describe \
 --group transformer-logstash-consumer | column -t -s ','
kafka-consumer-groups.sh --zookeeper 192.168.56.81:2181 --describe \
 --group thresh-metric | column -t -s ','

To restart the Monitoring API and the Log API, use the following command:

systemctl start apache2

Elasticsearch Database

For backing up and restoring your Elasticsearch database, use the Snapshot and Restore module of Elasticsearch.

To create a backup of the database, proceed as follows:

Make sure that curl is installed, zypper in curl.
Log in to the host where the Monitoring Service is installed.
Create a snapshot repository. You need the Elasticsearch bind address for all commands. run grep network.bind_host /etc/elasticsearch/elasticsearch.yml to find the bind address, and replace IP in the following commands with this address. For example:
```
curl -XPUT http://IP:9200/_snapshot/my_backup -d '{
  "type": "fs",
  "settings": {
       "location": "/mount/backup/elasticsearch1/my_backup",
       "compress": true
  }
}'
```
The example registers a shared file system repository ("type": "fs") that uses the /mount/backup/elasticsearch1 directory for storing snapshots.

Note

The directory for storing snapshots must be configured in the elasticsearch/repo_dir setting in the Monasca barclamp (see the Deployment Guide). The directory must be manually mounted before creating the snapshot. The elasticsearch user must be specified as the owner of the directory.

compress is turned on to compress the metadata files.

Check whether the repository was created successfully:

curl -XGET http://IP:9200/_snapshot/my_backup

This example response shows a successfully created repository:

{
  "my_backup": {
    "type": "fs",
    "settings": {
      "compress": "true",
      "location": "/mount/backup/elasticsearch1/my_backup"
    }
  }
}

Create a snapshot of your database that contains all indices. A repository can contain multiple snapshots of the same database. The name of a snapshot must be unique within the snapshots created for your database, for example:
```
curl -XPUT http://IP:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true
```
The example creates a snapshot named snapshot_1 for all indices in the my_backup repository.

To restore the database instance, proceed as follows:

Close all indices of your database, for example:
```
curl -XPOST http://IP:9200/_all/_close
```
Restore all indices from the snapshot you have created, for example:
```
curl -XPOST http://IP:9200/_snapshot/my_backup/snapshot_1/_restore
```
The example restores all indices from snapshot_1 that is stored in the my_backup repository.

For additional information on backing up and restoring an Elasticsearch database, refer to the Elasticsearch documentation.

InfluxDB Database

For backing up and restoring your InfluxDB database, you can use the InfluxDB shell. The shell is part of your InfluxDB distribution. If you installed InfluxDB via a package manager, the shell is, by default, installed in the /usr/bin directory.

To create a backup of the database, proceed as follows:

Log in to the InfluxDB database as a user who is allowed to run the influxdb service, for example:
```
su influxdb -s /bin/bash 
```
Back up the database, for example:
```
influxd backup -database mon /mount/backup/mysnapshot
```
Monasca is using mon as the name of the database The example creates the backup for the database in /mount/backup/mysnapshot.

Before restoring the database, make sure that all database processes are shut down. To restore the database, you can then proceed as follows:

If required, delete all files not included in the backup by dropping the database before you carry out the restore operation. A restore operation restores all files included in the backup. Files created or merged at a later point in time are not affected. For example:
```
influx -host IP -execute 'drop database mon;'
```
Replace IP with the IP address that the database is listening to. You can run influxd config and look up the IP address in the [http] section.
Stop the InfluxDB database service:
```
systemctl stop influxdb
```
Log in to the InfluxDB database as a user who is allowed to run the influxdb service:
```
su influxdb -s /bin/bash
```

Restore the metastore:

influxd restore -metadir /var/opt/influxdb/meta /mount/backup/mysnapshot

Restore the database, for example:
```
influxd restore -database mon -datadir /var/opt/influxdb/data /mount/backup/mysnapshot
```
The example restores the backup from /mount/backup/mysnapshot to /var/opt/influxdb/influxdb.conf.
Ensure that the file permissions for the restored database are set correctly:
```
chown -R influxdb:influxdb /var/opt/influxdb
```
Start the InfluxDB database service:
```
systemctl start influxdb
```

For additional information on backing up and restoring an InfluxDB database, refer to the InfluxDB documentation.

MariaDB Database

For backing up and restoring your MariaDB database, you can use the mysqldump utility program. mysqldump performs a logical backup that produces a set of SQL statements. These statements can later be executed to restore the database.

To back up your MariaDB database, you must be the owner of the database or a user with superuser privileges, for example:

mysqldump -u root -p mon > dumpfile.sql

In addition to the name of the database, you have to specify the name and the location where mysqldump stores its output.

To restore your MariaDB database, proceed as follows:

Log in to the host where the Monitoring Service is installed as a user with root privileges.
Make sure that the mariadb service is running:
```
systemctl start mariadb
```
Log in to the database you have backed up as a user with root privileges, for example:
```
mysql -u root -p mon
```
Remove and then re-create the database:
```
DROP DATABASE mon;
CREATE DATABASE mon;
```
Exit mariadb:
```
\q
```
Restore the database, for example:
```
mysql -u root -p mon < dumpfile.sql
```

For additional information on backing up and restoring a MariaDB database with mysqldump, refer to the MariaDB documentation.

Configuration Files

Below you find a list of the configuration files of the agents and the individual services included in the Monitoring Service. Back up these files at least after you have installed and configured SUSE OpenStack Cloud Monitoring and after each change in the configuration.

/etc/influxdb/influxdb.conf
/etc/kafka/server.properties
/etc/my.cnf
/etc/my.cnf.d/client.cnf
/etc/my.cnf.d/mysql-clients.cnf
/etc/my.cnf.d/server.cnf
/etc/monasca/agent/agent.yaml
/etc/monasca/agent/conf.d/*
/etc/monasca/agent/supervisor.conf
/etc/monasca/api-config.conf
/etc/monasca/log-api-config.conf
/etc/monasca/log-api-config.ini
/etc/monasca-log-persister/monasca-log-persister.conf
/etc/monasca-log-transformer/monasca-log-transformer.conf
/etc/monasca-log-agent/agent.conf
/etc/monasca-notification/monasca-notification.yaml
/etc/monasca-persister/monasca-persister.yaml
/etc/monasca-thresh/thresh.yaml
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/kibana/kibana.yml

Recovery

If you need to recover the configuration of one or more agents or services, the recommended procedure is as follows:

If necessary, uninstall the agents or services, and install them again.
Stop the agents or services.
Copy the backup of your configuration files to the correct location according to the table above.
Start the agents or services again.

Dashboards

Kibana can persist customized log dashboard designs to the Elasticsearch database, and allows you to recall them. For details on saving, loading, and sharing log management dashboards, refer to the Kibana documentation.

Grafana allows you to export a monitoring dashboard to a JSON file, and to re-import it when necessary. For backing up and restoring the exported dashboards, use the standard mechanisms of your file system. For details on exporting monitoring dashboards, refer to the Getting Started tutorial of Grafana.

Prev	Up	Next
Log File Handling	Home	Chapter 4. Monitoring