It is always a good idea to keep backups, and it is possible to back up both the Juju client environment and the Juju state server (bootstrap node) to be able to reconnect to running cloud environments. Both the backing up and restore procedures for each are detailed below.
It is easy to forget that your client environment needs to be backed up. Aside
from things like the 'environments.yaml' file, which you may be able to
recreate, it contains unique files such as Juju's SSH keys, which are vital to
be able to connect to a running instance. In order to also save any additional
configuration files (such as '.jenv' files for running environments, or
credentials) it is usually best to simply back up the entire Juju directory
As the Juju directory is simply just another directory on your filesystem, you
may wish to include it in any regular backup procedure you already have. For a
simple backup on an Ubuntu system, you can just use
tar to create an archive
of the directory:
cd ~ tar -cpzf juju-client-$(date "+%Y%m%d-%H%M%S").tar.gz .juju
This command datestamps the created file for easy identification, you may of course, call it what you wish.
If you have used the local provider, you will need to run the command as a superuser:
cd ~ sudo tar -cpzf juju-client-$(date "+%Y%m%d-%H%M%S").tar.gz .juju
This is because the local provider makes use of superuser privileges, and
consequently some of the files belong to 'root' and cannot be read by a normal
user. You will also find that the backup archives are substantially bigger.
As the local provider stores everything in '~/.juju', you are effectively
backing up the entire local environment also. If you aren't concerned about the
local user, you can omit the local provider files by supplying the
--exclude-path= switch as many times as necessary and specifying the name
of your local environment(s):
sudo tar -cvpzf juju-client-$(date "+%Y%m%d-%H%M%S").tar.gz --exclude=.juju/local --exclude=.juju/local2 .juju
For Ubuntu, restoring your Juju client settings and environments is a simple matter of extracting the archive you created above.
cd ~ tar xzf juju-yymmdd-hhmmss.tar.gz
In the case of any local environment, this will also be restored providing the actual service itself (LXC) has remained running.
Juju provides commands for recovering the Juju state server (bootstrap node) in case of failure. The included commands allow you to create, restore and manage backup files containing the state server configuration, keys, and environment data. If the state server or its host machine later fails, you can create a new state server from the backup file. For environments with "High availability" enabled, also see the relevant sections below
create command to create a new backup file:
juju backups create [-e ENV] [--quiet ] [--filename=FILENAME | --no-download]
create command creates an archive file for the state server in an
environment, along with metadata about that file. By default the current
environment is used. It may be explicitly identified by the -e option
JUJU_ENV shell environment variable.
As a convenience, the state server stores both the file and the
juju backups list to see stored files), however, it is up to the
user to download and safely store these files for when they are needed.
juju backups create will download the new backup file, though you
can use the --no-download option to disable this.
If the --filename option is not provided to
create then the backup
archive is downloaded with a default name that incorporates the date and
time, such as
juju-backup-20151103-1408.tar.gz. The filename is displayed
once the backup completes. Unless the --quiet option is used, the backup
metadata is also output.
juju backups create juju backups create --no-download juju backups create -e my-env juju backups create --filename backup-19.tgz
Note that creating a backup may take a long time.
Each time a backup is created, it is also stored on the state-server for
safekeeping, along with associated metadata giving it a unique id value and a
timestamp. This means that you can manage backups from whatever client you can
connect from, and fetch previous backups if the originally downloaded file has
gone astray. For managing backups, the
juju backups command has an additional
set of subcommands:
juju backups list [--brief]
list subcommand will display all the backups currently available on the
state server. There are two options for the output:
juju backups list
will display a complete list of all backup metadata for all the available
backups. As such, it can output a lot of data, so you may consider using
--brief switch on the command, which will return just a list of the
backup ID values:
juju backups list --brief
should return something like:
20150910-150614.f41b1639-09b3-4eb7-8fbf-0927558cdf44 20150911-192138.f41b1639-09b3-4eb7-8fbf-0927558cdf44 20150912-100824.f41b1639-09b3-4eb7-8fbf-0927558cdf44 20150914-160331.f41b1639-09b3-4eb7-8fbf-0927558cdf44
The appended alphanumeric string is not actually random gibberish, but is an identifier for the instance the backup was created on. This information is not normally useful to the end user, but it does help give all backup files a unique name.
juju backups download [options] <ID>
download command will fetch the backup specified from the environment
and download it for local storage. If the
--filename switch is not used, the
backup is downloaded to the current directory and the filename used (derived
from a timestamp and the ID) is output to the console.
juju backups download 20150914-150614.f41b1639-09b3-4eb7-8fbf-0927558cdf44
juju backups info [options] <ID>
This command outputs the complete metadata record for the specified backup. For example:
juju backups info 20150914-150614.f41b1639-09b3-4eb7-8fbf-0927558cdf44
will result in output such as:
backup ID: "20150914-150614.f41b1639-09b3-4eb7-8fbf-0927558cdf44" checksum: "gECSwsOc2QtVG2RNrzk1bzCb+JU=" checksum format: "SHA-1, base64 encoded" size (B): 32583286 stored: 2015-09-14 15:06:30 +0000 UTC started: 2015-09-14 15:06:14.433210757 +0000 UTC finished: 2015-09-14 15:06:23.634473665 +0000 UTC notes: "" environment ID: "f41b1639-09b3-4eb7-8fbf-0927558cdf44" machine ID: "0" created on host: "juju-f41b1639-09b3-4eb7-8fbf-0927558cdf44-machine-0" juju version: 1.24.5 20150914-150614.f41b1639-09b3-4eb7-8fbf-0927558cdf44
juju backups remove [options] <ID>
If you wish to remove a particular backup file from the state server (perhaps to save space!), you can use the remove subcommand with the appropriate ID:
juju backups remove 20150914-160331.f41b1639-09b3-4eb7-8fbf-0927558cdf44
After a short time, you should see a confirmation:
successfully removed: 20150914-160331.f41b1639-09b3-4eb7-8fbf-0927558cdf44
juju backups upload [options] <filename>
As well as downloading backups from the Juju state server, it is also possible to upload them. This can be useful either to break up the process of restoring from a backup (upload the file, then restore using the ID), or in the case where backups have been removed from the state server.
By default, this command will return the new ID for the uploaded backup file. It
is possible to return instead the complete metadata (using
nothing at all (using
juju backups upload mybackup.tar.gz juju backups upload juju-backup-20150817-122042.tar.gz --quiet juju backups upload juju-backup-20150817-122042.tar.gz --verbose
The metadata of uploaded files will reflect the time it was stored, but should also determine the correct dates for when the backup was started/completed.
You can get usage information from any of the backup commands by running:
juju backups help <command>
juju backups restore --id=<ID> | --file=<filname> [-b] [--constraints=<string>]
If the state server for the environment is still operational it can be restored from one of the stored backups by specifying the ID:
juju backups restore --id=20150914-160331.f41b1639-09b3-4eb7-8fbf-0927558cdf44
It is also possible to restore from a local backup file by instead specifying the filename. This will then be uploaded to the state server and used to restore it:
juju backups restore --file=juju-backup-23.tgz
In the case that the original state server no longer exists, it is possible to re-bootstrap the environment and restore the backup to the new state-server. To do this, use the '-b' switch:
juju backups restore -b --file=juju-backup-23.tgz
In this case it is also possible to specify constraints for the newly created bootstrap node, for example:
juju backups restore -b --constraints="mem=4G" --file=juju-backup-23.tgz
Read the constraints reference page for more information on the constraints which may be used.
As stated in the Juju HA documentation, High Availability in
general terms means that a Juju environment has 3 or more (up to 7) redundant
state servers. In the normal course of operation, having multiple, redundant
state servers means that requiring a backup is less likely. As long as one of
the original state servers remains, the others can be replaced by simply
juju ensure-availability command again.
The contemplated case for HA backup/restore is when you have lost all your state
servers and need to recover a basic setup in order to be able to perform the
ensure availability step again.
When you perform a
backup on a Juju installation which has multiple redundant
state-servers, the initial state-server will be chosen to perform the backup.
As an example, the following environment has 3 active state-servers. Running the command:
... will return something similar to:
environment: jujutest machines: "0": agent-state: started agent-version: 1.24.5 dns-name: 188.8.131.52 instance-id: i-a4fc2707 instance-state: running series: trusty hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M availability-zone=us-east-1a state-server-member-status: has-vote "1": agent-state: started agent-version: 1.24.5 dns-name: 184.108.40.206 instance-id: i-a7814672 instance-state: running series: trusty hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M availability-zone=us-east-1b state-server-member-status: has-vote "2": agent-state: started agent-version: 1.24.5 dns-name: 220.127.116.11 instance-id: i-2b1db88b instance-state: running series: trusty hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M availability-zone=us-east-1c state-server-member-status: has-vote ...
Performing a backup on this environment, will be based on the first state-server, machine 0:
juju backups create
backup ID: "20150907-154511.19f1b8b2-29cc-43de-866d-48ef88cf2f17" checksum: "9NttJ4ELrA9XBlBFBX0ToxmrlNM=" checksum format: "SHA-1, base64 encoded" size (B): 33013934 stored: 2015-09-07 15:45:59 +0000 UTC started: 2015-09-07 15:45:11.831515387 +0000 UTC finished: 2015-09-07 15:45:48.842386549 +0000 UTC notes: "" environment ID: "19f1b8b2-29cc-43de-866d-48ef88cf2f17" machine ID: "0" created on host: "ip-10-180-194-91" juju version: 1.24.5 20150907-154511.19f1b8b2-29cc-43de-866d-48ef88cf2f17 downloading to juju-backup-20150907-154511.tar.gz
As with backing up a non-HA environment, the backup file is stored on the state server and automatically downloaded, or you can specify further options as stated above.
Please note that a restore must take place when you have lost all your redundant
state-servers. If that is not the case, simply issuing the
juju ensure-availability command will be enough to create
a new state-server replica on your environment.
For performing a
restore, the only check performed by the utility is to make
sure that the initial state-server is not up.
To restore an initial bootstrap environment, the procedure is the same as for non-HA environments:
juju backups restore -b --file=juju-backup-23.tgz
Once this step is completed, you will have a single state-server running. To recover the rest of the state-server replicas, all that remains is to reissue the command:
juju ensure-availability -n 3
This will create additional state-servers based on the restored one.