Using Failover with Replication in Qumulo Core 2.12.0 (or Higher)

REQUIREMENTS

Administrative privileges
Source and target clusters, both with Qumulo Core 2.12.0 (or higher)

If you run a version of Qumulo Core lower than 2.12.0, see Replication: Failover and Failback with 2.11.4 and Below.

DETAILS

There are two types of failover, planned and unplanned. Replication helps you recover from either scenario. The following sections explain performing planned and unplanned failover on your clusters.

To Perform a Planned Failover to a Secondary Cluster

Notes:

For a planned failover, set the source directory on the primary cluster to read-only and wait for the new replication job to begin and finish (so that no writes are lost).
If your relationship uses Snapshot Policy Replication mode, after you set the source directory to read-only, change the replication mode to Snapshot Policy with Continuous Replication. To ensure that any changes that occurred (since Qumulo Core took the last policy snapshot) replicate to the target cluster, this process initiates a new replication job.
After the job finishes (or after a fail-back), you can reconfigure the relationship to Snapshot Policy Replication Mode.

The following steps explain how you can perform a planned failover with replication.

To ensure that the target directory is in a point-in-time consistent state that matches the last source snapshot, do the following.
1. Set the source directory on your primary cluster to read-only.
2. To ensure that no writes are lost, wait for a new replication job to start and finish.
3. To ensure that a job is finished, verify that any blackout windows are disabled and then check the relationship's recovery point time, ensuring that is after the time that you set the source directory to read-only.
In the Web UI of your secondary cluster, click Cluster > Replication.
On the Replication page, under Actions, click > Make Target Writable...
In the Make Target Writable dialog box, click Yes, Make Target Writable.
Wait for your directory to revert to its last recovery point.
To monitor progress, click Details.
When the Status column displays the following message, the target directory is consistent and writable.
Disconnected and target writable. Reverse or reconnect using the relationship actions under the actions menu.
If the following configuration doesn't already exist on your secondary cluster, migrate it:
- NFS exports
- SMB shares
- AD/LDAP servers
- Snapshot policies
- Quotas
Remount any clients that require access to your secondary cluster and were previously connected to your primary cluster.

The planned failover process is complete.

To Perform an Unplanned Failover to a Secondary Cluster

If the previous replication was incomplete before the primary cluster became unavailable, it might have left the target directory in an indeterminate state with some files truncated or inconsistent.

The following steps explain how you can perform an unplanned failover with replication by bringing the target directory to a determinate state using the last successful replication recovery point to let you use it as the source directory.

In the Web UI of your secondary cluster, click Cluster > Replication.
On the Replication page, under Actions, click > Make Target Writable...
In the Make Target Writable dialog box, click Yes, Make Target Writable.
Wait for your directory to revert to its last recovery point.
To monitor progress, click Details.
When the Status column displays the following message, the target directory is consistent and writable.
Disconnected and target writable. Reverse or reconnect using the relationship actions under the actions menu.
If the following configuration doesn't already exist on your secondary cluster, migrate it:
- NFS exports
- SMB shares
- AD/LDAP servers
- Snapshot policies
- Quotas
Remount any clients that require access to your secondary cluster and were previously connected to your primary cluster.