IP Failover with Qumulo Core

REQUIREMENTS

Cluster running Qumulo Core
Command line (CLI) tools installed through API & Tools tab

For details on configuring your DNS server to point to the floating IPs, check out the Set Explicit DNS Hostnames and IP Mappings article.

PROCESS

IP Failover is a high-availability feature that allows a node's virtual IP address(es) to be reassigned to other nodes in the cluster should a node go offline for any reason.

In addition to the fixed IP range assigned to a cluster, an administrator can set a floating range of addresses that can be reshuffled amongst online nodes as necessary. When using IP Failover, it is recommended that the cluster's client-facing DNS record be pointed at these floating IPs as opposed to the fixed range.

Clients mount the cluster using one of the following recommended mount options:

Mac Client

mount -t nfs -o \
rsize=65536,wsize=65536,intr,hard,tcp,rdirplus,readahead=128 \
your.qumulo.ip:/share /path/to/mountpoint

Mac Client with Local Locking Enforced

mount -t nfs -o \
rsize=65536,wsize=65536,intr,hard,tcp,locallocks,rdirplus,readahead=128 \
your.qumulo.ip:/share /path/to/mountpoint

Note: Mac Client mount works only on NFSv3.

Linux Client

Note: Modern Linux distributions auto-negotiate a 1MB read/write block size (rsize/wsize of rsize=1048576).

mount -t nfs -o intr,hard,tcp your.qumulo.ip:/share \
/path/to/mountpoint

Linux Client with Local Locks Enforced

mount -t nfs -o intr,hard,tcp,local_lock=all your.qumulo.ip:/share \
/path/to/mountpoint

In a node outage scenario, any IP in the floating range that is assigned to the offline node moves to another available node ensuring that connected clients can continue writing and reading to/from the cluster. Typically, the time period during which an IP address fails over to another node causes only a momentary interruption for any running workloads.

Note: Although certain clients (such as those using SMB) might require a new TCP connection, the retry mechanism in most operating systems provides a sufficiently fast failover.

We recommend assigning enough floating IP addresses so that each node has the total number of nodes minus one for the number of floating IP addresses per client-facing VLAN. The math to use is (N-1)*N where N is the total number of nodes in the cluster. Assuming many client connections, this best practice helps to evenly distribute the connections from the lost node onto the remaining nodes as needed. For example, in a 4 node cluster when 1 node goes offline, its 3 virtual IPs would then float to each of the remaining 3 nodes.

Note: The number of floating IPs is dependent on your workflow and clients connecting to the cluster, with a minimum of 2 floating IPs per node per client-facing VLAN, but no more than 10 floating IPs per node.

Configure IP Failover using the Web UI

Log in to your cluster's Web UI as admin.
Hover your mouse over the Cluster menu and select Network Configuration.
On the Network Configuration page, click on Edit Static Settings.
In the fields for Persistent IPv4 Addresses and Floating IPv4 Addresses, input the fixed and floating ranges.
Click Save.

Configure IP Failover using the QQ CLI

Using a node's IP address, ssh to the cluster as admin.

ssh admin@10.101.1.101

Login as root:

sudo -s

Run the following qq command replacing the IP range with your preferred floating range:

qq network_conf_mod --floating-ip-ranges 10.100.1.201-204

Once IP failover is configured, confirm that your clients are using the new floating IPs. If a client already has a mount point based on a previous static IP, you must unmount and then remount the Qumulo cluster, ensuring the new mount picks up the floating IP set. The DNS record must be updated to list the floating IPs and not the static IPs for the cluster before remounting.