Qumulo in AWS: Build a Multi-Instance Cluster with CloudFormation

IN THIS ARTICLE

Outlines how to create a multi-instance cluster configuration using templates provided by CloudFormation in AWS

REQUIREMENTS

• Amazon Web Services (AWS) account
  • AWS account number and target region
  • IAM permissions for full access to EC2 and CloudFormation
  • SSH key-pair for accessing Qumulo instance
    
  • Virtual Private Cloud (VPC) with at least one subnet configured in the target region
    

• Amazon Machine Image (AMI) of Qumulo Core obtained from AWS Marketplace or Qumulo
  • Modifying the type or size of the EBS volumes in the Qumulo AMI will render the software not functional. Please use the block device layout provided in the original AMI.

NOTE: If you are deploying to special AWS regions or zones like AWS Outposts or GovCloud (US), you can proceed to build a multi-instance cluster using the steps below. There is no difference in deployment. For more info, check out the details provided in the Qumulo in AWS: Getting Started article on Qumulo Care. 

IAM PERMISSIONS

The table below lists the required IAM permissions for creating a Qumulo cloud cluster in AWS with CloudFormation.

cloudformation:CreateStack	cloudformation:DeleteStack	cloudwatch:DeleteAlarms
cloudwatch:PutMetricAlarm	ec2:AuthorizeSecurityGroupEgress	ec2:AuthorizeSecurityGroupIngress
ec2:CreateSecurityGroup	ec2:CreateTags	ec2:DeleteSecurityGroup
ec2:DescribeInstances	ec2:DescribeKeyPairs	ec2:DescribeSecurityGroups
ec2:DescribeSubnets	ec2:DescribeVpcs	ec2:RevokeSecurityGroupEgress
ec2:RevokeSecurityGroupIngress	ec2:RunInstances	ec2:TerminateInstances

If you are using EBS encryption, the following permissions are also required:

kms:CreateGrant	kms:Decrypt	kms:DescribeKey
kms:GenerateDataKeyWithoutPlaintext	kms:ReEncryptFrom	kms:ReEncryptTo

IMPORTANT! The s3:GetObject IAM permission will also be required if you are using a CloudFormation Template hosted in S3.

DETAILS

Deploying Qumulo in AWS involves creating unique EC2 instances backed by EBS storage and clustering them together into a distributed file system. Generally, the number of instances in your cluster relates to performance where the higher instance counts equate to higher throughput and IOPS. Before creating your Qumulo in AWS cluster, consider the amount of storage and number of instances that you need.

CloudFormation templates are especially helpful in that they can be customized to meet any size or type requirements and can be deployed in any region.

Launch a Qumulo Cloud Cluster from the AWS Marketplace

Use the steps below  to launch a Qumulo Cloud cluster directly from the AWS Marketplace. If you have already been provided with a CloudFormation Template directly from Qumulo, refer to Launch a Qumulo Cloud Cluster with a Provided CloudFormation Template below for appropriate steps.

1. Navigate to the AWS Marketplace.
2. Log in to your AWS account.
   
   
   
   
3. Use the Search Bar at the top of the screen to find “Qumulo”.
   
   
   
   
4. Select the solution that meets your deployment's capacity and performance needs.
5. Click Continue to Subscribe after verifying the details provided.
   
   
   
   
6. Verify that the Delivery Method field specifies the most recent CloudFormation Template available.
   
   
   
   
7. Confirm all other details are correct.
8. Click Continue to Launch.
9. Use the Choose Action drop-down menu to select Launch CloudFormation.
   
   
   
   
10. Click Launch.

Proceed to the Configure your Qumulo Cloud Cluster Stack section below to continue to stack configuration.

Launch a Qumulo Cloud Cluster with a Provided CloudFormation Template

Follow the steps below if you have a CloudFormation Template file directly from Qumulo.

1. Navigate to the AWS Marketplace.
2. Log in to your AWS account.
3. Click the Services drop-down menu.
4. Click the CloudFormation link in the Management Tools section.
   

   

5. Click Create new stack to open the Create Stack page.
   

   

6. Follow the appropriate step below:
   • If you have a URL for the template file, click Amazon S3 URL and enter the URL that was provided at purchase

     

   • If you have a downloaded template file, select Upload a template file, then Choose File to select your local template file

     

7. Click Next to finish.

Configure your Qumulo Cloud Cluster Stack

1. Enter the required information for the fields displayed (stack name, network configuration, etc.). 
   

   

   Note: If you do not already have an SSH key pair, you can obtain one by clicking the Services menu and selecting Key Management Service. 
2. Enter the ARN of your own SNS topic into the Instance recovery alarm SNS topic text box (available in 3.2.0 or above) in the SNS configuration section if you would like to be notified when your instances stop or recover from a failed status check.
3. When finished, click Next.
4. Review the configuration details you have provided. If they are correct, click Create stack.

You may need to refresh the page in order to see progress updates. When it has finished, the Status column will display CREATE_COMPLETE.

Before proceeding, make sure that the creation process completes for the entire cluster, not just the individual nodes. In the screenshot above, you can see the process complete for four nodes, and then the cluster completion (sm-test) at the very top.

IMPORTANT! It is not recommended to run a Qumulo cluster in AWS for production data without a replication or backup strategy.

Launch Your New Qumulo Cloud Cluster

1. After the creation has finished, click the Outputs tab to locate your instance's details.
   

   

2. Click IP address link provided in the LinkToManagement field to open it in a new browser tab.
   
   • Note: Qumulo takes your security seriously. As a result, the template only creates a private IP address for each of the instances. To complete cluster creation you will need to access the instance in one of two ways: 
     • Ensure the browser you are using is running on a machine with access to the VPC, or
     • Update the firewall of the VPC to allow access to the instance using a public IP address. This is most commonly accomplished using the AWS Internet Gateway Service.
3. Enter admin for the user name.
4. Go back to the AWS browser tab and copy the value for TemporaryPassword from the table in the Outputs pane.
   
5. Enter this value for your password in the Qumulo instance.
   
6. Click Login to view your new Qumulo cluster.
7. Change your password after logging in for security purposes.

RESOLUTION

You should now be able to successfully create a multi-instance cloud cluster for Qumulo in AWS

ADDITIONAL RESOURCES

Qumulo in AWS: Getting Started

Qumulo in AWS: Add a Node to an Existing Cloud Cluster

QQ CLI: Cluster Configuration

 

Like what you see? Share this article with your network!