Qumulo in AWS: Build a Multi-Instance Cluster with CloudFormation

IN THIS ARTICLE

This article explains how you can create a multi-instance cluster configuration by using CloudFormation templates in AWS.

REQUIREMENTS

• Amazon Web Services (AWS) account
  • AWS account number and target region
  • IAM permissions for full access to EC2 and CloudFormation
  • SSH key pair for accessing Qumulo instance
    
  • Virtual Private Cloud (VPC) with at least one subnet configured in the target region
• Amazon Machine Image (AMI) of Qumulo Core obtained from AWS Marketplace or Qumulo
  Important: Modifying the type or size of the EBS volumes in the Qumulo AMI renders the software not functional. You must use the block device layout from the original AMI.

Note: If you deploy to special AWS regions or zones like AWS Outposts or GovCloud (US), you can use the following steps to build a multi-instance cluster. There is no difference in the deployment process deployment. For more info, see Qumulo in AWS: Getting Started.

IAM PERMISSIONS

The following are the IAM permissions required for creating a Qumulo cloud cluster in AWS with CloudFormation:

• cloudformation:CreateStack
• cloudformation:DeleteStack
• cloudwatch:DeleteAlarms
• cloudwatch:PutMetricAlarm
• ec2:AuthorizeSecurityGroupEgress
• ec2:AuthorizeSecurityGroupIngress
• ec2:CreateLaunchTemplate
• ec2:CreateSecurityGroup
• ec2:CreateTags
• ec2:DeleteLaunchTemplate
• ec2:DeleteSecurityGroup
• ec2:DescribeInstances
• ec2:DescribeKeyPairs
• ec2:DescribeLaunchTemplates
• ec2:DescribeSecurityGroups
• ec2:DescribeSubnets
• ec2:DescribeVpcs
• ec2:RevokeSecurityGroupEgress
• ec2:RevokeSecurityGroupIngress
• ec2:RunInstances
• ec2:TerminateInstances

If you use EBS encryption, the following IAM permissions are also required:

• kms:CreateGrant
• kms:Decrypt
• kms:DescribeKey
• kms:GenerateDataKeyWithoutPlaintext
• kms:ReEncryptFrom
• kms:ReEncryptTo

If you ue a CloudFormation Template hosted in S3, the s3:GetObject IAM permission is also required.

DETAILS

Deploying Qumulo in AWS requires creating unique EC2 instances backed by EBS storage and clustering them together into a distributed file system. Generally, the number of instances in your cluster relates to performance, where higher instance counts equate to higher throughput and IOPS. Before you create your Qumulo in AWS cluster, consider the amount of storage and number of instances that you need .

Launch a Qumulo Cloud Cluster from the AWS Marketplace

Use the following steps to launch a Qumulo Cloud cluster directly from the AWS Marketplace. If you already have  a CloudFormation Template directly from Qumulo, see the CloudFormation section.

1. Navigate to the AWS Marketplace.
2. Log in to your AWS account.
   
3. Into the search bar, enter Qumulo.
   
4. Select the solution that meets your deployment's capacity and performance needs.
5. Verify the details and then click Continue to Subscribe.
   
6. Verify that the Delivery Method field specifies the most recent CloudFormation Template.
   
7. Confirm that all other details are correct.
8. Click Continue to Launch.
9. From the Choose Action list, choose Launch CloudFormation.
   
10. Click Launch.

Launch a Qumulo Cloud Cluster with a Provided CloudFormation Template

Use these steps if you have a CloudFormation Template file directly from Qumulo.

1. Navigate to the AWS Marketplace.
2. Log in to your AWS account.
3. Click the Services drop-down menu.
4. In the Management Tools section, click CloudFormation.
   
5. Click Create new stack.
   
6. Do one of the following:
   • If you have a URL for the template file, click Amazon S3 URL and then enter the URL that you received at time of purchase.
   • If you have a template file, click Upload a template file > Choose File.
7. Click Next.

Configure your Qumulo Cloud Cluster Stack

1. Enter information into all relevant fields .
   Note: If you don't already have an SSH key pair, you can obtain one by clicking the Services > Key Management Service. 
2. To get notified when your instances stop or recover from a failed status check, enter the ARN of your own SNS topic into the Instance recovery alarm SNS topic text box.
3. Click Next.
4. Review the configuration details and then click Create stack.

You might need to refresh the page to see progress updates. When it finishes, the Status column displays CREATE_COMPLETE.

Important:

• Before you continue, make sure that the creation process completes for the entire cluster, not just the individual nodes.
• We don't recommend running a Qumulo cluster in AWS for production data without a replication or backup strategy.

Launch Your New Qumulo Cloud Cluster

1. After the creation finishes, click Outputs to locate the details of your instance.
   
2. Click the IP address link in the LinkToManagement field.
   
   • Note: Qumulo takes your security seriously. As a result, the template creates only a private IP address for each of the instances. To complete cluster creation you must access the instance in one of two ways: 
     • Ensure the browser you use is running on a machine with access to the VPC.
     • Update the VPC firewall to allow access to the instance by using a public IP address, for example by using the AWS Internet Gateway Service.
3. For the username, enter admin.
4. From the Outputs pane in your AWS session, copy the value for TemporaryPassword, and enter it into the Qumulo instance.
   
5. Click Login.
6. Change your password.

ADDITIONAL RESOURCES

Qumulo in AWS: Getting Started

Qumulo in AWS: Add a Node to an Existing Cloud Cluster

QQ CLI: Cluster Configuration

Like what you see? Share this article with your network!