Print Email PDF

Qumulo in AWS: Audit Logging with CloudWatch

IN THIS ARTICLE

Outlines how to send Audit Logs to CloudWatch for your Qumulo cloud cluster in AWS

REQUIREMENTS

  • AWS Qumulo cloud cluster
    • All the cluster’s node instance have an IAM role with the appropriate privileges assigned to them (detailed later)
    • Running Qumulo Core 3.1.1 and above
  • AWS Console or API access to AWS CloudWatch
  • Admin privileges to the cloud cluster
  • Command line (qq CLI) tools installed via API & Tools in the Web UI

NOTE: Modifying the type or size of the EBS volumes in the Qumulo AMI will render the software not functional. Please use the AMI volume layout provided in the original image. 

IAM PERMISSIONS

The table below lists the required IAM permissions to configure audit logging with CloudWatch on a Qumulo cloud cluster in AWS.

CloudWatchLogs::PutLogEvents

CloudWatchLogs::CreateLogStream

To learn more about IAM roles, check out the IAM roles for Amazon EC2  and Assign an Existing IAM Role AWS support articles.

DETAILS

Audit logging in Qumulo Core provides a mechanism for tracking filesystem operations. As connected clients issue requests to the cluster, log messages are generated describing each attempted operation. These log messages are then sent over the network to the CloudWatch logs service.

Each audit log message body consists of a JSON object with multiple string values:

{
"ip_address": "1.2.3.4",
"user": "AD\alice",
"protocol": "nfs",
"operation": "nfs_read_data”,
"result": "ok",
"object_id": "123",
"path_1": "/.snapshot/1_snapshot1225/dir/",
"path_2": ""
}

Create a Log Group for CloudWatch Audit Logging

To use the CloudWatch audit log feature, you will need to create a log group (required) and ensure that all the nodes’ instances have the correct permissions to write to CloudWatch. To provide the nodes’ instances the permissions to write to CloudWatch, an IAM role with the required permissions must be attached to all of the instances. The following action must be allowed by the IAM role for the CloudWatch audit feature to work:

  1. logs:CreateLogStream
  2. logs:PutLogEvents

TIP! To learn more about how to set-up and attach IAM roles, see Using IAM Policies for CloudWatch Logs and Attaching an IAM role to an instance AWS documents.

Each node sends its audit logs to a specific log stream that is grouped by a log group. Note that you can create a log group per cluster if you'd like to keep your clusters’ audit logs separated. Alternatively, you can create a single log group for all or a subset of your Qumulo clusters, with the rest of your infrastructure using another log group to keep them separated.

To create a log group via the CloudWatch Dashboard in the AWS Console:

  1. Click on Log group in the left-hand menu.
  2. Choose Actions, and then choose Create log group.
  3. Enter the log group’s name.
  4. Click Create log group.

To create a log group via the AWS CLI Tool, run the following command:

aws logs create-log-group --log-group-name example_log_group

IMPORTANT! The log group name is required when enabling CloudWatch with the QQ CLI as outlined below.

Send Audit Logs to CloudWatch via QQ CLI

Now that you have a new log group, you can send audit logs to CloudWatch. Run the following qq command, including the log group name we created and the region to send the logs to:

qq audit_set_cloudwatch_config --enable --log-group-name example_log_group --region us-west-2
{
"enabled": true,
"log_group_name": "example_log_group",
"region": "us-west-2"
}

NOTE: You can send your audit logs to a different region instead of where your cluster lives.

Once CloudWatch audit logging is enabled, each node will create a log stream (e.g., qumulo-CLUSTER_NAME-CLUSTER_UUID-node-NODE_ID) in the log group. The node will then upload its audit logs in JSON format to that log stream. 

To disable audit logging with CloudWatch, use the following command:

qq audit_set_cloudwatch_config --disable

If needed, you can run the following command to wipe the previous configuration by setting the parameters to be empty strings:

qq audit_set_cloudwatch_config --disable --log-group-name='' --region=''

To review the current configuration for CloudWatch audit logging, run the following:

qq audit_get_cloudwatch_config
{
"enabled": false,
"log_group_name": "",
"region": ""
}

View and Search the Logs in CloudWatch Logs

  1. Click Log groups in the left-hand navigation pane on your AWS CloudWatch Dashboard.
  2. Click the log group that was used to configure the CloudWatch Audit feature on your Qumulo cluster from the list.

You'll find one log stream for each node in your cluster named in the following format in the selected log group:

qumulo-CLUSTER_NAME-CLUSTER_UUID-node-NODE_ID

Each log stream will have a timeline of log messages formatted in JSON:

CloudWatch_Log.png

You can match and filter these log messages using a special syntax. In the picture example above, you could type “{ $.operation = fs_read_metadata }” into the search bar to match on the first message only. Another useful search is “{ $.result = *error* }” which helps you find problematic operations performed by that node. Similarly, you could use “$.ip_address” to look at the operations requested by a specific client or “$.user” for a specific user.

Troubleshooting via the Status API

Use the qq command below to check the current state of the connection with the CloudWatch service:

qq audit_get_cloudwatch_status
{
"node_statuses": {
"1": {
"last_seen_error": null
},
"2": {
"last_seen_error": null
},
"3": {
"last_seen_error": null
},
"4": {
"last_seen_error": null
}
}
}

Each node has a “last_seen_error” attribute with the last error from the CloudWatch API. When the configuration is enabled and all nodes return a “null” error, the most recent CloudWatch requests on each node have succeeded.

Example Errors

If you forgot to give one of your instances permission to call the PutLogEvents API, you would see the following:

qq audit_get_cloudwatch_status
{
"node_statuses": {
"1": {
"last_seen_error": {
“error_message”:
“Access denied sending logs to CloudWatch. Do
your instances have the appropriate IAM roles
associated?”,
“error_details”: “...”
}
}, ...
}
}

If you forgot create a log group prior to configuration, the following error would occur:

qq audit_get_cloudwatch_status
{
"node_statuses": {
"1": {
"last_seen_error": {
“error_message”: “Log group does not exist”,
“error_details”: “...”
}
}, ...
}
}

If you see the following error or any other persistent error cannot be resolved, please note what displays in the “error_details” and contact Qumulo Care.

qq audit_get_cloudwatch_status
{
"node_statuses": {
"1": {
"last_seen_error": {
“error_message”:
“Encountered unrecognized error. Contact Qumulo
Support.”,
“error_details”: “...”
}
}, ...
}
}

Considerations

  • When the CloudWatch audit log feature is disabled, audit logs that have not been sent to CloudWatch will be dropped.
  • If CloudWatch becomes inaccessible from the cluster, audit logs will be dropped shortly thereafter depending on your configuration and workflow. When CloudWatch comes back the cluster will resume sending audit logs.
  • CloudWatch Logs have per-stream and account-wide limits on the number of messages sent per second, per account, per Region. If other applications are using CloudWatch Logs in an account, message throughput may be impacted in both the Qumulo cluster and other applications.
  • If CloudWatch’s limits are exceeded, audit logs will be dropped depending on your current workflow and configuration.

RESOLUTION

You should now be able to successfully send Audit Logs to CloudWatch for your Qumulo cloud cluster in AWS

ADDITIONAL RESOURCES

Qumulo Core Audit Logging

Qumulo in AWS: Configure CloudWatch Alarms

Qumulo in AWS: Create a CloudWatch Dashboard

QQ CLI: Cluster Configuration

 

Like what you see? Share this article with your network!

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.

Have more questions?
Open a Case
Share it, if you like it.