Qumulo Core Audit Logging

IN THIS ARTICLE

Outlines how to use audit logging in Qumulo Core

REQUIREMENTS

• Admin privileges required
• Cluster running Qumulo Core 2.12.0 and above
• Syslog server configured to receive audit logs over TCP

IMPORTANT! You must set up a syslog server that implements syslog protocol using any one of the numerous programs available to receive audit logs. This syslog server must be configured to receive logs over TCP on the port specified in your Qumulo’s audit logging configuration. 

DETAILS

Audit logging in Qumulo Core provides a mechanism for tracking filesystem operations. As connected clients issue requests to the cluster, log messages are generated describing each attempted operation. These log messages are then sent over the network to the remote syslog instance specified by the current audit configuration in compliance with RFC 5424.

Each audit log message body consists of a few fields in CSV format. Note that the user id and both file path fields are quoted since they may contain characters that need to be escaped in CSV (i.e., quotes and commas). In addition to escaping characters, we also strip all newline characters (“\n” or “\r”) from these three fields.

The fields are described below in the order that they display within the log message body:

127.0.0.1,"AD\alice",nfs,fs_read_data,ok,123,"/.snapshot/1_snapshot1225/dir/",""

User IP: the IP address of the user in IPV4/IPV6 format

User ID: the user that performed the action

• AD username
• Qumulo local username
• POSIX UID
• Windows SID
• Qumulo auth ID (if we fail to resolve the user ID to any of the previous types)

Protocol: the protocol that the user request came through

• nfs
• smb
• ftp
• api

File System Operation: the operation that the user attempted

• fs_create_directory
• fs_create_file
• fs_create_hard_link
• fs_create_symlink
• fs_create (a filetype other than one of the types captured above was created)
• fs_delete
• fs_fsstat
• fs_read_metadata
• fs_list_directory
• fs_open
• fs_read_data
• fs_read_link
• fs_rename
• fs_write_data
• fs_write_metadata

Management Operation (2.12.2 and above): any operation (common examples below) that modifies cluster configuration

• auth_create_user
• smb_create_share
• smb_login
• nfs_create_export
• nfs_mount
• snapshot_create_snapshot
• replication_create_source_relationship

Error Status: “ok” if the operation succeeded or a “<Qumulo-specific-error-code>” (common errors below) if the operation failed

• fs_access_denied_error
• fs_no_space_error
• fs_invalid_file_type_error

File id: the id of the file that the operation was on

File path: the path of the file that the operation was on

• When accessing a file through a snapshot, the path is prefixed with “/.snapshot/<snapshot-directory-name>” which is the same path prefix used to access snapshotted files via nfs and smb.

Secondary file path: any rename/move operations

IMPORTANT! In order to keep the volume of audit log messages to a minimum, similar operations performed in quick succession will be deduplicated. For example, if a user reads the same file 100,000 times in one minute, only one message, corresponding to the first read, will be generated.

Configure Audit Logging

To enable audit logging to send messages to a remote syslog instance, run the following qq command including a specific IP address (or hostname) and port number:

qq audit_set_config --enable --server-address <syslog-server-hostname> --server-port <port-number>

To disable audit logging, use the following command:

qq audit_set_config --disable

To review the current configuration for audit logging, run the following:

qq audit_get_config

Sample Output:

qq audit_get_config
{
 "enabled": true,
 "server_address": "syslog-server-hostname",
 "server_port": 514
}

Use the qq command below to check the current state of the connection with the remote syslog instance:

qq audit_get_status

The connection_status included in the output will be one of the following:

• "AUDIT_LOG_CONNECTED": the connection with the remote syslog instance has been established and all log messages should be successfully transmitted.
• "AUDIT_LOG_DISCONNECTED": there is no active connection to the remote syslog instance. The cluster will attempt to buffer all outgoing log messages until the connection can be reestablished, at which point the buffered messages will be sent to the remote syslog instance. If a power outage or reboot occurs, all unsent messages will be lost. If the message buffer fills up, all new messages will be thrown away.
• "AUDIT_LOG_DISABLED": audit logging has been explicitly disabled.

Check out the sample output of an established syslog instance below:

qq audit_get_status
{
 "connection_status": "AUDIT_LOG_CONNECTED"
}

Configuration Example with Rsyslog

Let’s walk through one example where we configure the cluster to send audit log messages to an Rsyslog process on a client machine. For this example our client will be running Ubuntu 18.04.

First, we need to update the global Rsyslog configuration to allow receiving syslog messages over TCP connections. Uncomment the following lines in /etc/rsyslog.conf to configure Rsyslog to listen for TCP connections on port 514.

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Now configure Rsyslog to capture Qumulo audit log messages in their own dedicated log file and format. Create a new file called /etc/rsyslog.d/10-qumulo-audit.conf with the following contents making sure to replace the bold fields:

# Log file name:"/directory-storing-the-logs/node-hostname.log"
template(name="QumuloFileName" type="list") {
 constant(value="/directory-storing-the-logs/")
 property(name="hostname")
 constant(value=".log")
 }

# Log message format: "timestamp,audit-msg-csv-fields..."
template(name="QumuloAuditFormat" type="list") {
 property(name="timestamp" dateFormat="rfc3339")
 constant(value=",")
 property(name="msg")
 constant(value="\n")
 }

# Filter to catch all Qumulo audit log messages.
if ($app-name startswith "qumulo") then {
 action(type="omfile" dynaFile="QumuloFileName" template="QumuloAuditFormat")
 stop
 }

NOTE: Syslog users must have permission to write to the directory storing the logs or an error will occur. If you do encounter an error, check your local Rsyslog log directory for details.

To apply the new Rsyslog configuration, run the following command on the client:

systemctl restart rsyslog

Lastly, run the following qq command to start logging audit messages to the client:

qq audit_set_config --enable --server-address <syslog-server-hostname> --server-port <port-number>

Audit log messages should now appear in a dedicated log file from each node in the cluster within the directory you specified.

Considerations

• If the connection to the remote syslog server is lost or the configuration of audit is modified, some audit log messages may be lost.
• Depending on your current workflow and configuration, the use of audit logging may potentially impact performance on your cluster.
• For operations that rename/move files, only the new path is displayed in the audit log. 
• Management operations will include specific ids related to the change in cluster configuration (snapshot id, replication relationship id, SMB share id, etc).

TIP! Audit logs can generate a lot of data. Use logrotate (or something similar) to manage the log files on your cluster.

RESOLUTION

You should now be able to successfully use audit logging in Qumulo Core

ADDITIONAL RESOURCES

QQ CLI: Comprehensive List of Commands

Rsyslog

logrotate

Like what you see? Share this article with your network!