Qumulo Core Audit Logging

IN THIS ARTICLE

Outlines how to use audit logging in Qumulo Core  

REQUIREMENTS

• Admin privileges required
• Cluster running Qumulo Core 2.12.0 and above
• Syslog server configured to receive audit logs over TCP
  • You must set up a syslog server that implements syslog protocol using any one of the numerous programs available to receive audit logs. This syslog server must be configured to receive logs over TCP on the port specified in your Qumulo’s audit logging configuration. 

TIP! Qumulo Core generates the audit logs but does not parse, analyze, index or visualize the data. For info on how to capture and correlate real-time data from the logs, check out the Qumulo Core Audit Logging with Splunk, Qumulo Core Audit Logging with Elasticsearch, and Qumulo in AWS: Audit Logging with CloudWatch articles (3.1.1 or above) so that you can search, monitor and examine the results via a web-style interface.  

DETAILS

Audit logging in Qumulo Core provides a mechanism for tracking filesystem operations. As connected clients issue requests to the cluster, log messages are generated describing each attempted operation. These log messages are then sent over the network to the remote syslog instance specified by the current audit configuration in compliance with RFC 5424.

Each audit log message body consists of a few fields in CSV format. Note that the user id and both file path fields are quoted since they may contain characters that need to be escaped in CSV (i.e., quotes and commas). In addition to escaping characters, we also strip all newline characters (“\n” or “\r”) from these three fields.

 The fields are described below in the order that they display within the log message body:

192.168.1.10,"AD\alice",nfs,fs_read_data,ok,123,"/.snapshot/1_snapshot1225/dir/",""

User IP: the IP address of the user in IPV4/IPV6 format

User ID: the user that performed the action

• AD username
• Qumulo local username
• POSIX UID
• Windows SID
• Qumulo auth ID (if we fail to resolve the user ID to any of the previous types)

Logins (2.12.4 and above): any successful or unsuccessful login attempt by the user for the operations below

• Session login via the Web UI 
• Session login via qq CLI
• SMB login
• NFS mount
• FTP login

Protocol: the protocol that the user request came through

• nfs
• smb
• ftp
• api

 File System Operation: the operation that the user attempted

• fs_create_directory
• fs_create_file
• fs_create_hard_link
• fs_create_symlink
• fs_create (a filetype other than one of the types captured above was created)
• fs_delete
• fs_fsstat
• fs_read_metadata
• fs_list_directory
• fs_open
• fs_read_data
• fs_read_link
• fs_rename
• fs_write_data
• fs_write_metadata

Management Operation (2.12.2 and above): any operation (common examples below) that modifies cluster configuration

• auth_create_user
• smb_create_share
• smb_login
• nfs_create_export
• nfs_mount
• snapshot_create_snapshot
• replication_create_source_relationship 

Error Status: “ok” if the operation succeeded or a Qumulo specific error status code (see table below) if the operation failed

• Keep in mind that error status codes are subject to change with new releases of Qumulo Core and may differ depending on the the version you have installed on your cluster. 

File id: the id of the file that the operation was on

File path: the path of the file that the operation was on

• When accessing a file through a snapshot, the path is prefixed with “/.snapshot/<snapshot-directory-name>” which is the same path prefix used to access snapshotted files via nfs and smb.

Secondary file path: any rename/move operations

IMPORTANT! In order to keep the volume of audit log messages to a minimum, similar operations performed in quick succession will be deduplicated. For example, if a user reads the same file 100,000 times in one minute, only one message, corresponding to the first read, will be generated.

Configure Audit Logging via the Web UI (2.12.5 and above)

1. Hover over the Cluster menu and click Audit.
   
   
   
   
2. Fill in the following fields:
   

   Remote Syslog Address: the IP address or URL of your audit data repository
   Port Number: the port number of your audit data repository (default is 514)

   
   
3. Click Save.
4. Verify that the Status shows as "Connected" and that all configuration details are correct on the Audit page.

To change the audit configuration, click the Edit button to modify the settings or select Delete to disable audit logging on your cluster.

Configure Audit Logging via QQ CLI  

Run the following qq command, including a specific IP address (or hostname) and port number, to enable Audit Logging:

qq audit_set_syslog_config --enable --server-address <syslog-server-hostname> --server-port <port-number>

 To disable audit logging, use the following command:

qq audit_set_syslog_config --disable

To review the current configuration for audit logging, run the following:

qq audit_get_syslog_config

Sample Output: 

qq audit_get_syslog_config
{
 "enabled": true,
 "server_address": "syslog-server-hostname",
 "server_port": 514
}

Use the qq command below to check the current state of the connection with the remote syslog instance: 

qq audit_get_syslog_status

The connection_status included in the output will be one of the following: 

• "AUDIT_LOG_CONNECTED": the connection with the remote syslog instance has been established and all log messages should be successfully transmitted.
• "AUDIT_LOG_DISCONNECTED": there is no active connection to the remote syslog instance. The cluster will attempt to buffer all outgoing log messages until the connection can be reestablished, at which point the buffered messages will be sent to the remote syslog instance. If a power outage or reboot occurs, all unsent messages will be lost. If the message buffer fills up, all new messages will be thrown away.
• "AUDIT_LOG_DISABLED": audit logging has been explicitly disabled.

Check out the sample output of an established syslog instance below:

qq audit_get_syslog_status
{
 "connection_status": "AUDIT_LOG_CONNECTED"
}

NOTE: The qq commands for audit logging configuration may be slightly different depending on what version of Qumulo Core you are running. Use the command below to verify the supported qq audit commands on your cluster:

qq -h

To Configure Cluster Auditing Using RSyslog

This section explains how to configure a cluster to forward audit log messages to an RSyslog process on a client machine. In this example, the client runs Ubuntu 18.04.

NOTE: Syslog users must have permission to write to the directory that contains the logs. If an error occurs, see your local RSyslog log directory for details.

1. To allow listening to TCP connections on port 514, ensure that the following lines are uncommented in the /etc/rsyslog.conf file on the client machine.

   # provides TCP syslog reception
   
   module(load="imtcp")
   
   input(type="imtcp" port="514")

2. To configure RSyslog to capture Qumulo audit log messages, create the /etc/rsyslog.d/10-qumulo-audit.conf file and add the log template to it. For example:

   # Log file name:"/<my-logs-directory>/<node-hostname>.log"
   
   template(name="QumuloFileName" type="list") {
   
     constant(value="/<my-logs-directory>/")
   
     property(name="hostname")
   
     constant(value=".log")
   
   }

3. (Optional) To capture Qumulo audit log messages in their own log file and format, add the following template and filter.

   # Log message format: "timestamp,audit-msg-csv-fields..."
   
   template(name="QumuloAuditFormat" type="list") {
   
     property(name="timestamp" dateFormat="rfc3339")
   
     constant(value=",")
   
     property(name="msg")
   
     constant(value="\n")
   
   }
   
   
   
   # Filter to catch all Qumulo audit log messages.
   
   if ($app-name startswith "qumulo") then {
   
     action(type="omfile" dynaFile="QumuloFileName" template="QumuloAuditFormat")
   
     stop
   
   }

4. To apply the new configuration, restart RSyslog on the client machine:

   systemctl restart rsyslog

5. To start forwarding audit messages to the client machine, run the following command on your cluster:

   qq audit_set_syslog_config --enable --server-address <syslog-server-hostname> --server-port <port-number>

Audit log messages from each node in the cluster appear in a dedicated log file in the directory you specified.

To Configure Cluster Auditing Using syslog-ng

This section explains how to configure a cluster to forward audit log messages to an syslog-ng process on a client machine.

NOTES:

• Syslog users must have permission to write to the directory that contains the logs. If an error occurs, see your local RSyslog log directory for details.
• To forward logs from Qumulo to syslog-ng (Open Source Edition 3.4 and higher), you must use the network source driver.

1. To allow listening to TCP connections on port 1514, and to set the destination for the log files, ensure that the following lines are uncommented in the /etc/syslog-ng/syslog-ng.conf file on the client machine.

   # provides TCP syslog reception
   
   source s_network {
   
       network(
   
           transport("tcp")
   
           port("1514")
   
           flags(syslog-protocol)
   
       );
   
   };
   
   
   
   # sets the destination for the log files
   
   destination d_syslog {
   
       file("/var/log/syslogprot");
   
   };
   
   
   
   # passes the configuration parameters
   
   log {
   
       source(s_network);
   
       destination(d_syslog);
   
   };

2. To apply the new configuration, restart syslog-ng on the client machine:

   service syslog-ng restart

3. To start forwarding audit messages to the client machine, run the following command on your cluster:

   qq audit_set_syslog_config --enable --server-address <syslog-server-hostname> --server-port <port-number>

Audit log messages from each node in the cluster appear in a dedicated log file in the directory you specified.

Considerations

• If the connection to the remote syslog server is lost or the audit configuration changes, some audit log messages might be lost.
• Depending on your current workflow and configuration, using audit logging might impact your cluster's performance.
• For operations that rename or move files, the audit log shows only the new path. 
• Management operations include specific IDs related to the cluster configuration changes (a snapshot ID, replication relationship ID, SMB share ID, and so on).

TIP! Audit logs can generate a lot of data. To manage the log files on your cluster, use a program such as logrotate.

RESOLUTION

You should now be able to successfully use audit logging in Qumulo Core

ADDITIONAL RESOURCES

Qumulo in AWS: Audit Logging with CloudWatch

Qumulo Core Audit Logging with Splunk

Qumulo Core Audit Logging with Elasticsearch

QQ CLI: Comprehensive List of Commands

Rsyslog 

logrotate 

Like what you see? Share this article with your network!