Qumulo Core Audit Logging

REQUIREMENTS

Admin privileges
Cluster running Qumulo Core 2.12.0 and above
Syslog server configured to receive audit logs over TCP
- You must set up a syslog server that implements syslog protocol using any one of the numerous programs available to receive audit logs. This syslog server must be configured to receive logs over TCP on the port specified in your Qumulo audit logging configuration.

TIP! Qumulo Core generates the audit logs but does not parse, analyze, index or visualize the data. For info on how to capture and correlate real-time data from the logs, check out the Qumulo Core Audit Logging with Splunk, Qumulo Core Audit Logging with Elasticsearch, and Qumulo in AWS: Audit Logging with CloudWatch articles (3.1.1 or above) so that you can search, monitor and examine the results via a web-style interface.

DETAILS

Audit logging in Qumulo Core provides a mechanism for tracking filesystem operations. As connected clients issue requests to the cluster, log messages are generated describing each attempted operation. These log messages are then sent over the network to the remote syslog instance specified by the current audit configuration in compliance with RFC 5424.

Each audit log message body consists of a few fields in CSV format. Note that the user id and both file path fields are quoted since they may contain characters that need to be escaped in CSV (i.e., quotes and commas). In addition to escaping characters, we also strip all newline characters (“\n” or “\r”) from these three fields.

The fields are described below in the order that they display within the log message body:

192.168.1.10,"AD\alice",nfs,fs_read_data,ok,123,"/.snapshot/1_snapshot1225/dir/",""

User IP: the IP address of the user in IPv4/IPv6 format

User ID: the user that performed the action

AD username
Qumulo local username
POSIX UID
Windows SID
Qumulo auth ID (if we fail to resolve the user ID to any of the previous types)

Logins (2.12.4 and above): any successful or unsuccessful login attempt by the user for the operations below:

Session login via the Web UI
Session login via qq CLI
SMB login
NFS mount
FTP login

Protocol: the protocol that the user request came through

nfsv3
nfsv4.1
smb
ftp
api

File System Operation: the operation that the user attempted

fs_create_directory
fs_create_file
fs_create_hard_link
fs_create_symlink
fs_create (a filetype other than one of the types captured above was created)
fs_delete
fs_fsstat
fs_read_metadata
fs_list_directory
fs_open
fs_read_data
fs_read_link
fs_rename
fs_write_data
fs_write_metadata

Management Operation (2.12.2 and above): any operation (common examples below) that modifies cluster configuration

auth_create_user
smb_create_share
smb_login
nfs_create_export
nfs_mount
snapshot_create_snapshot
replication_create_source_relationship

Error Status: “ok” if the operation succeeded or a Qumulo specific error status code (see table below) if the operation failed

Keep in mind that error status codes are subject to change with new releases of Qumulo Core and may differ depending on the the version you have installed on your cluster.

Error Status	Details
ok	The operation was successful.
fs_no_such_path_error	The directory portion of the path contains a name that doesn't exist.
fs_no_space_error	The file system had no available space. Your cluster is 100% full.
fs_invalid_file_type_error	The operation isn't valid for this filetype.
fs_not_a_file_error	The operation (e.g. read) is only valid for a file.
fs_sharing_violation_error	The file or directory is opened by another party in an exclusive manner.
fs_no_such_entry_error	The directory of file or inode name doesn't exist in the file system.
fs_access_denied_error	The user does not have access to perform the operation.
fs_access_perm_not_owner_error	The user would need superuser or owner access to perform the operation.
fs_entry_exists_error	A file system object by the given name already exists.
fs_directory_not_empty_error	The directory cannot be removed, because it is not empty.
fs_no_such_inode_error	The file system object does not exist.
http_unauthorized_error	The user does not have access to perform the management operation.
share_fs_path_doesnt_exist_error	The directory does not exist on Qumulo.
decode_error	Invalid json was passed to the API.

File id: the id of the file that the operation was on

File path: the path of the file that the operation was on

When accessing a file through a snapshot, the path is prefixed with “/.snapshot/<snapshot-directory-name>” which is the same path prefix used to access snapshotted files using nfsv3 and smb.

Secondary file path: any rename/move operations

IMPORTANT! In order to keep the volume of audit log messages to a minimum, similar operations performed in quick succession will be deduplicated. For example, if a user reads the same file 100,000 times in one minute, only one message, corresponding to the first read, will be generated.

Configure Audit Logging via the Web UI (2.12.5 and above)

Hover over the Cluster menu and click Audit.
Fill in the following fields:

Remote Syslog Address: the IP address or URL of your audit data repository
Port Number: the port number of your audit data repository (default is 514)
Click Save.
Verify that the Status shows as "Connected" and that all configuration details are correct on the Audit page.

To change the audit configuration, click the Edit button to modify the settings or select Delete to disable audit logging on your cluster.

Configure Audit Logging via QQ CLI

Run the following qq command, including a specific IP address (or hostname) and port number, to enable Audit Logging:

qq audit_set_syslog_config \
  --enable
  --server-address <syslog-server-hostname>
  --server-port <port-number>

To disable audit logging, use the following command:

qq audit_set_syslog_config \
  --disable

To review the current configuration for audit logging, run the following:

qq audit_get_syslog_config

Sample Output:

qq audit_get_syslog_config

{
  "enabled": true,
  "server_address": "syslog-server-hostname",
  "server_port": 514
}

Use the qq command below to check the current state of the connection with the remote syslog instance:

qq audit_get_syslog_status

The connection_status included in the output will be one of the following:

AUDIT_LOG_CONNECTED: the connection with the remote syslog instance has been established and all log messages should be successfully transmitted.
AUDIT_LOG_DISCONNECTED: there is no active connection to the remote syslog instance. The cluster will attempt to buffer all outgoing log messages until the connection can be reestablished, at which point the buffered messages will be sent to the remote syslog instance. If a power outage or reboot occurs, all unsent messages will be lost. If the message buffer fills up, all new messages will be thrown away.
AUDIT_LOG_DISABLED: audit logging has been explicitly disabled.

Check out the sample output of an established syslog instance below:

qq audit_get_syslog_status

{
  "connection_status": "AUDIT_LOG_CONNECTED"
}

NOTE: The qq commands for audit logging configuration may be slightly different depending on what version of Qumulo Core you are running. Use the command below to verify the supported qq audit commands on your cluster:

qq -h

To Configure Cluster Auditing Using RSyslog

This section explains how to configure a cluster to forward audit log messages to an RSyslog process on a client machine. In this example, the client runs Ubuntu 18.04.

NOTE: Syslog users must have permission to write to the directory that contains the logs. If an error occurs, see your local RSyslog log directory for details.

To allow listening to TCP connections on port 514, ensure that the following lines are uncommented in the /etc/rsyslog.conf file on the client machine.
```
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
```

To configure RSyslog to capture Qumulo audit log messages, create the /etc/rsyslog.d/10-qumulo-audit.conf file and add the log template to it. For example:

# Log file name:"/<my-logs-directory>/<node-hostname>.log"
template(name="QumuloFileName" type="list") {
  constant(value="/<my-logs-directory>/")
  property(name="hostname")
  constant(value=".log")
}

(Optional) To capture Qumulo audit log messages in their own log file and format, add the following template and filter.

# Log message format: "timestamp,audit-msg-csv-fields..."
template(name="QumuloAuditFormat" type="list") {
  property(name="timestamp" dateFormat="rfc3339")
  constant(value=",")
  property(name="msg")
  constant(value="\n")
}

# Filter to catch all Qumulo audit log messages.
if ($app-name startswith "qumulo") then {
  action(type="omfile" dynaFile="QumuloFileName" template="QumuloAuditFormat")
  stop
}

To apply the new configuration, restart RSyslog on the client machine:
```
systemctl restart rsyslog
```

To start forwarding audit messages to the client machine, run the following command on your cluster:

qq audit_set_syslog_config \
  --enable \
  --server-address <syslog-server-hostname> \
  --server-port <port-number>

Audit log messages from each node in the cluster appear in a dedicated log file in the directory you specified.

To Configure Cluster Auditing Using syslog-ng

This section explains how to configure a cluster to forward audit log messages to an syslog-ng process on a client machine.

NOTES:

Syslog users must have permission to write to the directory that contains the logs. If an error occurs, see your local RSyslog log directory for details.
To forward logs from Qumulo to syslog-ng (Open Source Edition 3.4 and higher), you must use the network source driver.

To allow listening to TCP connections on port 1514, and to set the destination for the log files, ensure that the following lines are uncommented in the /etc/syslog-ng/syslog-ng.conf file on the client machine.

# provides TCP syslog reception
source s_network {
  network(
    transport("tcp")
    port("1514")
    flags(syslog-protocol)
  );
};

# sets the destination for the log files
destination d_syslog {
  file("/var/log/syslogprot");
};

# passes the configuration parameters
log {
  source(s_network);
  destination(d_syslog);
};

To apply the new configuration, restart syslog-ng on the client machine:
```
service syslog-ng restart
```

To start forwarding audit messages to the client machine, run the following command on your cluster:

qq audit_set_syslog_config \
  --enable \
  --server-address <syslog-server-hostname> \
  --server-port <port-number>

Audit log messages from each node in the cluster appear in a dedicated log file in the directory you specified.

Considerations

If the connection to the remote syslog server is lost or the audit configuration changes, some audit log messages might be lost.
Depending on your current workflow and configuration, using audit logging might impact your cluster's performance.
For operations that rename or move files, the audit log shows only the new path.
Management operations include specific IDs related to the cluster configuration changes (a snapshot ID, replication relationship ID, SMB share ID, and so on).

Tip: Audit logs can generate a lot of data. To manage the log files on your cluster, use a program such as logrotate.

RESOLUTION

You can successfully use audit logging in Qumulo Core

ADDITIONAL RESOURCES

Qumulo in AWS: Audit Logging with CloudWatch

Qumulo Core Audit Logging with Splunk

Qumulo Core Audit Logging with Elasticsearch

QQ CLI: Comprehensive List of Commands

Rsyslog

logrotate

Like what you see? Share this article with your network!

Was this article helpful?

0 out of 0 found this helpful

Comments

1 comment

Stefan Radtke
April 19, 2019 08:59

Hi Lindsie,

you may consider to mention in the beginning that using a syslog-server as a receiver is one option (in fact, the most basic one). There are many more options like SIEM tools, Splunk, logstash. They all typically can receive syslog messages and do much more with them like indexing, doing analysis , visualisation, alarms etc.

0

Please sign in to leave a comment.