Alternate Data Streams over SMB

IN THIS ARTICLE 

Outlines how to use Alternate Data Streams over SMB in Qumulo Core

REQUIREMENTS

• Cluster running Qumulo Core version 2.12.2 or above

OVERVIEW

Alternate Data Stream (ADS) is the SMB-centric name for a feature that stores more than one file-data object in a single file-object, or where one or more file-data object is stored in a directory object. Common use-cases of ADS include storing small amounts of extended metadata (e.g., stash display icons for the file) about a particular file or directory so that it stays alongside the standard data, but not act as a part of it.

Most documentation recommends that mission-critical data should not be stored in an ADS since support for the feature varies from one platform to the next. However, Qumulo Core provides full support for the feature, so this is not a concern—data integrity is assured.

Common Use Cases

Windows Use Case: A user downloads a file from the internet via Microsoft Edge onto an SMB share on their Qumulo cluster. This action will result in one named stream being applied to the downloaded file that Windows uses to identify whether or not the file is safe based on its source.

Mac Use Case: A user creates a file on an SMB share stored on their Qumulo cluster. They then open Finder, right-click on the file, and select a colored tag from the menu. Since the Mac Finder uses named streams for tracking, this action will result in two named streams being applied to the file.

Alternate Data Stream Details via the QQ CLI

To view your aggregated data for ADS, use the following qq command and include the directory path, file ID, or Snapshot ID:

qq fs_read_dir_aggegates --path PATH

Details for the total streams and stream data is included in the output:

qq fs_read_dir_aggregates --path /grumpquat

{
 "id": "328580640904",
 "path": "/grumpquat",
 "total_capacity": "16442669170688",
 "total_data": "16185325985792",
 "total_directories": "2260438",
 "total_files": "58247449",
 "total_meta": "257343184896",
 "total_named_stream_data": "209817600",
 "total_named_streams": "28",
 "total_other_objects": "333818",
 "total_symlinks": "104557"
}

TIP! In Qumulo Core 2.12.6 and above, a Named Streams field is available on multiple pages in the Web UI. Use the pop up info displayed when hovering over a bar on the Capacity Explorer page or reference the left-hand summary section on the Integrated Analytics page to see the ADS counts for a specific file or directory.

To list all named streams on a file or directory (2.12.5 and above), use the following qq command and include the system path or file ID:

qq fs_list_named_streams --path PATH
qq fs_list_named_streams --id ID

Remove an existing ADS stream on a file or directory (2.12.5 and above) using the system path or file ID and either the stream ID or stream name as outlined in the commands below:

qq fs_remove_stream --path PATH --stream-name STREAM-NAME
qq fs_remove_stream --path PATH --stream-id STREAM-ID
qq fs_remove_stream --id ID --stream-name STREAM-NAME
qq fs_remove_stream --id ID --stream-id STREAM-ID

Specifying a named stream on a file or directory (2.12.5 and above) is supported with the following list of commands using the optional stream name or stream ID argument:

OPTIONAL ARGUMENTS
  --stream-id STREAM-ID
  --stream-name STREAM-NAME

COMMANDS
  qq fs_file_set_attr
  qq fs_file_get_attr
  qq fs_write
  qq fs_read
  qq fs_punch_hole

CONSIDERATIONS

• Alternate Data Streams will be replicated if both the source and target clusters are running 2.12.2 or above. If both clusters are not on 2.12.2 or above, the Replication Relationship will fail and not resume until both clusters are upgraded. In order to maintain cross Replication protocol version compatibility, please contact your CSM prior to upgrading either cluster to 2.12.2 or above.
• Replicating Alternate Data Streams will be supported without intervention when upgrading from 2.13.0 to the following future quarterly release of Qumulo Core.
• Qumulo Core limits the number of Alternate Data Streams on any file/directory to 31.
• ADS names (includes stream name + stream type over SMB) have a max character length of 255.
• While allowed on NTFS, Qumulo Core does not allow ADS to be created on or be a target of a symlink and does not permit the default (un-named) stream to be renamed.

RESOLUTION 

You should now have an overall understanding of Alternate Data Streams (ADS) over SMB in Qumulo Core

ADDITIONAL RESOURCES

[MS-FSCC] Alternate Data Streams 

NTFS and Alternate Data Streams

QQ CLI: File System

Like what you see? Share this article with your network!