azure_blob_storage
Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the container field.
Introduced in version 3.36.0.
In order to have a different path for each object you should use function interpolations described here, which are calculated per message of a batch.
Supports multiple authentication methods but only one of the following is required:
-
storage_connection_string -
storage_accountandstorage_access_key -
storage_accountandstorage_sas_token -
storage_accountto access via DefaultAzureCredential
If multiple are set then the storage_connection_string is given priority.
If the storage_connection_string does not contain the AccountName parameter, please specify it in the
storage_account field.
Tags
The tags field allows you to attach key/value pairs to blobs as blob index tags, where the values support interpolation functions:
output:
azure_blob_storage:
container: TODO
path: ${!counter()}-${!timestamp_unix_nano()}.txt
tags:
Environment: production
Source: ${!@kafka_topic}
Keys and values can contain only alphanumerics, spaces, and the characters + - . / : = _. Only values support interpolation. Keys are static and are checked against Azure’s limits at startup. Values are resolved and checked per message, and a message with an invalid value fails before any upload.
If authenticating with storage_sas_token, the token must include the t (tags) permission.
When blob_type is APPEND the tags describe the blob rather than any single message, so they are applied only when this output creates the blob, and are not re-sent with subsequent appends. Two consequences are worth knowing:
-
A blob that already exists is never tagged, including one created by an earlier run of the pipeline. Appending to a long-lived blob therefore only picks up tags if this output created it.
-
Interpolated tag values resolve from the message that creates the blob.
Performance
This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field max_in_flight.
-
Common
-
Advanced
output:
label: ""
azure_blob_storage:
storage_account: ""
storage_access_key: ""
storage_connection_string: ""
storage_sas_token: ""
container: "" # No default (required)
path: ${!counter()}-${!timestamp_unix_nano()}.txt
max_in_flight: 64
output:
label: ""
azure_blob_storage:
storage_account: ""
storage_access_key: ""
storage_connection_string: ""
storage_sas_token: ""
container: "" # No default (required)
path: ${!counter()}-${!timestamp_unix_nano()}.txt
tags: {}
blob_type: BLOCK
public_access_level: PRIVATE
max_in_flight: 64
Fields
blob_type
Block and Append blobs are made up of blocks, and each blob can support up to 50,000 blocks. The default value is BLOCK.
This field supports interpolation functions.
Type: string
Default: BLOCK
Options: BLOCK, APPEND
container
The container for uploading the messages to.
This field supports interpolation functions.
Type: string
# Examples:
container: messages-${!timestamp("2006")}
max_in_flight
The maximum number of messages to have in flight at a given time. For outputs that send messages in batches, this limit applies to message batches. Increase this value to improve throughput.
Type: int
Default: 64
path
The path of each message to upload.
This field supports interpolation functions.
Type: string
Default: ${!counter()}-${!timestamp_unix_nano()}.txt
# Examples:
path: ${!counter()}-${!timestamp_unix_nano()}.json
# ---
path: ${!meta("kafka_key")}.json
# ---
path: ${!json("doc.namespace")}/${!json("doc.id")}.json
public_access_level
The container’s public access level. The default value is PRIVATE.
This field supports interpolation functions.
Type: string
Default: PRIVATE
Options: PRIVATE, BLOB, CONTAINER
storage_access_key
The access key for the storage account. Use this field along with storage_account for authentication. This field is ignored when the storage_connection_string field is populated.
Type: string
Default: ""
storage_account
The storage account to access. This field is ignored when the storage_connection_string field is populated, unless the connection string does not contain the AccountName parameter.
Type: string
Default: ""
storage_connection_string
The connection string for the storage account. This field is required if storage_account is not set.
If the storage_connection_string field does not contain the AccountName parameter value, specify it in the storage_account field.
|
Type: string
Default: ""
storage_sas_token
The SAS token for the storage account. Use this field along with storage_account for authentication. This field is ignored when either the storage_connection_string or storage_access_key fields are populated.
Type: string
Default: ""
tags
Key/value pairs to store with the blob as blob index tags. A maximum of 10 tags are permitted, tag keys must be between 1 and 128 characters, and tag values must be between 0 and 256 characters. Keys and values are case-sensitive and only support string values. Not supported on storage accounts with hierarchical namespace (Data Lake Gen2).
This field supports interpolation functions.
Requires version 4.118.0 or later.
Type: object<string>
Default: {}
# Examples:
tags:
Environment: production
Source: ${!@kafka_topic}