Daily Defender News
Aboutmerill.net
Microsoft Sentinel
Cloud and workloads

Connect Logstash Data Connection Rules

4 September 2026Feature updateAction required+119 / −76 lines

In brief

The documentation now references output plugin v2.5.0, revises supported Logstash versions, adds version 9.0.8, and flags required security updates for listed versions.

What Defender admins need to know

Administrators using the listed Logstash versions should apply the required security updates.

Summaries are generated from the documentation change itself.

Documentation change

View full page diffView on Microsoft Learn

The comparison below shows only the changed extract. Use the full-page view for complete context.

  • Output plugins: Customized sending of collected and processed data to various destinations.

The plugin sends JSON-formatted data to your Log Analytics workspace using the Logs Ingestion API. The data is ingested into custom logs or a standard table.

To set up the plugin, follow these steps:

  • Review the prerequisitesLogstash plugin prerequisites

  • Install the plugin

  • Create a sample file

  • Create the required DCR-related resources

  • Install a supported version of Logstash. The plugin supports the following Logstash versions:

    • 7.0 - 7.17.13
    • 8.0 - 8.9 (these versions require a security update, according to Logstash)
    • 8.11 - 8.15 (these versions require a security update, according to Logstash)
    • 8.19.2 (this version requires a security update, according to Logstash)
    • 9.0.8 (this version requires a security update, according to Logstash)
    • 9.1.10 (this version requires a security update, according to Logstash)
    • 9.2.4 - 9.2.5 (these versions require a security update, according to Logstash)
    • 9.3.3
    • 9.4.0

Install the plugin

The Microsoft Sentinel output plugin is available in the Logstash collection on RubyGemsLogstash collection on RubyGems.

  1. Make sure the referenced file path already exists, then start Logstash.

    The plugin writes ten records to a sample file named sampleFile<epoch seconds>.json in the configured path once there are 10 events to sample or when the Logstash process exits gracefully. For example: c:\temp\sampleFile1648453501.json. Here is part of a sample file that the plugin creates: } }

  2. Copy the output plugin configuration below to your Logstash configuration file. 

    output {
    }
}

  3. Make sure the file path already exists, then start Logstash.

    The plugin writes ten records to a sample file named sampleFile<epoch seconds>.json in the configured path once there are 10 events to sample or when the Logstash process exits gracefully. For example: c:\temp\sampleFile1648453501.json. Here is part of a sample file that the plugin creates:

To ingest the data to a custom table, follow these steps (based on the Send data to Azure Monitor Logs using REST API (Azure portal) tutorial):

  1. Review the prerequisites.

  2. Configure the application.

  3. Add a custom log table.

  4. Parse and filter sample data using the sample file you created in the previous section.

  5. Collect information from the DCR.

  6. Assign permissions to the DCR.

    Skip the Send sample data step.

If you come across any issues, see the Logs Ingestion API troubleshooting steps.

Create DCR resources for ingestion into a standard table

To ingest the data to a standard table like Syslog or CommonSecurityLog, you use a process based on the Send data to Azure Monitor Logs using REST API (Resource Manager templates) tutorial. While the tutorial explains how to ingest data into a custom table, you can easily adjust the process to ingest data into a standard table. The steps below indicate relevant changes in the steps.

  1. Review the prerequisites.

  2. Collect workspace details.

  3. Configure an application.

    Skip the Create new table in Log Analytics workspace step. This step isn't relevant when ingesting data into a standard table, because the table is already defined in Log Analytics.

  4. Create the DCR. In this step:

    • Provide the sample file you created in the previous section.Create a sample file.
    • Use the sample file you created to define the streamDeclarations property. Each of the fields in the sample file should have a corresponding column with the same name and the appropriate type (see the example below).
    • Configure the value of the outputStream property with the name of the standard table instead of the custom table. Unlike custom tables, standard table names don't have the _CL suffix.
    • The prefix of the table name should be Microsoft- instead of Custom-. In this example, the outputStream property value is Microsoft-Syslog.

  5. Assign permissions to a DCR.

    Skip the Send sample data step.

If you come across any issues, see the Logs Ingestion API troubleshooting steps.

Example: DCR that ingests data into the Syslog table

Service principal authentication

To configure the Logstash configuration file to ingest the logs into a custom table using service principal authentication, retrieve thesethe following values:client_id, client_secret, tenant_id, data_collection_endpoint, dcr_id, and stream_name.

Field How to retrieve
client_app_Idclient_id The Application (client) ID value you create in step 3 when you create the DCR resources,create the DCR resources, according to the tutorial you used in this section.Azure portal tutorial or Resource Manager templates tutorial.
client_app_secretclient_secret The client secret value you create in step 5 when you create the DCR resources,create the DCR resources, according to the tutorial you used in this section.Azure portal tutorial or Resource Manager templates tutorial.
tenant_id Your subscription's tenant ID. You can find the tenant ID under Home > Microsoft Entra ID > Overview > Basic Information.
data_collection_endpoint The value of the logsIngestion URI in step 3 when you create the DCR resources,create the DCR resources, according to the tutorial you used in this section.Azure portal tutorial or Resource Manager templates tutorial.
dcr_immutable_iddcr_id The value of the DCR immutableId in step 6 when you create the DCR resources,create the DCR resources, according to the tutorial you used in this section.Azure portal tutorial or Resource Manager templates tutorial.
dcr_stream_namestream_name For custom tables, as explained in step 6 when you create the DCR resources, go to the JSON view of the DCR, and copy the dataFlows > streams property. See the dcr_stream_namestream_name in the example below.Service principal output plugin configuration example. For standard tables, the value is Custom-SyslogStream.

After you retrieve the required values: 

output {
    microsoft-sentinel-log-analytics-logstash-output-plugin {
      client_app_Idclient_id => "<enter your client_app_idclient_id value here>"
      client_app_secretclient_secret => "<enter your client_app_secretclient_secret value here>"
      tenant_id => "<enter your tenant id here>"
      data_collection_endpoint => "<enter your logsIngestion URI here>"
      dcr_immutable_iddcr_id => "<enter your DCR immutableId here>"
      dcr_stream_namestream_name => "<enter your stream name here>"
      create_sample_file=> false
      sample_file_path => "c:\\temp"
    }

#### Managed identity authentication (passwordless)

When `managed_identity` is set to `true`you don't provide service principal credentials (`client_id`, `client_secret`, and `tenant_id`), the plugin authenticates withoutby using [`DefaultAzureCredential`](/azure/developer/java/sdk/authentication/credential-chains#defaultazurecredential-overview) from the Azure SDK. `DefaultAzureCredential` tries a client secret. The plugin automatically detectssequence of authentication methods and uses the appropriate identity mechanism at runtimefirst one that succeeds. In a server environment, the relevant methods are attempted in the followingthis order:

1. **AKS Workload Identity*Environment variables** — If the: Reads credentials from environment variables such as `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, and `AZURE_CLIENT_SECRET` to authenticate as a service principal.
1. **Workload identity**: If the plugin runs on an Azure host with workload identity enabled (for example, AKS with the `AZURE_FEDERATED_TOKEN_FILE` are present (set automatically by AKS)environment variable set), the plugin performs an OIDC token exchange.
2.1. **Azure Arc*Managed identity** —: If the Azure Connected Machine Agent (`azcmagent`) is detected on the host,host has a managed identity enabled, the plugin usesauthenticates by using that identity. This method covers Azure VMs, Virtual Machine Scale Sets, and Azure Arc-enabled servers.

For the full sequence of credentials that `DefaultAzureCredential` attempts, see [Credential chains in the Azure Arc managed identity endpointIdentity library for hybrid and on-premises servers.
3. **IMDS** — Otherwise, the plugin falls back to the Azure Instance Metadata Service (IMDS) for Azure VMs and VMSS.Java](/azure/developer/java/sdk/authentication/credential-chains#defaultazurecredential-overview).

Required configuration for managed identity:

| Field | Description |
| --- | --- |
| `managed_identity` | Boolean, `false` by default. Set to `true` to enable passwordless authentication. |
| `data_collection_endpoint` | String. The logsIngestion URI for your DCE. |
| `dcr_immutable_id``dcr_id` | String. The DCR immutableId. |
| `dcr_stream_name``stream_name` | String. The name of the data stream. |

| `managed_identity_object_id` | Optional. String, empty by default. The object ID of a user-assigned managed identity. Required when the VM has multiple user-assigned identities. Omit for system-assigned managed identity. |

##### Example: System-assigned managedManaged identity

output { microsoft-sentinel-log-analytics-logstash-output-plugin { managed_identity => true data_collection_endpoint => "" dcr_immutable_iddcr_id => "" dcr_stream_name => "" } }


##### Example: User-assigned managed identity

output { microsoft-sentinel-log-analytics-logstash-output-plugin { managed_identity => true managed_identity_object_id => "" data_collection_endpoint => "" dcr_immutable_id => "" dcr_stream_namestream_name => "" } }

Optional configuration

FieldKeyDefault DescriptionDefault value
azure_cloud Used to specify the name of the AzurePublicCloudAzure cloud that is being used. Available values are: AzureCloud, AzureChinaCloud, and AzureUSGovernment.AzureCloud
key_namesAn array of strings. Provide this field if you want to send a subset of the columns to Log Analytics.None (field is empty)
plugin_flush_intervalDefines the maximal time difference (in seconds) between sending two messages to Log Analytics.5
retransmission_timeSets the amount of time in seconds for retransmitting messages once sending failed.10
retransmission_delayThe delay in seconds between each retry attempt when sending log data fails. Increase this value to reduce request rate during throttling (HTTP 429) scenarios.2
compress_dataWhen this field is True, the event data is compressed before using the API. Recommended for high throughput pipelines.Falseenvironment.
proxy Specify which(none)Optional. Base HTTP proxy URL applied to use for all API calls.None (fieldplugin traffic. Format: [http://][user:password@]host:port. When unset, no proxy is empty)used and behavior is unchanged.
proxy_aad Specify which(value of proxy)Optional. HTTP proxy URL to useused only for API calls to Microsoft Entra ID. Overrides theID authentication and token traffic. Falls back to proxy setting.None (field is empty)when unset.
proxy_endpoint Specify which(value of proxy)Optional. HTTP proxy URL to useused only for API callstraffic to the Data Collection Endpoint. Overrides theFalls back to proxy setting.when unset.
keys_to_keep(all)Array of field names to send (subset filtering).
max_retries_num3Max retry attempts for failed sends.
initial_wait_time_seconds1Initial backoff between retries.
connect_timeout_seconds15Timeout for establishing the connection to the ingestion endpoint. Bounds how long an upload can block in the connect phase; a resulting timeout is retried.
write_timeout_seconds60Timeout for sending the request body to the ingestion endpoint. Bounds how long an upload can block in the write phase; a resulting timeout is retried.
max_graceful_shutdown_time_seconds60Max wait for graceful shutdown.
max_waiting_time_for_batch_seconds10Max wait before flushing a batch.
max_waiting_for_unifier_time_seconds10Max wait before flushing the unifier.
max_batch_size10000Maximum number of events per batch. When a batch reaches this size, it's flushed immediately, regardless of the time window.
input_queue_capacity50000Maximum capacity of the input queue. Bounds memory usage under high-volume ingestion. When full, back-pressure is applied to the Logstash pipeline.
internal_queue_capacity500Maximum capacity of the internal queues between batcher, unifier, and sender workers. Bounds memory usage for in-flight batches.
worker_sleep_time_millis10Delay between worker iterations.
batcher_workers_count(auto)Number of batcher threads.
sender_workers_count(auto)Number of sender threads.
unifier_workers_count(auto)Number of unifier threads.
id None (field is empty)A custom identification tag to be added to sent-batches logs.

Restart Logstash

Plugin version history

2.5.0

  • Added optional per-plugin proxy configuration for authentication and ingestion traffic using proxy, proxy_aad, and proxy_endpoint.
  • Updated Netty handler, HTTP, HTTP/2, and DNS components from 4.1.133.Final to 4.1.136.Final.
  • Updated Jackson Databind and Jackson Core from 2.18.6 to 2.18.8.

2.4.0

  • Worker threads now run as bounded, executor-scheduled passes: recoverable exceptions are logged and the worker resumes on the next cycle; fatal JVM errors are logged and re-thrown.
  • Fixed graceful shutdown so in-flight batches are drained (batchers, then unifiers, then senders) before workers stop, bounded by max_graceful_shutdown_time_seconds.
  • Added configurable upload timeouts connect_timeout_seconds (default 15) and write_timeout_seconds (default 60); connect and write timeouts are retried.
  • Added thread ID, exception type, batch size, and DCR stream to batch failure logs.

2.3.3

  • Fixed loss of numeric and boolean type fidelity: fields backed by Logstash's internal JRuby types (for example, ports and byte counts) are now preserved as native JSON numbers and booleans instead of being converted to strings, ensuring reliable ingestion into DCRs with typed columns.

2.3.2

  • Fixed silent worker thread death caused by uncaught exceptions in the worker processing loop.
  • Fixed NullPointerException in SenderWorker when Azure returns a LogsUploadException with a null HTTP response.
  • Added resilient error handling with consecutive error tracking to reduce permanent worker failure.
  • Added optional id configuration value for telemetry.
  • Added DCR stream to sent-batches logging.

2.3.0

  • Enabled functionality with Logstash 9.4.
  • Bumped dependency versions for external libraries (azure-sdk-bom, logback, slf4j, Netty).

2.2.1

  • Adds an info-level logging line when batches are successfully sent.

2.2.0

  • Adds ability to use either new or old configuration values.

2.1.2

  • Documentation updates.

2.1.0

  • Fixed event normalization.

For more information, see JNR regression in Logstash 7.17.0 (Docker).

If your environment's event rate is low, increase the value of plugin_flush_intervalmax_waiting_time_for_batch_seconds and max_waiting_for_unifier_time_seconds to 60 or more. You can monitor the ingestion payload using DCR metrics. For more information on plugin_flush_interval,the waiting time variables, see the Optional configuration table.

Limitations

  • The columns of the input stream in the streamDeclarations property must start with a letter. If you start a column with other characters (for example @ or _), the operation fails.
  • The TimeGenerated datetime field is required. You must include this field in the KQL transform.
  • For additional possible issues, review the Logs Ingestion API troubleshooting steps in the tutorial..