Microsoft Sentinel
Cloud and workloads

Transformation Filter Split

In brief

The documentation now requires Microsoft Sentinel data lake onboarding and an Analytics-tier table for split transformations. It also clarifies DCR interactions and warns that deleting a rule immediately stops its data processing.

What Defender admins need to know

Administrators configuring split transformations must meet the new prerequisites and review active ingestion before deleting transformation rules.

Summaries are generated from the documentation change itself.

Documentation change

The comparison below shows only the changed extract. Use the full-page view for complete context.

#Customer intent: As a security engineer, I want to filter and split incoming data during ingestion so that I can reduce noise, optimize costs, and route data to the appropriate storage tier.

As security data volumes continue to grow, organizations face the challenge of balancing cost-effective retention of telemetry used for AI, compliance, and investigations while ensuring that only necessary data is retained in high-performance storage tiers. Use filter and split data transformations in Microsoft Sentinel to address this challenge by modifying data at ingestion time to optimize your data retention strategy.

This article describes how to configure filter and split data transformations without the need to manually create custom Data Collection Rule (DCR) configurations. By tailoringWhen you tailor data ingestion,ingestion by using filter and split data transformationstransformations, you can improve performance and reduce noise.

By using data transformations, you can optimize your security data pipeline by controlling what data is stored and in which tier. Using filter and split transformations provides the following benefits:

  • In the Microsoft Defender portal with unified role-based access control (RBAC), Data (manage) permissions under the Data operations permissions group.

  • For the Microsoft Sentinel workspace, you need the following permissions:

  • Log Analytics Contributor role to provide:role, which provides:

    • Microsoft.OperationalInsights/workspaces/write
    • Microsoft.OperationalInsights/workspaces/tables/write permissions to the Log Analytics workspace.
  • To configure a split transformation:

Supported tables

Filter and split transformations have different table support requirements:requirements. Data Collection Rules (DCRs) determine how data is collected and transformed during ingestion.

  • Filtering: Supported on any table that supports Data Collection Rules (DCRs).DCRs.
  • Splitting: Supported on any table that supports Analytics only ingestion, Data lake only ingestion, and Data Collection Rules (DCRs).

To verify whether a connector's tables support DCRs, see Find your Microsoft Sentinel data connector. Your enterprise ingests millions of firewall log entries daily for threat detection and compliance. Your SOC team needs real-time access to recent logs for active investigations, but must also retain historical logs for regulatory audits. Create a split transformation rule to route real-time data to the Analytics tier and historical data to the Data lake tier.

Configure filter transformation rules

To manage existing rules, select the table and then select either Split rule or Filter rule depending on the rule type you want to manage.

  • To disable a rule, select the Rule status switch to turn off the rule, and then select Save.
  • To delete a rule, note that deleting a transformation rule immediately stops data processing for that rule and can affect active ingestion. To proceed, select Delete.

Verify rules by running KQL queries to confirm that data is ingested correctly and routed to the correct tier.