Workload groups

• 1: Query consistency policy
• 2: Request limits policy
• 3: Request queuing policy
• 4: Request rate limit policy
• 5: Request rate limits enforcement policy
• 6: Workload groups
• 7: Request classification policy
• 7.1: Request classification policy
• 8: Workload group commands

1 - Query consistency policy

A workload group’s query consistency policy allows specifying options that control the consistency mode of queries.

The policy object

Each option consists of:

• A typed Value - the value of the limit.
• IsRelaxable - a boolean value that defines if the option can be relaxed by the caller, as part of the request’s request properties. Default is true.

The following limits are configurable:

Example

"QueryConsistencyPolicy": {
  "QueryConsistency": {
    "IsRelaxable": true,
    "Value": "Weak"
  },
  "CachedResultsMaxAge": {
    "IsRelaxable": true,
    "Value": "05:00:00"
  }
}

Monitoring

You can monitor the latency of the metadata snapshot age on nodes serving as weak consistency service heads by using the Weak consistency latency metric. For more information, see Query metrics.

Related content

• Workload groups
• Query consistency
• .alter-merge workload_group
• .create-or-alter workload_group command
• .drop workload_group
• .show workload_group command

2 - Request limits policy

A workload group’s request limits policy allows limiting the resources used by the request during its execution.

The policy object

Each limit consists of:

• A typed Value - the value of the limit.
• IsRelaxable - a boolean value that defines if the limit can be relaxed by the caller, as part of the request’s request properties.

The following limits are configurable:

CPU resource usage

Queries can use all the CPU resources within the cluster. By default, when multiple queries are running concurrently, the system employs a fair round-robin approach to distribute resources. This strategy is optimal for achieving high performance with ad-hoc queries. Queries can use all the CPU resources within the Eventhouse. By default, when multiple queries are running concurrently, the system employs a fair round-robin approach to distribute resources. This strategy is optimal for achieving high performance with ad-hoc queries.

However, there are scenarios where you might want to restrict the CPU resources allocated to a specific query. For instance, if you’re running a background job that can accommodate higher latencies. The request limits policy provides the flexibility to specify a lower percentage of threads or nodes to be used when executing distributed subquery operations. The default setting is 100%.

The default workload group

The default workload group has the following policy defined by default. This policy can be altered.

{
  "DataScope": {
    "IsRelaxable": true,
    "Value": "All"
  },
  "MaxMemoryPerQueryPerNode": {
    "IsRelaxable": true,
    "Value": < 50% of a single node's total RAM >
  },
  "MaxMemoryPerIterator": {
    "IsRelaxable": true,
    "Value": 5368709120
  },
  "MaxFanoutThreadsPercentage": {
    "IsRelaxable": true,
    "Value": 100
  },
  "MaxFanoutNodesPercentage": {
    "IsRelaxable": true,
    "Value": 100
  },
  "MaxResultRecords": {
    "IsRelaxable": true,
    "Value": 500000
  },
  "MaxResultBytes": {
    "IsRelaxable": true,
    "Value": 67108864
  },
  "MaxExecutiontime": {
    "IsRelaxable": true,
    "Value": "00:04:00"
  }
}

Example

The following JSON represents a custom requests limits policy object:

{
  "DataScope": {
    "IsRelaxable": true,
    "Value": "HotCache"
  },
  "MaxMemoryPerQueryPerNode": {
    "IsRelaxable": true,
    "Value": 2684354560
  },
  "MaxMemoryPerIterator": {
    "IsRelaxable": true,
    "Value": 2684354560
  },
  "MaxFanoutThreadsPercentage": {
    "IsRelaxable": true,
    "Value": 50
  },
  "MaxFanoutNodesPercentage": {
    "IsRelaxable": true,
    "Value": 50
  },
  "MaxResultRecords": {
    "IsRelaxable": true,
    "Value": 1000
  },
  "MaxResultBytes": {
    "IsRelaxable": true,
    "Value": 33554432
  },
  "MaxExecutiontime": {
    "IsRelaxable": true,
    "Value": "00:01:00"
  }
}

Related content

• Workload groups
• Client request properties
• .alter-merge workload_group
• .create-or-alter workload_group
• .drop workload_group
• .show workload_group command

3 - Request queuing policy

A workload group’s request queuing policy controls queuing of requests for delayed execution, once a certain threshold of concurrent requests is exceeded.

Queuing of requests can reduce the number of throttling errors during times of peak activity. It does so by queuing incoming requests up to a predefined short time period, while polling for available capacity during that time period.

The policy might be defined only for workload groups with a request rate limit policy that limits the max concurrent requests at the scope of the workload group.

Use the .alter-merge workload group management command, to enable request queuing.

The policy object

The policy includes a single property:

• IsEnabled: A boolean indicating if the policy is enabled. The default value is false.

Related content

• Workload groups
• .show workload_group command
• .create-or-alter workload_group command

4 - Request rate limit policy

The workload group’s request rate limit policy lets you limit the number of concurrent requests classified into the workload group, per workload group or per principal.

Rate limits are enforced at the level defined by the workload group’s Request rate limits enforcement policy.

The policy object

A request rate limit policy has the following properties:

Concurrent requests rate limit

A request rate limit of kind ConcurrentRequests includes the following property:

When a request exceeds the limit on maximum number of concurrent requests:

• The request’s state, as presented by System information commands, will be Throttled.
• The error message will include the origin of the throttling and the capacity that’s been exceeded.

The following table shows a few examples of concurrent requests that exceed the maximum limit and the error message that these requests return:

• The HTTP response code will be 429. The subcode will be TooManyRequests.
• The exception type will be QueryThrottledException for queries, and ControlCommandThrottledException for management commands.

Resource utilization rate limit

A request rate limit of kind ResourceUtilization includes the following properties:

When a request exceeds the limit on resources utilization:

• The request’s state, as presented by System information commands, will be Throttled.
• The error message will include the origin of the throttling and the quota that’s been exceeded. For example:

The following table shows a few examples of requests that exceed the resource utilization rate limit and the error message that these requests return:

• The HTTP response code will be 429. The subcode will be TooManyRequests.
• The exception type will be QuotaExceededException.

How consistency affects rate limits

With strong consistency, the default limit on maximum concurrent requests depends on the SKU of the cluster, and is calculated as: Cores-Per-Node x 10. For example, a cluster that’s set up with Azure D14_v2 nodes, where each node has 16 vCores, will have a default limit of 16 x 10 = 160.

With weak consistency, the effective default limit on maximum concurrent requests depends on the SKU of the cluster and number of query heads, and is calculated as: Cores-Per-Node x 10 x Number-Of-Query-Heads. For example, a cluster that’s set up with Azure D14_v2 and 5 query heads, where each node has 16 vCores, will have an effective default limit of 16 x 10 x 5 = 800. With strong consistency, the default limit on maximum concurrent requests depends on the SKU of the eventhouse, and is calculated as: Cores-Per-Node x 10. For example, a eventhouse that’s set up with Azure D14_v2 nodes, where each node has 16 vCores, will have a default limit of 16 x 10 = 160.

With weak consistency, the effective default limit on maximum concurrent requests depends on the SKU of the eventhouse and number of query heads, and is calculated as: Cores-Per-Node x 10 x Number-Of-Query-Heads. For example, a eventhouse that’s set up with Azure D14_v2 and 5 query heads, where each node has 16 vCores, will have an effective default limit of 16 x 10 x 5 = 800.

For more information, see Query consistency.

The default workload group

The default workload group has the following policy defined by default. This policy can be altered.

[
  {
    "IsEnabled": true,
    "Scope": "WorkloadGroup",
    "LimitKind": "ConcurrentRequests",
    "Properties": {
      "MaxConcurrentRequests": < Cores-Per-Node x 10 >
    }
  }
]

Examples

The following policies allow up to:

• 500 concurrent requests for the workload group.
• 25 concurrent requests per principal.
• 50 requests per principal per hour.

[
  {
    "IsEnabled": true,
    "Scope": "WorkloadGroup",
    "LimitKind": "ConcurrentRequests",
    "Properties": {
      "MaxConcurrentRequests": 500
    }
  },
  {
    "IsEnabled": true,
    "Scope": "Principal",
    "LimitKind": "ConcurrentRequests",
    "Properties": {
      "MaxConcurrentRequests": 25
    }
  },
  {
    "IsEnabled": true,
    "Scope": "Principal",
    "LimitKind": "ResourceUtilization",
    "Properties": {
      "ResourceKind": "RequestCount",
      "MaxUtilization": 50,
      "TimeWindow": "01:00:00"
    }
  }
]

The following policies will block all requests classified to the workload group:

[
  {
    "IsEnabled": true,
    "Scope": "WorkloadGroup",
    "LimitKind": "ConcurrentRequests",
    "Properties": {
      "MaxConcurrentRequests": 0
    }
  },
]

Related content

• Workload groups
• Request rate limits enforcement policy
• .alter-merge workload_group command
• .create-or-alter workload_group command
• .drop workload_group command
• .show workload_group command

5 - Request rate limits enforcement policy

A workload group’s request rate limits enforcement policy controls how request rate limits are enforced.

The policy object

A request rate limit policy has the following properties:

Request rate limits enforcement level

Request rate limits can be enforced at one of the following levels:

• Cluster:

  • Rate limits are enforced by the single cluster admin node.

• Database:

  • Rate limits are enforced by the database admin node that manages the database the request was sent to.
  • If there are multiple database admin nodes, the configured rate limit is effectively multiplied by the number of database admin nodes.

• QueryHead:

  • Rate limits for queries are enforced by the query head node that the query was routed to.
  • This option affects queries that are sent with either strong or weak query consistency.
    • Strongly consistent queries run on the database admin node, and the configured rate limit is effectively multiplied by the number of database admin nodes.
    • For weakly consistent queries, the configured rate limit is effectively multiplied by the number of query head nodes.
  • This option doesn’t apply to management commands.

• Cluster:

  • Rate limits are enforced by the single Eventhouse admin node.

• Database:

  • Rate limits are enforced by the database admin node that manages the database the request was sent to.
  • If there are multiple database admin nodes, the configured rate limit is effectively multiplied by the number of database admin nodes.

• QueryHead:

  • Rate limits for queries are enforced by the query head node that the query was routed to.
  • This option affects queries that are sent with either strong or weak query consistency.
    • Strongly consistent queries run on the database admin node, and the configured rate limit is effectively multiplied by the number of database admin nodes.
    • For weakly consistent queries, the configured rate limit is effectively multiplied by the number of query head nodes.
  • This option doesn’t apply to management commands.

Examples

Setup

• The cluster has 10 nodes as follows:

  • one cluster admin node.
  • two database admin nodes (each manages 50% of the cluster’s databases).
  • 50% of the tail nodes (5 out of 10) can serve as query heads for weakly consistent queries.

• The default workload group is defined with the following policies:

    "RequestRateLimitPolicies": [
        {
            "IsEnabled": true,
            "Scope": "WorkloadGroup",
            "LimitKind": "ConcurrentRequests",
            "Properties": {
                "MaxConcurrentRequests": 200
            }
        }
    ],
    "RequestRateLimitsEnforcementPolicy": {
        "QueriesEnforcementLevel": "QueryHead",
        "CommandsEnforcementLevel": "Database"
    }

Effective rate limits

The effective rate limits for the default workload group are:

• The maximum number of concurrent cluster-scoped management commands is 200.

• The maximum number of concurrent database-scoped management commands is
  2 (database admin nodes) x 200 (max per admin node) = 400.

• The maximum number of concurrent strongly consistent queries is
  2 (database admin nodes) x 200 (max per admin node) = 400.

• The maximum number of concurrent weakly consistent queries is
  5 (query heads) x 200 (max per query head) = 1000.

• The maximum number of concurrent eventhouse-scoped management commands is 200.

• The maximum number of concurrent database-scoped management commands is
  2 (database admin nodes) x 200 (max per admin node) = 400.

• The maximum number of concurrent strongly consistent queries is
  2 (database admin nodes) x 200 (max per admin node) = 400.

• The maximum number of concurrent weakly consistent queries is
  5 (query heads) x 200 (max per query head) = 1000.

Related content

• Workload groups
• System information
• .alter-merge workload_group command
• .create-or-alter workload_group command
• .drop workload_group command
• .show workload_group command

6 - Workload groups

Workload groups allow you to group together sets of management commands and queries based on shared characteristics, and apply policies to control per-request limits and request rate limits for each of these groups.

Together with workload group policies, workload groups serve as a resource governance system for incoming requests to the cluster. When a request is initiated, it gets classified into a workload group. The classification is based on a user-defined function defined as part of a request classification policy. The request follows the policies assigned to the designated workload group throughout its execution.

Workload groups are defined at the cluster level, and up to 10 custom groups can be defined in addition to the three built-in workload groups. Together with workload group policies, workload groups serve as a resource governance system for incoming requests to the eventhouse. When a request is initiated, it gets classified into a workload group. The classification is based on a user-defined function defined as part of a request classification policy. The request follows the policies assigned to the designated workload group throughout its execution.

Workload groups are defined at the eventhouse level, and up to 10 custom groups can be defined in addition to the three built-in workload groups.

Use cases for custom workload groups

The following list covers some common use cases for creating custom workload groups:

• Protect against runaway queries: Create a workload group with a requests limits policy to set restrictions on resource usage and parallelism during query execution. For example, this policy can regulate result set size, memory per iterator, memory per node, execution time, and CPU resource usage.

• Control the rate of requests: Create a workload group with a request rate limit policy to manage the behavior of concurrent requests from a specific principal or application. This policy can restrict the number of concurrent requests, request count within a time period, and total CPU seconds per time period. While your cluster comes with default limits, such as query limits, you have the flexibility to adjust these limits based on your requirements.

• Create shared environments: Imagine a scenario where you have 3 different customer teams running queries and commands on a shared cluster, possibly even accessing shared databases. If you’re billing these teams based on their resource usage, you can create three distinct workload groups, each with unique limits. These workload groups would allow you to effectively manage and monitor the resource usage of each customer team.

• Control the rate of requests: Create a workload group with a request rate limit policy to manage the behavior of concurrent requests from a specific principal or application. This policy can restrict the number of concurrent requests, request count within a time period, and total CPU seconds per time period. While your eventhouse comes with default limits, such as query limits, you have the flexibility to adjust these limits based on your requirements.

• Create shared environments: Imagine a scenario where you have 3 different customer teams running queries and commands on a shared eventhouse, possibly even accessing shared databases. If you’re billing these teams based on their resource usage, you can create three distinct workload groups, each with unique limits. These workload groups would allow you to effectively manage and monitor the resource usage of each customer team.

• Monitor resources utilization: Workload groups can help you create periodic reports on the resource consumption of a given principal or application. For instance, if these principals represent different clients, such reports can facilitate accurate billing. For more information, see Monitor requests by workload group.

Create and manage workload groups

Use the following commands to manage workload groups and their policies:

• .alter-merge workload_group
• .create-or-alter workload_group
• .drop workload_group
• .show workload_group

Workload group policies

The following policies can be defined per workload group:

• Request limits policy
• Request rate limit policy
• Request rate limits enforcement policy
• Request queuing policy
• Query consistency policy

Built-in workload groups

The pre-defined workload groups are:

• internal workload group
• default workload group
• $materialized-views workload group

Default workload group

Requests are classified into the default group under these conditions:

• There are no criteria to classify a request.
• An attempt was made to classify the request into a non-existent group.
• A general classification failure has occurred.

You can:

• Change the criteria used for routing these requests.
• Change the policies that apply to the default workload group.
• Classify requests into the default workload group.

To monitor what gets classified to the default workload group, see Monitor requests by workload group.

Internal workload group

The internal workload group is populated with requests that are for internal use only.

You can’t:

• Change the criteria used for routing these requests.
• Change the policies that apply to the internal workload group.
• Classify requests into the internal workload group.

To monitor what gets classified to the internal workload group, see Monitor requests by workload group.

Materialized views workload group

The $materialized-views workload group applies to the materialized views materialization process. For more information on how materialized views work, see Materialized views overview.

You can change the following values in the workload group’s request limits policy:

• MaxMemoryPerQueryPerNode
• MaxMemoryPerIterator
• MaxFanoutThreadsPercentage
• MaxFanoutNodesPercentage

Monitor requests by workload group

System commands indicate the workload group into which a request was classified. You can use these commands to aggregate resources utilization by workload group for completed requests.

The same information can also be viewed and analyzed in Azure Monitor insights.

Related content

• Requests classification policy

7 - Request classification policy

7.1 - Request classification policy

The classification process assigns incoming requests to a workload group, based on the characteristics of the requests. Tailor the classification logic by writing a user-defined function, as part of a cluster-level request classification policy. The classification process assigns incoming requests to a workload group, based on the characteristics of the requests. Tailor the classification logic by writing a user-defined function, as part of an Eventhouse-level request classification policy.

In the absence of an enabled request classification policy, all requests are classified into the default workload group.

Policy object

The policy has the following properties:

• IsEnabled: bool - Indicates if the policy is enabled or not.
• ClassificationFunction: string - The body of the function to use for classifying requests.

Classification function

The classification of incoming requests is based on a user-defined function. The results of the function are used to classify requests into existing workload groups.

The user-defined function has the following characteristics and behaviors:

• If IsEnabled is set to true in the policy, the user-defined function is evaluated for every new request.
• The user-defined function gives workload group context for the request for the full lifetime of the request.
• The request is given the default workload group context in the following situations:
  • The user-defined function returns an empty string, default, or the name of nonexistent workload group.
  • The function fails for any reason.
• Only one user-defined function can be designated at any given time.

Requirements and limitations

A classification function:

• Must return a single scalar value of type string. That is the name of the workload group to assign the request to.
• Must not reference any other entity (database, table, or function).
  • Specifically - it might not use the following functions and operators:
    • cluster()
    • database()
    • table()
    • external_table()
    • externaldata
• Has access to a special dynamic symbol, a property-bag named request_properties, with the following properties:

Examples

A single workload group

iff(request_properties.current_application == "Kusto.Explorer" and request_properties.request_type == "Query",
    "Ad-hoc queries",
    "default")

Multiple workload groups

case(current_principal_is_member_of('aadgroup=somesecuritygroup@contoso.com'), "First workload group",
     request_properties.current_database == "MyDatabase" and request_properties.current_principal has 'aadapp=', "Second workload group",
     request_properties.current_application == "Kusto.Explorer" and request_properties.request_type == "Query", "Third workload group",
     request_properties.current_application == "Kusto.Explorer", "Third workload group",
     request_properties.current_application == "KustoQueryRunner", "Fourth workload group",
     request_properties.request_description == "this is a test", "Fifth workload group",
     hourofday(now()) between (17 .. 23), "Sixth workload group",
     "default")

Management commands

Use the following management commands to manage a cluster’s request classification policy.

Related content

• Workload groups
• Request properties

8 - Workload group commands