Sandbox policy

• 1: Sandbox policy
• 2: Sandboxes

1 - Sandbox policy

Certain plugins run within sandboxes whose available resources are limited and controlled for security and for resource governance.

Sandboxes run on the nodes of your cluster. Some of their limitations are defined in sandbox policies, where each sandbox kind can have its own policy.

Sandbox policies are managed at cluster-level and affect all the nodes in the cluster.

Permissions

You must have AllDatabasesAdmin permissions to run this command.

The policy object

A sandbox policy has the following properties.

• SandboxKind: Defines the type of the sandbox (such as, PythonExecution, RExecution).
• IsEnabled: Defines if sandboxes of this type may run on the cluster’s nodes.
  • The default value is false.
• InitializeOnStartup: Defines whether sandboxes of this type are initialized on startup, or lazily, upon first use.
  • The default value is false. To ensure consistent performance and avoid any delays for running queries following service restart, set this property to true.
• TargetCountPerNode: Defines how many sandboxes of this type are allowed to run on the cluster’s nodes.
  • Values can be between one and twice the number of processors per node.
  • The default value is 16.
• MaxCpuRatePerSandbox: Defines the maximum CPU rate as a percentage of all available cores that a single sandbox can use.
  • Values can be between 1 and 100.
  • The default value is 50.
• MaxMemoryMbPerSandbox: Defines the maximum amount of memory (in megabytes) that a single sandbox can use.
  • For Hyper-V technology sandboxes, values can be between 200 and 32768 (32 GB). The default value is 1024 (1 GB). The maximum memory of all sandboxes on a node (TargetCountPerNode * MaxMemoryMbPerSandbox) is 32768 (32 GB).
  • For legacy sandboxes, values can be between 200 and 65536 (64 GB). The default value is 20480 (20 GB).

If a policy isn’t explicitly defined for a sandbox kind, an implicit policy with the default values and IsEnabled set to true applies.

Example

The following policy sets different limits for PythonExecution and RExecution sandboxes:

[
  {
    "SandboxKind": "PythonExecution",
    "IsEnabled": true,
    "InitializeOnStartup": false,
    "TargetCountPerNode": 4,
    "MaxCpuRatePerSandbox": 55,
    "MaxMemoryMbPerSandbox": 8192
  },
  {
    "SandboxKind": "RExecution",
    "IsEnabled": true,
    "InitializeOnStartup": false,
    "TargetCountPerNode": 2,
    "MaxCpuRatePerSandbox": 50,
    "MaxMemoryMbPerSandbox": 10240
  }
]

Related content

• Sandbox policy management commands

2 - Sandboxes

Kusto can run sandboxes for specific flows that must be run in a secure and isolated environment. Examples of these flows are user-defined scripts that run using the Python plugin or the R plugin.

Sandboxes are run locally (meaning, processing is done close to the data), with no extra latency for remote calls.

Prerequisites and limitations

• Sandboxes must run on VM sizes supporting nested virtualization, which implemented using Hyper-V technology and have no limitations.
• The image for running the sandboxes is deployed to every cluster node and requires dedicated SSD space to run.
  • The estimated size is between 10-20 GB.
  • This affects the cluster’s data capacity, and may affect the cost of the cluster.

Runtime

• A sandboxed query operator may use one or more sandboxes for its execution.
  • A sandbox is only used for a single query and is disposed of once that query completes.
  • When a node is restarted, for example, as part of a service upgrade, all running sandboxes on it are disposed of.
• Each node maintains a predefined number of sandboxes that are ready for running incoming requests.
  • Once a sandbox is used, a new one is automatically made available to replace it.
• If there are no pre-allocated sandboxes available to serve a query operator, it will be throttled until new sandboxes are available. For more information, see Errors. New sandbox allocation could take up to 10-15 seconds per sandbox, depending on the SKU and available resources on the data node.

Sandbox parameters

Some of the parameters can be controlled using a cluster-level sandbox policy, for each kind of sandbox.

• Number of sandboxes per node: The number of sandboxes per node is limited.
  • Requests that are made when there’s no available sandbox will be throttled.
• Initialize on startup: if set to false (default), sandboxes are lazily initialized on a node, the first time a query requires a sandbox for its execution. Otherwise, if set to true, sandboxes are initialized as part of service startup.
  • This means that the first execution of a plugin that uses sandboxes on a node will include a short warm-up period.
• CPU: The maximum rate of CPU a sandbox can consume of its host’s processors is limited (default is 50%).
  • When the limit is reached, the sandbox’s CPU use is throttled, but execution continues.
• Memory: The maximum amount of RAM a sandbox can consume of its host’s RAM is limited.
  • Default memory for Hyper-V technology is 1 GB, and for legacy sandboxes 20 GB.
  • Reaching the limit results in termination of the sandbox, and a query execution error.

Sandbox limitations

• Network: A sandbox can’t interact with any resource on the virtual machine (VM) or outside of it.
  • A sandbox can’t interact with another sandbox.

Errors

VM Sizes supporting nested virtualization

The following table lists all modern VM sizes that support Hyper-V sandbox technology.