Worker configuration

This section explains the in-depth configuration of workers.

info

Updating the configuration of a worker in Taskurai will trigger the start of new instances of your container and stopping of old instances.

Each running container will receive a terminate signal (see CancellationToken in the command) to shut down gracefully.

Worker deployment configuration

A worker is deployed using a configuration section in the solution yaml.

Reference worker configuration

This is a reference worker configuration sample:

workers:
  TestWorker:
    container:
      imageName: taskurai-worker-sample
      image: mycontainerregistry.azurecr.io/taskurai-worker-sample:latest
      resourceAllocation: Cpu_0_25_Memory_0_5Gi
      server: mycontainerregistry.azurecr.io
      userName: taskuraipulltoken
      passwordSecretReference: containerregistrypassword
    scaling:
      scaleOutTaskCount: 100
      minInstances: 0
      maxInstances: 2
    secrets: 
      - name: containerregistrypassword
        secretReference: containerregistrypassword
    environment:
      - name: EnviromentValue
        value: test123
    options:
      hub: DefaultHub
      autoRegisterCommands: true
      cleanupCompletedTasksAfterSeconds: 2592000
      queueReaderIntervalMsec: 250
      queueReaderIdleIntervalMsec: 1000
      commandTimeoutSeconds: 300
      commandStaleTimeoutSeconds: 60
      staleTasksAfterSeconds: 3600

Worker configuration

Property	Type	Required	Additional requirements	Description
name	string	No	1-20 alphanumeric characters and hyphens	Override worker name (default: worker key)
container	Container	Yes		Container configuration
scaling	Scaling	Yes		Scaling configuration
secrets	List of secrets	No		List of secrets
environment	List of environment variables	No		List of environment variables
commands	List of commands (string)	No		List of commands (Not required, defaults to auto register)
options	Worker options	No		Worker options

Container image configuration

Property	Type	Required	Description
imageName	string	Yes	Container image name
image	string	Yes	Container image
resourceAllocation	Resource allocation	Yes	Container image
server	string	No	The container registry server
userName	string	No	Container Registry Username
passwordSecretReference	string	No	A secret should be made within the worker and referenced here

Taskurai supports:

• Any Linux-based x86-64 (linux/amd64) container image
• Containers from any public or private container registry

Features include:

• There's no required base container image.
• If a container crashes, it automatically restarts.

When using a container image from a private container registry, the following settings are required:

• server
• userName
• passwordSecretReference

tip

When you want to put a new container in usage, you need to:

• Update the image when the tag has changed.
• Deploy the worker again.

Resource allocation configuration

Each worker can be configured to a size as needed. The following settings are supported:

Worker size	Description
Cpu_0_25_Memory_0_5Gi	0.25 vCPU cores / 0.5Gi Memory
Cpu_0_50_Memory_1_0Gi	0.5 vCPU cores / 1Gi Memory
Cpu_0_75_Memory_1_5Gi	0.75 vCPU cores / 1.5Gi Memory
Cpu_1_00_Memory_2_0Gi	1 vCPU core / 2Gi Memory
Cpu_1_25_Memory_2_5Gi	1.25 vCPU cores / 2.5Gi Memory
Cpu_1_50_Memory_3_0Gi	1.5 vCPU cores / 3Gi Memory
Cpu_1_75_Memory_3_5Gi	1.75 vCPU cores / 3.5Gi Memory
Cpu_2_00_Memory_4_0Gi	2 vCPU cores / 4Gi Memory

Scaling configuration

Taskurai manages automatic horizontal scaling of workers, depending on the number of tasks waiting for a worker.

When a worker scales, a new instance of that worker is created. When a worker instance is no longer needed, it is automatically stopped.

You can configure the scaling behavior of a worker using these settings:

Property	Type	Required	Description
scaleOutTaskCount	integer	Yes	The number of jobs waiting for the worker. Each scaleOutTaskCount delta will engage a new instance.
minInstances	integer	Yes	The minimum number of instances (0-400)
maxInstances	integer	Yes	The maximum number of instances (1-400)

Keep the following items in mind:

• Taskurai billing works with RU (request and resource usage), increasing the number of instances, will increase RU usage.
• The Azure resources created in your Azure subscription are charged like this:
  • You aren't billed usage charges for the containers if your worker scales to zero.
  • Instances that are running idle but remain in memory are billed at a lower "idle" rate.
  • If you want to ensure that an instance of a worker is always running (for better response times), set the minimum number of instances to 1 or higher.
  • For the latest information, always check the pricing details.

caution

Consider the use case of your commands and whether concurrent execution of tasks is supported.

In the case your worker commands are calling services that are rate-limited, for example mailing services, consider the correct scaling setting for each worker.

Secret configuration

Each worker can contain secrets to be used in environment variables and the container registry password.

Secret properties:

Property	Type	Required	Description
name	string	Yes	Secret name
secretReference	string	No	Global secret reference (recommended)
value	string	No	Secret value (not recommended)

While it is technically possible to define a secret value directly in the worker configuration, it is highly recommended to use globally defined secrets in Taskurai and reference them in the worker.

options:
  ...
secrets:
  - myglobalcontainerregistrypassword
  - myglobalsecret
workers:
  TestWorker:
    container:
      ...
      passwordSecretReference: containerregistrypassword
      ...
    secrets: 
      - name: containerregistrypassword
        secretReference: myglobalcontainerregistrypassword
      - name: mycontainersecret
        secretReference: myglobalsecret
    environment:
      - name: ENV_WITH_SECRET
        secretReference: mycontainersecret
      - name: ENV_WITH_VALUE
        value: myvalue
      ...

Notice that the global Taskurai secret can only be referenced in the container secret configuration. To use secrets within the container itself, the local secrets defined in the worker should be used.

Environment configuration

Environment variables can be set to be used in containers.

Environment variable properties:

Property	Type	Required	Description
name	string	Yes	Variable name
secretReference	string	No	Reference to a worker secret
value	string	No	Variable value

An environment variable can either be a plain text value or can reference a container secret.

options:
  ...
secrets:
  - myglobalcontainerregistrypassword
  - myglobalsecret
workers:
  TestWorker:
    ...
    secrets: 
      ...
      - name: mycontainersecret
        secretReference: myglobalsecret
    environment:
      - name: ENV_WITH_SECRET
        secretReference: mycontainersecret
      - name: ENV_WITH_VALUE
        value: myvalue
      ...

Command configuration

Each worker should have at least one command. A worker name must be unique in Taskurai and not be shared in different workers.

By default, commands are auto-registered. If you want to manually register commands, you can do so by specifying the command names in the worker configuration and specifying the autoRegisterCommands option as false.

Command names are case-insensitive and should match the workers setup:

workers:
  TestWorker:
    ...
    commands:
      - testCommand
      - echo
    ...

tip

Depending on your scenario, a worker can contain one or more commands.

It is recommended to balance the following principles:

• Single responsibility: Keep workers dedicated to a single task, separating concerns. When a command fails, the impact on the whole system is limited.
• Keep it simple: Workers should be designed to be simple to set up and easy to maintain.
• Fit for purpose: Keep workers small and fast for short-running tasks that are in high demand (sending emails, generating invoices, etc.). Long-running jobs (aggregating data, etc.) can be beefier.
• Shared resources: In some cases, it can be useful for a worker to contain multiple commands that share similar purposes, like sending notification emails of all kinds.

In short, don't design a new monolithic worker. Taskurai is designed to handle many workers at scale.

Worker options

The following settings can be configured:

Property	Type	Required	Default	Description
hub	string	Yes		The name of the hub where the worker must be deployed.
autoRegisterCommands	boolean	No	true	Auto register commands.
cleanupCompletedTasksAfterSeconds	integer	No		Cleanup tasks when complete after seconds.
cleanupStateAfterSeconds	integer	No		Cleanup state after seconds.
commandStaleTimeoutSeconds	double	No	60	Default time in seconds before the command is considered stale, time starts after a command is timed out (not responding to timeout) (default 300, max 7 days).
commandTimeoutSeconds	double	No	300	Default time in seconds before a continuously running command is in timeout (default 300, max 7 days). The time only considers when the command is running continuously (not being suspended).
defaultStateStore	string	No		Default state store name.
inlineStepRetryDelayThresholdSec	double	No	60	When an inline step must be retried, how long may the command wait inline before retrying. When retry timeout exceeds the threshold, the command is suspended and the task will be restarted afterwards (default 60 sec.).
maxConcurrentTasks	integer	No	1	The maximum number of concurrent tasks that are started in single worker instance.
queueReaderIdleIntervalMsec	integer	No	1000	The interval to retrieve new tasks from the queue (milliseconds), -1 for no delay (default 1000 msec). The delay happens between the time no more new tasks can be found and the next lookup operation. While there are no new tasks available, the QueueReaderIdleInterval delay is used.
queueReaderIntervalMsec	integer	No	-1	The interval to retrieve new tasks from the queue (milliseconds), -1 for no delay (default -1). The delay happens between the time new tasks can be found and the next lookup operation. While there are still tasks available, this QueueReaderInterval delay is used.
staleTasksAfterSeconds	integer	No	3600	Default time in seconds where a continuously running task is considered stale and is marked as timed-out (-1 = no stale detection). Should be larger than sum of commandTimeoutSeconds + commandStaleTimeoutSeconds.

Worker runtime configuration

The runtime configuration is used when running the worker locally.

C#

Each worker can be configured using the appsettings.json file:

appsettings.json

{
  ...
  "Taskurai": {
    "IsolationMode": false,
    "WorkerName": "TestWorker",    
    "CommandTimeoutSeconds": 300,
    "MaxConcurrentTasks": 10
  }
  ...
}

For local development, you should use the Development configuration file:

appsettings.Development.json

{
  ...
  "Taskurai": {
    "IsolationMode": true,
    "WorkerName": "TestWorker",    
    "CommandTimeoutSeconds": 300,
    "MaxConcurrentTasks": 10
  }
  ...
}

The following settings can be configured:

Property	Type	Required	Default	Description
HubName	string	Yes		The name of the hub
WorkerName	string	Yes		The name of the worker (same as the deployment name)
IsolationMode	boolean	No	false	Run the worker in isolation mode. This is recommended when developing locally
IsolationKey	string	No		Isolation key used in the isolation mode
CleanupCompletedTasksAfterSeconds	integer	No		Cleanup tasks when complete after seconds.
CleanupStateAfterSeconds	integer	No		Cleanup state after seconds.
CommandStaleTimeoutSeconds	double	No	60	Default time in seconds before the command is considered stale, time starts after a command is timed out (not responding to timeout) (default 300, max 7 days).
CommandTimeoutSeconds	double	No	300	Default time in seconds before the command is in timeout (default 300, max 7 days).
InlineStepRetryDelayThresholdSec	double	No	60	When an inline step must be retried, how long may the command wait inline before retrying. When retry timeout exceeds the threshold, the command is suspended and the task will be restarted afterwards (default 60 sec.).
MaxConcurrentTasks	integer	No	1	The maximum number of concurrent tasks
QueueReaderIdleIntervalMsec	integer	No	1000	The interval to retrieve new tasks from the queue (milliseconds), -1 for no delay (default 1000 msec). The delay happens between the time no more new tasks can be found and the next lookup operation. While there are no new tasks available, the QueueReaderIdleInterval delay is used.
QueueReaderIntervalMsec	integer	No	-1	The interval to retrieve new tasks from the queue (milliseconds), -1 for no delay (default -1). The delay happens between the time new tasks can be found and the next lookup operation. While there are still tasks available, this QueueReaderInterval delay is used.

Command lifetime

Command timeouts are crucial for managing continuously long-running tasks. The CommandTimeoutSeconds setting is the maximum time a command can run before it's considered timed out.

The command timeout applies to the command continuously runtime, not the total time a task is allowed to run. To configure the maximum task duration, use the Execution options in the task configuration.

When a command times out, it's essential to handle the timeout gracefully (respecting cancellation tokens). The command should be designed to be idempotent, meaning it can be safely retried without causing side effects. Consider writing complex commands (having multiple actions) using the Taskurai's Durable Steps & Workflows API (Steps), to make commands fault-tolerant and durable out-of-the-box.

The command should respond to the CancellationToken passed to the command, allowing it to be canceled when the timeout is reached.

Once a command is considered timed out, the CommandStaleTimeoutSeconds is used as a grace period before the command is considered stale. When a command is stale, it's considered to have failed and is no longer processed or responding to cancellation requests.

The command will be marked as timed-out, the worker will be reported as unhealthy and will be restarted.

When a command is expected to run longer than the default timeout, it can be extended at runtime using the Progress or ProgressAsync method on the task context.

Load leveling

Queue-based load leveling uses a buffer between tasks and services to smooth heavy workloads. This is especially beneficial for external downstream services vulnerable to failures during peaks or subject to rate limits.

It helps prevent additional costs by managing occasional spikes without requiring pricier plans.

The approach ensures system reliability, cost-effectiveness, and responsiveness to user demands.

How Taskurai provides load leveling

Taskurai implements load leveling on the level of a worker. Commands that have different load leveling requirements, should be organized in different workers.

Load leveling should be configured using a combination of:

• Maximum worker instances
• Maximum concurrent tasks per worker
• Scale out task count
• Queue reader interval (both in active and idle mode)

Load leveling configuration

info

The values is somewhat arbitrary and should be fine-tuned based on actual workload patterns, the processing time of tasks, and the behavior of the downstream service's rate limiting. Monitoring and adjusting based on real-world performance is crucial for optimal configuration.

The following example configures a worker to handle a rate limit of a downstream service of maximum 2 request / second (120 RPM), handled by maximum two workers.

workers:
  TestWorker:
    ...
    scaling:
      scaleOutTaskCount: 15 # Scale out from 1 -> 2 instances when more than 15 tasks are queued
      minInstances: 1 # Minimum one worker instance
      maxInstances: 2 # Maximum two worker instances
    ...
    options:
      maxConcurrentTasks: 1 # Only allow one concurrent task
      queueReaderIntervalMsec: 1000 # Only poll once a second for a new task (when actively receiving tasks from the queue)
      queueReaderIdleIntervalMsec: 1000 # Only poll once a second for a new task (in idle mode, when no tasks are scheduled in the last polling request)
    ...