Skip to main content

Log monitoring

Taskurai is integrated with Azure Monitor Log Analytics to monitor and analyze the worker logs.

There are two types of logs for Taskurai workers:

  • Console logs: These are emitted by the worker.
  • System logs: These are emitted by the container service.

Prerequisites​

Console Logs​

The Taskurai service provides console log messages at the worker level.

The console log data is accessible by querying the TaskuraiConsoleLogs table. Common columns in the table are:

ColumnTypeDescription
MessageStringMessage
TimestampRealTimestamp
TimeGeneratedDateTimeTime generated (UTC)
TimeDateTimeTime (UTC)
TaskIdStringTask ID
TaskRunCountLongTask run count
TaskCorrelationIdStringTask correlation ID
CommandStringCommand name
WorkerNameStringWorker name
WorkerRevisionStringWorker revision
ContainerImageStringContainer Image
SourceContextStringSource context
PlainMessageStringPlain message
CustomColumnsDynamicUser defined columns (structured logging)

Console Logs​

The Taskurai service provides system log messages at the worker level.

The system log data is accessible by querying the TaskuraiSystemLogs table. Common columns in the table are:

ColumnTypeDescription
MessageStringMessage
TimestampRealTimestamp
TimeGeneratedDateTimeTime generated (UTC)
TimeDateTimeTime (UTC)
CountLongCount
TypeStringMessage type
LevelStringLog level
ReasonStringReason
ContainerAppNameStringContainer app name
WorkerRevisionStringWorker revision
WorkerInstanceStringWorker instance

Querying Taskurai's logs​

The Kusto Query Language is used to query and analyse log data. Learn more about the Kusto Query Language (KQL).

In your application, you can create one or more tasks to offload workloads to asynchronous commands. You can store the returned task ID to provide updates to the end-user about the progress and outcome of the task.

API reference:

Required scopes:

  • Taskurai.Log.Query.Read

Querying logs​

var logsClient = taskurai.GetLogsClient();

var query = "TaskuraiConsoleLogs | summarize Count = count() by Command | top 10 by Count | project Command, Count";

var listMessagesResponse = await logsClient.QueryMessagesAsync(query, new ListLogOptions()
{
    OutputJson = true,
    Timerange = new LogTimeRange(TimeSpan.FromDays(5))
});

if (listMessagesResponse.Messages != null)
{
    Console.WriteLine(JsonSerializer.Serialize(listMessagesResponse.Messages));
}
else
{
    Console.WriteLine(listMessagesResponse.PlainText);
}

Next, run the console application to validate:

dotnet run

Sample output:

[
  {
    "message": "[\"testCommand@2\",1228]",
    "customColumns": {
      "command": "testCommand@2",
      "count": 1228
    }
  }
]

Using custom columns​

When custom columns are introduced in structured logging, these columns can be used in your Kusto queries:

TaskuraiConsoleLogs | extend TaskuraiSDKVersion = CustomColumns[\"TaskuraiSDKVersion_s\"] | where TaskuraiSDKVersion == '1.10.0.0'

Listing console logs​

Use this command to easily list the console logs of the Taskurai environment.

API reference:

Required scopes:

  • Taskurai.Log.Console.Read
var logsClient = taskurai.GetLogsClient();

var listConsoleMessagesResponse = await logsClient.ListConsoleMessagesAsync(new ListLogOptions()
{
    OutputJson = true,
    Timerange = new LogTimeRange(TimeSpan.FromDays(5))
});

Console.WriteLine(JsonSerializer.Serialize(listConsoleMessagesResponse.Messages));

Next, run the console application to validate:

dotnet run

Sample output:

  ...
  {
    "taskRunCount": 1,
    "taskId": "7ad2187d-7640-4a20-86a6-a147968e3cb5",
    "command": "testCommand@2",
    "taskCorrelationId": "7ad2187d-7640-4a20-86a6-a147968e3cb5",
    "sourceContext": "Taskurai.Worker.TaskuraiWorkerParallel",
    "workerName": "TestWorker",
    "containerImage": "taskuraisample.azurecr.io/taskurai-worker-sample:latest",
    "plainMessage": "",
    "timeGenerated": "2024-04-29T20:22:30.7906401+00:00",
    "message": "Handling task \"7ad2187d-7640-4a20-86a6-a147968e3cb5\" - command \"testCommand@2\" worker \"TestWorker\" started, run 1. (TaskuraiSDKVersion \"1.10.7.0\" - TaskuraiSDKVersion = \"1.10.0.0\")",
    "time": "2024-04-29T20:22:25.4626141+00:00",
    "timestamp": 1714422145.46261,
    "workerRevision": "testworker--dxysjnpuexihm",
    "customColumns": {
      "taskuraiSDKVersion": "1.10.0.0",
      "taskuraiWorkerSDKVersion": "1.10.7.0"
    },
    "logTableName": "TaskuraiConsoleLogs"
  },
  ...

Listing system logs​

Use this command to easily list the system logs of the Taskurai environment.

API reference:

Required scopes:

  • Taskurai.Log.System.Read
var logsClient = taskurai.GetLogsClient();

var listSystemMessagesResponse = await logsClient.ListSystemMessagesAsync(new ListLogOptions()
{
    OutputJson = false
});

Console.WriteLine(listSystemMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

[2024-05-01T10:20:03.492Z] Stopping container taskurai-worker-sample
[2024-05-01T10:20:03.492Z] Deactivated apps/v1.Deployment k8se-apps/testworker--h3eqhvj5btdwq from 1 to 0
[2024-05-01T10:15:43.256Z] KEDA is stopping the watch for revision 'testworker--ggwqd7okgntyc' to monitor scale operations for this revision
[2024-05-01T10:15:13.035Z] Successfully updated containerApp: testworker
[2024-05-01T10:15:13.035Z] Successfully provisioned revision 'testworker--h3eqhvj5btdwq'
[2024-05-01T10:15:13.035Z] Deactivating old revisions for ContainerApp 'testworker'
[2024-05-01T10:15:13.035Z] Updating revision : testworker--h3eqhvj5btdwq
[2024-05-01T10:15:13.035Z] Updating containerApp: testworker
[2024-05-01T10:15:08.375Z] Started container taskurai-worker-sample
[2024-05-01T10:15:08.375Z] Created container taskurai-worker-sample
[2024-05-01T10:15:08.375Z] Successfully pulled image "taskuraisample.azurecr.io/taskurai-worker-sample:latest" in 143.616098ms (143.630129ms including waiting)
[2024-05-01T10:15:08.375Z] Pulling image "taskuraisample.azurecr.io/taskurai-worker-sample:latest"

Listing task console logs​

Use this command to easily list the console logs of a specified task.

API reference:

Required scopes:

  • Taskurai.Log.Task.Console.Read
var logsClient = taskurai.GetLogsClient();

var taskId = "7528b696-36dc-4b59-9cfb-c06ebc759704";

var listTaskConsoleMessagesResponse = await logsClient.ListTaskConsoleMessagesAsync(taskId, new ListLogOptions()
{
    OutputJson = false,
    Timerange =  LogTimeRange.Indefinite
});

Console.WriteLine(listTaskConsoleMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

[2024-04-18T16:23:34.010Z] Handling task "7528b696-36dc-4b59-9cfb-c06ebc759704" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-18T16:23:29.744Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-18T16:23:29.744Z] Selected command "testCommand@2" - Version 2
[2024-04-18T16:23:29.744Z] Handling task "7528b696-36dc-4b59-9cfb-c06ebc759704" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")

Listing task history console logs​

Use this command to easily list the console logs of a specified task and run.

API reference:

Required scopes:

  • Taskurai.Log.Task.Console.Read
var logsClient = taskurai.GetLogsClient();

var taskId = "7528b696-36dc-4b59-9cfb-c06ebc759704";
var runCount = 1;

var listTaskHistoryConsoleMessagesResponse = await logsClient.ListTaskHistoryConsoleMessagesAsync(taskId, runCount, new ListLogOptions()
{
    OutputJson = false,
    Timerange =  LogTimeRange.Indefinite
});

Console.WriteLine(listTaskHistoryConsoleMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

[2024-04-18T16:23:34.010Z] Handling task "7528b696-36dc-4b59-9cfb-c06ebc759704" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-18T16:23:29.744Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-18T16:23:29.744Z] Selected command "testCommand@2" - Version 2
[2024-04-18T16:23:29.744Z] Handling task "7528b696-36dc-4b59-9cfb-c06ebc759704" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")

Listing task correlation console logs​

Use this command to easily list the console logs using the task correlation id. In the default scenario, the task correlation id is equal to the task id. It is possible to override the task correlation id when creating a new task.

API reference:

Required scopes:

  • Taskurai.Log.TaskCorrelation.Console.Read
var logsClient = taskurai.GetLogsClient();

var correlationId = "7528b696-36dc-4b59-9cfb-c06ebc759704";

var listTaskCorrelationConsoleMessagesResponse = await logsClient.ListTaskCorrelationConsoleMessagesAsync(correlationId, new ListLogOptions()
{
    OutputJson = false,
    Timerange =  LogTimeRange.Indefinite
});

Console.WriteLine(listTaskCorrelationConsoleMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

[2024-04-18T16:23:34.010Z] Handling task "7528b696-36dc-4b59-9cfb-c06ebc759704" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-18T16:23:29.744Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-18T16:23:29.744Z] Selected command "testCommand@2" - Version 2
[2024-04-18T16:23:29.744Z] Handling task "7528b696-36dc-4b59-9cfb-c06ebc759704" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")

Listing task history correlation console logs​

Use this command to easily list the console logs using the task correlation id for a specific run. In the default scenario, the task correlation id is equal to the task id. It is possible to override the task correlation id when creating a new task.

API reference:

Required scopes:

  • Taskurai.Log.TaskCorrelation.Console.Read
var logsClient = taskurai.GetLogsClient();

var correlationId = "7528b696-36dc-4b59-9cfb-c06ebc759704";
var runCount = 1;

var listTaskHistoryCorrelationConsoleMessagesResponse = await logsClient.ListTaskHistoryCorrelationConsoleMessagesAsync(correlationId, runCount, new ListLogOptions()
{
    OutputJson = false,
    Timerange =  LogTimeRange.Indefinite
});

Console.WriteLine(listTaskHistoryCorrelationConsoleMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

[2024-04-18T16:23:34.010Z] Handling task "7528b696-36dc-4b59-9cfb-c06ebc759704" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-18T16:23:29.744Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-18T16:23:29.744Z] Selected command "testCommand@2" - Version 2
[2024-04-18T16:23:29.744Z] Handling task "7528b696-36dc-4b59-9cfb-c06ebc759704" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")

Listing command console logs​

Use this command to easily list the console logs using a command name. In case versioned commands are used, the full command name and version should be specified.

API reference:

Required scopes:

  • Taskurai.Log.Command.Console.Read
var logsClient = taskurai.GetLogsClient();

var command = "testCommand@2"; // Filter on messages of the command testCommand with version 2

var listCommandConsoleMessagesResponse = logsClient.ListCommandConsoleMessages(command, new ListLogOptions()
{
    OutputJson = false,
    Timerange =  new LogTimeRange(TimeSpan.FromDays(10))
});

Console.WriteLine(listCommandConsoleMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

...
[2024-04-26T09:31:33.167Z] Handling task "bc520a6e-a557-4501-982f-c4fb5a327ac1" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-26T09:31:24.583Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-26T09:31:24.583Z] Selected command "testCommand@2" - Version 2
[2024-04-26T09:31:24.583Z] Handling task "db9fd546-ab69-426c-85c7-0e84399556a6" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
[2024-04-26T09:31:33.167Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-26T09:31:33.167Z] Selected command "testCommand@2" - Version 2
[2024-04-26T09:31:33.167Z] Handling task "6d22176e-3e7b-492a-854c-be44681d2f0d" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
...

Listing worker console logs​

Use this command to easily list the worker console logs.

API reference:

Required scopes:

  • Taskurai.Log.Worker.Console.Read
var logsClient = taskurai.GetLogsClient();

var workerName = "TestWorker";

var listWorkerConsoleMessagesResponse = logsClient.ListWorkerConsoleMessages(workerName, new ListLogOptions()
{
    OutputJson = false,
    Timerange =  new LogTimeRange(TimeSpan.FromDays(3))
});

Console.WriteLine(listWorkerConsoleMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

...
[2024-04-29T20:22:53.931Z] Handling task "1fb483fe-5417-4f04-806f-e2e6c3438b55" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-29T20:22:53.931Z] Handling task "2de1ad63-8c9d-4adf-95f1-82597f513cb1" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-29T20:22:53.931Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-29T20:22:53.931Z] Selected command "testCommand@2" - Version 2
[2024-04-29T20:22:53.931Z] Handling task "9e0b7271-bc10-412e-b6ee-9928ef1e9d88" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
[2024-04-29T20:22:52.463Z] Handling task "75f0ea42-6542-49c5-b736-ed96369beb7f" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-29T20:22:52.463Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-29T20:22:52.463Z] Selected command "testCommand@2" - Version 2
[2024-04-29T20:22:52.463Z] Handling task "478fd7f0-adb6-45aa-83d0-8a7920f37c1f" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
[2024-04-29T20:22:52.463Z] Handling task "129b3b55-f8b1-422a-80da-a26fb0292013" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-29T20:22:52.463Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-29T20:22:52.463Z] Selected command "testCommand@2" - Version 2
[2024-04-29T20:22:52.463Z] Handling task "0be4352b-1a84-4f74-9eb5-49d8f0ee3076" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
...

Listing worker system logs​

Use this command to easily list the worker system logs.

API reference:

Required scopes:

  • Taskurai.Log.Worker.System.Read
var logsClient = taskurai.GetLogsClient();

var workerName = "TestWorker";

var listWorkerSystemMessagesResponse = logsClient.ListWorkerSystemMessages(workerName, new ListLogOptions()
{
    OutputJson = false,
    Timerange =  new LogTimeRange(TimeSpan.FromDays(1))
});

Console.WriteLine(listWorkerSystemMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

...
[2024-05-02T16:10:09.278Z] Scaled apps/v1.Deployment k8se-apps/testworker--usaaexct6fkle from 0 to 1
[2024-05-02T16:10:09.278Z] Started container taskurai-worker-sample
[2024-05-02T16:10:09.278Z] Created container taskurai-worker-sample
[2024-05-02T16:10:09.278Z] Successfully pulled image "taskuraisample.azurecr.io/taskurai-worker-sample:latest" in 582.270231ms (582.285562ms including waiting)
[2024-05-02T16:10:09.278Z] Pulling image "taskuraisample.azurecr.io/taskurai-worker-sample:latest"
[2024-05-02T16:10:09.278Z] Replica 'testworker--usaaexct6fkle-5b9c8f6c89-2n74x' has been scheduled to run on a node.
[2024-05-02T16:03:02.100Z] Scaled apps/v1.Deployment k8se-apps/testworker--usaaexct6fkle from 0 to 1
[2024-05-02T16:03:02.100Z] Started container taskurai-worker-sample
[2024-05-02T16:03:02.100Z] Created container taskurai-worker-sample
[2024-05-02T16:03:02.100Z] Successfully pulled image "taskuraisample.azurecr.io/taskurai-worker-sample:latest" in 582.270231ms (582.285562ms including waiting)
[2024-05-02T16:03:02.100Z] Pulling image "taskuraisample.azurecr.io/taskurai-worker-sample:latest"
[2024-05-02T16:03:02.100Z] Replica 'testworker--usaaexct6fkle-5b9c8f6c89-2n74x' has been scheduled to run on a node.
[2024-05-02T15:54:15.425Z] Scaled apps/v1.Deployment k8se-apps/testworker--usaaexct6fkle from 0 to 1
[2024-05-02T15:54:15.425Z] Started container taskurai-worker-sample
[2024-05-02T15:54:15.425Z] Created container taskurai-worker-sample
[2024-05-02T15:54:15.425Z] Successfully pulled image "taskuraisample.azurecr.io/taskurai-worker-sample:latest" in 582.270231ms (582.285562ms including waiting)
[2024-05-02T15:54:15.425Z] Pulling image "taskuraisample.azurecr.io/taskurai-worker-sample:latest"
[2024-05-02T15:54:15.425Z] Replica 'testworker--usaaexct6fkle-5b9c8f6c89-2n74x' has been scheduled to run on a node.
[2024-05-02T15:47:56.696Z] Scaled apps/v1.Deployment k8se-apps/testworker--usaaexct6fkle from 0 to 1
[2024-05-02T15:47:56.696Z] Started container taskurai-worker-sample
[2024-05-02T15:47:56.696Z] Created container taskurai-worker-sample
[2024-05-02T15:47:56.696Z] Successfully pulled image "taskuraisample.azurecr.io/taskurai-worker-sample:latest" in 582.270231ms (582.285562ms including waiting)
[2024-05-02T15:47:56.696Z] Pulling image "taskuraisample.azurecr.io/taskurai-worker-sample:latest"
[2024-05-02T15:47:56.696Z] Replica 'testworker--usaaexct6fkle-5b9c8f6c89-2n74x' has been scheduled to run on a node.
[2024-05-02T15:38:41.562Z] Created container taskurai-worker-sample
[2024-05-02T15:38:41.562Z] Scaled apps/v1.Deployment k8se-apps/testworker--usaaexct6fkle from 0 to 1
[2024-05-02T15:38:41.562Z] Started container taskurai-worker-sample
...

Listing worker system logs​

Use this command to easily list the console logs that are generated by a certain container image.

API reference:

Required scopes:

  • Taskurai.Log.ContainerImage.Console.Read
var logsClient = taskurai.GetLogsClient();

var containerImage = "taskuraisample.azurecr.io/taskurai-worker-sample:latest";

var listContainerImageConsoleMessagesResponse = logsClient.ListContainerImageConsoleMessages(containerImage, new ListLogOptions()
{
    OutputJson = false,
    Timerange =  new LogTimeRange(TimeSpan.FromDays(7))
});

Console.WriteLine(listContainerImageConsoleMessagesResponse.PlainText);

Next, run the console application to validate:

dotnet run

Sample output:

...
[2024-04-26T09:31:24.583Z] Handling task "5dc417f8-4718-40d1-997b-f76d7509f12d" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
[2024-04-26T09:31:33.167Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-26T09:31:33.167Z] Selected command "testCommand@2" - Version 2
[2024-04-26T09:31:33.167Z] Handling task "985945cc-1270-46d4-bba2-b7f2ad07e9be" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
[2024-04-26T09:31:33.167Z] Handling task "b5857904-1d63-43f3-981b-cd7b84854f19" - command "testCommand@2" worker "TestWorker", run 1 finished.
[2024-04-26T09:31:24.583Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-26T09:31:24.583Z] Selected command "testCommand@2" - Version 2
[2024-04-26T09:31:24.583Z] Handling task "6717289a-04a0-420f-870e-ac7805b16814" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
[2024-04-26T09:31:33.167Z] TestCommandV2 - RequestedVersion: "2"
[2024-04-26T09:31:33.167Z] Selected command "testCommand@2" - Version 2
[2024-04-26T09:31:33.167Z] Handling task "fd85d446-0c5e-42fe-8d2c-913546f753d8" - command "testCommand@2" worker "TestWorker" started, run 1. (TaskuraiSDKVersion "1.10.7.0" - TaskuraiSDKVersion = "1.10.0.0")
...