Azure

How to run scheduled processes in Azure?

Kalle Marjokorpi

• 6 min read

I discussed with a colleague before the summer holiday about how to schedule background processes in Azure. There are a lot of options available for many use cases. This blog post introduces some Azure services that can be used to execute scheduled background processes.

A scheduled background process refers to a task or operation that is automatically executed at specific times or intervals and typically it doesn't require user intervention. Scheduled processes are useful when the system needs to execute something "behind the scenes" which is invisible to the end-user. Typically scheduled background processes are related to maintenance or operational tasks. Scheduled background processes are typically part of almost any system because especially maintenance-related tasks or operations can decrease infrastructure costs.

A few samples of maintenance-related scheduled background processes:

  1. Cleaning unused user sessions from the database. 
  2. Cleaning Azure resources from test environments and re-provisioning resources by schedule.
  3. Execute health checks

Operations tasks that can be executed in scheduled background processes:

  1. Aggregate data from multiple sources and generate reports.
  2. Train AI model.
  3. Data synchronization between different internal or external systems. 

Scheduling services provided by Azure

Azure Automation Account

Azure Automation Account is a process automation service that enables the running of Python and PowerShell scripts (Runbooks) in Azure by a schedule. Azure Automation Account is useful, especially for people who maintain the IT infrastructure and are already familiar with PowerShell. PowerShell and Python basically enable to creation of almost anything in scripts if necessary. Azure Automation Account supports Managed Identity and provides role-based access control and a secure store for variables & credentials & certificates. 

Execution of process automation (=running scheduled scripts) jobs are billed per minute. Currently (7/23) 500 minutes per month are provided for free which is a nice amount of time to execute some basic light tasks or operations.

Azure Automation Account is a good choice if you want to create scheduled operations or tasks with PowerShell and you're running long-running tasks (see Checkpoints).

Use cases for scheduling: 

  • Scaling down SQL server during night and weekends in a test environment to save costs
  • Execute SQL database maintenance tasks

Azure Function

Azure Function is a serverless computing platform provided by Azure that enables you to write less code, maintain less infrastructure, and save costs. Azure Function runtime supports multiple languages i.e. C#, Javascript, F#, Java, Powershell, Python, and Typescript. Azure Function widely supports event triggers from multiple Azure services i.e. Service Bus and Storage Account. Timer Trigger is also supported which enables to execute Function on a schedule. When using Azure Functions in the Consumption plan you need to remember that default timeout is 5 minutes and it can be extended to 10 minutes. A more expensive Premium plan provides a longer timeout.

Azure Function is billed by the time and total amount of executions. 

Azure Function has wide support for multiple languages so there is a suitable language available for all to create scheduled background processes. Azure Function is a good choice especially when creating something more complex and the process is more advanced than just a script with a few lines of code. The serverless platform enables great scalability, but you should know the limits of timeouts.

Use cases for scheduling: 

  • Execute fraud detection model twice a day
  • Execute complex business rules to detect which users should be notified by email
[FunctionName("ScheduledBackgroundTrigger")]
public static void Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer, ILogger log)
{
    // Business logic
}

Azure Logic Apps

Azure Logic Apps is an integration platform as a service (iPaaS) where workflows can be created with a small amount of coding. Powerful visual designer of Azure Logic Apps enables quick creation of workflows using drag-and-drop type of working. Azure Logic Apps platform supports a large number of connectors that provide integrations to multiple services provided by Microsoft or external (ex. Salesforce). Azure Logic Apps support recurrence trigger which enables to creation of varied scheduling rules i.e. repeat weekly on specific days etc.

Azure Logic Apps is a powerful tool to create scheduled background processes via visual designer and take advantage of great scalability. Azure Logic App is a great match for simpler scheduled tasks or operations because many scenarios can be created without writing any code. If business logic becomes large then the maintenance of Azure Logic App might become hard and other solutions i.e. Azure Functions might be better.

Use cases for scheduling: 

  • Send email notifications for admin users about failed orders in Service Bus Dead-Letter

Azure App Service with scheduled WebJobs

WebJobs are tightly coupled with Azure App Service. WebJobs supports scheduling tasks or operations based on CRON expression. With version 3.x of the Azure WebJobs SDK, you can create and publish WebJobs as .NET Core console apps. If you deploy WebJobs alongside another application in App Service, you should remember that WebJob consumes the same CPU and memory capacity. WebJobs also has limits with timeout and timeout can be adjusted via WEBJOBS_IDLE_TIMEOUT configuration in App Settings.

There has been some discussion that WebJobs will be deprecated, but this hasn't been confirmed by Microsoft. I would prefer Azure Functions or Azure App Service with .NET hosted service for all new scheduled solutions which require more advanced handling and business logic.

public class ScheduledBackgroundWebJob
{
    public async Task TimerTriggerOperation([TimerTrigger("0 */5 * * * *", RunOnStartup = true)] TimerInfo timerInfo, ILogger logger, CancellationToken cancellationToken)
    {
        // Business logic
    }
}

Azure App Service with .NET hosted service

.NET supports to create and run background operations or tasks using Hosted Services. Basically, a hosted service is a class that implements the IHostedService interface. IHostedService requires you to implement StartAsync() method which is called at startup and StopAsync() method which executes in shutdown. Hosted Services are registered in the dependency injection. You can schedule background operations in StartUp() using i.e. Timer class.

Hosted service is a good scheduling solution like Azure Function for more advanced use cases which require more business logic and handling. Hosted services like WebJobs consume the same.

Use cases for scheduling: 

  • Clean up expired user sessions from the database
  • Process order queue
public class ScheduledBackgroundHostedService : IHostedService
{
    private readonly ILogger _logger;
    private Timer _timer;

    public ScheduledBackgroundHostedService(ILogger<ScheduledBackgroundHostedService> logger)
    {
        _logger = logger;
    }

    public Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Timed Hosted Service running.");

        _timer = new Timer(DoWork, null, TimeSpan.Zero, 
            TimeSpan.FromSeconds(5));

        return Task.CompletedTask;
    }

    private void DoWork(object state)
    {
        _logger.LogInformation("Timed Hosted Service is working...");
        
        // Business logic
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Timed Hosted Service is stopping...");
        _timer?.Change(Timeout.Infinite, 0);
        return Task.CompletedTask;
    }
}

Azure Batch

Azure Batch enables to running of large-scale parallel and high-performance computing batch jobs in Azure. Azure Batch is a good solution i.e. nighly data processing, testing, or analysis operations that require high performance and scalability. Behind the scenes Azure Batch creates and manages a pool of compute nodes (virtual machines), installs the applications you want to run, and schedules jobs to run on the nodes. 

Use cases for scheduling:

  • AI model training
  • Processing large volumes of image data

Azure Data Factory

Azure Data Factory is a cloud-based ETL (Extract Transform Load) and data integration service. It natively integrates with a large number of Azure services and enables the creation workflows of for orchestrating the data movement and transformation. Azure Data Factory supports scheduled triggers for pipelines which enables the scheduling based on your rules.

Use cases for scheduling:
- Moving audit log data from Blob Storage to another long-term Blob Account (archive)
- Create AVRO files from the latest order rows and send them to the SFTP server

Jobs in Azure Container Apps

Jobs in Azure Container Apps (preview) enable to creation of containerized tasks that are executed by the schedule (CRON expression). Jobs supports Manual, Schedule, and Event types of triggers. If you're already working in a containerized environment, Jobs are a good solution to consider for scheduled background processes.

Use cases for scheduling:

  • Notify users by email about abandoned shopping carts

Scheduled pipelines in Azure DevOps

Pipelines in Azure DevOps support executing different kinds of scripts (Powershell etc.) which can do i.e. maintenance type of operations. Pipelines can be scheduled to execute at the right moment.
the
Use cases for scheduling:

  • Nightly Security check of the environment
  • Nightly execution of UI automation tests
  • Scaling down Azure resources during the night and weekends in a test environment
  • Scaling up Azure resources before office hours in a test environment
  • Cleaning up sandbox Azure environments

Comments