Skip to main content

 

OutSystems Platform Timers and Asynchronous Processes

 

OutSystems

OutSystems Platform Timers and Asynchronous Processes

A Timer is an OutSystems tool that allows executing application logic periodically on a scheduled time. These are also known as batch jobs.

Different Timers can be executed at the same time, but the same Timer never has more than one execution at a time.

Here follows some common scenarios where you can use Timers:

Scenario Example
Scheduled Jobs Execute the same job every day at the same time. For example, send every day at 4 a.m. e-mails to subscribers with digest news.

Create a Timer to execute an Action that does the job of sending e-mails to subscribers every day at 4 a.m.
Executing Long Running Actions Execute application logic that usually takes long time to finish. For example, at 2 a.m. of the 1st day of every month, the system must archive a lot of database records. It takes about 2 hours.

Create a Timer to execute an Action that archives records and set it to run at 2 a.m. of 1st day of every month and with the default timeout of 150 minutes (this value can be adjusted in Service Center).

Architecture

The following table lists the OutSystems elements that relate to Timers:

Element Description
OutSystems Scheduler Service This is the service that has the responsibility of checking for Timers to be executed. It's a multi-threaded service that allows having different Timers executing at the same time.
Runtime Database The Runtime Database contains all System Entities for managing Timers, such as:
- The record of all existing Timers.
- The schedule for executing Timers.
- Current execution of Timers: when started, their timeout, or next execution.
Log Database When a Timer is executed, an entry is created in the Log Database.
Configuration Tool This is the tool that allows configuring the maximum number of Timers that may execute at the same time in each Front-end Server node.
OutSystems Log Service In Service Center, it's possible to have access the information about Timers. The Log Service has the responsibility of writing this data in the Log Database.
Deployed module (Application) The deployed application contains the code for the Timer. It has the application logic designed in the Action executed by the Timer, and also some stub code needed to get the system database up-to-date about the state of the Timer.

Runtime and Log databases

This section describes the database entities that support Timers and their functionality.

Entity Meta_Cyclic_Job

This entity contains the definitions of Timers created in modules which are stored in records in the ossys_Meta_Cyclic_Job database table. These records are created and managed by OutSystems and the Effective_Timeout is the only attribute that you can change.

Here's the complete description of the attributes:

Attribute Description
Id The primary key.
Espace_Id The module where the Timer is defined.
Name The Timer name, as defined in the module.
Default_Schedule The default scheduled time for the Timer, as defined in the module.

This is a string with the time and the frequency the Timer is executed. It has a special editor in Service Studio to set its value.

This value is used to set the Schedule property in the Cyclic_Job_Shared entity.
Priority The Timer priority, as defined in the module. It's a value between 1 (Highest) and 4 (Lowest).

When the number of Timers being executed at the same time is greater than the maximum allowed for the Front-end Server, Timers with highest priority are executed first.
Is_Active Indicates whether the Timer is active.
SS_Key The Timer key in Service Studio.
Timeout The Timer default timeout, as defined in the module. The value is in minutes.

If the Timer doesn't finish the job in this time limit, the Timer execution is aborted by the system.
Effective_Timeout If the value is different from zero, it overrides the above Timer default timeout.

Use it to adjust a new timeout, which is done in Service Center, in the Timer's detail page.
IsShared When True, it indicates that instances of the Timer are created in the Cyclic_Job_Shared entity, otherwise they're created in the Cyclic_Job entity.

Entity Cyclic_Job

This entity contains the information about all Timers to be executed by the Scheduler Service in multi-tenant modules. You can browse this information in Service Center (see Monitoring the execution of Timers).

As there may be multiple Front-end Server nodes executing Timers, each with its own time, the time reference for executing Timers is always the database clock.

Records of this entity are stored in the database in the ossys_Cyclic_Job table. They're created and managed by OutSystems.

The Schedule and Next_Run are the only attributes of this entity that you can change. However, don’t set them when the Timer is being executed, which can be verified by checking the value in the Is_Running_Since attribute:

  • If set: the Timer is running.

  • If not set: the Timer isn't running.

Here's the complete description of the attributes:

Attribute Description
Meta_Cyclic_Job_Id A reference to the Meta_Cyclic_Job entity.
Tenant_Id The reference to the tenant where the Timer is running.
Schedule After the Timer has finished its execution, this attribute is used to calculate the time of the next execution.

For example: Schedule: "02:00 10:00 18:00" (runs every day at 2 a.m., 10 a.m., and 6 p.m.). If a Timer finished on the 10-23-2012 at 18:00:50 then the Next_Run is set to 10-24-2012 at 2:00:00.

This property is initially set by the value defined in the Default_Schedule property of the Meta_Cyclic_Job entity, but it can be changed in Service Center.
Last_Run The date and time at which the Timer started its last execution (using the database clock).
Next_Run The date and time of the next execution (using the database clock).

The Scheduler Service executes Timers which have Next_Run >= current date time.
Last_Duration The time in seconds the Timer took to finish its last execution.
Is_Running_Since The date and time when the Schedule Service started executing the Timer (database clock time).

This attribute is also used to synchronize the several Front-end Server nodes because the Scheduler Service doesn't execute Timers with this attribute set (unless in error recovering actions).

When the execution of the Timer finishes, this attribute is cleaned.
Is_Running_By This attribute has the name of the Front-end Server node that's executing the Timer. The Is_Running_Since attribute is also set.
Number_Of_Tries The number of consecutive times the Timer was executed without success.

Entity Cyclic_Job_Shared

This entity contains the information of Timers to be executed by the Scheduler Service in single-tenant modules. You can browse this information in Service Center (see Monitoring the execution of Timers).

As there may be multiple Front-end Server nodes executing Timers, each with its own time, the time reference for executing Timers is always the database clock.

Records of this entity are stored in the database in the ossys_Cyclic_Job_Shared table. They're created and managed by OutSystems.

The Schedule and Next_Run are the only attributes of this entity that you can change. However, don’t set them when the Timer is being executed, which can be verified by checking the value in the Is_Running_Since attribute:

  • If set: the Timer is running.

  • If not set: the Timer isn't running.

Here's the complete description of the attributes:

Attribute Description
Meta_Cyclic_Job_Id A reference to the Meta_Cyclic_Job entity.
Schedule After the Timer has finished its execution, this attribute is used to calculate the time of the next execution.

For example: Schedule: "02:00 10:00 18:00" (runs every day at 2 a.m., 10 a.m., and 6 p.m.). If a Timer finished on the 10-23-2012 at 18:00:50 then the Next_Run is set to 10-24-2012 at 2:00:00.

This property is initially set by the value defined in the Default_Schedule property of the Meta_Cyclic_Job entity, but it can be changed in Service Center.
Last_Run The date and time at which the Timer started its last execution.
Next_Run The date and time of the next execution.

The Scheduler Service executes Timers which have Next_Run >= current date time.
Last_Duration The time in seconds the Timer took to finish its last execution.
Is_Running_Since The date and time when the Schedule Service started executing the Timer (database clock time).

This attribute is also used to synchronize the several Front-end Server nodes because the Scheduler Service doesn't execute Timers with this attribute set (unless in error recovering actions).

When the execution of the Timer finishes, this attribute is cleaned.
Is_Running_By This attribute has the name of the Front-end Server node that's executing the Timer. The Is_Running_Since attribute is also set.
Number_Of_Tries The number of consecutive times the Timer was executed without success.

Tables oslog_Cyclic_Job_< N >

These tables store all logging information about executed Timers. You can browse it in Service Center (see Checking the Timers log).

Here's the complete description of the attributes:

Attribute Description
Instant The date and time when the Timer execution started ('Time of Log' in Service Center).

This is the IIS clock date and time.
Duration The time in seconds the Timer took to finish the execution.
Cyclic_Job_Key The Timer key in Service Studio.
Espace_Id The reference to the module where the Timer was created.
Tenant_Id The reference to the tenant where the Timer was executed.
Executed_By The name of the Front-end Server node that executed the Timer ('Server' in Service Center).
Error_Id The reference to the [oslog_]Error_< N > entity.

If set, it means there was an error executing the application logic of the Timer and in Service Center Timers Log page displays a red link called Error in the row of the Timer.
Should_Have_Run_At The date and time when this Timer should have been executed.

Note that if a system is very loaded, this value can be quite different than the Instant attribute. This may happen because the same module has many tenants and all have the same schedule, other Timers are taking too much time to execute, etc.

This is the database clock date and time.
Next_Run Is a date time representing when this Timer should start again (based on schedule, as explained above).
Cycle Log cycle number for internal use.

How Timers are executed

The Scheduler Service is responsible for executing all Timers. This service is multi-threaded, therefore, it allows executing different Timers at the same time. For simplicity, the steps below describe the execution of a single Timer.

The steps to execute Timers are as follows:

  1. The Scheduler Service cyclically fetches the database for Timers to be executed (Next_Run).

  2. The Scheduler Service launches a thread to execute the Timer that calls a Web Service in the module where the Timer is defined for executing it.

  3. The Web Service checks first whether the Timer is already executing in any Front-end Server node. If not, the Is_Runnining_Since and Is_Runnining_By attributes of the Cyclic_Job_Shared entity are updated. This locks the Timer from executing in any other Front-end Server node.

  4. The module executes the action associated with the Timer.

  5. After the action execution is finished, the Is_Runnining_Since and Is_Runnining_By attributes are cleaned, which frees the Timer for a new execution. The Next_Run attribute is recalculated based on the Schedule attribute and the current date and time.

    Regarding the Next_Run attribute, it's updated only if it didn't change during the execution of the action. This can happen if, for example, the action updates itself that attribute.

  6. A record with the information about the Timer execution is sent to the Log Service that stores it in the Log Database.

  7. The Web Service returns the control to the Scheduler Service, which now becomes ready to execute another Timer (step 1.).

Why a timeout in Timers?

As described before, the Scheduler Service executes a Timer by calling a Web Service built in the module where the Timer was created. This means that there is a Web Request involved in Timers execution and, like in any Web Request, there is a maximum time allowed for it to execute in the server. This is the reason why Timers have the 'Timeout in Minutes' property in Service Studio, which by default it's set to 20 minutes, but you may adjust it to each case.

The role of the timeout in unexpected errors

If the execution of a Timer terminates due to an unexpected error, the Scheduler Service recovers the Timer and synchronize the database only after the timeout time plus 20% has passed. Until then, the data about Timers in the database may be inconsistent.

It's also important to implement the login in the Timer action so that you make sure that data is kept consistent in case of the Timer execution terminates unexpectedly.

Retrying after an execution error

Whenever a Timer has an execution error, OutSystems may execute the Timer again for a number of retries. You can set this number of retries in Service Center, in the Environment Configuration screen, under the Administration folder. By default, it's set to 3 retries.

Changing the schedule of a Timer dynamically

Besides setting regular and fixed executions for your Timers, it's also possible to schedule the execution of your Timers dynamically, based on the logic you want. For that, simply edit the Timer action, calculate the new date and time of execution, and update it on the entity holding the Timer execution information.

To have access to all necessary entities to set dynamic executions, proceed as follows:

  1. In the module with the Timer action, add a reference to the following System entities:

    • Meta_Cyclic_Job
    • Cyclic_Job_Shared
  2. Calculate the date and time of the next execution.

  3. Update the Next_Run attribute of Cyclic_Job_Shared with the value calculated in step 2.

Example:

Consider the CleanUp Timer, which periodically cleans up data from the database, and that's to be executed 2 hours after ending the last execution.

Open the action of the Timer and at the end of the flow add the following logic:

  1. Create a Query to get the Cyclic_Job_Shared record of the Timer. The Query has the module identifier (eSpaceId), the Timer name, and input parameters:

  2. Set the query parameters with the following values:

    • eSpaceId: GetOwnerEspaceIdentifier()
    • TimerName: "CleanUp"

  3. Calculate the next execution date and time to 2 hours from now and set it to the Next_Run attribute of the Cyclic_Job_Shared record: AddHours(CurrDateTime(),2)

  4. Update the Cyclic_Job_Shared record in the database.

  5. Commit the transaction to clear the lock over the updated record.

The action flow should end like this:

Managing Timers in Service Center

Service Center provides a set of functionality that allows you to manage your Timers, namely:

  • Monitor the execution of Timers

  • Edit the Timer settings

  • Force the execution of a Timer

  • Deactivate/Activate a Timer

  • Browse the logs of previous executions

Monitoring the execution of Timers

In Service Center, you may monitor the execution of your Timers in the Environment Health option, under Monitoring. The page has a Timers section with a list that shows the order by which Timers are executed.

The criteria to sort Timers in the list are based on the following:

  • The Timer priority.

  • The time a Timer takes to be executed, based on its previous duration. Timers with faster execution times are executed first.

  • The time a Timer is waiting to be executed. As a Timer waiting time grows, it tends to ascend in the list.

The next picture shows an example of the Timers section in two moments, with 10 minutes of difference.

Taking the Send Notifications Timer in the example, you can see that:

  • It was scheduled to be executed at 14:15:00.
  • It was in fact executed at 14:14:10.

The conclusion is that the Timer was executed before what was expected. This may be due to either being explicitly awakened by the application or someone forced its execution in Service Center.

Checking the Timers health

In Service Center, you may check the Timer threads the Scheduler Service is executing, in the Environment Health option, under Monitoring.

In the Front-end Servers section, click the detail link in the Scheduler column to display the page with the details of the service. In this case, focus on Timer-related threads.

In example above, the Scheduler Service has the following threads related with Timers:

  • The Timers Fetcher is a thread that awakes from time to time to fetch Timer jobs to be executed and puts them in the queue to be picked. At the current time, it was sleeping.

  • There are three Timer Processor threads that execute pick Timer jobs from the queue and execute them. The number of Timer Processor threads is configured in the Configuration Tool.

You can also conclude that:

  • No Timer job was being executed, as all Timer Processor threads were in Idle status.

  • The Queue has no entries, which means that there are no Timer jobs to be executed at the time.

Editing a Timer

You can edit a Timer by clicking the Timer name in one of following places in Service Center:

  • Module: edit the module where the Timer is defined, select the Timers tab, and a list of Timers is displayed.

  • Timers Log: go to the Timers log page where each line in the list has the Timer name.

  • Environment Health: go to the Environment Health page, check the Timers section where each line in the list has the Timer name.

In the example below, you can see that:

  • The timer runs every 15 minutes.

  • The priority is set to Normal.

  • The timeout isn't set, so it's the default set in Configuration Tool (20 min).

  • The last execution took place on 08-01-2013 at 18:15:09.

  • The next execution is set to 08-01-2013 at 18:30:00.

Forcing the execution of a Timer

To force a Timer to be executed without waiting for the scheduled time of the next run, press the Run Now button at the bottom of the page.

Deactivating/Activating a Timer

To stop a Timer from being executed (fetched by the Scheduler Service and consume resources), press the Deactivate button at the bottom of the page. The Timer will no longer be fetched by the Scheduler Service.

If the Timer is deactivated, the button shows Activate and you should press it to get the Timer back to work again.

Checking the Timers log

In Service Center, you may look into how Timers execution has been going. For that, select the Timers option, under Monitoring. In case of errors, you can click the Error link to see the details.