Skip to main content
OutSystems

Troubleshooting the OutSystems Platform Server

This article describes known techniques for troubleshooting OutSystems Platform in the presence of system malfunction. You will learn how to obtain error information and, for every known error, what was the cause and which recovery actions should be taken.

How to Obtain Error Information

Effective error tracking, monitoring and alert strategies can be put in practice using the standard OutSystems Platform Server configuration.

Error tracking is achieved via Service Center, accessing Monitoring tab:

  • Screen Log - Includes an entry for each access to a Screen, either WEB, Mobile Web or SMS;
  • SMS Log - Includes an entry for each SMS sent or received;
  • Timer Log - Includes an entry for each executed Scheduled Job;
  • Error Log - Includes an entry, every time a platform or application error occurs;
  • General Log - Reserved for application specific logging or tracing. Some platform informative entries are also registered (e.g. slow queries);
  • Web Service Log - Includes an entry for each Web Service invocation;
  • Web Reference Log - Includes an entry for each Web Reference invocation;

The same information can be accessed directly in the OutSystems Log Database, accessing data in tables named as os_log_log_kind_n (e.g. os_log_screen_1). Due to rotation purposes, there are 10 distinct tables for each kind of log. Nevertheless, a specific view exists for current data, as presented in Service Center. These views are named, simply as os_log_log_kind.

Error monitoring is achieved via Service Center, accessing page Online Monitoring in Monitoring tab:

  • Front-end Servers (Hub Nodes) - Display online information about every running Server, the status and queue of each service;
  • SMS Gateway - Display online information about the SMS Gateway, the status and queues of each connection; 
  • Timers - Display online information about the Scheduled Job activity.

Error alerts shall be based on the Windows Event Viewer entries. The OutSystems Platform Server produces Windows Event Viewer entries in the Application Log every time a severe error condition or every time any other type of error occurs several times in a period of time. Error alerts shall be issued based on its source and description.

In one of the sections it is presented a list of errors that can be found either in the Monitoring --> Logs section in Service Center (or corresponding database tables) or directly in the Event Viewer.

Windows Performance Counters

OutSystems Platform produces its own performance information, compliant with the Microsoft Windows System Performance Monitor, thus allowing integrated system performance monitoring in every Server. Along with common CPU, memory and hard disk access information, OutSystems Platform Performance Monitor provides accurate information about:

  • Incoming SMS
  • Outbound SMS
  • Asynchronous Jobs Execution
  • Application Error Rates
  • Web and Mobile Web Screen Access
  • Web Service Access

Errors, Causes and Recovery Actions

OutSystems Deployment Controller Service Errors

"OutSystems Deployment Controller Service Stopped. Please Restart"

"OutSystems Deployment Controller Service Status: No connection could be made because the target machine actively refused it" 

Cause The "OutSystems Deployment Controller Service" is stopped at the Deployment Controller Server (Hub Controller).
Resolution / Recovery Action
  1. Login to Deployment Controller Server (Hub Controller server) and go to Windows Services.
  2. Confirm that "OutSystems Deployment Controller Service" is not Started. Start it.
  3. If it is running, check the network and communication conditions between the Deployment Controller Server (Hub Controllerserver) and every Front-end Server (Hub Node server).

OutSystems Deployment Service Errors

"OutSystems Deployment Service Stopped. Please Restart"

"OutSystems Deployment Service Status: No connection could be made because the target machine actively refused it" 

Cause The "OutSystems Deployment Service" at the Front-end Server (Hub Node server) referred in the message is stopped.
Resolution / Recovery Action
  1. Login to the corresponding Front-end Server (Hub Node server) and go to Windows Services;
  2. Confirm that "OutSystems Deployment Service" is not Started. Start it. 

OutSystems Scheduler Service Errors

"OutSystems Scheduler Service Stopped. Please restart"

"OutSystems Scheduler Service Status: No connection could be made because the target machine actively refused it" 

Cause The "OutSystems Scheduler Service" at the Front-end Server (Hub Node server) referred in the message is stopped. 
Resolution / Recovery Action
  1. Login Front-end Server (Hub Node server) and go to Windows Services;
  2. Confirm that "OutSystems Scheduler Service" is not Started. Start it. 

 

"OutSystems Scheduler Service Status: Warning: Job Worker is in status Processing for more than n seconds"

Cause The "OutSystems Scheduler Service" is waiting for the Front-end Server (Hub Node server) to complete the execution of a scheduled job. n may vary according to execution conditions. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding Front-end Server (Hub Node server) Scheduler indicator in Frontend Servers (Hub Nodes) section for details;
  3. Check all "Job Worker" thread status and click on the Refresh Data link to confirm that the statuses are maintained.
  4. If there are "Job Worker" threads not in "Processing" status, keep clicking on the Refresh Data link, checking thread status, until no threads are in "Processing" status for more than n seconds;
  5. If they all become "Processing" for more than n seconds, check the detail columns for eSpace and timer info.
  6. If the error is specific of a single eSpace, report the problem to the application owner;
  7. If the error is observed in more than one eSpace:
    1. Login to the corresponding Front-end Server (Hub Node server), go to Windows Services and restart the "OutSystems Scheduler Service";
    2. In console mode execute "iisreset";
  8. Go back to Service Center's page Online Monitoring and check in the Scheduler details if the problem persists. 

 

"OutSystems Scheduler Service Status: Error: Job Worker is in status Error" 

Cause The "OutSystems Scheduler Service" could not start the Job execution. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding Front-end Server (Hub Node server) Scheduler indicator in Frontend Servers (Hub Nodes) section for details;
  3. Check all "Job Worker" thread statuses and click on the Refresh Data link to confirm that the statuses are maintained.
  4. If status is still "Error" for all:
    1. Login to the corresponding Front-end Server (Hub Node server), go to Windows Services and restart the "OutSystems Scheduler Service";
    2. In console mode execute "iisreset";
  5. Go back to Service Center's page Online Monitoring and check in the Scheduler details if the problem persists.

 

"OutSystems Scheduler Service Status: Error: Jobs Producer is in status Error"

Cause The "OutSystems Scheduler Service" cannot determine next jobs to execute. Possible error accessing database. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding Front-end Server (Hub Node server) Scheduler indicator in Frontend Servers (Hub Nodes) section for details;
  3. Check all "Jobs Producer" thread status and click on the Refresh Data link to confirm that the statuses are maintained.
  4. If status is still "Error" for "Jobs Producer":
    1. Login to the corresponding server and check network configuration and communication conditions between the Front-end Server (Hub Node) and the Platform Database (Hub Database) Servers.
    2. Login to Platform Database (Hub Database) Server and check SQL Server availability and runtime conditions (CPU, memory, etc.);
    3. Execute any specific recovery action;
  5. If the problem persists:
    1. Login to the corresponding Front-end Server (Hub Node server), go to Windows Services and restart the "OutSystems Scheduler Service";
  6. Go back to Service Center's page Online Monitoring and check in the Scheduler details if the problem persists.

 

"OutSystems Scheduler Service Status: No connection could be made because the target machine actively refused it"

"OutSystems Scheduler Service Status: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond" 

Cause The "OutSystems Scheduler Service" is waiting for the Front-end Server (Hub Node) to complete the execution of a scheduled job or cannot access the database.
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding Front-end Server (Hub Node server) Scheduler indicator in Frontend Servers (Hub Nodes) section for details;
  3. Check "Jobs Producer" and "Jobs Worker" threads status and click on the Refresh Data link to confirm that the statuses are maintained.
  4. If status are still "Error" for "Jobs Producer" or for any "Jobs Worker":
    1. Login to the corresponding Front-end Server (Hub Node server) and check network configuration and communication conditions between the Front-end Server (Hub Node) and the Platform Database (Hub Database) Servers.
    2. Login to Platform Database (Hub Database) Server and check SQL Server availability and runtime conditions (CPU, memory, etc.);
    3. Execute any specific recovery action;
  5. If the problem persists:
    1. Login to the corresponding Front-end Server (Hub Node server), go to Windows Services and restart the "OutSystems Scheduler Service";
  6. Go back to Service Center's page Online Monitoring and check in the Scheduler details if the problem persists.

 

"Scheduled for execution at date time. Suspected to be blocked"

Cause The scheduled timer execution was not started at the predefined date/time.
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Check if the job is still pending for execution in the Timers section. Pending jobs will show a warning indicator;
  3. If the problem persists, click on every Scheduler indicator for details about each of them;
  4. Check all "Job Worker" thread statuses and click on the Refresh Data link to confirm that the statuses are maintained.
  5. If there are "Job Worker" threads not in "Error" status, go back to the Online Monitoring page, refreshing data until the job is executed;
  6. If they all become "Error":
    1. Login to the corresponding Front-end Server (Hub Node server), go to Windows Services and restart the "OutSystems Scheduler Service";
    2. Execute "iisreset" in console mode.
  7. Go back to Service Center's page Online Monitoring and check in the Scheduler details if the problem persists.

OutSystems Log Service Errors

"OutSystems Log Service Stopped. Please restart"

"OutSystems Log Service Status: No connection could be made because the target machine actively refused it" 

Cause The "OutSystems Log Service" at the referred Front-end Server (Hub Node) or Deployment Controller Server (Hub Controllerserver) is stopped. 
Resolution / Recovery Action
  1. Login to the Front-end Server (Hub Node) or Deployment Controller Server (Hub Controller server) and go to Windows Services;
  2. Confirm that the "OutSystems Deployment Controller Service" is not Started. Start it.

 

"OutSystems Log Service Status: Warning: Log Queue has n items"

Cause There are pending log items not yet in the database, in the "OutSystems Log Service" queue.
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding Front-end Server (Hub Node) server Log indicator in Front-end Servers (Hub Nodes) section for details;
  3. Check Queue "log" size and "Logger Listener" thread status and click on the Refresh Data link to confirm that the statuses are maintained.
  4. If the queue size does not decrease and the status is not "Waiting for messages":
    1. Login to the corresponding server, go to Windows Services and restart the "OutSystems Log Service";
    2. Restart "Message Queuing" service;
  5. Go back to Service Center's page Online Monitoring and check in the Log details if the problem persists. 

 

"OutSystems Log Service Status: Error: Logger Listener is in status SQL Exception"

"OutSystems Log Service Status: Error: Logger Listener is in status Error connecting to database"

"OutSystems Log Service Status: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"

Cause The "OutSystems Log Service" cannot access the database. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding Front-end Server (Hub Node server) Log indicator in Front-end Servers (Hub Nodes) section for details;
  3. Check "Logger Listener" thread status and click on the Refresh Data link to confirm that the statuses are maintained.
  4. If the status is still "SQL Exception" or "Error":
    1. Login to the corresponding server and check network configuration andcommunication conditions between the Front-end Server (Hub Node) and the Platform Database (Hub Database) Servers;
    2. Login to the Platform Database (Hub Database) Server and check SQL Server availability and runtime conditions (CPU, memory, etc.);
    3. Execute any specific recovery action;
  5. If the problem persists:
    1. Login to the corresponding server, go to Windows Services and restart the "OutSystems Log Service";
  6. Go back to Service Center's page Online Monitoring and check in the Log details if the problem persists. 

OutSystems SMS Connector Errors 

"OutSystems SMS Connector Service Status: No connection could be made because the target machine actively refused it"

"OutSystems SMS Connector Service Stopped. Please restart" 

Cause The "OutSystems SMS Connector Service" at the referred server is stopped.
Resolution / Recovery Action
  1. Login to the Front-end Server (Hub Node Server) and go to Windows Services;
  2. Confirm that the "OutSystems SMS Connector Service" is not Started. Start it.

 

"OutSystems SMS Connector Service Status: Warning: Process Inbound is in status Processing for more than n seconds"

"SMS Connector status: Warning: Inbound queue has n items"

Cause  The "OutSystems SMS Connector Service" is waiting to deliver one or more SMS to the Platform Server.
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding server SMS Connector indicator in Front-end Servers (Hub Nodes) section for details;
  3. Check all "Process Inbound" thread statuses and click on the Refresh Data link to confirm that the statuses are maintained or that the "inbound" queue size is rapidly increasing;
  4. If any of the above is verified:
    1. Login to the corresponding server, go to Windows Services and restart the "OutSystems SMS Connector Service";
    2. In console mode, execute "iisreset" command.
  5. Go back to Service Center's page Online Monitoring and check in the SMS Connector details if the problem persists.

 

"SMS Connector status: Warning: Process Outbound is in status Processing for more than n seconds"

"SMS Connector status: Warning: Outbound queue has n items"

Cause The "OutSystems SMS Connector Service" is waiting to deliver on or more SMS to the SMS Gateway.
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding server SMS Connector indicator in Front-end Servers (Hub Nodes) section for details;
  3. Check all "Process Outbound" thread statuses and click on the Refresh Data link to confirm that the statuses are maintained or that the "outbound" queue size is rapidly increasing;
  4. If any of the above is verified:
    1. Check the SMS Gateway execution conditions:
      1. Login to the SMS Gateway Server as "smsgtw";
      2. Execute "ps -ax | grep smsgateway". At least one process must be running;
      3. Execute "ps -ax | grep smsbox". At least one process must be running;
      4. If no processes are running, (re)start the SMS Gateway.
    2. If the SMS Gateway was executing ok:
      1. Login to the corresponding Front-end Server (Hub Node server), go to Windows Services and restart the "OutSystems SMS Connector Service"; 5
  5. Go back to Service Center's page Online Monitoring and check in the SMS Connector details if the problem persists.

 

"SMS Connector status: Warning: inbound failed queue has n items"

Cause There are pending SMS messages not yet delivered by the "OutSystems SMS Connector Service" to the applications. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding server SMS Connector indicator in Front-end Servers (Hub Nodes) section for details;
  3. Check the "Process Inbound Failed" thread status and click on the Refresh Data link to confirm that the statuses are maintained and that the "inbound failed", "internal mobile originated failed", "internal delivery report failed" queues size is rapidly increasing;
  4. If any of the above is verified:
    1. Login to the corresponding server, go to Windows Services and restart the "OutSystems SMS Connector Service";
    2. In console mode, execute "iisreset" command.
  5. Go back to Service Center's page Online Monitoring and check in the SMS Connector details if the problem persists.

Note: Under severe error conditions, inbound failed queues may take several days to become empty. This procedure should only be executed if their size is growing fast.

 

"SMS Connector Service Status: Warning: internal mobile terminated failed queue has n items"

Cause There are pending SMS messages not yet delivered by the "OutSystems SMS Connector Service" to the SMS Gateway Server. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Click on the corresponding server SMS Connector indicator in Front-end Servers (Hub Nodes) section for details; 
  3. Check the "Process Outbound Failed" thread status and click on the Refresh Data link to confirm that the statuses are maintained and that the "outbound failed", "internal mobile terminated failed" queues is rapidly increasing;
  4. If any of the above is verified:
    1. Check the SMS Gateway execution conditions:
      1. Login to the SMS Gateway Server as "smsgtw";
      2. Execute "ps -ax | grep smsgateway". At least one process must be running;
      3. Execute "ps -ax | grep smsbox". At least one process must be running;
      4. If no processes are running, (re)start the SMS Gateway.
    2. If the SMS Gateway was executing ok:
      1. Login to the corresponding Front-end Server (Hub Node server), go to Windows Services and restart "OutSystems SMS Connector Service";
  5. Go back to Service Center's page Online Monitoring and check in the SMS Connector details if the problem persists.

SMS Gateway Errors

"SMS Gateway Connection" and "status: dead"

Cause Connectivity to a large account was lost/is dead.
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Check the SMS Gateway section and the corresponding Connection status, and click on the Refresh Data link to confirm that the status is maintained;
  3. If status is still dead:
    1. Login to the SMS Gateway Server;
    2. Execute command "lynx -dump "http://localhost:13000/start-smsc?pa...msc=connection";
  4. Go back to Service Center's page Online Monitoring and check if the problem persists

Note: the admin-password is defined in the SMS Gateway configuration file ("/etc/smsgateway.conf") , group core, property admin-password. Connection is the name of the connection as presented in the Service Center monitoring page.

 

"SMS Gateway status: The operation has timed-out"

"SMS Gateway status: The underlying connection was closed: Unable to connect to the remote server"

Cause Possible Network problems between a Front-end Server (Hub Node) and the SMS Gateway server. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Check the SMS Gateway section and verify its status and click on the Refresh Data link to confirm that the status is maintained;
  3. If this situation continues:
    1. Login to the corresponding server and to the SMS Gateway Server and check network configurations and communications between them;
    2. Login to the SMS Gateway Server as "smsgtw";
    3. Execute "ps -ax | grep smsgateway". At least one process must be running;
    4. Execute "ps -ax | grep smsbox". At least one process must be running;
    5. If no processes are running, (re)start the SMS Gateway.
    6. Execute any specific recovery action;
  4. Go back to Service Center's page Online Monitoring and check if the problem persists.

Windows Server Base Service Errors

"Message Queue Service is not available"

Cause Microsoft Message Queue Service is not available. The "OutSystems SMS Connector Service" and the "OutSystems Log Service" may stop working.
Resolution / Recovery Action
  1. Login to the corresponding server and go to Windows Services;
  2. Confirm that the "Messaging Queuing" service is not Started. Start it.

 

"IIS Status:The operation has timed-out"

"IIS Stopped. Please restart" 

Cause Microsoft Internet Information Server at the Front-end Server (Hub Node server) referred in the message is not responding or is stopped. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Online Monitoring in Monitoring tab;
  2. Check the corresponding Front-end Server (Hub Node server) IIS indicator status, in Frontend Servers (Hub Nodes) section, and click on the Refresh Data link to confirm that the status is maintained;
  3. If indicator is not green:
    1. Login to the corresponding Front-end Server (Hub Node server);
    2. In console mode execute "iisreset";
  4. Go back to Service Center's page Online Monitoring and check if the problem persists. 

Common Application Errors 

"SQL Exception: Timeout expired"

"OleDbException: Timeout expired"

"Error in advanced query" SQL statement ": timeout expired"

Cause The Application timed out trying to access the database. 
Resolution / Recovery Action
  1. Login to Service Center and go to page Error Log in Monitoring tab;
  2. Refresh the page to check if the same error occurs several times;
  3. If the error occurs repeatedly:
    1. Login to the corresponding server and check network configuration and communication conditions between the Front-end Server (Hub Node) and the Platform Database (Hub Database) Servers;
    2. Login to the Platform Database (Hub Database) Server and check SQL Server availability and runtime conditions (CPU, memory, etc.);
    3. Execute any specific recovery action;
  4. Go back to Service Center's page Error Log, in Monitoring tab, and check if the problem persists.

 

"Error executing child request for node"

"Internal Server Error"

Cause If any of the above is reported consecutively for every request to a specific application, possibly the ASP.NET Temporary Files are corrupted. (Microsoft known defect)
Resolution / Recovery Action
  1. Login to Service Center and go to page Error Log in Monitoring tab;
  2. Refresh the page to check if the same error occurs several times in a single server
  3. If the error occurs repeatedly:
    1. In Service Center, go to tab Factory and select the corresponding eSpace from the eSpaces list;
    2. Press "Redeploy Published Version" button;
    3. Go to the Online Monitoring page in Monitoring tab;
    4. Check the Deploy.Controller status and click on the Refresh Data link until the indicator turns green;
    5. If the indicator doesn't turn green, go directly to step 5.1.
  4. Go to page Error Log again and refresh it;
  5. If the error occurs repeatedly:
    1. Login to the corresponding server and go to Windows Services;
    2. Stop all "OutSystems" services;
    3. In "Control Panel"->"Administrative Tools"->" Internet Services Manager", stop the "Default Web Site";
    4. In console mode, run the command "iisreset"
    5. Delete all the contents of "C:\WINNT\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files" folder
    6. In "Control Panel"->"Administrative Tools"->" Internet Services Manager", start the "Default Web Site";
    7. Start all OutSystems services;
  6. Go back to Service Center, refresh Error log page and check if the problem persists.