One of the following becomes slower after an update / upgrade to OutSystems version 11.7.x
Solution publishing in ServiceCenter
Application publishing in ServiceCenter
LifeTime staging (via UI or API) to a target environment running 11.7.x or higher
When compared to publishing prior to the upgrade, the duration increased by 20% or more; this is especially felt in the deployment phase of the publishing operation.
The problem manifests if the following combination of factors occurs in the target environment (all conditions must apply):
Running Platform Server 11.7.x;
Farm installations (with 2 or more front-ends);
The deployment zone address is the load balancer's hostname
PaaS customers are not affected
Confirm that you are affected
- Confirm the bottleneck on the Deployment phase
Analyze the detailed publishing log (in Service Center or LifeTime) to confirm that each individual step of the Deployment phase takes longer than before (consult a prior deployment log if needed). Moreover, for each module (not extensions) the deployment takes approximately 1 minute and 30 seconds, as depicted below, instead of a few seconds as previously.
Updating tenant views of module ‘AdminLead_cs’. Updating tenant views of module ‘AutomatedEmails_cs’. Deploying module ‘RichWidgets’. (Last Step took 315). Deploying module ‘Users’. (Last Step took 1m 27s). Deploying module ‘W_Theme_UI’. (Last Step took 1m 285). Deploying module ‘Countries’. (Last Step took 1m 28s). Deploying module ‘WorldTimezone’. (Last Step took 1m 28s). Deploying module ‘DiffEntityDataToText’. (Last Step took Im 28s).
- Identify the slow RefreshISAPIFilters action calls
Once you confirm the previous pattern, filter the Service Center's General logs for SLOWEXTENSION and identify multiple entries like the below, during the deployment period for each module deployed, with durations of approximately 90.000 ms:
OMLProcessor.RefreshISAPIFilters took 88765 ms [actual value may vary]
- Identify the errors connecting to Deployment Service
After confirming the pattern above, locate errors like the below in Service Center's Error log - approximately one for each slow RefreshISAPIFilters in the same time period. Note the entry referring to RefreshPathRules - this is the important part. If you don’t see this, it’s not the relevant error:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.21.8:12001 (...) Exception rethrown at : at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(...) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(...) at OutSystems.HubEdition.IBroadcastListener.MessageTransmission(...) at OutSystems.HubEdition.DeploymentController.Compiler.RefreshPathRules(...) at cs#pbxmkwit.InnerExecute() at OutSystems.HubEdition.ServerCommon.Tasks.AbstractTask.Execute()
The problem occurs because the deployment controller service is trying to communicate with the deployment service through the IP address of the deployment zone address, instead of the front-end server address.
In situations where that communication is not possible and the network connection is not explicitly rejected, the deployment controller will wait until a timeout is attained. This wait occurs for each module being deployed.
A workaround is already available for this problem in the most frequent farm configuration (farms in which the deployment controller is also a front-end), in the form of a reconfiguration of the platform. Environments with pure deployment controllers may opt to promote their deployment controller to also be a front-end, and then apply the workaround.
This workaround can only be applied when the controller server is also a front-end, and it has the deployment service running.
To overcome this situation, reconfigure the platform so that the deployment controller communicates with its own deployment service on IP 127.0.0.1.
To do that, the required steps are:
1. Execute the following query in the environment's database:
insert into ossys_parameter (name, VAL) VALUES ('Compiler.UseFrontEndAddressRefreshPaths', 'False')
2. Restart the deployment controller service in the controller server.