Skip to main content

RabbitMQ management and troubleshooting

OutSystems

RabbitMQ management and troubleshooting

This article applies to:OutSystems 11

The default RabbitMQ batch scripts folder mentioned in the sections below is the following:

C:\<OutSystems Install Folder>\thirdparty\RabbitMQ Server\rabbitmq_server-<version>\sbin

Adjust it if necessary.

Enable the RabbitMQ management plugin

The best way to monitor and manage RabbitMQ is by using the management plugin included in the installation. To enable it, do the following:

  1. Open a command-line console (run as Administrator) and change to the RabbitMQ batch scripts folder;

  2. Execute the following command:
    rabbitmq-plugins.bat enable rabbitmq_management

The management dashboard will be available at http://localhost:15672/. You can login using the administrator credentials that have been set during service configuration.

Check the service status

You can use the CLI tools provided by RabbitMQ to check the service status.

Do the following:

  1. Open a command-line console (run as Administrator) and change to the RabbitMQ batch scripts folder;

  2. Execute the following command:
    rabbitmqctl.bat status

Troubleshooting

Service installation issues

If you find any issues while installing the Cache Invalidation Service, which uses RabbitMQ as its underlying technology, follow these troubleshooting suggestions in order:

Check if Erlang was correctly installed on your machine

A) Check the installed version of Erlang by following these steps:

  1. Run the Erlang shell by executing the following command:
    C:\<OutSystems Installation Folder>\thirdparty\Erlang\bin\werl.exe

  2. Checking the version information at the top of the screen; it should match the following:
    Erlang/OTP 20 [erts-9.3] [64-bit]

B) Check if the ERLANG_HOME environment variable is defined and pointing to the Erlang installation folder.

Check the Erlang cookies

A) Make sure that the .erlang.cookie files in %USERPROFILE% and in the Windows system profile folders (C:\Windows\System32\config\systemprofile, C:\Windows\SysWOW64\config\systemprofile, C:\Windows\SysNative\config\systemprofile) are synchronized, i.e. they have the exact same content.

If they're not the same, copy the contents of one file to the other, replacing its contents.

Check the RabbitMQ service status

A) Check if the RabbitMQ service is installed and running by opening the Task Manager.

If it's not, do the following:

  1. Open a command-line console (run as Administrator) and change your directory to the RabbitMQ batch scripts folder.

  2. Execute the following commands to remove the currently running service, install a new one, and start it:
    rabbitmq-service.bat remove
    rabbitmq-service.bat install
    rabbitmq-service.bat start

Check if RabbitMQ port is being blocked

A) If you are using multiple front-end servers, check that the port used by RabbitMQ (port 5672, by default) is not being blocked by the firewall.

To change the port used by the RabbitMQ service:

  1. Set the RABBITMQ_NODE_PORT environment variable to the desired port number;
  2. Reinstall the RabbitMQ service, execute the same commands described in the previous issue ("Check the RabbitMQ service status").
Check the logs for any errors
A) Check if there are any errors in the RabbitMQ service log files. The log files are stored in %ALLUSERSPROFILE%\RabbitMQ\log.

Common issues

The RabbitMQ service fails to start

A) Make sure that the %ALLUSERSPROFILE% folder is accessible by the user running Configuration Tool.

B) If the %ALLUSERSPROFILE% folder is located in a network share, make sure that the "Workstation" Windows service is running.

C) Check the log file (located in %ALLUSERSPROFILE%\RabbitMQ\log) for any fatal errors.

The RabbitMQ service is not installed and its installation/removal procedure throws an error

Description: The RabbitMQ service does not appear in the Windows Services list. Running rabbitmq-service.bat install or rabbitmq-service.bat remove throws an error: "The handle is invalid".

A) Delete the RabbitMQ variables from the registry, located at HKEY_LOCAL_MACHINE\SOFTWARE\Ericsson\Erlang\ErlSrv\1.1\RabbitMQ, and reinstall the service.

The RabbitMQ broker is not starting

A) Ensure that the Erlang cookies are synchronized (follow the instructions in "Check the Erlang cookies" above).

B) Ensure that the ERLANG_HOME variable is defined and pointing to the Erlang folder.

RabbitMQ crashes soon after failing and there's an error in the log file saying {error,{already_started,<...>}}

A) Ensure that there is no TLS listener using the same port as a non-TLS listener. Listeners can be configured in %ALLUSERSPROFILE%\RabbitMQ\rabbitmq.conf or %ALLUSERSPROFILE%\RabbitMQ\advanced.config.

B) Ensure that RABBITMQ_NODE_PORT environment variable, which defines the default port used by non-TLS listeners, is not set to the same port as an TLS listener.

Applications cannot reach the RabbitMQ service. The "Test Connection" operation in Configuration Tool (Cache tab) fails.

A) Check if the RabbitMQ service is running.

B) Check that there is a firewall exception for port number 5672.

C) Check if there is a RabbitMQ node running by following these steps:

  1. Open a command-line console (run as Administrator) and change to the RabbitMQ batch scripts folder.

  2. Run the following command:
    rabbitmqctl.bat status

  3. If the rabbitmq@<machine_name> node is not running, run the following command to start it:
    rabbitmqctl.bat start_app

D) Check if the RabbitMQ user was successfully created by following these steps:

  1. Open a command-line console (run as Administrator) and change to the RabbitMQ batch scripts folder.

  2. Run the following command:
    rabbitmqctl.bat list_users

  3. If the user it's not listed in the command output, run the following command to create it:
    rabbitmqctl.bat add_user <admin_username> <admin_password>

E) Check if the RabbitMQ user has permissions for the virtual host (vhost) defined on installation script (/outsystems by default) by following these steps:

  1. Run the following command:
    rabbitmqctl.bat list_permissions

  2. If the user doesn't have permissions, run the following command:
    rabbitmqctl.bat set_permissions -p /outsystems <admin_username> ".*" ".*" ".*"

F) If the issue persists, escalate the case.

Event Viewer contains messages like "Error executing PubSub jobs, X consecutive Jobs failed."

Description: A job failed to complete due to a misconfiguration or inconsistent state. Jobs are: creating queues, creating exchanges, register a queue into an exchange.

A) If it happens consistently and the number of consecutive jobs failed is high, try to redeploy the module.

B) If the issue persists, escalate the case.

Event Viewer contains messages like "Error executing PubSub jobs, X consecutive connections failed."

Description: The module cannot reach the RabbitMQ service.

A) Follow the instructions provided in "Applications cannot reach the RabbitMQ service" to troubleshoot this issue.