Troubleshooting

Determining Controller Status

The controller is bundled with a status script that checks different components of the controller installation. The script is installed to /opt/undeployme/status.sh and must be run with root privileges.

The output from a healthy controller appears similarly to the following:

OK: vermilion-core is running, pid=26852.
OK: cobalt is running, pid=26944.
OK: rabbitmq-server is running, pid=3914.
OK: rabbitmq-server shovels: 8 running.
OK: rabbitmq-server <-> vermilion-core connection established.
OK: rabbitmq-server <-> cobalt connection established.
OK: mongod is running, pid=26192.
OK: mongod <-> vermilion-core total connections: 1.
OK: mongod <-> cobalt total connections: 33.

Monitoring Service Status

The controller is bundled with a watchdog application, Monit, which will restart the controller after an abnormal termination or out-of-memory event. The following commands can be used to control Monit’s behavior:

  • monit status to view the services’ current state
  • monit unmonitor all to disable service monitoring
  • monit monitor all to re-enable service monitoring
  • /etc/init.d/monit stop to shut down Monit

Note

Reinstalling or upgrading the controller will re-enable service monitoring.

Note

Monitoring will be re-enabled automatically if the corresponding service (cobalt or vermilion-core) is restarted manually using an init.d script or if the operating system reboots.

Log Locations

  • MongoDB: /var/log/mongo/*
  • RabbitMQ: /var/log/rabbitmq/*
  • Cobalt: /opt/qubell/cobalt/logs/*
  • Vermilion-core: /opt/qubell/core/logs/*
  • Monit: /var/log/messages

Help

If a system malfunction occurs, please email support@tonomi.com. Please include the following information in your support request:

  1. All logs from the bundled applications (see “Log Locations,” above).
  2. Output from the rabbitmqctl report command.
  3. Output from the following commands:
/etc/init.d/rabbitmq-server status
/etc/init.d/mongod status
/etc/init.d/vermilion-core status
/etc/init.d/cobalt status
/usr/bin/monit status
  1. Output from the /opt/qubell/status.sh script.
  2. Error description, along with output/screenshot (if applicable).