Why doesn’t this work? You have everything installed and set up, but something is not working as expected. This section presents some issues you may encounter and how to solve them. Where to start Start troubleshooting by checking the following: Appliance status Minion status Appliance events Appliance/Minion logs Appliance Service status Minion container configuration You can also do the following: Sign in to the terminal user interface (TUI) to run a health check, view system settings and logs, and list running containers and container images. Run Karaf commands to diagnose issues in the Minion container. Appliance Service status You can view the status of the OpenNMS Appliance Service and subscribe to updates. If you subscribe to updates, you will receive email notifications whenever The OpenNMS Group creates, updates, or resolves an incident: https://status.opennms.com/ Global status mapping States are determined by a combination of indicators: connectivity to the Appliance Service (CLOUD), status of the Minion container (Minion), and connectivity with OpenNMS (ONMS). The following table provides a high-level summary of global statuses and how they map to CLOUD/Minion/ONMS indicators. Global Cloud Minion ONMS (OpenNMS) Scenario Up Online Up Up OpenNMS ecosystem is up. Minion up and running and connected, and can connect to OpenNMS/ActiveMQ or Kafka. Down Online Up Down Minion up and running and connected, but loses connectivity to OpenNMS/ActiveMQ or to Kafka. Down Online Down N/A Minion container fails to spawn due to snap issue or configuration issue (failure to reach Docker Hub via HTTP proxy). Unknown Online Unknown Unknown Transient state. May be taking a long time to download Minion container. Misconfiguration issues. Appliance Service-side infrastructure issues. Initializing Online Grace period Grace period Usually transient state where Minion is restarting. Can be a misconfiguration issue. Offline Offline Unknown Unknown Appliance created and never connected to Appliance Service. Ready Online Not configured N/A Appliance does not have Minion container installed, but is connected to Appliance Service. Download Minion container configuration You can download the configuration that the Minion container uses to help troubleshoot or work with support. The Minion must be running when you submit a request to download the Minion container configuration. Navigate to Appliances and locate the appliance whose Minion container configuration you want to download. Click the vertical ellipsis (⋮) to view the drop-down menu, and click Appliance Utilities. Beside Minion Container Configuration, click Download. Follow any download prompts your browser displays to save the file locally. The file is in the format <appliance-name>-Minion-config.yaml (for example, new-york-Minion-config.yaml). Run Karaf commands You can run some Karaf commands directly in the OpenNMS Portal to help troubleshoot issues in the Minion container. This feature works only with appliances that are online. Note that this is not a full Karaf shell, and limitations include the following: Interactive commands such as ttop or log:tail do not work (ttop returns the shell prompt. log:tail returns a timeout expired message.). shell:exec commands do not work. Output for some commands may be truncated (display is limited to 100KB). Some configuration-related commands do not persist; any changes will be gone the next time the container restarts. Some commands may cause a container restart. Sample commands include the following: metrics-display (display health metrics for events, flows, and Minion heartbeats) list-collectors (list all available collectors) list-detectors (list all available detectors) list-monitors (list all available monitors) health-check (test communication with OpenNMS core instance) For more details on Karaf commands, see the Karaf CLI cheat sheet on Discourse. Only users with a thorough understanding of Karaf commands should use this advanced troubleshooting method. Navigate to Appliances and locate the appliance you want to troubleshoot. Click the vertical ellipsis (⋮) to view the drop-down menu and click Appliance Utilities. In the Karaf Command area, type the command and click Run Command. The command and its results will appear in the terminal screen. Sample error messaging The following table provides sample scenarios and associated messaging. Scenario Appliance Service Status Appliance (offline/online) Minion (UP, UNKNOWN, GRACE_PERIOD) OpenNMS (UP, UNKNOWN, GRACE_PERIOD) Global Events All working Online UP UP Up Info: Spawned Minion with Id 'X' in location 'Y' Date ONMS FQDN resolution failure Online UNKNOWN UNKNOWN Unknown Generated every minute: Warning Failed to communicate with the OpenNMS instance associated with Minion with Id 'X' due to 'io.micronaut.http.client.exceptions.HttpClientException: Connect Error: IP address: Name or service not known'. ONMS IP not reachable Online UNKNOWN UNKNOWN Unknown Generated every minute: Warning Failed to communicate with the OpenNMS instance associated with Minion with Id 'X' due to 'io.micronaut.http.client.exceptions.HttpClientException: Connect Error: No route to host: /IP address'. ONMS stopped Online Respawn loop Initializing Warning: Failed to communicate with the OpenNMS instance associated with Minion with Id 'X' due to 'io.micronaut.http.client.exceptions.HttpClientException: Connect Error: Connection refused: /IP address'. [date] [time] ONMS ports blocked by FW Online UNKNOWN UNKNOWN Unknown As above REST HTTP port not listening Online UNKNOWN UNKNOWN Unknown Generated every minute: Warning Failed to communicate with the OpenNMS instance associated with Minion with Id 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientException: Connect Error: Connection refused: /IP address'. HTTPS on non-secure Online UNKNOWN UNKNOWN Unknown Generated every minute: Warning Failed to communicate with the OpenNMS instance associated with Minion with Id 'kiwi1' due to 'java.lang.RuntimeException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 485454502f312e312034303020496c6c6567… HTTP on secure Online UNKNOWN UNKNOWN Unknown Generated every minute: Warning Failed to communicate with the OpenNMS instance associated with Minion with Id 'kiwi1' due to 'io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout'. Wrong name Online UNKNOWN UNKNOWN Unknown Generated every minute: Warning Failed to communicate with the OpenNMS instance associated with Minion with Id 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientResponseException: <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Bad credentials</title> </head> <body><h2>HTTP ERROR 401 Bad credentials</h2> <table> <tr><th>URI:</th><td>/opennms/rest/info</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Bad credentials</td></tr> <tr><th>SERVLET:</th><td>cxfRestServlet</td></tr> </table> <hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.34.v20201102</a><hr/> </body> </html> '. Wrong password Online UNKNOWN UNKNOWN Unknown Generated every minute: Warning Failed to communicate with the OpenNMS instance associated with Minion with Id 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientResponseException: <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Bad credentials</title> </head> <body><h2>HTTP ERROR 401 Bad credentials</h2> <table> <tr><th>URI:</th><td>/opennms/rest/info</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Bad credentials</td></tr> <tr><th>SERVLET:</th><td>cxfRestServlet</td></tr> </table> <hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.34.v20201102</a><hr/> </body> </html> '. 12/22/2020 08:50:23. ROLE_MINION not assigned to user Online UP* UP* Certificate - missing CA Online UNKNOWN UNKNOWN UNKNOWN Every minute: Warning Failed to communicate with the OpenNMS instance associated to Minion with Id 'kiwi1' due to 'java.io.IOException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target'. Active MQ (scenarios below assume that the HTTP section is properly configured to access REST) OpenNMS not listening on port Online Respawn loop Initializing No events to indicate what the problem might be apart from failing health check. Minion is in a restart loop. Warning: Restarting Minion with Id 'kiwi1' due to failed health check. SSL on non-secure Online Respawn loop Initializing No events to indicate what the problem might be apart from failing health check. Minion is in a restart loop. Non-Secure on secure Online Respawn loop Initializing No events to indicate what the problem might be apart from failing health check. Minion is in a restart loop. Warning Restarting Minion with Id 'kiwi1' due to failed health check. Wrong name Online UP GRACE_PERIOD DOWN loop Initializing, Down No events to indicate what the problem might be. Wrong password Online UP GRACE_PERIOD DOWN loop Initializing, Down No events to indicate what the problem might be. ROLE_MINION not assigned to user Online UP GRACE_PERIOD DOWN loop Initializing, Down No events to indicate what the problem might be. Certificate - missing CA Online Respawn loop Initializing, Down No events to indicate what the problem might be. Portal Operations (from active) Stop OpenNMS OnlineLast Status Update: <frequency?>Last Appliance activity: <frequency?> UP DOWN Down Warning: Failed to communicate with the OpenNMS instance associated with Minion with Id 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientException: ConnectError: Connection refused: /192.168.1.51:8443'. Starting OpenNMS Online UP UP Up No events … just that the events about "connectionrefused" stop. Restarting OpenNMS Online UP DOWN/UP Down, Up Same as stop OpenNMS Pull ethernet on OpenNMS Online UP DOWN Down Warning: Failed to communicate with the OpenNMS instance associated with Minion with Id 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientException: ConnectError: No route to host: /192.168.1.51:8443'.12/18/2020 12:47:19 Every minute Plug ethernet back on OpenNMS Online UP UP Up No events …. just that the events about "connectionrefused" stop. Deleting a Minion Online N/A N/A Ready Info: De-spawning Minion with Id 'kiwi1' due to it being deleted.12/18/2020 12:54:22 Readying a Minion Online UNKNOWN→UP UNKNOWN→UP Ready→Up Info: Spawned Minion with Id 'kiwi1' in location 'Home Secure IP'12/18/2020 12:57:27InfoSpawning Minion with Id 'kiwi1' for newly discovered instance in location 'Home Secure IP'.12/18/2020 12:57:26 Deleting appliance N/A N/A N/A Deleting location Online Minion Deleted N/A Unknown? Minion Deleted N/A Info: De-spawning Minion with Id 'kiwi1' due to it being deleted.12/18/2020 14:32:17 Deleting connectivity profile Online Minion Deleted N/A Minion Deleted N/A Unknown? Info: De-spawning Minion with Id 'kiwi1' due to it being deleted.12/18/2020 15:41:25 Deleting feature profile Online UNKNOWN/GRACE_PERIOD In respawn loop UNKNOWN/GRACE_PERIOD Unknown, Initializing Warning: Failed to communicate with the OpenNMS instance associated with Minion with Id 'kiwi1' due to 'java.lang.NullPointerException: Parameter specified as non-null is null: method org.opennms.dominion.minion.docker.BaseLocalDockerClientKt.addConfigs, parameter config'.12/18/2020 15:05:32InfoSpawning Minion with Id 'kiwi1' for newly discovered instancein location 'Home Secure IP'.12/18/2020 15:05:32After re-adding a feature profile under LocationInfo Spawned Minion with Id 'kiwi1' in location 'Home Secure IP'12/18/2020 15:14:41InfoSpawning Minion with Id 'kiwi1' for newly discovered instancein location 'Home Secure IP'.12/18/2020 15:14:40InfoConfiguration changed for Minion with Id 'kiwi1'. Minion replacement in progress(de-spawning and spawning).12/18/2020 15:14:40 Remove certificate from connectivity profile Online UNKNOWN/GRACE_PERIOD In respawn loop UNKNOWN/GRACE_PERIOD In respawn loop Unknown, Initializing Info Spawned Minion with Id 'kiwi1' in location 'Home Secure IP' 12/18/2020 16:01:05 Delete certificate from UI Online UNKNOWN/GRACE_PERIOD In respawn loop UNKNOWN/GRACE_PERIOD In respawn loop Unknown, Initializing Same as above Subscription expiry Online UNKNOWN UNKNOWN Unknown Info De-spawning Minion with Id 'kiwi1' due to it being deleted. Subscription deallocation Online DOWN UNKNOWN Unknown Info De-spawning Minion with Id 'kiwi3' due to it being deleted. 01/19/2021 07:59:31 Assigning Minion to a different instance Online UP→UNKNOWN→GRACE_PERIOD→UP UP→UNKNOWN→GRACE_PERIOD→UP Up→Unknown→Initializing→Up Info: Spawned Minion with Id 'kiwi1' in location 'Home Secure IP' 12/21/2020 14:46:17 Info: Spawning Minion with Id 'kiwi1' for newly discovered instance in location 'Home Secure IP'. 12/21/2020 14:46:15 Info: Configuration changed for Minion with Id 'kiwi1'. Minion replacement in progress (de-spawning and spawning). 12/21/2020 14:46:14 Upgrading OpenNMS Online UP→UNKNOWN→GRACE_PERIOD→UP UP→UNKNOWN→GRACE_PERIOD→UP Up→Unknown→Initializing→Up Info Handling version change for Minion with Id '2c216971-3585-41ea-b274-54efefbf6cab' from 'opennms/minion:27.0.0' to 'opennms/minion:27.0.2'. Minion replacement in progress (de-spawning and spawning). 12/10/2020 15:07:13 Info: Spawned Minion with Id '2c216971-3585-41ea-b274-54efefbf6cab' in location 'Home Not Secure IP' Snap upgrade with restart Online Info Successfully connected to the cloud. 12/23/2020 15:38:03 Portal Administration Security and Firewall Rules