Troubleshooting Sometimes, you may have everything installed and set up, but something does not work as expected. This section presents some issues that you may encounter, and how to solve them. You can start troubleshooting your environment by checking these items: Appliance status. Minion status. Minion Appliance events. Appliance logs. Minion logs. Appliance Service status. Minion container configuration. You can also use the following tools: The terminal user interface (TUI). You can use it to run a health check, view system settings and logs, and list running containers and container images. Karaf commands can diagnose issues in the Minion container. Appliance Service status You can view the status of the OpenNMS Appliance Service, and subscribe to updates. If you choose to subscribe to updates, you will receive email notifications when The OpenNMS Group creates, updates, or resolves an incident. You can also check the status at https://status.opennms.com/. Global status mapping Statuses are determined by a combination of factors: Connectivity to the Appliance Service (CLOUD). Status of the Minion container (Minion). Connectivity with OpenNMS (ONMS). This table provides a high-level summary of global statuses and how they map to these indicators: Global CLOUD Minion ONMS Scenario Up Online Up Up The OpenNMS ecosystem is up. The Minion is up, running, and connected, and can connect to OpenNMS, ActiveMQ, or Kafka. Down Online Up Down The Minion is up, running, and connected, but loses connectivity to OpenNMS, ActiveMQ, or Kafka. Down Online Down None The Minion container does not spawn due to a snap or configuration issue. This could also indicate a failure to reach Docker Hub via an HTTP proxy. Unknown Online Unknown Unknown Indicates a transient state. For example, it may be taking a longer time than usual to download the Minion container. This could also indicate configuration issues or Appliance Service-side infrastructure issues. Initializing Online Grace period Grace period This usually indicates a transient state where Minion is restarting. It could also indicate a configuration issue. Offline Offline Unknown Unknown The appliance was created and never connected to the Appliance Service. Ready Online Not configured None The appliance does not have a Minion container installed, but is connected to the Appliance Service. Download Minion container configuration These instructions are intended for those who are familiar and comfortable with the OpenNMS Portal. We recommend that you use the Appliance Utilities tools only if you are an advanced user. You can download the configuration that the Minion container uses to help the troubleshooting process, or to work with support. The Minion must be running when you submit a request to download the container configuration. To download the configuration, follow these steps: Find the appliance whose Minion container configuration you want to download. Click Download beside Minion Container Configuration. Follow any prompts that your browser displays to save the file locally. The file name is in the following format: <appliance-name>-Minion-config.yaml (for example, new-york-Minion-config.yaml). Karaf commands You can run some Karaf commands directly in the OpenNMS Portal to help troubleshoot issues in the Minion container. This feature works only with appliances that are online. Note that the OpenNMS Portal Karaf shell has the following limitations: Interactive commands (ttop, log:tail, and so on) do not work. shell:exec commands do not work. Output for some commands may be truncated (display is limited to 100KB). Some configuration-related commands do not persist; any changes will revert the next time the Minion container restarts. Some commands may cause a container restart. Sample Karaf commands include the following: metrics-display: Display health metrics for events, flows, and Minion heartbeats. list-collectors: List all available collectors. list-detectors: List all available detectors. list-monitors: List all available monitors. health-check: Test communication with your OpenNMS Meridian instance. For more details on Karaf commands, see the Karaf CLI cheat sheet on Discourse. Run Karaf commands Only users with a thorough understanding of Karaf commands should use this method. To troubleshoot using Karaf commands, follow these steps: Click Appliances in the left navigation menu. Locate the appliance that you want to troubleshoot, and click the vertical ellipsis (⋮) beside its name. Select Appliance Utilities. In the Karaf Command area, type the command and click Run Command. The command and its results are displayed in the terminal. Sample error messaging This table provides sample scenarios and their associated messaging: Scenario Appliance Minion OpenNMS Global Events Appliance Service Status All working. Online Up Up Up Info: Spawned Minion with ID <ID number> in location <location name> <date> ONMS FQDN resolution failure. Online Unknown Unknown Unknown Generated every minute: Warning: Failed to communicate with the Meridian instance associated with Minion with ID <ID> due to 'io.micronaut.http.client.exceptions.HttpClientException: Connect Error: IP address: Name or service not known'. ONMS IP not reachable. Online Unknown Unknown Unknown Generated every minute: Warning: Failed to communicate with the Meridian instance associated with Minion with ID <ID> due to 'io.micronaut.http.client.exceptions.HttpClientException: Connect Error: No route to host: /IP address'. ONMS stopped. Online Respawn loop Empty Initializing Warning: Failed to communicate with the Meridian instance associated with Minion with ID <ID> due to 'io.micronaut.http.client.exceptions.HttpClientException: Connect Error: Connection refused: /IP address'. <date> <time> ONMS ports blocked by firewall. Online Unknown Unknown Unknown See "ONMS stopped". REST HTTP port not listening. Online Unknown Unknown Unknown Generated every minute: Warning: Failed to communicate with the Meridian instance associated with Minion with ID 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientException: Connect Error: Connection refused: /IP address'. HTTPS on non-secure. Online Unknown Unknown Unknown Generated every minute: Warning: Failed to communicate with the Meridian instance associated with Minion with ID 'kiwi1' due to 'java.lang.RuntimeException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 485454502f312e312034303020496c6c6567… HTTP on secure. Online Unknown Unknown Unknown Generated every minute: Warning: Failed to communicate with the Meridian instance associated with Minion with ID 'kiwi1' due to 'io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout'. Wrong name Online Unknown Unknown Unknown Generated every minute: Warning: Failed to communicate with the Meridian instance associated with Minion with ID 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientResponseException: <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Bad credentials</title> </head> <body><h2>HTTP ERROR 401 Bad credentials</h2> <table> <tr><th>URI:</th><td>/opennms/rest/info</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Bad credentials</td></tr> <tr><th>SERVLET:</th><td>cxfRestServlet</td></tr> </table> <hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.34.v20201102</a><hr/> </body> </html>'. Wrong password Online Unknown Unknown Unknown Generated every minute: Warning: Failed to communicate with the Meridian instance associated with Minion with ID 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientResponseException: <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Bad credentials</title> </head> <body><h2>HTTP ERROR 401 Bad credentials</h2> <table> <tr><th>URI:</th><td>/opennms/rest/info</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Bad credentials</td></tr> <tr><th>SERVLET:</th><td>cxfRestServlet</td></tr> </table> <hr><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.34.v20201102</a><hr/> </body> </html> '. <date> <time> ROLE_MINION not assigned to user account. Online Up * Up * Empty None Certificate - missing CA. Online Unknown Unknown Unknown Generated every minute: Warning: Failed to communicate with the Minion instance associated to Minion with ID 'kiwi1' due to 'java.io.IOException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target'. ActiveMQ These scenarios assume that the HTTP section is properly configured to access REST. OpenNMS not listening on port. Online Respawn loop Empty Initializing No events to indicate what the problem might be other than a failing health check. The Minion is in a restart loop. Warning: Restarting Minion with ID 'kiwi1' due to failed health check. SSL on non-secure. Online Respawn loop Empty Initializing No events to indicate what the problem might be other than a failing health check. The Minion is in a restart loop. Non-secure on secure. Online Respawn loop Empty Initializing No events to indicate what the problem might be other than a failing health check. The Minion is in a restart loop. Warning: Restarting Minion with ID 'kiwi1' due to failed health check. Wrong name Online Up Grace period/Down loop Initializing/Down No events to indicate what the problem might be. Wrong password Online Up Grace period/Down loop Initializing/Down No events to indicate what the problem might be. ROLE_MINION not assigned to user account. Online Up Grace period/Down loop Initializing/Down No events to indicate what the problem might be. Portal Operations Stop OpenNMS. Online Up Down Down Warning: Failed to communicate with the Meridian instance associated with Minion with ID 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientException: ConnectError: Connection refused: /192.168.1.51:8443'. Starting OpenNMS. Online Up Up Up No events; indicates that "connectionrefused" events have stopped. Restarting OpenNMS. Online Up Down/Up Down/Up See "Stop OpenNMS". Pull ethernet on OpenNMS. Online Up Down Down Generated every minute: Warning: Failed to communicate with the Meridian instance associated with Minion with ID 'kiwi1' due to 'io.micronaut.http.client.exceptions.HttpClientException: ConnectError: No route to host: /192.168.1.51:8443'. <date> <time> Plug ethernet back on OpenNMS. Online Up Up Up See "Starting OpenNMS". Deleting a Minion. Online None None Ready Info: De-spawning Minion with ID 'kiwi1' due to it being deleted. <date> <time> Readying a Minion. Online Unknown > Up Unknown > Up Ready > Up Info: Spawned Minion with ID 'kiwi1' in location 'Home Secure IP'12/18/2020 12:57:27InfoSpawning Minion with Id 'kiwi1' for newly discovered instance in location 'Home Secure IP'. <date> <time> Deleting appliance. Empty Empty Empty Empty None Deleting location. Online Empty Unknown? Empty Info: De-spawning Minion with ID 'kiwi1' due to it being deleted. <date> <time> Deleting connectivity profile. Online Empty Empty Unknown? Info: De-spawning Minion with ID 'kiwi1' due to it being deleted. <date> <time> Deleting feature profile. Online Unknown/Grace period loop Unknown/Grace period Unknown/Initializing Warning: Failed to communicate with the Meridian instance associated with Minion with ID 'kiwi1' due to 'java.lang.NullPointerException: Parameter specified as non-null is null: method org.opennms.dominion.minion.docker.BaseLocalDockerClientKt.addConfigs, parameter config'. <date> <time> Info: Spawning Minion with ID 'kiwi1' for newly discovered instance in location 'Home Secure IP'. <date> <time> After re-adding a feature profile under Location: Info: Spawned Minion with ID 'kiwi1' in location 'Home Secure IP'. <date> <time> Info: Spawning Minion with ID 'kiwi1' for newly discovered instance in location 'Home Secure IP'. <date> <time> Info: Configuration changed for Minion with ID 'kiwi1'. Minion replacement in progress (de-spawning and spawning). <date> <time> Remove certificate from connectivity profile. Online Unknown/Grace period loop Unknown/Grace period loop Unknown/Initializing Info: Spawned Minion with ID 'kiwi1' in location 'Home Secure IP'. <date> <time> Delete certificate from UI. Online Unknown/Grace period loop Unknown/Grace period loop Unknown/Initializing See "Remove certificate from connectivity profile". Subscription expiry. Online Unknown Unknown Unknown Info: De-spawning Minion with ID 'kiwi1' due to it being deleted. Subscription deallocation. Online Down Unknown Unknown Info: De-spawning Minion with ID 'kiwi3' due to it being deleted. <date> <time> Assigning Minion to a different instance. Online Up > Unknown > Grace period > Up Up > Unknown > Grace period > Up Up > Unknown > Initializing > Up Info: Spawned Minion with ID 'kiwi1' in location 'Home Secure IP'. <date> <time> Info: Spawning Minion with ID 'kiwi1' for newly discovered instance in location 'Home Secure IP'. <date> <time> Info: Configuration changed for Minion with ID 'kiwi1'. Minion replacement in progress (de-spawning and spawning). <date> <time> Upgrading OpenNMS. Online Up > Unknown > Grace period > Up Up > Unknown > Grace period > Up Up > Unknown > Initializing > Up Info: Handling version change for Minion with ID '2c216971-3585-41ea-b274-54efefbf6cab' from 'opennms/minion:27.0.0' to 'opennms/minion:27.0.2'. Minion replacement in progress (de-spawning and spawning). <date> <time> Info: Spawned Minion with ID '2c216971-3585-41ea-b274-54efefbf6cab' in location 'Home Not Secure IP'. Snap upgrade with restart. Online None None None Info: Successfully connected to the cloud. <date> <time> Appliance Administration Security and Firewall Rules