If there are errors related to the SELinux modules for the ANMS containers during installation of the ANMS on the local host, as discussed in Section 3.2, add the following line to the Puppet common.yaml file, typically found at puppet/data/common.yaml, and redeploy.

selinux::mode: permissive

This will result in the host being in permissive mode which allows activity not defined in SELinux modules but records those events to the system audit log. See Section 3.6.2 for details on observing the audit log events.

If the Grafana panels in the Monitor tab displays Connection was reset errors, the Grafana container may not have started successfully.

Restart the container with podman compose up grafana (run from within the anms/ folder).

If restarting the container does not resolve the problem, and the Grafana startup contains errors related to only having read-only access to the database, permissions on various files in the source code will need to be updated for Grafana to run.

The following permissions example is for typical a Docker system with its privileged daemon. Rootless podman may require different permissions.

For both the docker_data/grafana_vol/ folder and the docker_data/grafana_vol/grafana.db file, change the group to docker and the permissions to 777:

$ sudo chgrp docker docker_data/grafana__vol
$ sudo chgrp docker docker_data/grafana_vol/grafana.db
$ sudo chmod 777 docker_data/grafana_vol
$ sudo chmod 777 docker_data/grafana_vol/grafana.db

After changing these permissions, run podman compose up grafana again, and the Grafana container should start successfully.

If an Agent is not present in the Agents tab on start up, it is likely due to an error in one of the ION containers and their connection to the underlying database.

To resolve the issue, restart the ION containers using podman compose restart n1 n2.

If registering a new Agent does not result in an update to the displayed Agents in the ANMS Agent tab, check that it has been registered to the Manager via the nm-manager CLI. The nm-manager CLI is accessible from a terminal, and this check can be done using a command such as:

podman compose -p anms exec amp-manager journalctl -f --unit refdm-proxy

If the results confirm that the Agent is registered but it still does not show on the Agents tab of the ANMS, there may be an issue with connection between the Manager and ANMS database.

This can be manually resolved by adding the Agent via the adminer DB tool that is deployed as part of the docker-compose tool at http://localhost/. The connection information is described in Section 4.1.2.4.

To see what is present in the underlying AMP database, use the adminer access point.

With ANMS running, go to localhost:8080 and log in to the database with: - System: PostgreSQL - Server: postgres - Username: root - Password: root - Database amp_core

This error may indicate that the anms-ui docker is experiencing issues receiving HTTP requests. This is most likely related to the host or bind address specified in anms-ui/server/shared/config.py, an environment variable that overrides this, or a firewall issue.

If http://hostname:9030 (replace hostname with the server’s hostname) displays the ANMS UI, but http://hostname does not render the same page, this indicates an issue with the frontend HTTP router.

Check the status of the authnz container in the compose services list. It may be necessary to restart the container using: sudo podman compose -p anms restart authnz.

Port numbers can be overridden through environment variables (see .env file). Check that ports are mapped to the expected ports and are not being blocked by your system’s firewall (if applicable).

When running rootless podman the container may fail to start if the user does not have permission to bind on the configured port(s). Users typically cannot bind to low numbered ports, including 80 (http) and 443 (https). If this is the issue, try to set AUTHNZ_PORT and AUTHNZ_HTTPS_PORT to higher values and test at the specified port (i.e., In .env set AUTHNZ_PORT=9080 and test at http://hostname:9080).

Check that the authnz container is running with podman compose ps.

Logs can be viewed with podman compose logs authnz to identify potential issues.

Following an upgrade or failed installation step, it is possible for the system to be in an inconsistent state resulting in containers failing to start, initialize properly, or using outdated caches.

As a first debug step, it is recommended to fully restart the host system and all containers. If that fails to resolve the issue, backup any existing data and proceed with a full reset of your installation files. See Section 3.4 for details and ensure you restart the host system prior to rebuilding.

It is sometimes necessary to override configuration files or scripts built into the container environment. Volume mounts can be used override select files or directories without rebuilding containers.

For example, to override the ION NM Manager configuration file add the following to the volumes section of `ion-manager in docker-compose.yml. Adjust the source file to any relative or absolute path desired. For this change to take effect, you must restart all containers with podman compose down and then restart with podman compose up. Simply restarting the affected container may not refresh changes made to the compose file.

      - ./ion/configs/simple-2-node/mgr.rc:/etc/ion.rc

This approach can also be used to override sample agent configurations (see agent-compose.yml), extend authentication settings in the selected auth container, or for developers to quickly test changes.

If services cannot be accessed from remote machines, verify that it is not being blocked by your system’s firewall.

In rare cases, it may be useful to utilize Wireshark or Tshark to verify network traffic within the containers, particularly if debugging NM agents within the container network. The following is a quick guide to installing and running these tools within the container if you do not have the ability to view activity from the host system. This example is for the ion-manager container, but can be adapted to any.

To temporarily install in the ion-manager container run podman compose exec ion-manager yum install tshark. Alternatively, modify the ion/Dockerfile to permanently add it to your installation and rebuild. It can then be run with podman compose exec ion-manager tshark -i any, with tshark arguments modified as needed.

To use the GUI wireshark, install with podman compose exec ion-manager yum install wireshark. To easily run, create a file novnc-compose.yml with the definition below. Start it with podman compose -f novnc-compose.yml up. From the shell run podman compose exec ion-manager bash followed by DISPLAY=novnc:0.0 wireshark to launch wireshark in the VNC session accessible from http://hostname:9081

# noVNC provides browser access to network to aide debugging.
# This container is NOT intended for production usage.

networks:
  # This network is created by docker-compose.yml
  default:
    name: ${DOCKER_CTR_PREFIX}anms
    external: true

services:
  novnc:
    image: chrome-novnc
    build:
      dockerfile: novnc.Dockerfile
    environment:
      # Adjust to your screen size
      - DISPLAY_WIDTH=1920
      - DISPLAY_HEIGHT=1080
      - RUN_XTERM=yes
    ports:
      - "9081:8080"

See Section 4.1.3.1 for an example of volume mounting a local directory for easier saving of captures for later analysis. In this case, you may wish to mount a directory instead of a single file, such as - ./logs:/logs and append to the tshark command -w /logs/log.pcap