The subsystems (containers) of the ANMS are illustrated as gray blocks within the "ANMS Instance" group in the diagram of Figure 1.1, “ANMS Components with Protocol Associations”. The entire ANMS instance is made to be run on a single host, with future plans to allow installing in a more distributed environment. Currently the ANMS provides security at the boundary of the instance but not between containers (see Section 1.6, “Security” for details), which would be required for a distributed installation.

A higher-level logical view of the ANMS is shown in Figure 1.2, “ANMS Primary Components with Logical Associations” where some of the internal infrastructure containers (e.g., PostgreSQL database, MQTT broker) are removed for clarity.

The User Agents in both diagrams are how a user can interact with the ANMS, which is solely via HTTP exchanges. Most of the ANMS API follows RESTful principals of stateless service interactions, while some of the API is more browser-oriented to provide UI visuals.

Figure 1.1. ANMS Components with Protocol Associations

Figure 1.2. ANMS Primary Components with Logical Associations

1.1.1. Light Components

A lightweight ("light") configuration of the ANMS involves removing containers related to web browser user interfaces, including the anms-ui file server and its redis cache, the grafana plotting UI and its renderer, the adminer database inspection UI, and the OpenSearch dashboard UI.

This is depicted in the set of containers in the diagram of Figure 1.3, “ANMS Light Components with Protocol Associations”. The expectation of this light configuration is that there will not be a web browser user agent interacting with the ANMS, but instead some HTTP client application operated by another operations/management system external to ANMS.

In the light configuration, the ANMS will still support loading of ADMs needed to operate the transcoder service, sending immediate execution messages to agents, receiving and archiving report messages from agents, and allowing direct access to report contents as well as the postgres data warehouse (which would normally be used by the Grafana plotting UI).

Figure 1.3. ANMS Light Components with Protocol Associations

The target host will be running the RedHat Enterprise Linux (RHEL) version 9 (RHEL-9) with network interfaces configured, and IP addressing and DNS configured along with a running local firewall.

The ANMS is intended to be deployed using the Puppet orchestrator, either from a local Puppet apply execution or configured from a central Puppet server. Part of the ANMS source is a Puppet module "anms" to automate the configuration of an ANMS deployment. Specific procedures for performing an installation using a local Puppet apply are in Section 3.2, “Installation”.

Conditions for installing the ANMS are a host with packages identified in Table 1.1, “Target host packages”, at least 7 GiB of filesystem space for container image storage, and additional space for long-term data warehouse storage. The total amount of storage needed depends on the mission use of reports, specifically the average size and rate of reported data.

The ANMS is designed to operate on a network where the MGSS Common Access Manager (CAM) is used to manage user accounts and a CAM Gateway is used as a reverse proxy within the ANMS installation to enforce user login sessions and access permissions. The ANMS has been exercised with CAM v5.1.0 in a test environment outside of the MGSS environment. To deploy the ANMS in an environment without a CAM instance available (or without using it) the ANMS can be built with a CAM Gateway emulator as described in Section 1.2.1, “Using a CAM Gateway Emulator”. In any case, deployment and configuration of CAM itself is outside the scope of this document and is described in detail in the CAM Product Guide.

The containers defined by the ANMS compose configuration in Section 1.4, “Filesystem” are as follows in alphabetical order. Associations between these containers are illustrated in Figure 1.1, “ANMS Components with Protocol Associations”.

adminer

Administrative access to the PostgreSQL database, which requires special authorization. Exposes TCP port 8080 for HTTP.

authnz

The CAM Gateway reverse proxy for authentication, authorization, and auditing (AAA); also the endpoint of user agent TLS connections. This container uses the external ammos-tls volume for TLS configuration (see Section 3.2.1, “TLS Configuration Volume”). This provides HTTP routing access is to anms-ui, anms-core, and grafana containers. Exposes TCP port 443 for HTTPS and 80 for HTTP, both mapped to the same host port numbers. This container can be remapped to use the cam-gateway or demo (basic http auth) configuration.

anms-core

The ANMS backend REST services. Exposes TCP port 5555 for HTTP.

anms-ui

The ANMS frontend REST services and browser client UI. Exposes TCP port 9030 for HTTP.

grafana

The data warehouse plotting engine. This uses the grafana-data volume for storage. Exposes TCP port 3000 for HTTP.

grafana-image-renderer

Image renderer for the grafana subsystem. Exposes TCP port 8081 for internal APIs.

ion-manager

A combination of the AMP Manager used by ANMS and the BP Agent used for message transport. Exposes UDP port 1113 for LTPCL and port 4556 for UDPCL, and TCP port 8089 for HTTP API; the CL ports are mapped to the same host port numbers.

aricodec

A service to convert ARI between text and compressed binary form based on available ADMs.

mqtt-broker

The broker host for MQTT pub-sub messaging. Exposes TCP port 1883 for MQTT.

opensearch

Log aggregator for the ANMS. This uses the opensearch volume for storage. Exposes TCP port 9200 and 9600 for internal APIs.

opensearch-dashboards

A user interface for accessing the opensearch logs. Exposes TCP port 5601 for HTTP.

postgres

Persistent database for the ANMS. This uses the postgres-data volume for storage. Exposes TCP port 5432 for PSQL.

redis

A database for state keeping from the ANMS UI. Exposes TCP port 6379 for redis API.

transcoder

An intermediate service to bookkeep transcoding requests from the ANMS to the ARI CODEC engine.

Because the ANMS is deployed as a Compose configuration, the only primary files present on the host are to configure the containerized project, its use as a system service, and the system firewall.

The principal directories and files used by ANMS are:

/etc/systemd/system

Directory for systemd local configuration units. Its contents for ANMS are:

podman-compose@.service

A template systemd service for all compose projects represented as files below.

/etc/containers/compose/projects

Compose project top-level service configuration files. Each file is used by the podman-compose@ template service. Its contents for ANMS are:

anms.env

The ANMS service configuration.

testenv.env

The test environment service configuration.

/ammos/anms

The project-specific deployment path for compose configurations. Its contents are:

.env

The compose-level environment configuration for all of the ANMS and testenv containers, including host port mapping options (see Section 1.3, “Containers”).

anms-compose.yml

Compose configuration for the ANMS containers, volumes, and networks. The compose runtime is managed by the podman-compose@anms template service.

testenv-compose.yml

Compose configuration for the test environment containers and network. The compose runtime is managed by the podman-compose@testenv template service.

/run/anms

The temporary file directory used at runtime. Its contents are:

proxy.sock

The transport proxy socket needed by the AMP Manager in order to message to and from agents.

Secondary files related to the ANMS deployment are:

/etc/containers/containers.conf

Configured to enable compose provider and SELinux for containers.

/etc/containers/compose/projects/anms.env

The systemd environment for the anms project, used by the systemd service podman-compose@anms. This points to /ammos/anms/ for its compose configuration.

/etc/systemd/system/podman-compose@.service

A parameterized service interface which reads compose environment from under /etc/containers/compose/projects/

/ammos/etc/pki/tls

Host-level PKI security configuration from which the ANMS user-agent TLS configuration is derived. Its contents for ANMS are:

private/ammos-server-key.pem

The private key for the local TLS endpoint.

certs/ammos-server-cert.pem

The corresponding public key certificate for the local TLS endpoint.

certs/ammos-ca-bundle.crt

The trusted CA bundle for all TLS endpoints (including users).

/var/cache/puppet/puppet-selinux/modules

The containing directory for SELinux modules for the ANMS containers (see Section 1.6, “Security”).

The target host will be running RHEL-9 with network interfaces configured, and IP addressing and DNS configured along with a running local firewall.

The container network configuration (under /ammos/anms) for the ANMS includes host port forwarding for the following services:

HTTPS

Default port 443 forwarded to the authnz container for HTTP use.

As depicted in Figure 1.1, “ANMS Components with Protocol Associations”, the ANMS deployment itself does not include a BP Agent and does not either contain or require any specific configuration of a BP Agent or its associated convergence layers.

The host on which the ANMS instance runs is expected to have FIPS-140 mode enabled and SELinux enabled and in enforcing mode. Part of the ANMS deployment includes an SELinux module for each of the component containers which allow all necessary inter-service communication. If issues with SELinux are suspected in a deployment, follow the procedures in Section 3.6.2, “SELinux Audit Events” to find any audit events related to the ANMS.

The host is also expected to have a running OS-default firewall which will be configured by the Puppet module to allow HTTPS for user agent access.

The interface between ANMS and its User Agents is TLS-secured HTTP with a PKIX certificate supplied by the host network management and chained to the CA hierarchy of the network.

The ANMS uses an internal PostgreSQL database for following purposes, all within the same schema amp_core:

User Configuration

Each user account authorized to access the ANMS can have parameters associated with their account, mostly related to UI parameters.

ADM configuration

The most static configuration of the ANMS is the set of ADMs available to all agents managed by that ANMS instance.

Agent configuration

The Agent configuration consists of agents which are known to, and managed by, the ANMS which are parameterized by their AMP messaging BP EID and their associated CL parameters (network name/address and port).

Reported Data Warehouse

When reports arrive from managed agents and are associated with known ADMs they are disassembled and stored as object-values in the historical data warehouse.

TBD

This procedure builds images in the root user’s podman context and then uses puppet to install to the local host.

More TBD

1. Images for all of the ANMS production and test containers can be built with the following:

sudo AUTHNZ_EMU=authnz-emu podman compose -p anms -f docker-compose.yml --podman-build-args='--format docker' build
sudo podman compose -p agents -f agent-compose.yml --podman-build-args='--format docker' build

1. The deployment configuration is set by editing the file puppet/data/override.yaml to contain similar to:

   anms::docker_image_prefix: "" # Left empty to use locally built images
   anms::docker_image_tag: "latest"

2. Pull the necessary upstream Puppet modules with:

   ./puppet/prep.sh

3. Perform a dry-run of the puppet apply with:

   sudo PATH=/opt/puppetlabs/bin:$PATH ./puppet/apply_local.sh --test --noop

   and verify that there are no unexpected changes.

4. Perform the actual puppet apply with:

   sudo PATH=/opt/puppetlabs/bin:$PATH ./puppet/apply_local.sh --test

The ANMS source is composed of a top-level repository anms and a number of submodule repositories; all of them are required for building the ANMS.

Before beginning, ensure that either Docker or Podman (preferred) is installed and functional on your system along with docker-compose. A hello-world image can be run to verify functionality, i.e. docker run --rm hello-world or podman run --rm hello-world. Compose can be verified with podman compose ps - If compose is not properly installed it will report 'unrecognized command'

1. The top-level checkout can be done with:

   git clone --recursive --branch <TAGNAME> https://github.com/NASA-AMMOS/anms.git

2. Optional: switching to a different tag or branch can be done with the sequence:

   git checkout <TAGNAME>
   git submodule update --init --recursive

3. If running containers as the root user (not recommended. Podman defaults to rootless), it may be necessary to add the local user to the docker access group with the following. :

   sudo usermod -a -G docker ${USER}

4. Update .env file as needed. Fields that may need to be updated include DOCKER_IMAGE_PREFIX and for rootless podman the port mappings for AUTHNZ may need to be un-commented to avoid permissions issues

5. Build images using either: (Note: The podman-build-args flag is required for health checks to function under podman).

   docker compose -f docker-compose.yml -f testenv-compose.yml build
   podman compose --podman-build-args='--format docker' -f docker-compose.yml -f testenv-compose.yml build

   ⓘ

   Note

   To build an ANMS that uses an emulator for the CAM Gateway (which means that the ANMS will not require a CAM server), have the following environment set in the build step above. (Note: The default value that uses the CAM Gateway is cam-gateay):

   export AUTHNZ_EMU=demo

The ANMS uses Puppet version 7 [puppet-agent] to install requisite system packages and configure system files and services. In addition, Bolt [puppet-bolt] is needed to install needed Puppet modules and run the Puppet agent remotely.

To install the ANMS on the local host perform the following:

1. A TLS configuration must be embedded in a volume mounted by the authnz frontend container with contents described in Section 3.2.1, “TLS Configuration Volume”. This can be done with a boilerplate test-only CA and certificates by running:

   ./create_volume.sh ./puppet/modules/apl_test/files/anms/tls

2. The deployment configuration is set by editing the file puppet/data/override.yaml to contain similar to:

   anms::docker_image_prefix: "" # Matching the DOCKER_IMAGE_PREFIX from build procedure
   anms::docker_image_tag: "latest" # Matching the tag name from build procedure
   anms::docker_registry_user: ""
   anms::docker_registry_pass: ""

3. Pull the necessary upstream Puppet modules with:

   ./puppet/prep.sh

4. Perform a dry-run of the puppet apply with:

   sudo PATH=/opt/puppetlabs/bin:$PATH ./puppet/apply_local.sh --test --noop

   and verify that there are no unexpected changes.

5. Perform the actual puppet apply with:

   sudo PATH=/opt/puppetlabs/bin:$PATH ./puppet/apply_local.sh --test

Because the ANMS is deployed as a series of containers managed by compose with associated environment variables and configuration, an upgrade involves rebuilding and restarting affected containers.

An upgrade can be performed using the same procedure as Section 3.2, “Installation”, where Puppet will make any required changes for the upgrade and restart services and containers as necessary. Individual ANMS releases may identify pre-upgrade or post-upgrade steps in their specific Release Description Document (RDD).

To force containers and volumes (containing long-term database files) to be cleared, a maintainer can run the following from the host.

podman stop --all
podman system prune --all --volumes

To verify artifacts have been removed you may use the following. The prune command (or object-specific prune command) may need to be reissued if not all relevant artifacts have been cleared.

podman image ls
podman container ls
podman volume ls
podman network ls

After clearing containers and volumes, the normal apply_local step of Section 3.2, “Installation” should be performed to re-install and start the containers.

To enable Wireshark logging with patched AMP dissector, run similar to the following:

wireshark -i br-anms -f 'port 1113 or port 4556' -k

Although the docker volume anms_postgres contains the raw database state, this will not allow backup of or transferring that state to other hosts.

To perform an online backup (keeping the database running) run the following on the host:

podman compose -p anms exec -T postgres pg_dump -Ft -d amp_core | gzip -c >~/anms-backup.tar.gz

which can the later be restored using:

gunzip -c <~/anms-backup.tar.gz | podman compose -p anms exec -T postgres pg_restore --clean -d amp_core

podman compose -p anms [action] ...

podman compose -p anms ps

podman compose -p anms logs [service-name]

sudo sealert -l '*'

sudo semanage dontaudit off

Each of the following checkout procedures makes progressively more detailed and more normal-operations-like tests of the external interfaces with the ANMS.

In many fault cases, the procedure will work for the first steps and then fail on a specific step and thereafter. This is taken advantage of for the purposes of troubleshooting and failure reporting; the specific procedure(s) run and step(s) that fail are valuable to include in issue reports related to the ANMS.

To make the procedures more readable, the ANMS host is assumed to have the resolveable host name anms-serv. For checkout steps occurring on a "client host" it is assumed to be running RHEL-9 or equivalent from the perspective of commands available.

docker compose -p anms exec -T amp-manager journalctl --unit refdm-proxy | grep -B4 Connected

The ANMS is hosted on a GitHub repository [anms-source] with submodule references to several other repositories. There is a CONTRIBUTING.md document in the ANMS repository which describes detailed procedures for submitting tickets to identify defects and suggest enhancements.

Separate from the source for the ANMS proper, the ANMS Product Guide and User Guide are hosted on a GitHub repository [anms-docs], with its own CONTRIBUTING.md document for submitting tickets about either the Product Guide or User Guide.

While the GitHub repositories are the primary means by which users should submit detailed tickets, other inquiries can be made directly via email to the the support address dtnma-support@jhuapl.edu.