Expose Logos node metrics to Prometheus

Serve the blockchain, storage, and delivery metrics of a running Logos node on one OpenMetrics endpoint and scrape it with Prometheus.

Version

This document is accurate for Testnet v0.3.

This procedure covers adding the openmetrics module to a Logos node and pointing Prometheus at it. openmetrics runs a small HTTP server inside the node. On every scrape, it asks each module you list for its metrics and returns them as one OpenMetrics document, with a module label on every series. It is intended for node operators who want dashboards and alerts for a node they already run.

The steps continue from Run a Logos node with blockchain, storage, and delivery and use the same logos user, paths, and logosctl session.

Prerequisites

• A Logos node set up with Run a Logos node with blockchain, storage, and delivery, with the daemon running and the blockchain, storage, and delivery modules started.
• logosctl installed.
• A Prometheus server that can reach the node over TCP, on the same host or on your private network.

What to expect

• You can scrape blockchain, storage, and delivery metrics from a single /metrics endpoint on the node.
• You can see the node as a healthy target in Prometheus and query its series by module label.
• You can keep the endpoint private and bring it back automatically after the node restarts.

Step 1: Install the openmetrics module

Install the module into the running node. As root, open a shell as the logos user, and run every logosctl command in this procedure from it:

runuser -u logos -- env HOME=/var/lib/logos-node bash

1. Install the pinned openmetrics package from the module catalogue:

   logosctl package install openmetrics \
   --version 0.1.2 \
   --root-hash 15b22b2020bf83f775b6d55f2e62ceb85582d1720c5d56099b8edd33928a13bb \
   --yes

   • The running node picks up the new package without a restart.

2. Load the module:

   logosctl module load openmetrics

   • The output must include "module":"openmetrics","status":"ok".

Step 2: Turn on the blockchain node's metrics

The blockchain node keeps its metrics off unless its configuration turns them on, and it reads that setting only when the module starts. The user_config.yaml that generate_user_config writes has them off.

1. Turn the metrics on in the node configuration:

   sed -i 's/enable_open_metrics: false/enable_open_metrics: true/' /var/lib/logos-node/user_config.yaml
   grep enable_open_metrics /var/lib/logos-node/user_config.yaml

   • The output must show enable_open_metrics: true. The setting is under tracing.metrics.
   • If you are setting up a new node, make this change after generate_user_config and before the first start call, and skip the next step.

2. Restart the blockchain module so it reads the new setting:

   logosctl module unload blockchain_module
   logosctl module load blockchain_module
   logosctl call blockchain_module start /var/lib/logos-node/user_config.yaml ""

   • The node keeps its state and resumes from the blocks it already stored.
   • The start call can return RPC call failed, or call to 'blockchain_module.start' timed out after 20000ms, while the node replays its stored blocks. The node still starts. Wait a few minutes, then check logosctl call blockchain_module get_cryptarchia_info.

3. Check that the node reports metrics:

   logosctl call blockchain_module collectOpenMetricsText

   • The result is an OpenMetrics document with # TYPE lines. An empty result means the metrics are still off or the node is not running.

Step 3: Start the metrics endpoint

Tell openmetrics which port to listen on and which modules to collect from.

1. Create the endpoint configuration:

   mkdir -p /var/lib/logos-node/openmetrics
   cd /var/lib/logos-node/openmetrics
   cat > config.json <<'EOF'
   {
     "port": 9464,
     "modules": [
       { "name": "blockchain_module", "format": "text" },
       "storage_module",
       { "name": "delivery_module", "format": "text" }
     ]
   }
   EOF

   Field	Purpose
   port	TCP port of the HTTP server. Pick a free port; 9090 is the default port of Prometheus itself
   modules	Modules to collect from on each scrape
   modules[].name	Module name, as shown by logosctl module ls --loaded
   modules[].format	data (the default, also used for a bare name) or text for modules that already produce OpenMetrics text

   • storage_module returns structured metrics, so a bare name is enough.
   • blockchain_module and delivery_module return OpenMetrics text and need "format": "text". Listed as a bare name, they contribute no metrics.

2. Start the endpoint:

   logosctl call openmetrics start @config.json

   • "result":1 means the server is listening. "result":0 means it did not start; see Troubleshooting.

3. Confirm it is serving:

   logosctl call openmetrics getInfo

   • The result must contain \"running\":true and \"port\":9464.

Step 4: Check the endpoint

1. Check that the server answers:

   curl http://127.0.0.1:9464/health

   • The response is ok.

2. Fetch the metrics:

   curl -s http://127.0.0.1:9464/metrics | grep -E 'consensus_tip_height|consensus_peers_connected|libp2p_peers|kad_routing_table_peers|repostore_bytes_used'

   • Expected result: gauges from all three modules, each tagged with the module that reported it. Your values differ:

   # TYPE consensus_peers_connected gauge
   consensus_peers_connected{module="blockchain_module",otel_scope_name="logos-blockchain-node"} 127
   # TYPE consensus_tip_height gauge
   consensus_tip_height{module="blockchain_module",otel_scope_name="logos-blockchain-node"} 3481
   # HELP libp2p_peers total connected peers
   # TYPE libp2p_peers gauge
   libp2p_peers{module="delivery_module"} 15.0
   # HELP logos_storage_kad_routing_table_peers total peers in routing table
   # TYPE logos_storage_kad_routing_table_peers gauge
   logos_storage_kad_routing_table_peers{module="storage_module"} 21.0
   # HELP logos_storage_repostore_bytes_used storage repostore bytes used
   # TYPE logos_storage_repostore_bytes_used gauge
   logos_storage_repostore_bytes_used{module="storage_module"} 31522926.0

   • The full document holds a few hundred metric families and ends with # EOF. Storage metrics start with logos_storage_ and delivery metrics with logos_delivery_. The delivery module also reports its libp2p, discovery, and mix network metrics.
   • Blockchain metrics have no common prefix (consensus_, chainsync_, mempool_, network_, storage_, and others) and no # HELP lines. Filter them by module="blockchain_module".
   • A blockchain series appears once the node has recorded it, so a node that just started reports fewer families.
   • A module that is not running, or that fails to answer, is left out of the scrape. The other modules are still reported.

Step 5: Keep the endpoint private

The metrics server listens on all network interfaces, not only on 127.0.0.1. The metrics reveal peer counts, traffic, and storage use, so expose the port only to your Prometheus server.

1. Confirm the listening address:

   ss -lntp | grep ':9464'

   • The output shows 0.0.0.0:9464.

2. Make sure the host firewall does not open port 9464/tcp to the internet. If Prometheus runs on another host, allow only that host. For example, with ufw, replace <prometheus-ip> with the address of your Prometheus server:

   sudo ufw allow from <prometheus-ip> to any port 9464 proto tcp

   • Run this from an account with sudo access, not from the logos user shell.

Step 6: Scrape the node with Prometheus

1. Add a scrape job for the node to your Prometheus configuration (prometheus.yml). Add it as a new item under the existing scrape_configs: key. A second scrape_configs: key makes Prometheus reject the file. Replace <node-address> with 127.0.0.1 if Prometheus runs on the node, or with the node's private address otherwise:

2. Reload or restart Prometheus so it reads the new job.

3. Check that Prometheus scrapes the node. In the Prometheus UI, open Status > Target health, or query the API from the Prometheus host (default port 9090):

   curl -s --get http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=up{job="logos-node"}'

   • The logos-node target is up, and the query returns the value "1".

4. Query the node's series, for example the number of series each module reports:

   count by (module) ({job="logos-node"})

   • The result lists blockchain_module, storage_module, and delivery_module. A few series without a module label, such as up, are added by Prometheus itself.

Step 7: Start the endpoint with the node

The endpoint does not persist across restarts: when the node daemon restarts, the openmetrics module is unloaded and the server stops.

1. If you run the node with the systemd service, add these commands to the module bootstrap script, after the blockchain, storage, and delivery modules start:

   logosctl module load openmetrics
   logosctl call openmetrics start @/var/lib/logos-node/openmetrics/config.json

   • Collection only reports modules that are running, so start openmetrics after the modules it scrapes.
   • The blockchain setting from Step 2 is stored in user_config.yaml, so the blockchain module keeps reporting metrics after a restart.

2. To stop the endpoint without stopping the node, run:

   logosctl call openmetrics stop

Troubleshooting

logosctl call openmetrics start returns "result":0

The server did not start. The usual causes are:

• The server is already running. Check with logosctl call openmetrics getInfo, and run logosctl call openmetrics stop before starting it with a new configuration.
• Another process holds the port. Check with ss -lntp | grep ':9464', then pick a free port in config.json.
• config.json is not valid JSON, or port is missing or not between 1 and 65535.

A module's metrics are missing from /metrics

• Check that the module is loaded and started with logosctl module ls --loaded and the module's own status call, for example logosctl call storage_module isRunning.
• Check that blockchain_module and delivery_module are listed with "format": "text". As bare names they contribute no metrics.
• If blockchain_module reports nothing, run logosctl call blockchain_module collectOpenMetricsText. An empty result means enable_open_metrics is not true in user_config.yaml, the module was not restarted after the change, or the node is not running. Repeat Step 2.