How to set up Prometheus metrics (deb/3.0/CLI)
MAAS services can provide Prometheus endpoints for collecting performance metrics. These include five endpoints of particular interest to MAAS users:
- TFTP server file transfer latency
- HTTP requests latency
- Websocket requests latency
- RPC calls (between MAAS services) latency
- Per request DB queries counts
All available metrics are prefixed with
maas_, to make it easier to look them up in Prometheus and Grafana UIs.
Three questions you may have:
- How do I enable Prometheus endpoints?
- How do I configure Prometheus endpoints?
- How can I deploy Prometheus and Grafana?
Whenever you install the
python3-prometheus-client library, Prometheus endpoints are exposed over HTTP by the
regiond processes under the default
Currently, prometheus metrics are shared when rack and region controllers are running on the same machine, even though each service provides its own port. You can safely only query one of the two ports if you’re running both controllers.
For a Debian-based MAAS installation, install the library and restart MAAS services as follows:
sudo apt install python3-prometheus-client sudo systemctl restart maas-rackd sudo systemctl restart maas-regiond
MAAS also provides optional stats about resources registered with the MAAS server itself. These include four broad categories of information:
- The number of nodes by type, arch, …
- Number of networks, spaces, fabrics, VLANs and subnets
- Total counts for machines CPU cores, memory and storage
- Counters for VM host resources
After installing the
python3-prometheus-client library as describe above, run the following to enable stats:
maas $PROFILE maas set-config name=prometheus_enabled value=true
/metrics endpoint is available in MAAS services, Prometheus can be configured to scrape metric values from these. You can configure this by adding a stanza like the following to the prometheus configuration:
- job_name: maas static_configs: - targets: - <maas-host1-IP>:5239 # for regiond - <maas-host1-IP>:5249 # for rackd - <maas-host2-IP>:5239 # regiond-only - <maas-host3-IP>:5249 # rackd-only
If the MAAS installation includes multiple nodes, the
targets entries must be adjusted accordingly, to match services deployed on each node.
If you have enabled MAAS stats, you must add an additional Prometheus job to the config:
- job_name: maas metrics_path: /MAAS/metrics static_configs: - targets: - <maas-host-IP>:5240
In case of a multi-host deploy, adding a single IP for any of the MAAS hosts running
regiond will suffice.
Grafana and Prometheus can be easily deployed using Juju.
The MAAS performance repo repository provides a sample
deploy-stack script that will deploy and configure the stack on LXD containers.
First, you must install juju via:
sudo snap install --classic juju
Then you can run the script from the repo:
To follow the progress of the deployment, run the following:
watch -c juju status --color
Once you deploy everything, the Grafana UI is accessible on port
3000 with the credentials
grafana. The Prometheus UI will be available on port
The repository also provides some sample dashboard covering the most common use cases for graphs. These are available under
grafana/dashboards. You can import them from the Grafana UI or API.