How to enable high availability (snap/2.9/UI)
This page describes how to provide high availability (HA) for MAAS at both region and rack levels. In the context of MAAS, there are four types of HA:
- BMC (node power cycling)
- API services
Nine questions you might have:
- How do I make rack controllers highly available?
- How do I enable highly-available BMC?
- How do I enable highly-available DHCP services?
- How do I make region controllers highly available?
- How do I enable highly-available PostgreSQL?
- How do I enable highly-available API services?
- How do I enable load balancing for API services?
- How do I move a rack controller from one MAAS instance to another?
- What are the potential dangers of moving a rack controller?
You need to install multiple rack controllers to achieve real high availability. Once that’s done, you automatically gain highly-available BMC control, and you can also enable highly-available DHCP.
HA for BMC control (node power cycling) is provided out-of-the-box, once a second rack controller is present. MAAS will automatically identify which rack controller is responsible for a BMC and set up communication accordingly.
DHCP HA affects the way MAAS manages node, including enlistment, commissioning and deployment. It enables primary and secondary DHCP instances to serve the same VLAN. This VLAN replicates all lease information is between rack controllers. MAAS-managed DHCP is a requirement for DHCP HA.
If you are enabling DHCP for the first time after adding a second rack controller, please read Enabling DHCP. On the other hand, if you have already enabled DHCP on your initial rack controller, you’ll need to reconfigure DHCP.
Access the appropriate VLAN (via the ‘Subnets’ page) and choose action ‘Reconfigure DHCP’. There, you will see the second rack controller in the ‘Secondary controller’ field. All you should have to do is press the ‘Reconfigure DHCP’ button:
MAAS will automatically discover and track all reachable region controllers in a single cluster of rack controllers It will also attempt to automatically connect to them if the one in use becomes inaccessible. Administrators can alternatively specify multiple region-controller endpoints for a single rack controller by adding entries to
/var/snap/maas/current/rackd.conf. For example:
- http://<ip 1>:/MAAS/
- http://<ip 2>:/MAAS/
The setup of highly-available DHCP is now complete. Note that, for HA purposes, DHCP provisioning will take into account multiple DNS services when there is more than one region controller on a single region.
Implementing highly-available region control involves setting up two highly-available services:
- PostgreSQL HA
- Secondary API server(s)
Load balancing is optional, but is highly recommended.
MAAS stores all state information in the PostgreSQL database. It is therefore recommended to run it in HA mode. Configuring HA for PostgreSQL is external to MAAS. You will, therefore, need to study the PostgreSQL documentation and implement the variant of HA that makes you feel most comfortable.
A quick treatment of PostgreSQL HA: hot standby is provided here for convenience only. This summary will give you an idea of the command line implementation of HA with PostgreSQL.
Each region controller uses up to 40 connections to PostgreSQL in high load situations. Running two region controllers requires no modifications to the
postgresql.conf. More than two region controllers require that
max_connections be adjusted to add 40 more connections per added region controller.
Setting up high-availability using snaps is relatively easy:
Set up PostgreSQL for high-availability as explained above. PostgreSQL should run outside of the snap.
Install the MAAS snap on each machine you intend to use as a rack or region controller. You’ll need the MAAS shared secret, located here,
/var/snap/maas/common/maas/secret, on the first region controller you set up.
Initialise the snap as a
regioncontroller. Note that if you intend to use a machine as a region controller, you’ll need to tell MAAS how to access your PostgreSQL database host with the following four arguments:
You can add load balancing with HAProxy load-balancing software to support multiple API servers. In this setup, HAProxy provides access to the MAAS web UI and API.
If you happen to have Apache running on the same server where you intend to install HAProxy, you will need to stop and disable
apache2, because HAProxy binds to port 80.
sudo apt install haproxy
Configure each API server’s load balancer by copying the following into
/etc/haproxy/haproxy.cfg (see the upstream configuration manual (external link) as a reference). Replace $PRIMARY_API_SERVER_IP and $SECONDARY_API_SERVER_IP with their respective IP addresses:
frontend maas bind *:80 retries 3 option redispatch option http-server-close default_backend maas backend maas timeout server 90s balance source hash-type consistent server localhost localhost:5240 check server maas-api-1 $PRIMARY_API_SERVER_IP:5240 check server maas-api-2 $SECONDARY_API_SERVER_IP:5240 check
maas-api-2 are arbitrary server labels.
Now restart the load balancer to have these changes take effect:
sudo systemctl restart haproxy
The configuration of region controller HA is now complete.
The API server(s) must be now be referenced (e.g. web UI, MAAS CLI) using port 80 (as opposed to port 5240).
In effect, there is no such action as moving a rack controller, although you can delete a rack controller from one MAAS and reinstantiate the same controller (binary-wise) on another MAAS instance. First, delete the rack controller. In the “Controllers” tab in the UI, select the rack controller you with to delete, choose “Take action” and select “Delete.” You will be asked to confirm with a red button, entitled “Delete 1 controller.”
Next, you must register a new rack controller, which is always done from the command line.
For this exercise, we’re assuming you are using the already installed rack controller code that was previously running on the “from” MAAS instance. All that’s necessary is that you register a new rack controller with the “to” MAAS instance, like this:
sudo maas init rack --maas-url $MAAS_URL_OF_NEW_MAAS --secret $SECRET_FOR_NEW_MAAS
where the secret is found in
Note that in the UI, if you go to the “Controllers” tab and press the button entitled, “Add rack controller,” at the top of the Controllers screen, MAAS will give you a complete command string, including the correct URL and secret values. Simply cut and paste that string to move the rack controller, paying attention to whether you are using snap or package build modes.
There are dangers associate with moving a rack controller – dangers that may generate errors, get you into a non-working state, or cause you significant data loss. These dangers are precipitated by one caveat and two potential mistakes:
Using the same system as a rack controller and a VM host: While not forbidden or inherently dangerous, using the same machine as both a rack controller and a VM host may cause resource contention and poor performance. If the resources on the system are not more than adequate to cover both tasks, you may see slowdowns (or even apparent “freeze” events) on the system.
Moving a rack controller from one version of MAAS to another: MAAS rack controller software is an integral part of each version of MAAS. If you delete a rack controller from, say, a 2.6 version of MAAS, and attempt to register that 2.6 version of the rack controller code to, say, a 2.9 version of MAAS, you may experience errors and potential data loss. Using the above example, if you are running both a VM host and a rack controller for MAAS 2.6 on one system, and you suddenly decide to delete that rack controller from 2.6 and attempt to register the same code to a 2.9 MAAS, the VM host may fail or disappear. This will possibly delete all the VMs you have created or connected to that VM host – which may result in data loss. This action is not supported.
Connecting one instance of a rack controller to two instances of MAAS, regardless of version: Trying to connect a single rack controller to two different instances of MAAS can result in all sorts of unpredictable (and potentially catastrophic) behavior. It is not a supported configuration.
Take these warnings to heart. It may seem like a faster approach to “bridge” your existing rack controllers from one MAAS to another – or from one version of MAAS to another – while they’re running. Ultimately, though, it will probably result in more work than just following the recommended approach.