What's new in MAAS (deb/2.9/cli)

2.7 2.8 2.9 3.0
Snap CLI ~ UI CLI ~ UI CLI ~ UI CLI ~ UI
Packages CLI ~ UI CLI ~ UI CLI ~ UI CLI ~ UI

MAAS 2.9.2 release notes

We have released MAAS 2.9.2, which contains two new features, and some notable bug fixes. The two new features are:

  1. Proxmox driver: A driver has been added to MAAS 2.9.2 which intereacts with the Proxmox API. Only one URL is needed, though a username and credentials are required. Credentials can be either a password or an API token. Note that if you use a token, you have to configure the permissions for the token. Newly-created Proxmox tokens don’t assign any permissions by default, so you must add power on, power off, and query power permissions to the token before using it.

  2. Power driver Webhook: A webhook was added to 2.9.2, which allows MAAS to interface with another web service that’s running the power commands. This webhook is provided for interacting with objects that MAAS does not support, that is, the MAAS team supports the driver itself, but whatever is interfacing to the driver is not supported. This webhook as three URLs, one each for power on, power off, and power query. Optionally, this webhook also supports a power user and password or token (RFC 6717). This gives you a way to add your own power drivers without waiting for the driver to be added to MAAS. There is a video tutorial available on this new feature.

You can also find a digest of the 2.9.2 bug fixes below.

MAAS 2.9.1 release notes

Building upon MAAS 2.9, we have released 2.9.1, which contains some notable bug fixes. You can find a digest of these fixes below.

MAAS 2.9 release notes

Following on from MAAS 2.8, we are happy to announce that MAAS 2.9 is now available.

What are the new features & fixes for MAAS 2.9 and MAAS 2.9.1?

  1. Focal Fossa (20.04) as default commissioning/deployment release
  2. Support for OpenVswitch bridge type
  3. Support for NUMA, SR-IOV, and hugepages
  4. Improved performance for large MAAS installations
  5. New release notifications
  6. IPMI configuration screens
  7. Descriptions when marking machines broken
  8. Curtin 20.2 now included
  9. HTTP boot disabled
  10. BMC/IPMI default parameter additions
  11. New global IPMI configuration options
  12. Addition of IPMI config options to UI
  13. New MAAS CLI power command
  14. Commissioning speed improvements
  15. BMC improvements
  16. IPMI power driver upgrades
  17. Enlistment script improvements
  18. Commissioning script improvements
  19. Commissioning script reordering
  20. Reader Adaptive Documentation
  21. Offline documentation

Six other questions you may have:

  1. What known issues should I be aware of?
  2. How do I install MAAS 2.9?
  3. How do I upgrade my MAAS 2.8 snap to a MAAS 2.9 snap?
  4. How do I install MAAS 2.9 from packages?
  5. How do I upgrade MAAS 2.8 to MAAS 2.9 using packages?
  6. What bugs are fixed so far in this release?

Focal Fossa (Ubuntu 20.04 LTS) as default release

Ubuntu 20.04 LTS (Focal Fossa) is now the default commissioning and deployment release for new MAAS installations. Machines deployed with Focal may now be registered as KVM hosts.

Support for OpenVswitch bridge type

MAAS 2.9 allows you to create an OpenVswitch bridge type when creating a bridge.

Support for NUMA, SR-IOV, and hugepages

MAAS 2.9 adds extensive optimisation tools for using NUMA with virtual machines. You can now see how many VMs are allocated to each NUMA node, along with the allocations of cores, storage, and memory. You can quickly spot a VM running in multiple NUMA nodes, and optimise accordingly, with instant updates on pinning and allocations. You can also tell which VMs are currently running. Using the CLI, you can also pin nodes to specific cores, and configure hugepages for use by VMs.

Specifically, there are five new features available to support NUMA, SR-IOV, and hugepages:

  1. You can examine resources on a per-NUMA-node basis.
  2. You can pin nodes to specific cores (CLI only).
  3. You can see resources for VM hosts supporting NUMA nodes.
  4. You can see the alignment between VM host interfaces and NUMA nodes.
  5. You can configure and use hugepages (configurable in CLI only).

This functionality comes with an enhanced panel in the “KVM” details section:

See the VM hosting page for more details, and be sure to use the menu at the top of that page to select your desired build method and interface, so that you’ll see the most relevant instructions.

Improved performance for large MAAS installations

MAAS 2.9 includes changes to the machine batch size that the UI loads. Previously the UI loaded machines in batches of 25; it now pulls in 25 for the first call, then 100 at a time in subsequent batches.

You can see the results of the investigation in this video podcast.

New release notifications

MAAS now includes new release notifications for users and administrators. These appear when a new release is available:

Both regular and administrative users can snooze these notifications for two weeks at a time. Administrative users can opt out of new release notifications completely, preventing notifications for any user of that MAAS.

IPMI configuration screens

MAAS now includes UI panels corresponding to the IPMI power driver upgrades mentioned earlier:

This screen can be reached from Settings | Configuration | Commissioning.

Descriptions when marking machines broken

When marking a machine broken, a description can now included:

This description appears in that machine’s row on the machine list.

Curtin 20.2 now included

A number of MAAS issues have actually been issues with an older version of Curtin. MAAS now includes Curtin 20.2, which fixes many of these issues, including MAAS is changing my boot order!.

HTTP boot disabled

MAAS 2.9 disables HTTP boot. There are known issues with HTTP boot in MAAS, as well as known issues for HTTP boot with grub (e.g. https://bugs.launchpad.net/maas/+bug/1899581) This shouldn’t affect machine boot, as machines will normally try PXE as a fallback boot method if HTTP boot fails. Be aware, though, that machine boot will fail if the BIOS is configured to boot only over HTTP; those machines need to be reconfigured to use PXE.

30-maas-01-bmc-config parameter additions

Four new parameters have been added for IPMI BMC configuration. These parameters will pull from the global defaults, eliminating the need to set the corresponding parameter in each instance.

  1. maas_auto_ipmi_user - The username for the MAAS created IPMI user. Default comes from the global configuration setting.
  2. maas_auto_ipmi_user_password - The password for the MAAS created IPMI user, by default a random password is generated.
  3. maas_auto_ipmi_k_g_bmc_key - he IPMI K_g pre-shared encryption key to be set when adding the MAAS IPMI user. Note not all IPMI BMCs support setting the k_g key, if MAAS is unable to set the key commissioning will fail. Default comes from the global configuration setting. If an IPMI K_g key is set but the key is rejected by the BMC MAAS will automatically retry without the K_g key. This works around an edge case where some BMCs will allow you to set an K_g key but don’t allow it to be used.
  4. maas_auto_ipmi_user_privilege_level - The IPMI user privilege level to use when adding the MAAS IPMI user. Possible options are USER, OPERATOR, or ADMIN. Default comes from the global configuration setting.

Note that MAAS will not capture the BMC MAC address when detecting IPMI BMCs.

New global IPMI configuration options

Two new global IPMI configuration options have been added:

  1. maas_auto_ipmi_k_g_bmc_key - sets a global default IPMI BMC key.
  2. maas_auto_ipmi_user_privilege_level - sets a global default IPMI BMC user privilege level.

Addition of IPMI config options to UI

You may now set the global configuration options maas_auto_ipmi_user, maas_auto_ipmi_k_g_bmc_key, and maas_auto_ipmi_user_privilege_level on the “Settings” page in the UI under “Commissioning.”

New maas.power CLI command

Available in all MAAS 2.9 releases is the new maas.power CLI command. This command interfaces directly with the supported MAAS power drivers. This command can be used to control the power on a machine before it has been added to MAAS, for all maas supported power drivers. You can get power status, turn machines on or off, and cycle power. The maas.power --help shows usage details, including syntax for naming each power type (consistent with other MAAS CLI commands).

IPMI BMC detection improvements (RAD)

This release adds two improvements to IPMI BMC detection capability:

  1. The IPMI cipher suite ID will now be automatically detected. MAAS tries to find the most secure cipher suite available. Preference order is 17, 3, 8, 12. If detection fails MAAS will fall back to using freeipmi-tool default, 3, which is what previous versions of MAAS use.
  2. The IPMI K_g BMC key will now be automatically detected if previously set.

Reader Adaptive Documentation (RAD)

This release features Reader Adaptive Documentation, which allows you to adapt individual pages to your install method (Snap vs. Deb), version (2.7/2.8/2.9), and preferred interface (CLI/UI).

Offline documentation

This release will include offline documentation for those users whose MAAS installations reside behind firewalls, unable to access the online documentation.

BMC improvements

Three substantial improvements to BMC usage have been released:

  1. IPMI, HP Moonshot, and Facebook Wedge BMC detection and configuration scripts have been migrated to the commissioning script 30-maas-01-bmc-config .
  2. BMC detection and configuration are now logged to commissioning results.
  3. If BMC configuration is skipped a ScriptResult will log this result, and indicate which user chose to skip the configuration step.

IPMI power driver upgrades

Three new configuration options have been added to the IPMI power driver:

  1. K_g - The BMC Key of the IPMI device. Used to encrypt all traffic to and from the device during communication.
  2. Cipher Suite ID - The cipher suite to use when communicating with the IPMI BMC. Only 3, 8, 12, and 17 are available as only those enable ciphers for authentication, integrity, and confidentiality. Defaults to 3, freeipmi-tools default. See http://fish2.com/ipmi/bp.pdf for more information.
  3. Privilege Level - The IPMI privilege level to use when communicating with the BMC. Defaults to OPERATOR.

See the 2.9 UI or 2.9 CLI power management pages for details.

Improvements in enlistment scripting

Script flow and capabilities have been improved in three ways:

  1. `maas-run-remote-scripts` can now enlist machines.
  2. Enlistment `user_data` scripts have been removed.
  3. The metadata endpoints `http://:5240//` and `http://:5240//meta-data/` are now available anonymously for use during enlistment.

Major improvements to commissioning script capabilities

Seven major improvements were made to commissioning script flow and capabilities:

  1. Commissioning scripts can now send BMC configuration data
  2. Commissioning scripts can now be used to configure BMC data.
  3. The environment variable BMC_CONFIG_PATH is passed to serially run commissioning scripts.
  4. These scripts may write BMC power credentials to BMC_CONFIG_PATH in a YAML format where each key is the power parameter.
  5. If the commissioning script returns 0, it will be sent to MAAS.
  6. The first script to write BMC_CONFIG_PATH is the only script that may configure the BMC, allowing you to override MAAS's builtin BMC detection.
  7. All builtin commissioning scripts have been migrated into the database.

Commissioning script reordering

Commissioning scripts have been reordered and some are now set to run in parallel. You can now easily set a script to run before the builtin MAAS commissioning scripts. There are nine significant changes:

  1. 00-maas-03-install-lldpd -> 20-maas-01-install-lldpd
  2. 00-maas-05-dhcp-unconfigured-ifaces -> 20-maas-02-dhcp-unconfigured-ifaces
  3. 99-maas-05-kernel-cmdline -> maas -kernel-cmdline
  4. 00-maas-00-support-info -> maas-support-info(now runs in parallel)
  5. 00-maas-01-lshw -> maas-lshw(now runs in parallel)
  6. 00-maas-04-list-modaliases -> maas-list-modaliases(now runs in parallel)
  7. 00-maas-06-get-fruid-api-data -> maas-get-fruid-api-data(now runs in parallel)
  8. 00-maas-08-serial-ports -> maas-serial-ports(now runs in parallel)
  9. 99-maas-01-capture-lldp -> maas-capture-lldp(now runs in parallel)

See the commissioning logs page for more details on these changes.

Improvements in commissioning speed and logging

Four improvements have been made to speed up the commissioning process, mostly by running scripts in parallel (see above):

  1. Commissioning should now take 60s.
  2. Logging has been added to 20-maas-01-install-lldpd (commissioning log output).
  3. Logging added to 20-maas-02-dhcp-unconfigured-ifaces (commissioning log output).
  4. `user_data` can now be input directly into the UI.

Bug fixes

Bugs fixed in 2.9.2 release

  1. In the MAAS UI, ARM servers based on the Hi1620 ARM SoC appear as an “Unknown model”. A fix was added to lxd-4.11, released 2021-02-05.

  2. Debian package installs of MAAS reached an “impossible situation” trying to install the MAAS region controller. This is caused because of an unsupported move from the transitional MAAS PPA to the latest PPA. The workaround is to purge the MAAS packages (and the snap, if installed), and install clean with the latest PPA enabled, which will install the correct versions.

  3. CentOS/RHEL 7+ ship with an unsigned version of GRUB which breaks UEFI secure boot. This bug is believed to be fixed in curtin version 21.1, which is now supported by MAAS 2.9.2.

  4. Debug could not be properly enabled for MAAS snap version 2.9.1. This has been remedied.

  5. The MAAS Backup doc article was not clearly written with respect to stopping critical services. The article has been reworked to make clear in what order steps should be performed so that services are not stopped before appropriate data has been retrieved for backup.

  6. Deselecting all architectures in the Ubuntu extra architectures repo blocks all deployments. The default architectures have been changed to prevent this issue.

  7. MAAS does not allow FQDNs to be used in place of IPs when a BMC extracts the address from the power_address. This incorrect behaviour was changed in 2.9.2.

  8. The Proxmox driver uses a hard-coded port that cannot be customised. This port is now customisable in 2.9.2.

Bugs fixed in 2.9.1 release

  1. It is now possible to delete an LXD VM in an offline state.
  2. MAAS now handles multiple NUMA nodes even when there are gaps in the numbering.
  3. A snap install issue was fixed.
  4. The way MAAS handles gateways WRT DHCP was adjusted.
  5. A majority of the document headings have been converted to links for easy bookmarking.

Bugs fixed in 2.9 release

  1. MAAS 2.9 includes a fix for Bug #1894727: Admin uses cannot change other user’s passwords via the UI.

Known issues

RAD LHS menu

There is a known issue with the Reader Adaptive Documentation left-hand-side menu (navigation), in that the menu links cannot currently be adapted to the RAD parameters. This means that selecting a different page in the LHS menu will take you the the RAD for the current recommended version (in this case, Snap/2.8/UI). Every page that is different in RAD, though, should present you with a top menu, so that you can choose the RAD parameters matching your own preferences.

Erroneous message about “missing migration”

When upgrading to any release above 2.8, using packages, you may receive a warning about missing migration(s) – specifically something that looks like this:

Setting up maas-common (2.8.3~rc1-8583-g.9ddc8051f-0ubuntu1~18.04.1) ...
Setting up maas-region-api (2.8.3~rc1-8583-g.9ddc8051f-0ubuntu1~18.04.1) ...
Setting up maas-region-controller (2.8.3~rc1-8583-g.9ddc8051f-0ubuntu1~18.04.1) ...
Operations to perform:
  Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

This warning message has no effect on the installation or operation of MAAS, so it can be safely ignored.

Cumulative summary of new features in MAAS 3.0 Beta

  1. PCI and USB devices are now modelled in MAAS
  2. IBM Z DPM partition support
  3. Proxmox support
  4. LXD projects support
  5. PCI and USB device tabs in machine details
  6. Workload annotations
  7. Fixed status bar
  8. Registering a machine as a VM host during deployment
  9. Improvements to MAAS CLI help UX

MAAS 3.0 bug fixes

MAAS 3.0 incorporates a large number of bug fixes, summarized in the sections below. Please feel free to validate these fixes at your convenience and give us feedback if anything doesn’t seem to work as presented in the bug request.

One particular bug, #1916860, involves failures in the IPMI cipher suite in MAAS 2.9.2 and up, on the Lenovo x3650 M5 (and others). This particular bug is a not a MAAS bug, but a firmware issue with the subject machines. While the MAAS team can’t fix this (hence the assignment of “Won’t Fix”), the team did provide a easy workaround which helps circumvent this issue.

MAAS 3.0 Beta 5 bug fixes

Here are the bugs that have been Fix Released in MAAS 3.0 Beta 5:

Number Description Importance
#1925784 Processing LXD results failure with loopback Critical
#1923871 LXD vmhost project usage includes usage for other projects High
#1815084 MAAS web ui should perform Save action when Enter/Return is pressed Medium
#1923867 Commissioning fails if NIC gets different PCI address Medium

MAAS 3.0 Beta 4 bug fixes

Here are the bugs that have been Fix Released in MAAS 3.0 Beta 4:

Number Description Importance
#1923246 Unable to compose LXD VM with multiple NICs High
#1918963 Controllers page out of sync with nodes Undecided
#1923685 Unable to deploy LXD VM host on S390X Undecided
#1923687 LXD VM host refresh failure is ignored Undecided
#1774529 Cannot delete some instances of model ‘Domain’ because they are referenced through a protected foreign key High
#1914762 test network configuration broken with openvswitch bridge High
#1919001 Unable to network boot VM on IBM Z DPM Partition High
#1917963 Add chassis lowers the case of added machines Low
#1915087 2.9 UI is broken, seems to loop between user intro and machines pages endlessly High
#1923842 Can’t use action menu on machine details page High
#1917667 Commissioning/testing scripts no longer show ETA or progress Undecided
#1917669 No way to view previous commissioning or testing script results Undecided
#1917670 Storage and interface tests not assoicated with a device Undecided
#1917671 Commissioning/testing scripts not updated after starting commissioning or testing Undecided
#1917794 Unable to view full history of events in UI Undecided
#1918964 UI shows action unavailable after performing action Undecided
#1918966 Tabs aren’t always underscorred Undecided
#1918971 UI does not autofill size on storage tab Undecided
#1923524 Unable to delete LXD composed machine on KVM page Undecided

MAAS 3.0 Beta 3 bug fixes

Here are the bugs that have been Fix Released in MAAS 3.0 Beta 3:

Number Description Importance
#1922569 Create KVM fails in MAAS 3.0 Beta with a project error High
#1923251 Creating an LXD VM host now requires a project name High
#1809939 dhcp snippet create fail when dhcp subnet is relayed Medium
#1913460 Add option to pick whether to keep or decompose machines in a VM host Undecided
#1922787 make “LXD” the default VM host in MAAS UI (rather than virsh) Undecided
#1922876 Deploy KVM hosts with LXD by default Undecided
#1922972 MAAS 3.0 Beta2 UI says “machine cannot be deployed” while successfully deploying machine Undecided
#1923719 MAAS 3.0 : snap refresh maas from 3.0.0~beta2-9826-g.13cc184d5 Undecided

MAAS 3.0 Beta 2 bug fixes

Here are the bugs that have been Fix Released in MAAS 3.0 Beta 2:

Number Description Importance
#1922107 Hugepages/pinning available for virsh and lack validation High
#1922433 Machine resources path set incorrectly in rackd when using snap High

MAAS 3.0 Beta 1 bug fixes

Here are the bugs that have been Fix Released in MAAS 3.0 Beta 1:

Number Description Importance
#1896199 API docs link is not offline Critical
#1904245 MAAS Snap fails to build on PPC64 on Launchpad Critical
#1912727 KVM Page Fails to load with error “An unexpected error has occurred, please try refreshing your browser window.” Critical
#1915869 maas snap cli renders SyntaxWarning in the stderr Critical
#1916093 Unable to add more than 3 Promox VMs Critical
#1883824 Support LXD projects in power control High
#1884276 Terrible user experience adding existing LXD host High
#1902425 Failed to allocate the required AUTO IP addresses after 2 retries High
#1908087 Reverse DNS for non-maas RFC1918 zones fails inside maas High
#1908356 Owner data websocket methods are not working High
#1908434 Can’t delete LXD VM in offline state High
#1913323 /MAAS/docs/ leads to 404 page High
#1914588 Enabling debug from snap traceback High
#1915021 Mapping subnet doesn’t work from the MAAS snap High
#1915022 The MAAS snap doesn’t include nmap High
#1915715 LXD VM additional disks all show 10Gb size High
#1915970 Facebook Wedge BMC detection fails on non-x86 architectures High
#1918997 MAAS does not set snap proxy High
#1919000 Unable to connect MAAS to an LXD VM host High
#1887797 Impossible to delete zombie LXD VM Medium
#1894116 Machines can’t be deployed after deselecting all archs in the “Ubuntu extra architectures” package repo Medium
#1897946 hi1620-based ARM Servers are shown as “Unknown model” Medium
#1906212 timeout in testing scripts ignores the days if set to greater than 24 hours Medium
#1911825 Unable to use FQDN as power_address Medium
#1914165 Proxmox does not allow custom port Medium
#1917652 30-maas-01-bmc-config failing on commissioning Cisco UCSC-C220-M4L Medium
#1335175 maas does not combine kernel_opts when nodes have multiple tags with kernel options Low
#1915359 make sampledata can’t find machine-resources Low
#1916844 Removing a machine that is a vm host tells you to remove the “pod” Low
#1920019 maas_remote_syslog_compress is unnecessarily chatty Low
#1887558 Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. Wishlist
#1901944 tags field in machine edit page overtakes other fields Undecided
#1909985 Add commission timestamp to machine websocket api Undecided
#1913464 Drop RSD pods UI Undecided
#1914590 Support composing LXD VMs with multiple disks in the UI Undecided
#1915970 Facebook Wedge BMC detection fails on non-x86 architectures Undecided
#1916073 MAAS should install qemu-efi-aarch64 on arm64 KVM pods Undecided
#1916317 UI is using API to request scripts with full content Undecided
#1919381 typo “veryiying” in info message in smartctl-validate Undecided

Last updated 10 days ago.