As my fellow Stackers return home after yet another fantastic summit in Barcelona, the topic of upgrades is timely.
During a few of my speaking sessions and in overall hallway buzz were many questions about how best to upgrade an OpenStack cloud.
For those new to the Rackspace OpenStack product line, it’s important to know Rackspace Private Cloud powered by OpenStack, or RPC-O, includes an out-of-the-box, in-place upgrade option with minimal downtime to the applications running on your cloud. This option allows you to upgrade from a previous major release to the current major release of OpenStack available for RPC-O. By design, the x.1 release of RPC-O is intended to focus solely on the upgrade process.
With our newest release of RPC-O, version 13.1, we’re proud to announce the ability to upgrade from the previous release of Liberty to the current release of Mitaka. Below are details on some of the great features included in this upgrade release. The core scope was to automate some of the pre- and post-upgrade activities as part of the upgrade process. Plus v13.1 includes all the traditional upgrade goodness the previous releases contained.
Click for the RPC-O upgrade guide, which outlines all the prerequisites and steps required execute a successful upgrade. Please make sure to read thru the Upgrade overview of the guide BEFORE you get started. It has some considerations you must be made aware of before you begin the upgrade.
Below is a short version of items changed or included in the RPC-O v13.1 release. (And here’s the full list of what has changed).
- Two new playbooks were introduced to capture the state of the environment pre- and post-upgrade: rpc-pre-upgrades.yml and rpc-post-upgrades.yml.
- Upgrades the legacy version of ElasticSearch and Kibana to the latest versions.
- ssh_delay OSA variable default value changed to 60, verses the old default of 5.
- By default, security hardening is applied on an upgrade. This feature can be disabled if required.
- Since the Mitaka release of OpenStack-Ansible, the configuration file layout has changed. Any configuration overrides from previous versions must be manually migrated. A script is supplied to handle this migration.
I wanted to spend a bit more time reviewing the new features specifically introduced in the RPC-O v13.1 release. Just like with any upgrade process, there are steps the upgrader should perform before and after the upgrade. The focus normally is on making sure you capture the current state of your environment before and after the upgrade.
Since cloud is all about automation, it felt right to script (aka automate) those steps as much as possible to ease the overall upgrade process. The rpc-pre-upgrade.yml and rpc-post-upgrades.yml playbooks handle those standard tasks for you. See the breakdown below for the tasks each of the playbooks executes.
rpc-pre-upgrades.yml playbook executes:
- lists RabbitMQ cluster status
- lists MySQL cluster status
- gathers Nova, Cinder and Neutron service status
- gathers OpenStack endpoints
- gathers running instances from MySQL
- gathers instance-volume mappings from MySQL
- gathers swift recon status
- rotates OpenStack aggregate logs
- cleans up rsyslog forward and state files
rpc-post-upgrades.yml playbook executes:
- verifies wsrep_cluster_size is equal to the number of Galera hosts in inventory
- verifies wsrep_cluster_status for all Galera hosts is Primary
- verifies the RabbitMQ cluster does not have network partitions
- verifies Nova, Cinder and Neutron services are reporting
- verifies the md5sums of the Swift rings and configuration files have no errors
- verifies Elasticsearch indexes are reporting green as their health
Finally, since I am all about the hidden gems, I wanted to share something I came across while testing the latest upgrade release. For those of you who may be interested in automating your upgrade process, there is an additional script named test-upgrade.sh.
It helps streamline the upgrade steps into one script you can use to automate the upgrade process after performing all the necessary prerequisites. Give it a go — just keep in mind it’s an early release and so still requires extensive testing before using it against any production environments.