Inside My Home Rackspace Private Cloud, OpenStack Lab, Part 3: Installing A High Availability Rackspace Private Cloud With Chef Cookbooks

In the first two posts I covered the basics: what hardware is involved and the basic network services that form the basis of my Rackspace Private Cloud install. In this post, I set up Rackspace Private Cloud to give an OpenStack environment consisting of highly available Controllers running as a pair with services such as the OpenStack APIs, Neutron, Glance and Keystone and three compute servers allowing me flexibility to do some testing.

To recap, this is my environment showing the Rackspace Private Cloud. Click on the picture to view a bigger version.

To install Rackspace Private Cloud using the Rackspace Chef Cookbooks, we need to first have Chef installed. In this environment, to make most use of the hardware, I’ll be installing Chef onto the first of my hosts, openstack1, which we will refer to as a controller to facilitate the installation of RPC via the Chef Cookbooks. If you have Chef already installed in your environment, you can skip these steps and head straight to step No. 7 below. In Rackspace Private Cloud, Controllers are the servers that run the APIs and OpenStack services such as Glance, Keystone and Neutron. We run MySQL and RabbitMQ on these too. Through the use of our cookbooks, when we utilize two of these Controllers, we end up with an HA pair, giving highly available Controller services.

Rackspace Private Cloud provides scripts to make installation of an OpenStack-powered private cloud very easy – including the initial setup of Chef and underlying services. Head over to the Rackspace Private Cloud website for more information on this. This blog post pulls out those steps to show complete transparency and give greater control over my installation.

Installation of RabbitMQ

As we will be running Chef alongside other OpenStack services, we need to do some initial setup and configuration of services, such as RabbitMQ, to ensure common services and ports don’t conflict and operate seamlessly with one another.

1. We first install some pre-requisite packages:

apt-get update
apt-get install -y python-dev python-pip git erlang erlang-nox erlang-dev curl lvm2

2. Then we install and set up RabbitMQ on openstack1 ( Both Chef and OpenStack will be set up to use the same RabbitMQ service.

# Ensure our Rabbit environment doesn't lose it's settings later on
mkdir -p /var/lib/rabbitmq
echo -n "ANYRANDOMSTRING" > /var/lib/rabbitmq/.erlang.cookie
chmod 600 /var/lib/rabbitmq/.erlang.cookie
wget -O /tmp/rabbitmq.asc ${RABBITMQ_KEY};
apt-key add /tmp/rabbitmq.asc
wget -O /tmp/rabbitmq.deb ${RABBITMQ}
dpkg -i /tmp/rabbitmq.deb

3. Now that we have RabbitMQ installed, we need to configure it so that Chef can utilize it. To do this we create a vhost and appropriate user on here as follows:

rabbitmqctl add_vhost /chef
rabbitmqctl add_user chef $CHEF_RMQ_PW
rabbitmqctl set_permissions -p /chef chef '.*' '.*' '.*'

Installation of Chef Server

4. Now that we have RabbitMQ setup on one of our nodes (openstack1), we can install Chef Server. To do this, grab the package for Ubuntu 12.04 from the website as shown below:

wget -O /tmp/chef_server.deb ${CHEF}
dpkg -i /tmp/chef_server.deb

5. We can then configure Chef Server for our environment, where we set various configuration items such as the ports to run on and where RabbitMQ is, along with the password we created for the chef user in step No. 2. This is done by running the following commands:

RMQ_IP=""    # openstack1

mkdir -p /etc/chef-server
cat > /etc/chef-server/chef-server.rb <<EOF
erchef["s3_url_ttl"] = 3600
nginx["ssl_port"] = 4000
nginx["non_ssl_port"] = 4080
nginx["enable_non_ssl"] = true
rabbitmq["enable"] = false
rabbitmq["password"] = "${CHEF_RMQ_PW}"
rabbitmq["vip"] = "${RMQ_IP}"
rabbitmq['node_ip_address'] = "${RMQ_IP}"
chef_server_webui["web_ui_admin_default_password"] = "openstack"
bookshelf["url"] = "https://#{node['ipaddress']}:4000"

chef-server-ctl reconfigure

Installation of Chef Client

6. After this has pulled down packages and dependencies, we can configure the Chef client as shown below:

# Make sure knife can be found
ln -sf /opt/chef-server/embedded/bin/knife /usr/bin/knife 
ln -sf /opt/chef-server/embedded/bin/ohai /usr/bin/ohai
SYS_IP=$(ohai ipaddress | awk '/^ / {gsub(/ *"/, ""); print; exit}')
export CHEF_SERVER_URL=https://${SYS_IP}:4000
# Configure Knife
mkdir -p /root/.chef
cat > /root/.chef/knife.rb <<EOF
log_level :info
log_location STDOUT
node_name 'admin'
client_key '/etc/chef-server/admin.pem'
validation_client_name 'chef-validator'
validation_key '/etc/chef-server/chef-validator.pem'
chef_server_url "https://${SYS_IP}:4000"
cache_options( :path => '/root/.chef/checksums' )
cookbook_path [ '/opt/chef-cookbooks/cookbooks' ]

Upload Rackspace Private Cloud Cookbooks to Chef Server

7. With Chef Server running and the Chef Client configured, we can grab the Rackspace Private Cloud Cookbooks from GitHub and upload them to our Chef Server. We do this as follows:

COOKBOOK_VERSION="v4.2.2"     # Check versions here
mkdir -p /opt/
if [ -d "/opt/chef-cookbooks" ];then
    rm -rf /opt/chef-cookbooks
git clone /opt/chef-cookbooks
pushd /opt/chef-cookbooks
git checkout ${COOKBOOK_VERSION}
git submodule init
git submodule sync
git submodule update
# Upload all of the RCBOPS Cookbooks
knife cookbook upload -o /opt/chef-cookbooks/cookbooks -a

knife role from file /opt/chef-cookbooks/roles/*.rb

Configuration of Environment for Rackspace Private Cloud

With the cookbooks uploaded into Chef, we can now configure our environment ready for an installation. For this home lab we have two Controllers (openstack1 and openstack2) and three Computes (openstack3, openstack4 and openstack5). Each of the servers has two nics:

  • eth0 is on the LAN Subnet of and is our Management network.
  • eth1 has not been assigned an IP and will be used for Neutron and is known as the Provider network.

This means we will configure our environment so that eth1 will be used for Neutron, and when we get to use our environment (GUI or CLI), we will be accessing the environment using a 192.168.1.X/24 address – just like any other server or computer on this LAN.

The /etc/network/interfaces for these servers have the following contents:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback

# Host/Management
auto eth0
iface eth0 inet dhcp
# Neutron Provider interface
auto eth1
iface eth1 inet manual
  up ip link set $IFACE up
  down ip link set $IFACE down

8. After this we can create our Environment JSON file which describes our complete setup for Rackspace Private Cloud:

VIP_PREFIX="192.168.1"    # Home lab network is

cat > /opt/base.env.json <<EOF
  "name": "RPCS",
  "description": "Environment for Openstack Private Cloud",
  "cookbook_versions": {
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "default_attributes": {
  "override_attributes": {
  "monitoring": {
  "procmon_provider": "monit",
  "metric_provider": "collectd"
  "enable_monit": true,
  "osops_networks": {
    "management": "${VIP_PREFIX}.0/24",
    "swift": "${VIP_PREFIX}.0/24",
    "public": "${VIP_PREFIX}.0/24",
    "nova": "${VIP_PREFIX}.0/24"
  "rabbitmq": {
    "cluster": true,
    "erlang_cookie": "${ERLANG_COOKIE}"
  "nova": {
    "config": {
      "use_single_default_gateway": false,
      "ram_allocation_ratio": 1.0,
      "disk_allocation_ratio": 1.0,
      "cpu_allocation_ratio": 2.0,
      "resume_guests_state_on_host_boot": false
    "network": {
      "provider": "neutron"
    "scheduler": {
      "default_filters": [
    "libvirt": {
      "vncserver_listen": "",
      "virt_type": "kvm"
  "keystone": {
    "pki": {
      "enabled": false
    "admin_user": "admin",
    "tenants": [
    "users": {
      "admin": {
        "password": "secrete",
        "roles": {
          "admin": [
      "demo": {
        "password": "secrete",
        "default_tenant": "demo",
        "roles": {
          "Member": [
      "demo2": {
        "password": "secrete",
        "default_tenant": "demo2",
        "roles": {
          "Member": [
  "neutron": {
    "ovs": {
      "network_type": "gre",
      "provider_networks": [
          "bridge": "br-eth1",
          "vlans": "1024:1024",
          "label": "ph-eth1"
  "mysql": {
    "tunable": {
      "log_queries_not_using_index": false
    "allow_remote_root": true,
    "root_network_acl": ""
  "vips": {
    "horizon-dash": "${VIP_PREFIX}.${API_VIP}",
    "keystone-service-api": "${VIP_PREFIX}.${API_VIP}",
    "nova-xvpvnc-proxy": "${VIP_PREFIX}.${API_VIP}",
    "nova-api": "${VIP_PREFIX}.${API_VIP}",
    "nova-metadata-api": "${VIP_PREFIX}.${API_VIP}",
    "cinder-api": "${VIP_PREFIX}.${API_VIP}",
    "nova-ec2-public": "${VIP_PREFIX}.${API_VIP}",
    "config": {
      "${VIP_PREFIX}.${API_VIP}": {
        "vrid": 12,
        "network": "public"
      "${VIP_PREFIX}.${MYSQL_VIP}": {
        "vrid": 10,
        "network": "public"
      "${VIP_PREFIX}.${AMQP_VIP}": {
        "vrid": 11,
        "network": "public"
    "rabbitmq-queue": "${VIP_PREFIX}.${AMQP_VIP}",
    "nova-novnc-proxy": "${VIP_PREFIX}.${API_VIP}",
    "mysql-db": "${VIP_PREFIX}.${MYSQL_VIP}",
    "glance-api": "${VIP_PREFIX}.${API_VIP}",
    "keystone-internal-api": "${VIP_PREFIX}.${API_VIP}",
    "horizon-dash_ssl": "${VIP_PREFIX}.${API_VIP}",
    "glance-registry": "${VIP_PREFIX}.${API_VIP}",
    "neutron-api": "${VIP_PREFIX}.${API_VIP}",
    "ceilometer-api": "${VIP_PREFIX}.${API_VIP}",
    "ceilometer-central-agent": "${VIP_PREFIX}.${API_VIP}",
    "heat-api": "${VIP_PREFIX}.${API_VIP}",
    "heat-api-cfn": "${VIP_PREFIX}.${API_VIP}",
    "heat-api-cloudwatch": "${VIP_PREFIX}.${API_VIP}",
    "keystone-admin-api": "${VIP_PREFIX}.${API_VIP}"
  "glance": {
    "images": [
    "image": {
    "image_upload": false
  "osops": {
    "do_package_upgrades": false,
    "apply_patches": false
  "developer_mode": false

9. When we execute the above command, we end up with a file call /opt/base.env.json applicable to the home lab environment. We then load this into Chef as follows:

knife environment from file /opt/base.env.json

10. With the environment loaded, we can simply install Rackspace Private Cloud with just a few commands and a cup of coffee. In my environment I check that my proxy server is configured in /etc/apt/apt.conf.d/01squid as follows (as we’ll be installing a lot of the same packages on all the hosts):

Acquire::http { Proxy ""; };

Bootstrapping the Controllers (Installing Rackspace Private Cloud)

11. We are now ready to bootstrap the nodes (install Chef on each node), assign roles to them and install Rackspace Private Cloud to them. The first servers to do this on are the Controllers (openstack1 and openstack2). We assign the roles of ha-controller1 to openstack1 and ha-controller2 to openstack2, a well as the role of single-neutron to each which preps these for the roles assigned. We do this as follows:


knife bootstrap -E RPCS -r role[ha-controller1],role[single-network-node] ${CONTROLLER1}
knife bootstrap -E RPCS -r role[ha-controller2],role[single-network-node] ${CONTROLLER2}

12. This will fetch and install Chef Client and configure their roles within Chef Server. We can then run chef-client to apply the roles to the servers. As we are running a HA pair, we run chef-client in the following order: first on openstack1, then on openstack2, then finally on openstack1 again. This is because of the preparation and work to sync MySQL and RabbitMQ between the two to allow us to operate them in a HA (failover) capacity:

# On openstack1
# Configure Rabbit HA Policy
knife ssh -C1 -a ipaddress 'role:*controller*' 
    "rabbitmqctl set_policy ha-all '.*' '{"ha-mode":"all", "ha-sync-mode":"automatic"}' 0"
# SSH to openstack2 and run chef-client 
ssh openstack2 chef-client 

# Back on openstack1 

Bootstrapping the Computes

13. For the Computes it’s even simpler. We simply assign the role single-compute to them then execute chef-client across each of them as follows

for node in {103..105}; do
 knife bootstrap -E RPCS -r role[single-compute] 192.168.1.${node}
knife ssh "role:single-compute" chef-client

14. Lastly we add the eth1 device to the bridge, then reboot the cluster so everything starts up as expected each time

knife ssh "role:*" "ovs-vsctl add-port br-eth1 eth1&&reboot"

Congratulations! You now have a home lab ready for testing Rackspace Private Cloud!

I tend to wrap all these commands into a single script and execute from my first node where Chef is to be installed, openstack1 (recall that I grab a shell script ( as part of the Ubuntu installation process). This will run through all these steps fully automated. On my network and slow ADSL (and the N40L isn’t the fastest server on the planet!), I tend to have Rackspace Private Cloud up and running in about 30 minutes from boot.

In the next post we’ll look at configuring some basic Neutron Networking, which I set up after each installation of RPC on my home lab. This includes private networking and a Flat Network to allow me to access my instances from laptops on my home network.

Thanks to @cloudnull (Kevin Carter, Rackspace) for help fine-tuning these steps!


  1. Hello,

    Followed instructions in this blog, but now I’m blocked with this when runnning the knife bootstrap command:
    knife bootstrap -E RPCS -r role[ha-controller1],role[single-network-node] ${CONTROLLER1} -x dev –sudo

    Output errors are:

    No such file or directory – /etc/init.d/nova-scheduler restart
    No such file or directory – /etc/init.d/nova-conductor restart
    No such file or directory – /etc/init.d/nova-api-ec2 restart
    No such file or directory – /etc/init.d/nova-api-os-compute restart
    No such file or directory – /etc/init.d/nova-cert restart
    No such file or directory – /etc/init.d/nova-novncproxy restart
    No such file or directory – /etc/init.d/nova-consoleauth restart

    Don’t know why the recipie is trying to restart services using /etc/init.d rather by service command.

    Any help will be kindly appreciated


  2. hello,

    I’ve followed steps in this blog but I’m getting the I’m
    getting the following errors when executing the following knife
    bootstrap command:

    knife bootstrap -E RPCS -r role[ha-controller1],role[single-network-node] ${CONTROLLER1} -x dev –sudo

    The logs are: Errno::ENOENT No such file or directory – /etc/init.d/nova-scheduler restart No such file or directory – /etc/init.d/nova-conductor restart No such file or directory – /etc/init.d/nova-api-ec2 restart No such file or directory – /etc/init.d/nova-api-os-compute restart No such file or directory – /etc/init.d/nova-cert restart No such file or directory – /etc/init.d/nova-novncproxy restart No such file or directory – /etc/init.d/nova-consoleauth restart

    Any help will be very much appreciated



      • Thanks Arturo. Chef is quite memory intensive, and then add to that more components from OpenStack and you realise that memory requirements are important. Glad you’ve fixed your issue.

  3. I ran into an issue. I got all the way to step five. but when I go to run “sudo chef-server-ctl reconfigure” I get “[2014-05-11T19:31:08-06:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)”


    • Hi Paul, I’ve made some corrections to this post. I inadvertently included an installation of chef twice (one manually, as this post describes) then utilising the script provided by Rackspace to install it again. This, as you can imagine, causes a world of pain for Rabbit and Chef. Let me know if you have any problems with the corrected steps. I’m on Freenode #openstack as itarchitectkev if you need to me to troubleshoot.