{"id":1232,"date":"2016-03-08T20:14:56","date_gmt":"2016-03-08T19:14:56","guid":{"rendered":"https:\/\/elkano.org\/blog\/?p=1232"},"modified":"2016-03-09T17:20:47","modified_gmt":"2016-03-09T16:20:47","slug":"high-availability-rabbitmq-cluster-openstack","status":"publish","type":"post","link":"https:\/\/elkano.org\/blog\/high-availability-rabbitmq-cluster-openstack\/","title":{"rendered":"High Availability RabbitMQ cluster in OpenStack"},"content":{"rendered":"

The RabbitMQ service is the heart of the processes communication in OpenStack and in a PROD deployment you want to configure a rabbitmq cluster in order to achieve hight availability for the messages queues.<\/p>\n

There are two types of RabbitMQ nodes, disk nodes and ram nodes<\/a>. Ram nodes require less IOPS because the resource management is not written in disk, but in a cluster, at least, a disk node is required. <\/p>\n

In this post I’m going to configure a three node cluster: one disk node and two ram nodes.<\/p>\n

Installing the RabbitMQ server in all your nodes is as simple as running these commands:<\/p>\n

\r\n$ sudo echo "deb http:\/\/www.rabbitmq.com\/debian\/ testing main" > \/etc\/apt\/sources.list.d\/rabbitmq.list\r\n$ sudo wget -O - https:\/\/www.rabbitmq.com\/rabbitmq-signing-key-public.asc | sudo apt-key add - \r\n$ sudo apt-get update\r\n$ sudo apt-get install rabbitmq-server\r\n<\/pre>\n
To start from scratch stop the rabbitmq service in your nodes and reset the queues and configuration:<\/p>\n
\r\n$ sudo rabbitmqctl stop_app\r\n$ sudo rabbitmqctl reset\r\n$ sudorabbitmqctl start_app\r\n<\/pre>\n
Note:<\/strong> After the cluster creation the ram nodes can be reset without problems, but the disk node cannot be reset because it is the only disk node in the cluster, to reset it you can do it removing the data from the disk:<\/p>\n
\r\nrabbitmqctl stop_app\r\n\/etc\/init.d\/rabbitmq-server stop\r\nrm -rf \/var\/lib\/rabbitmq\/mnesia\/rabbit@node1*\r\n\/etc\/init.d\/rabbitmq-server start\r\nrabbitmqctl start_app\r\n<\/pre>\n
Now, if you run the cluster status command you can see the cluster running with your disk node only:<\/p>\n
\r\nrabbitmqctl cluster_status\r\nCluster status of node 'rabbit@node1' ...\r\n[{nodes,[{disc,['rabbit@node1']},\r\n         {ram,[]}]},\r\n {running_nodes,['rabbit@node1']},\r\n {cluster_name,<<"rabbit@node1">>},\r\n {partitions,[]}]\r\n<\/pre>\n
In the disk node create the user for the openstack services, set its permissions and set the cluster name:<\/p>\n
\r\n$ sudo rabbitmqctl add_user openstack openstack_pass\r\n$ sudo rabbitmqctl set_permissions -p \/ openstack ".*" ".*" "."\r\n$ sudo rabbitmqctl set_cluster_name openstack\r\n<\/pre>\n
Set the queues ha policy to ensure that all queues except those with auto-generated names are mirrored across all running nodes:<\/p>\n
\r\nrabbitmqctl set_policy ha-all '^(?!amq\\.).*' '{"ha-mode": "all"}'\r\n<\/pre>\n
To make a RabbitMQ cluster all the members have to have the same Erlang cookie, find the cookie in the first node and copy it to the other nodes. The cookie is located at:<\/p>\n
\r\n$ cat \/var\/lib\/rabbitmq\/.erlang.cookie\r\nSRITXWMZBCBIRFZMQOAQ\r\n<\/pre>\n
Join the other two nodes to the cluster as a ram nodes, to do that run the following command in node2 and in node3:<\/p>\n
\r\n$ sudo rabbitmqctl stop_app\r\n$ sudo rabbitmqctl join_cluster --ram rabbit@node1\r\n$ sudo rabbitmqctl start_app\r\n<\/pre>\n
The cluster is completed now:<\/p>\n
\r\n$ sudo rabbitmqctl cluster_status\r\nCluster status of node 'rabbit@node1' ...\r\n[{nodes,[{disc,['rabbit@node1']},\r\n         {ram,['rabbit@node2','rabbit@node3']}]},\r\n {running_nodes,['rabbit@node1','rabbit@node2',\r\n                 'rabbit@node3']},\r\n {cluster_name,<<"openstack">>},\r\n {partitions,[]}]\r\n<\/pre>\n
As an additional step you can enable the rabbitmq management plugin<\/a> in one or all of your nodes:<\/p>\n
\r\n$ sudo rabbitmq-plugins enable rabbitmq_management\r\n<\/pre>\n
Create a new user for the management interface:<\/p>\n
\r\n$ sudo rabbitmqctl add_user admin admin_pass\r\n$ sudo rabbitmqctl set_permissions -p \/ admin ".*" ".*" ".*"\r\n$ sudo rabbitmqctl set_user_tags  admin administrator\r\n<\/pre>\n
And finally go to your browser an and type:<\/p>\n
\r\nhttp:\/\/server-name:15672\r\nuser: admin\r\npass: admin_pass\r\n<\/pre>\n
Once you have configured your RabbitMQ cluster you can configure the OpenStack services to use the cluster and mirrored queues. Just in case you should configure check OpenStack documentation for each service:<\/p>\n
\r\n[oslo_messaging_rabbit]\r\nrabbit_hosts=node1:5672,node2:5672,node3:5672\r\nrabbit_retry_interval=1\r\nrabbit_retry_backoff=2\r\nrabbit_max_retries=0\r\nrabbit_ha_queues=true\r\nrabbit_userid = openstack\r\nrabbit_password = openstack_pass\r\namqp_auto_delete = true\r\namqp_durable_queues=True\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
The RabbitMQ service is the heart of the processes communication in OpenStack and in a PROD deployment you want to configure a rabbitmq cluster in order to achieve hight availability for the messages queues. There are two types of RabbitMQ nodes, disk nodes and ram nodes. Ram nodes require less IOPS because the resource management […]<\/p>\n","protected":false},"author":1,"featured_media":1235,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[128],"tags":[43,149,172],"_links":{"self":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1232"}],"collection":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/comments?post=1232"}],"version-history":[{"count":17,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1232\/revisions"}],"predecessor-version":[{"id":1259,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1232\/revisions\/1259"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/media\/1235"}],"wp:attachment":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/media?parent=1232"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/categories?post=1232"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/tags?post=1232"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}