Tutorial – RabbitMQ Mining Cluster – High availability

Tram Ho

  • I have just passed a fairly long time, to research and build a lab about building rabbitmq cluster. Here is the whole “diary” I note.

1. Introduction

  • My writing will not rewrite the basic concepts.
  • This article I think is probably suitable for:
    • Who wants to learn quickly the construction of rabbitmq cluster, to get an overview, before you want to study (or configure) more deeply.
    • Understand the thought, the problems of operating a Message Queue cluster in general. But I think I could become a background to reach out to another system.
  • Knowledge base to understand the article:
    • A little bit about Rabbitmq
    • A little bit about Docker
    • A little bit about Networking
  • The information is in the article:
    • A little theory about Quorum Queue in RabbitMQ
      • The reason of birth
      • Compare with Mirror Queue (support in the old version)
    • Constructing the cluster system
      • Design diagram
      • Step by step deployment
    • Lab some scenarios when the cluster has a problem
      • There are nodes down / reUp
      • Network Partitions – Split Brain

2. Quorum Queue

2.1 A few main ideas

  • Rabbitmq from version 3.8.0 onwards, there is a major change in feature support. In particular there is Quorum Queue. What was born to solve the problems of Mirror Queue in previous versions was encountered. (Quorum Queue is a “type”, không liên quan to Exchange Type: Direct, Topic, Fangout)
  • When creating Quorum Queue, there is no Durability option like Classic Queue. (When the node fails, or the problem restarts, the queue will be “reloaded” at startup, if non-durable it will be lost)
  • When using Quorum Queue, there is no need to upgrade the client. => backward compatible. The Quorum Queue setup is located at the server nodes. Client not participating.
  • When a node fails, then returns, it only synchronizes new messages. Without having to sync again from the beginning. And the sync of these new messages is not blocked.
  • If a broker fails to lose data, all the messge on that broker will be lost forever. When the broker is online again, it is not possible to synchronize data from the leader from the beginning.
  • When building a cluster to deploy the Quorum Queue, definitions like a master node, non-slave nodes, or replicates will no longer be true.

2.2 Compare Quorum Queue vs Mirror Queue

  • Mirror Queue => I think it has Depreciation (subjective view). Formerly Replicated queue
  • Quorum queue data messages forever on disk. As for Mirror Queue, the options for Durable Queue and Persistent Message will have different calculations:
    • Non-durable Queue + Non-Persistent message = lost Queue + lost Message (after Broker restart)
    • Durable queue + Non-Persistent message = And Queue + lost Message
    • Durable queue + Persistent message = Remaining Queue + Remaining Message
    • Mirrored queue + Persistent message = Remaining Queue + Remaining Message
  • When there is 1 faulty node, and then return to normal. With Mirror Queue, it will block the whole cluster. Because it needs to resynchronize the entire message during the incident. In contrast to Quorum Queue, it does not block. It only syncs new messages.
  • When having Network Partition problems. (networks of nodes are not connected to each other). With Mirror Queue, there will be split-brain situation. (1 queue / cluster> 1 master). With Quorum Queue, provide policies autoheal , pause_minority , pause_if_all_down , so that the administrator can configure the handling direction.
  • With Mirror Queue, the master node will receive all read / write requests. The mirror nodes will receive all messages from the master node and write to the disk. The mirror nodes are not responsible for offloading “read” / “write” to the master node. It just mirrors the message to serve the HA. With Quorum Queue, Queue A can master on Node 1. But Queue B can master on Node 1.

3. Building rabbitmq cluster system

3.1 Design diagram

  • Write a big, simple diagram:
    Diagram
  • I use 3 nodes. (should choose an odd number. To facilitate the algorithm to elect the leader)
  • These 3 nodes I installed Docker Swarm, with the leader is Node 1. And use the docker stack to deploy the cluster. (You absolutely can not use the Docker Swarm, but only the regular Docker container will do. In the step-by-step steps I will explain in detail so you can customize).
  • Should set up the number of nodes (broker) is odd. For example 3,5,7 to facilitate the leader election algorithm
  • Deployment environment:
    • Nodes running Ubuntu Server 18.04
    • The node installs Docker
    • Node1 runs rabbitMq, whose container hostname is rabbitmq1 . Similar to node2, node3 is rabbitmq2 , rabbitmq3 .

3.2 Step by step

first). Prepare

  • My lab uses 3 AWS EC2 instances. (for strong network, fast machine, don’t have to wait). You can replace it with any server, use docker-machine, or virtualize vmware. Not so important. As long as 3 nodes are connected to each other, and there is internet to download is.
    • Hostname information (in the docker-stack deployment step to be used).
      • Node1 = ip-172-31-11-205
      • Node2 = ip-172-31-3-230
      • Node3 = ip-172-31-1-3
      • (Get this information by ssh into the server and type command hostname .)
  • Install Docker Engineer
    • Google how to install or run this script I wrote for fast also

  • Install Docker Swarm on 3 nodes.
    • Select node1 as leader.

    • Node2 and Node3 join cluster

    • Result docker-swarm
    • For more details, please refer to the tutorial of Mr. xuanthulab.net

2). Dockerfile

  • Dockerfile

    • Note the RABBITMQ_ERLANG_COOKIE environment variable is very important. If the rabbitmq nodes want to communicate with each other, this cookie value needs to be the same in order to authen . (If this value is not set, the value of erlang.cookie will be randomly generated, and differ on each node.) For other environmental variables supported by rabbitmq, see https://www.rabbitmq.com/configure.html#customise-environment
  • Configuration file rabbitmq-qq.conf

    • listeners.tcp.default=5672 . port to the client to connect to the broker. Default port is 5672. Can be changed to another port if conflict
    • management.listener.port = 15672 . port to go to webadmin gui
    • There is 1 port epmd I do not put in the config file. By default, this port is 4369 . This port is very important, the nodes use this port to discovery each other. Force port numbers to be the same on the nodes
    • Information on ports can be found at https://www.rabbitmq.com/networking.html#ports
    • rabbitmq1 , rabbitmq2 , rabbitmq3 respectively the hostname of 3 nodes. (Note these 3 hostnames are different from the hostname of the ec2 instance, I wrote above). By default rabbitmq does not support FQDN. Want to use a long hostname. then set env RABBITMQ_USE_LONGNAME =true . More details https://www.rabbitmq.com/clustering.html#node-names
    • cluster_partition_handling=pause_minority : when there is a networking partition problem, rabbitmq provides 3 policies for handler, pause-minority , pause-if-all-down , autoheal . If you do not declare this configuration, it will ignore it by default and do nothing. After I consulted, I found that pause-minority probably will use the most. With this mode, the number of nodes on the lesser side will be down . Messages and queues will be sent to the side with more nodes. Information about this issue I will write in more detail below. You can refer to https://www.rabbitmq.com/partitions.html
  • File rabbitmq-qq-definitions.json

    • This file I just to define the login account
    • You can declare this file to declare queues, and more.
  • Can build docker image with command

3). Docker stack file

  • docker-compose.yml file

    • image: tungtv202/rabbitmq_ha_qq : docker image that I have built already
    • rabbitmq_volume : create volumes to persistent data. (If you have a problem with the lab, you should delete the volume and then recreate a new volume)
    • ports: route adds a port if some other public port is needed
    • replicas: 1 : only 1 container per node is enough.
    • Use constraints.constraints to specify deployed deployments spread over 3 separate nodes. Hostname information here is EC2 hostname, I reminded above.
    • Because I use docker-swarm deployment, so when sharing a network, the containers will understand each other’s hostnames. (rabbitmq1, rabbitmq2, rabbitmq3). In case you do not use docker swarm, you can edit the above file to compose docker file. And more information

    • At the end of the lesson, I have share source config file available 3 docker-compose.yml files to run independently on 3 instances. In case you are not using docker-swarm, docker stack
  • Run docker stack to deploy the service

4). Check the result

  • docker
    • node1
    • https://tungexplorer.s3.ap-southeast-1.amazonaws.com/rabbitmq/node1.JPG
    • node2
    • https://tungexplorer.s3.ap-southeast-1.amazonaws.com/rabbitmq/node2.JPG
    • node3 https://tungexplorer.s3.ap-southeast-1.amazonaws.com/rabbitmq/node3.JPG
  • Web admin
    • 172.31.11.205:15672
    • 172.31.3.230:15677
    • 172.31.1.3:15666
    • https://tungexplorer.s3.ap-southeast-1.amazonaws.com/rabbitmq/webadmin.JPG
    • Information about nodes in the cluster is shown in the Overview tab
  • Create Quorum Queue
    • Go to the Queues tab to create a queue, and experience switching nodes on and off. To test the Hight Avalibility of the queue https://tungexplorer.s3.ap-southeast-1.amazonaws.com/rabbitmq/quorumadmin.JPG
    • Note: Choose any 1 node to be the queue leader. (It does not matter, in the future, the cluster will automatically re-election the new leader)

5.) Some commands to debug errors during setup

  • Check log container
  • Access the container and use rabbitmqctl cli. Refer to https://www.rabbitmq.com/rabbitmqctl.8.html

3. Lab some scenarios

3.1 There are nodes that are down / and reUp

  • The script is quite simple, I find nothing complicated. You can stop the container. Or scale service = 0 to test.
  • The first example Queue A, has the master node [email protected] , then stops the container on node1. Then the master node is moved to node2, or node3. And the message is not lost

3.2 Network parttion

  • This scenario can be reproduced by “dropping the network” between node3 vs the other 2 nodes. I use aws ec2, so I should go to fix Secure Group. Or you can create a firewall on the nodes. To block, do not allow the network to connect.
  • (1) If I don’t have cluster_partition_handling = pause_minority in rabbitmq-qq.conf file, the following scenario will happen:
    • node3 thinks the other 2 nodes are down. It is itself the leader of that cluster.
    • cluster node1 + node2, think node3 down. These 2 guys voted each other as leaders themselves.
    • This problem is called split-brain
    • And when the client creates a new queue, or writes a message to the queue on node3. There will be no data synchronization corresponding to cluster node1 + node2. And vice versa.
    • At this point, both sides of node3, and node1 + node2 think that the other side is down. Not detected a network partition problem. Only when we reconnect the network between 3 nodes together. At this time the cluster has just been discovered. (Rabbitmq writes that they use the Mnesia database to detect this problem)
    • Webadmin screenshot of node1 + node2 https://tungexplorer.s3.ap-southeast-1.amazonaws.com/rabbitmq/node1_np.PNG
    • Snapshot webadmin of node3
    • https://tungexplorer.s3.ap-southeast-1.amazonaws.com/rabbitmq/node3_np.PNG
    • Hướng xử lý in this situation ?. You must choose 1 party as the standard. Then restart rabbitmq on the other side, leave rabbbitmq on the other side to rejoin the cluster. And synchronize the message from the standard side to. And accept the loss of data. Reference https://www.rabbitmq.com/partitions.html#recovering
  • (2) If I have the configuration of cluster_partition_handling = pause_minority in my rabbitmq-qq.conf file, the following scenario will occur:
    • node3 sees it as it is. The whole cluster declares 3 nodes. Because 1 is less than 2. So rabbitmq on node3 for always down. And cluster node1 + node2 still run normally. (note that rabbitmq is shutdown, but the container is still running normally. Can be checked using rabbitmqctl )
    • Now the route from the client will be redirected to cluster node1 + node2. (For this route to continue reading, part 4)

4. Setup Nginx

  • When I use java springboot to configure rabbitmq client. I just need to declare the list of brokers rabbitmq1, rabbitmq2, rabbitmq3. And the library automatically routes me to the “currently available” broker.
  • In case the library does not support, we need an external endpoint inspiration. And check before the route to the broker is available.
  • Can use nginx. With the following simple configuration

Bonus

(This one’s not lab yet)

  • Consider using Quorum Queue for Fanout Exchange. (Because the memory for storing messages is greatly increased => resource consuming)
  • The default message on Quorum Queue is stored on memory / disk forever. Limit setup (and 3rd monitoring system) is required so that when the threshold is reached, rabbitmq releases resources
    • x-max-in-memory-length sets a limit as a number of messages. Must be a non-negative integer.
    • x-max-in-memory-bytes sets a limit as the total size of message bodies (payloads), in bytes. Must be a non-negative integer.
  • Source code https://github.com/tungtv202/ops_rabbitmq_ha_qq
Share the news now

Source : Viblo