[TOC]

Why this Repo?

In this repository we provide an example deployment of Red Hat's AMQ-Broker Operator. It provides a strongly opinionated but simple example on how AMQ-Broker can be deployed in a High Available (HA) setup with a minimum of two Broker's who work independently yet still work together. My goal is to make it as easy as possible to deploy a reliable and secure AMQ-Broker Cluster deployment and include documentation that gets you up to speed quickly. Cause time is money, friend!

What is new in 7.10?

Happy to announce my latest release is compatible with the latest 7.10 AMQ-Broker release. This update includes:

• API version upgrades
• antiAffinity == More Reliability! More Here
• Alerting, see chapter 5 for more!
• Added Grafana Dashboard
• Updated documentation
• Added Yamllint

Happy Queue'in!

Open Tasks

• [x] Adding .yamllint
• [x] Check up on my reported Major Bug Fixed in 7.11
• [x] Check up on my second reported Minor Bug Fixed in 7.11
• [ ] Upgrade to AMQ-Broker 7.11 (LTS)
• [ ] Improve uppon documentation (use code snippets)

What is AMQ-Broker?

AMQ Broker is a pure-Java multiprotocol message broker. It’s built on an efficient, asynchronous core with a fast native journal for message persistence and the option of shared-nothing state replication for high availability.

• Publish and subscribe - Many to many dissemination in a fault tolerant, durable manner
• Long-term data retention - Efficiently stores data for immediate access in a manner limited only by disk space
• Advanced queueing - Last value queues, message groups, topic hierarchies, and large message support
• Replayable events - Serves as a repository for microservices to build in memory copies of source data, up to any point in time
• Partition messages for scalability - Allows for organizing messages to maximum concurrent access

AMQ Broker is based on the Apache ActiveMQ Artemis project.

Supported Libraries

RedHat supports the following wide range of libraries/packages in order to talk with AMQ-Broker: https://access.redhat.com/jbossnetwork/restricted/listSoftware.html?product=jboss.amq.clients&downloadType=distributions

Please note that there are quite a lot of libraries out there that can work with AMQP(S). But not all libraries support AMQP(S) over HTTP(S). If your library does not, it will not be able to communicate with the routes generated by the AMQ-Broker deployment. It will however be able to communicate by AMQP(S) directly on the service level. Please concider this before using any of the AMQP(S) compatible libraries.

Getting started

In this section I will try and explain the steps I have undertaken to be able to run this middleware properly, which includes High Availability en SSL security. As mentioned before, this is a strongly opinionated example of a minimal setup that should fit most use cases for when a Development team starts with AMQ-Broker.

Important

This setup and these templates are only compatible with AMQ-Broker Operator 7.10 If you are looking for the 7.9 version, please see the following release.

1. Installing the AMQ-Broker Operator

Install your AMQ-Broker Middelware from the OpenShift OperatorHub. For more information on this visit the following link.

If your development team adheres to the standard setup of multiple environments (for example development, test, acceptance, production) and you or the OpenShift platform team has access to the AMQ-Broker subscription? You are able to install the AMQ-Broker Operator within it's own independent namespace, for example “projectname-amq-broker”. But you can also choose to install the AMQ-Broker Operator within each respective environment aka namespace. It would not affect the subscription costs and the overhead is minimal consisting of one pod per installed operator that manages the AMQ-Broker instances.

So if your project within OpenShift would be called "example" and you adhere to the development within multiple environments aka namespaces. You'll have example-development, example-test, example-acceptance and example-production. Now you could choose to add another namespace (for example example-amqbroker) where you install manage all your AMQ-Broker Operator based deployments for each environment. But I suggest against this due to the fact that an update of the AMQ-Broker Operator could potentially effect all your AMQ-Broker deployments simultaneously. Therefore I suggest to install the AMQ-Broker Operator within each environment within you will use AMQ-Broker deployments. The templates I have provided within the "templates" folder can be processed and applied within each namespace that has the AMQ-Broker Operator installed.

2. Certificates

In order to be able to use SSL you will have to create your own certificate for each AMQ-Broker Cluster deployment. Before we get into what information that certificates needs to hold I will explain how AMQ-Broker automatically creates it’s route resources. As you are only allowed to use a max of 63 characters for your route, naming convention is rather important. Else you might run into problems later on.

First we need to understand how AMQ-Broker automatically exposes routes for each individual broker if you configure the deployment template to do so. If you do choose to expose all AMQ-Broker has to offer you will be provided with a route that can be accessed by the application and another route that allows you to access the broker’s web console.

The below route is automatically created if you set “expose: true” within your acceptor settings. Default paramater set to "false" within the template.

The below route is automatically created if you set “expose: true” within your console settings. Default paramater set to "false" within the template.

Here follows a short explanation for each field mentioned above:

• Name AMQ-Broker Cluster Deployment: The name you give the deployment. This name is based on the following two paramters: "${TEAM}-${ENVIRONMENT}".
• Name Acceptor: The name you give the Acceptor. This name is based on the following parameter: "${NAME_ACCEPTOR}".
• Amount of Replicas: The number for the amount of replicas within the cluster
• Namespace: The namespace the operator is located in
• Your Domain: The domain of your own company i.e. example.com
• Default: We have no say in this, the Operator places these sections within the route

In the AMQ-Broker Cluster example setup I chose not to automatically expose the application (AMQPS) routes for each broker in the cluster. This because I create my own Service and Route that contains all brokers within the AMQ-Broker Cluster deployment resulting in 1 route that is able to reach any of the brokers within the cluster. I did however choose to expose the web console routes for each broker so that we can access each broker’s web console respectively.

Now that you know the routes are build up you are able to create the certificate that you will need to be able to setup an AMQ-Broker Cluster deployment that includes SSL. As explained previously, each broker is able to each expose an AMQPS application route and a web console route. So if you have a cluster of two brokers and choose to expose them both you will have a minimum of 4 routes. If you want your AMQPS application routes to be able to access both brokers within the cluster that makes for another route coming to a total of 5 routes:

• 2 automatically generated AMQPS application routes
• 2 automatically generated web console routes
• 1 self generated AMQPS High Available application route

Now that you understand how the routes are build up and the amount of routes you need you should be able to create your SSL certificates. In this setup I suggest one certificate for each cluster. This means we will be using the SubjectAlternativeNames section within the certificate to be able to provide alternative endpoints so we can incorporate all 5 routes into one certificate for each cluster.

Prerequesits

To be able to follow the below steps you will need the following: - Either Windows with Git Bash or any Linux distro including OpenSSL - Please note that with Git Bash on Windows you need to prefix all the linux commands below with "winpty". - The Oracle JRE for the “keytool”

These are the steps I took to be able to create the necessary certificates that can be used by a single AMQ-Broker cluster:

1. Within your organization, request a certificate with a canonical name of your choosing and make sure you add the endpoint names of the before mentioned 2 automatically generated AMQPS application routes, the 2 automatically generated web console routes and the self generated AMQPS High Available application route. Make sure your container applications have setup the appropriate certificate chain within their containers to be able to trust these certificates. Else they will not be able to connect.
2. Within most companies you will generally be provided with a .PFX certificate that is secured with a password. This .PFX file contains everything we need to be able to create the .P12 and .JKS files that AMQ-Broker prefers.
3. The first step is to export both private and public keys from the .PFX by performing the following commands using the .PFX file as input. If you use GIT BASH on windows, instead of openssl perform the same commands but start with “winpty ” instead.

On linux:

# When Asked to provide a password, use the password that was set for the .PFX

## Obtaining the private key
openssl pkcs12 -in example-amq-broker-test.pfx -nocerts -out amq-broker-test.key
## Obtaining the public key
openssl pkcs12 -in example-amq-broker-test.pfx -clcerts -nokeys -out amq-broker-test.crt

1. Then use the java keytool to be able to create a .JKS (java trust store) file.

# When Asked to provide a new password, decide and confirm the password and 
# make sure you secure them somewhere so you can remember the trustStorePassword for later.

## Create the .JKS file
keytool -import -file amq-broker-test.crt -alias amq-broker-testCA - keystore amq-broker-test-truststore.jks 

1. Then create the .p12 keystore file.

On linux:

# When Asked to provide a new password, decide and confirm the password and 
# make sure you secure them somewhere so you can remember the keyStorePassword for later.

## Create the .p12 file
openssl pkcs12 -export -in amq-broker-test.crt -inkey amq-brokertest.key -certfile amq-broker-test.crt -name "amq-broker-test" -out amqbroker-test-keystore.p12

1. Now that you have both the .jks truststore and the .p12 keystore you can use them to create secrets within OpenShift. You will need to create two secrets.

One Secret that will be used for the the AMQPS application routes:

oc create secret generic example-test-ssl-secret -n example-test --from-file=broker.ks=amq-broker-test-keystore.p12 --from-file=client.ts=amqbroker-test-truststore.jks --from-literal=keyStorePassword=testtest --fromliteral=trustStorePassword=testtest

One secret that will be used for setting up the web console routes:

oc create secret generic console-example-test-ssl-secret -n example-test --from-file=broker.ks=amq-broker-test-keystore.p12 --from-file=client.ts=amq-broker-test-truststore.jks --from-literal=keyStorePassword=testtest --from-literal=trustStorePassword=testtest 

Unfortunately we cannot use the same secret for both the AMQPS application routes and the web console routes because AMQ-Broker will not accept this.

Make sure you have used the correct truststore and keystore passwords that you have decided upon when creating these .JKS and .p12 files. Now that you have the certificates setup within OpenShift you should be ready to deploy your first AMQ-Broker Cluster.

3. Deploying AMQ-Broker

There are two important Custom Resource Definitions (also known as CRDs) of interest for the use AMQ-Broker and those are "ActiveMQArtemis" and "ActiveMQArtemisAddress". Within the ActiveMQArtemis CRD you define everything that has to do with the AMQ-Broker deployment. For example the amount of pods, whether you expose (exposed) certain routes or whether you enable SSL or not. In addition, you have ActiveMQArtemisAddress CRD. Here you describe all your addresses, the type and queues. You list elements of the ActiveMQArtemisAddress CRD’s within your ActiveMQArtemis CRD in order to setup addresses and queue’s within your AMQ-Broker Cluster deployment. In this setup I use pre-defined addresses and queue’s using the ActiveMQArtemisAddress CRD.

3.1 amq-broker-deployment-template.yaml

To make it easier to explain the most important settings i have created some sections to be able to go through the more significant ones.

Lifecyclemanagement

These settings make it so that your AMQ-Broker Cluster will not update as soon as a new version becomes available. For example, when a new AMQ-Broker Operator version is installed. You will have to manually adjust this setting when you are ready to upgrade if needed.

Size, compute and antiAffinity management

Here you set a specific image (default: placeholder) so you know what image is being used for your deployments. You also set your cpu/memory resource and amount of storage. As 7.10, AMQ-Broker allows you to use antiAffinity rules so you can make sure the pods that your clusters consists of are spread amoungst multiple nodes. This allows for higher reliability.

Additional Settings

Here are some specific AMQ-Broker settings that allow you to expose your web console routes, setup SSL and also allow you to set a username and password for access to the web console.

addressSettings

Here you can list (either specifically or with REGEX) the ActiveMQArtemisAddress CRD’s that you want to use within your AMQ-Broker Cluster deployment. In this section we intentionally disabled the autoCreate options as we do not want to allow anyone to connect to the AMQPS application route to create rogue addresses and/or queues.

acceptors

This section allows you to decide what protocol to use when communicating with the AMQ-Broker Cluster deployment. When selecting an alternative port, do not use port 61616. This because AMQ-Broker pods use that port to communicate amongst the cluster of AMQ-Broker pods. This allows a listener to connect to any of the two brokers in the cluster and still be able to reach messages of of each of the AMQ-Broker pods.

Service

In this section we create our own service that incorporates the service of both brokers into one pool.

Route

In this section we create a route that uses the above mentioned service. This allows a listener or producer to get to either of the AMQ-Broker pods within the AMQ-Broker Cluster deployment.

3.2 amq-address-deployment-template.yaml

Here we will go into a bit more detail in regards to the ActiveMQArtemisAddress CRD.

anycast

See row 11-50. This is an example on how I make an anycast Address/Queue. In this setup I use anycast addresses in a one-to-one relationship with queues. Meaning one anycast address relates to one Anycast Queue. I use one or more of the same type of publisher(s) to put one type of message on these queue’s. And I use one or more of the same type of processor(s) to process these specific messages respectively. Each anycast type has an addressname, addressqueue, user/password and a routingtype. To make it clear that these are anycast addresses or queue’s I use the same name for both the addressname and the addressqueue.

multicast

See row 51-105. These is how I use multicast Addresses and Queue’s. The same as how you would commonly use Topic’s. In this case the address is used in a one-to-many queue’s relationship. Each multicast type has the same fields as an anycast type address/queue. An addressname, addressqueue, user/password and a routingtype. Each multicast address can have multiple queues as demonstrated between row 52-78. This way each multicast queue is nested under the same multicast address, making it a one-to-many relationship. As long as you make sure the routingtype is “multicast” and the addressname is the same, you can put as many queue’s under the address as you like. I use one or more of the same type of publisher(s) to put one type of message on the multicast address. And for each underlying queue I use one or more of the same type of processor(s) to process the messages. Each messages delivered to the multicast address will be sent out to each underlying queue’s respectively.

applyCrNames

This setting is found in both anycast and multicast type addresses/queue’s. This relates to the ActiveMQArtemis CRD’s “application” field. This allows the AMQ-Broker cluster deployment to collect the ActiveMQArtemisAddress CRD’s.

4. Metrics

If you enable enableMetricsPlugin within the amq-broker-deployment-template.yaml you will be able to gather a lot of useful metrics straight from the pod’s from the AMQ-Broker cluster. This can be enabled by setting the ${ENABLE_METRICS_PLUGIN} parameter to true.

Within OpenShift 4 you have the capability as DevOps team to utilize a by the existing OpenShift Platform Team Managed prometheus/thanos setup for user-defined projects. Using a ServiceMonitor template you will be able to gather the metrics from the pods within the AMQ-Broker cluster and visualize them with Grafana using either a paid or the Open-Source Grafana Operator within your project.

First you must be able to gather the metrics from your AMQ-Broker cluster pods in order to get them within the Prometheus time series database before you can query anything. In order to do that you have to create a service monitor. The metrics are exposed via the autogenerated webconsole routes if you decided to expose these endpoints. You can test to see if /metrics provided you with metrics by logging into one of the AMQ-Broker pod’s webconsoles by going to https://aut-generated-web-console-route.yourcompany.local/metrics.

After you made sure /metrics provides you with a page containing key valued pair metrics you can collect them using ServiceMonitors that point towards the autogenerated webconsole routes on /metrics. See an example within this git under the extras folder extras/amq-broker-service-monitor.yaml

Notice that it is important to use the “https” scheme. Also utilize the “insecureSkipVerify” option, unless you control the OpenShift 4 Cluster prometheus/thanos setup. Because you will need to import the certificates into Prometheus/Thanos to be able to query without “insecureSkipVerify”. I also defined two endpoints, one for each pod. The more pod’s you have within your AMQ-Broker Cluster, the more endpoints you’ll have to add to this template.

5. Alerts

Prometheus alert rules are a way to define alerts that are triggered based on specific conditions detected by Prometheus. These rules allow you to configure custom alerting logic based on the metrics collected by Prometheus. Within OpenShift 4 you have the capability as DevOps team to utilize a by the existing OpenShift Platform Team Managed prometheus/thanos setup for alerting as well as gathering Metrics. You can do this by creating PrometheusRule files.

See an example within this git under the extras folder extras/amq-broker-prometheus-rule.yaml. See the Summary/Description for each of the 3 example alerts on when they alert and why.

6. Grafana Dashboard

Collecting the metrics is one of the steps to be able to observe the workings of your AMQ-Broker Cluster deployments. Below I have a simple AMQ-Broker pre-made dashboard to get you started. You might want to add more or remove some variables that I have constructed in order to make it work for you. I added the “environment” option to be able to easily switch between environments as (in this opinionated install) I have decided to deploy an AMQ-Broker Cluster per environment within the OpenShift 4 project. This JSON Grafana Dashboard can be important within a new fresh Dashboard within Grafana (either standalone, external or the Opensource Grafana-Operator).

7. References

Check these links for more information:

• RedHat Documentation: Latest AMQ-Broker Operator documentation
• Artemiscloud.io: Opensource Artemiscloud Operator
• Apache ActiveMQ Artemis: Artemis Documentation

8. License

MIT License

Day 2 Operations

1. Adding or removing Addresses/Queue’s after initial setup

In this section i’ll explain on how to add anycast and/or multicast addresses/queue’s after you have already initially setup the AMQ-Broker Cluster with pre-existing addresses/queue’'s.

It is best practice to always create a branch before adjusting any code, even templates!

1.1 Anycast

Navigate to the amq-address-deployment-template.yaml and simply add another anycast address entry within the pre-existing list of address entries. Under Anycast Addresses, copy one of the pre-existing entries and place it within the Anycast Address section. See the example below:

  ## Anycast Addresses ##
  - kind: ActiveMQArtemisAddress
    apiVersion: broker.amq.io/v1beta1
    metadata:
      name: ${ENVIRONMENT}-${TEAM}-newanycastqueue
      namespace: ${NAMESPACE}
    spec:
      addressName: newanycastqueue
      queueName: newanycastqueue
      user: ${ENVIRONMENT}
      password: ${PASSWORD}
      routingType: anycast
      applyToCrNames:
        - ${TEAM}-${ENVIRONMENT}

Once you are done adding the additional queue’s. Commit your adjustments and re-apply the template. Existing records will remain whilst new anycast addresses/queue’s will be to the AMQ-Broker cluster accordingly.

1.2 Multicast

Navigate to the amq-address-deployment-template.yaml and simply add another multicast address entry within the pre-existing list of address entries. Under Multicast Addresses, copy one of the pre-existing entries and place it within the Multicast Address section. As these multicast addresses are probably going to be having multiple queue’s, I suggest to place it at the bottom so you can create a specific multicast section. See the example below:

  # newtopic # 
  - kind: ActiveMQArtemisAddress
    apiVersion: broker.amq.io/v1beta1
    metadata:
      name: ${ENVIRONMENT}-${TEAM}-newmulticastqueue
      namespace: ${NAMESPACE}
    spec:
      addressName: newtopic
      queueName: newmulticastqueue
      user: ${ENVIRONMENT}
      password: ${PASSWORD}
      routingType: multicast
      applyToCrNames:
        - ${TEAM}-${ENVIRONMENT}

Once you are done adding the additional queue’s. Commit your adjustments and re-apply the template. Existing records will remain whilst new anycast addresses/queue’s will be to the AMQ-Broker cluster accordingly.

1.3 Adding Expiry and/or Dead Letter Queue’s

Depending if you want an Anycast Dead Letter or Expiry queue, follow the examples listed for creating new anycast/multicast above in chapter 3.1.1 and 3.1.2. The way you then use those addresses/queue’s as a Dead Letter or Expiry queue is by matching the name of the queue to specific settings with the amq-broker-deployment-template.yaml. See the example below listed under “addressSettings”:

        addressSetting:
          - match: "addressIWantASpecificDLQOrEXPAddressFor"
            deadLetterAddress: "addressUsedAsDLQ::queueUsedAsDLQ"
            expireAddress: "addressUsedAsDLQ"

1.4 Adding ActiveMQArtemisSecurity

If you want want additional grained security access control on top of a Username/Password. ActiveMQArtemisSecurity has got you covered. Simply make sure you add this template to the other templates and deploy it within the same namespace as the address and amq-broker cluster templates. It is important to note that you must define users and subsequent roles for those users. Once defined, you can (like within other AMQ-Broker related templates) use the “match” option to define what address/queue you want these security settings to applied to. See a simplified ActiveMQArtemisSecurity example below:

kind: Template
apiVersion: template.openshift.io/v1
metadata:
  name: example-${ENVIRONMENT}-amq-security-deployment
  namespace: example-${ENVIRONMENT}
  labels:
    template: example-${ENVIRONMENT}-amq-security-deployment
    name: example-${ENVIRONMENT}-amq-security-deployment
    team: example
objects:
  - apiVersion: broker.amq.io/v1beta1
    kind: ActiveMQArtemisSecurity
    metadata:
      name: example-${ENVIRONMENT}-amq-security-deployment
      namespace: example-${ENVIRONMENT}
      application: example-${ENVIRONMENT}
      labels:
      team: example
      environment: ${ENVIRONMENT}
    spec:
      applyToCrNames:
        - example-${ENVIRONMENT}
    loginModules:
      propertiesLoginModules:
        - name: "user-login-module"
          users:
            - name: ${AMQ_example_USERNAME}
              password: ${AMQ_example_PASSWORD}
              roles:
                - example-producer
            - name: ${AMQ_GD_USERNAME}
              password: ${AMQ_GD_PASSWORD}
              roles:
                - angus-producer
            - name: ${AMQ_MWO_USERNAME}
              password: ${AMQ_MWO_PASSWORD}
              roles:
                - mcfive-producer
    securityDomains:
      brokerDomain:
        name: "activemq"
        loginModules:
          - name: "user-login-module"
            flag: "sufficient"
    securitySettings:
      broker:
        - match: "example-v1-topic"
          permissions:
            - operationType: send
              roles:
                - example-producer
        - match: "angus-v1-topic"
          permissions:
            - operationType: send
              roles:
                - angus-producer
        - match: "mcfive-v1-topic"
          permissions:
            - operationType: send
              roles:
                - mcfive-producer

AMQ-Broker Operator & Components Responsibility Matrix

AMQ-Broker can be deployed within each namespace you want to use AMQ-Broker in. Without increased costs in regards to subscriptions. This makes it easier to upgrade for each separate environment, reduces complexity and allows for a better integration with it’s environment.

In most companies there is a team that manages the OpenShift platform. In the below AMQ-Broker Responsibility Matrix I have illustrated on who is responsible for what. The team that manages the OpenShift platform is responsible for installing and possibly update the AMQ-Broker middleware (red). The DevOps team is responsible for the configuration of the AMQ-Broker cluster deployment (blue). This is merely an example on how to divide the responsibilities whilst still remaining flexible.