1. Introduction

When deciding to start using asynchronous communication in a software organization that is used to synchronous communication, it will take time before architects/analysts/programmers will be able to make the mental switch to design and implement truly idiomatic asynchronous interfaces.

Old habits die hard.

In order to help with this changeover, I have collected a list of questions that can be used during the review of designs and implementations of asynchronous interfaces. If you’re not entirely convinced a checklist is appropriate, check out the book 'Checklist Manifesto' by Atul Gawande.

2. FAQ - Frequently Asked Questions

A.k.a. The things I hoped you might ask

  1. Q: What technologies is this checklist covering?

    A: Originally the checklist was developed primarily with the RabbitMQ AMQP implementation in mind. I believe it is fair to claim the checklist would work with other AMQP and AMQP-like implementations such as Apache Qpid, Apache ActiveMQ, etc. In addition, it is largely applicable for Java Message service (JMS) implementations. And while some items are applicable to Apache Kafka, Apache Kafka might require a dedicated checklist.

  2. Q: This isn’t a checklist! The answers to the questions aren’t Yes/No

    A: Okay, it is true it does not resemble 'Oil pressure …​ Green' and 'Throttle …​ 1000 RPMN '. The idea of having an item as 'mark completed' would be to assess if the question was asked, alternatives considered, and a satisfactory answer provided and documented.

  3. Q: I found a grammar or spelling error(s)

    A: Please send me an email (or pull request) and I might correct it.

  4. Q: I would like to contribute to this checklist!

    A: Great! Check first if you agree with license, and if yes, feel free to fork this project and publish your own amended/extend/reduced version. When you drop me an email, I might add your version to the external references.

  5. Q: Can I use this list for commercial purposes?

    A: Check the license! You’ll see commercial use is allowed, provided you abide by the other rules. And when the checklist was useful, you can always buy me a beer! https://PayPal.Me/DennisVandePoel

  6. Q: This checklist seems to be devoid of security-related topics

    A: Security is too important for a it to be based on checklist that is not reviewed and maintained on a daily basis by dedicated people. A good start would be your CISO or https://owasp.org

  7. Q: What is the definition of a term used in the checklist?

    A: In future versions of this document I intend to add more precision to the questions, add a glossary, and add more detailed guidance on the interpretation. One day…​

3. The Checklist

Answer the below questions for each point-to-point channel. In case of a publish subscribe channel, consider each pub-sub combination as a point-to-point Channel; i.e. in case (X) publishes a message and (A) and (B) subscribe to the message; then consider this pub-sub as two point-to-point channels: (X) → (A) and (X) → (B).

The questions have been grouped by the different perspectives, i.e. from the side of the producer, from the side of the consumer, and a more holistic end-to-end view. You should assign these perspectives to the relevant team(s). The intention is that the responsible team’s documentation should have captured the answer. It does not mean the teams are responsible for providing the answers, but they should collect the answers, and each answer should be reflected in their documentation and guide their implementation processes.

3.1. End-to-end communication (e.g. Architect / System Analyst)

Note; For questions 1-3, consider the messages traveling between the endpoints. Repeat for each point-to-point channel;

  1. What happens if a message gets lost? Do you really need the 'at least once' delivery guarantee

    1. from POV of Business Process?

    2. from POV of Consumer side?

    3. from POV of Producer side?

  2. What happens if a message is delivered multiple times to the same consumer?

    1. Do you really need at most once/exactly once delivery guarantee?

    2. Can you design the message to be idempotent (e.g. UUID)?

  3. Is there any (implicit) ordering between messages?

    1. When ordering is violated, can this result in wrong business results in the consumer and its dependent systems?

    2. What happens if messages occur in the same period, arrive in order, but due to delay all in a separate 'period' (e.g. next day, next fiscal year)?

    3. Can you make the messages 'associative'?

    4. What happens if messages occur in the same period, arrive in order, but due to delay some of them in one period and some of them in a separate 'period' (e.g. next day, next fiscal year)?

    5. What happens if messages occur in the same period, arrive out-of-order, and due to delay some of them in one period and some of them in a separate 'period' (e.g. next day, next fiscal year)?

    6. Can the message structure contain ordering (so to offload ordering responsibility from the broker)?

3.2. Message Producer

  1. What happens if the (RabbitMQ) broker is unreachable/unavailable?

    1. Where will messages be stored during the duration of the outage?

    2. How will you ensure you do not cascade the (RabbitMQ) broker outage to the Producer’s clients? Where is the relevant Component test for this?

    3. What is the pressure valve in case of extended outage duration and a growth of messages to be produced? Where is the relevant Component test for this?

  2. Is there a need for the producer to be involved to execute a 'replay' in case of broker/consumer failure?

    1. Has this been verified with the Broker Infrastructure team?

    2. Has this been verified with the Consumer team?

    3. Is the producer capable of re-generating messages for a replay?

    4. How far back in time can the producer go back to do a replay?

  3. If an at least once delivery guarantee is required for a specific consumer, how will the producer verify that a specific binding is in place for that specific consumer (not just any binding to the exchange by any consumer).

  4. Does the producer expect the broker to provide durability/ persistence guarantees?

    1. If yes, has this been acknowledged by the team that provides broker?

3.3. Message Consumer

  1. What will be your acknowledgment of message strategy?

    1. Is it compatible with the delivery guarantee set out by the Architect / System Analyst?

    2. Is it not unnecessarily exceeding the delivery guarantee requirements required by all relevant point-to-point channels?

  2. What will a consumer do with messages that are not processable (e.g. not adhering to schema due to producer bug)?

    1. What team will monitor and handle this issue in Production? Has that team accepted responsibility?

  3. Will the consumer use the broker’s reject and/or requeue feature?

    1. Does the consumer distinguish between temporary message processing failure and permanent message processing failure? Where is the relevant Component test for this?

  4. How does the consumer handle 'duplicate messages' (caused by broker /consumer) and duplicate 'bodies of messages' (caused by producer)?

  5. Will there be competing consumers?

    1. What component test verifies correct behavior in case of a re-delivery of a non-idempotent message?

    2. To what extent can the broker group messages for efficiency?

  6. Are test messages expected in production?

  7. Are Correlation identifiers expected?

3.4. The Broker

  1. What is the time-to-live configuration for each queue?

    1. If not explicitly configured, what is the default value?

    2. Is this sufficient for normal operation

    3. Is this sufficient in case of consumer failure/outage?

    4. In case of overflow, where do the messages go?

    5. In case of event messages, how old can an event be before becoming irrelevant?

  2. What is the maximum queue size (bytes or number of messages) for each queue?

    1. If not explicitly configured, what is the default value?

    2. Is this sufficient for normal operation?

    3. Is this sufficient in case of consumer failure/outage?

    4. In case of overflow, where do the messages go?

  3. How are dead letter queues configured for exchanges and queues?

    1. What happens to the messages if the dead-letter queues time-to-live configuration is exceeded?

    2. What happens to the messages if the dead-letter queues maximum queue size configuration is exceeded?

    3. Is the underlying hardware (RAM/HDD/Network) sufficiently sized for this eventuality?

  4. Do consumers intend to use re-delivery?

  5. How often will the broker attempt to re-deliver a message?

    1. What happens to the message if the maximum number of re-delivery attempts has been reached?

    2. How do you ensure a message is not re-delivered indefinitely?

  6. Has a replay mechanism been agreed?

    1. Who are the relevant teams to involve in a replay?

    2. Is ordering of messages of any importance?

    3. Are any queue flushes required?

    4. Are there any performance considerations (e.g. a preference for LIFO handling)?

  7. Is the producer ensuring durability/persistence of the messages (underlying data) or does it expect the broker to provide durability/persistence guarantees?

4. Credits

  • Dennis Van de Poel

5. External References

6. License

CC

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.