A reliable message delivery platform is the key to engineering a data pipeline in a microservices architecture. Once a message is received by a messaging solution (brokers), it should guarantee the following:

• The loss of a single broker instance does not affect the availability of the message for consumers.
• If producers and consumers experience outages/restarts, certain delivery semantics are honored. Generally, the semantics are as follows:
  • At-least-once delivery: The messaging system guarantees that a message is delivered at least once to a consumer. It is possible that duplicate messages are received, and the consumer is responsible for the deduplication of messages.
  • At-most-once delivery: The messaging solution guarantees that messages are delivered at most once. Some messages might get dropped.
  • Exactly-once delivery: A message is guaranteed to be delivered exactly once for each consumer. Generally, this guarantee is difficult to engineer without some sort of consumer-broker coordination.

To build reliability, a key messaging design pattern is called the competing consumers pattern. Multiple concurrent consumers can consume messages from the same topic, thereby enabling redundancy—as shown here:

Besides the resiliency and availability benefit, such a pattern also enables the system to work at a higher throughput, and improves scalability (since the number of consumers can be increased/decreased on demand).

Another advantage of message-based interaction is the shock absorber behavior that queues bring in. With an API-based interaction, the consumer has to consume requests at the same rate at which the producer is making them. This impedance matching of all producers and consumers can be difficult to engineer in a non-trivial architecture. Message queues act as a buffer between the producer and the consumer, so that consumers can work at their own pace. The queues also smooth out intermittent heavy loads that otherwise could have caused failures.

Messaging architectures and related resiliency patterns are covered in detail in Chapter 6, Messaging.