Stop Calling Everything a "Message Queue"

If you're early in your backend career, you've probably run into this wall: RabbitMQ, Kafka, SQS, SNS, EventBridge, BullMQ — they all "send messages between services," so why do five different names exist?

The industry doesn't help you here. RabbitMQ is called a "message queue" but can do pub/sub. Kafka is called a "queue" in casual conversation but is architecturally a log. Redis is marketed as a cache but can do three of these five things at once.

So stop learning products first. Learn the underlying axes that separate these models. Once you have the axes, any new tool you encounter becomes "ah, it's X on this axis and Y on that one" instead of a brand-new thing to memorize.

What all of them have in common

Before splitting them apart, it's worth naming what doesn't change across any of these systems — this is the shared skeleton:

• A producer and a consumer, decoupled in time. The producer doesn't call the consumer directly. Something sits in between.

• Asynchronous by nature. The producer fires and moves on; it doesn't block waiting for the work to finish.

• A broker in the middle. Whether it's called a queue, a topic, a stream, or a bus, there's always an intermediary responsible for holding and moving the message.

• The goal is always decoupling. Producer and consumer don't need to be online at the same moment, run at the same speed, or even know about each other's existence.

• Every one of them has to answer the same five questions. What differs between a message queue, pub/sub, a streaming log, an event bus, and a job queue is just how each answers these five questions. That's the whole taxonomy — five axes, five different answer patterns.

Keep that shared skeleton in mind. Everything below is just variation on top of it.

Vocabulary you'll need before we go further

These terms show up in every family below, so it's worth having them straight before comparing styles. Four small groups:

How delivery is confirmed

• Acknowledgement (ack) — the consumer tells the broker "I'm done with this." No ack in time → the broker assumes failure and redelivers.

• Visibility timeout — the window a message stays hidden from other consumers while one is working on it. Set it too short, and two consumers can end up processing the same message.

• Delivery guarantees — at-most-once (may lose messages), at-least-once(may duplicate them), exactly-once (the marketing term). In practice you build on at-least-once and make your consumer idempotent — safe to run twice — using an idempotency key (a stable ID that lets it recognize "I already did this").

What happens when things fail

• Poison message — one that fails every single time it's processed, and would retry forever if nothing stopped it.

• Dead Letter Queue (DLQ) — where a message lands once retries are exhausted, so it stops blocking everything behind it. In streaming systems this becomes a retry topic plus a DLQ topic, since there's no per-message retry built in.

How work is spread across consumers

• Competing consumers — several workers pulling from the same queue, splitting the load between them.

• Consumer group — the streaming equivalent: a set of instances that together consume a topic exactly once, while a different group consuming the same topic is fully independent.

• Offset / partition / key — an offset is a consumer's position in the log; a partition is the unit of parallelism and ordering; the key is what decides which partition a given record lands in.

• Fan-out — one message, delivered to many destinations at once.

• Backpressure — whatever mechanism stops a fast producer from overwhelming a slow consumer.

• Consumer lag — how far behind a consumer's offset is from the latest record. It's the single most important thing to alert on in event streaming — queue depth is the equivalent metric for queues.

One reliability pattern worth knowing early

• Outbox pattern — write the database row and the outgoing message in the same transaction, then publish from that table afterward. This is the standard fix for the classic bug where the DB commit succeeds but the message never actually gets sent.

With that vocabulary in place, the five families below will read a lot more naturally.

The five axes

Axis 1 — Payload: command or event?

This single distinction explains most architecture debates you'll hear in a standup.

Axis 2 — Receivers: one consumer or everyone?

• Point-to-point (competing consumers) — many workers listen, but each message is handled by exactly one of them. Adding workers increases throughput; the total work stays the same.

• Fan-out (broadcast) — every subscriber gets its own copy. Adding subscribers increases total work done, not throughput of the same work.

Mixing these up is a classic early bug: deploy three replicas of a broadcasting service, and the same email goes out three times.

Axis 3 — After reading: destroy or retain?

• Destructive read (queue semantics) — the message is delivered, acknowledged, and deleted. The queue is a buffer of pending work; its natural size is close to zero. A growing queue means you're falling behind.

• Retained log (log semantics) — the record is appended to an ordered, immutable log and stays for a retention window regardless of who's read it. The log is the history. Its natural size is large and growing.

This is the deepest split in the whole taxonomy, and it decides whether replayis even possible.

Axis 4 — Read position: broker-owned or consumer-owned?

• Broker-owned — the broker tracks per-message state (pending, in-flight, acked, dead). Consumers stay stateless; redelivery is the broker's job. Cost: per-message bookkeeping caps throughput and makes strict ordering hard.

• Consumer-owned (offsets) — the broker only appends; each consumer remembers its own position ("I'm at record 1,482,113"). Cost: the consumer manages its own position, but the broker becomes extremely fast — basically sequential writes — and rewinding is just moving a number backwards.

Axis 5 — Routing: producer, broker rules, or consumer subscription?

• Producer decides — writes to a specific destination. Simple, tightly coupled.

• Consumer decides — subscribes to a topic or pattern; the producer stays ignorant of who's listening. This is what "decoupling" usually means in practice.

• Broker decides by rules — a routing layer inspects message content and routes by configured rules. Routing becomes configuration, not code.

The sixth thing worth knowing: push vs pull

This one is orthogonal to the five axes above, but it changes how a system behaves under load — and it's the most common thing people get backwards.

• Pull — the consumer asks the broker "do you have anything for me?" The broker never sends unprompted. SQS works this way: your consumer calls ReceiveMessage in a loop. Kafka consumers poll too.

• Push — the broker sends messages to the consumer on its own schedule. Some pub/sub systems (Redis Pub/Sub) and webhook-style integrations (event buses hitting an HTTP endpoint) work this way.

Pull is preferred almost everywhere because it gives natural backpressure — the consumer only asks for more when it's ready, so a slow consumer never gets flooded. Push hands control to the broker, which can overwhelm a consumer that can't keep up. Most systems marketed as "push" (Google Cloud Pub/Sub, for example) actually default to pull mode for exactly this reason, and only push to an HTTP endpoint when you explicitly configure it that way — which is closer to a webhook than a queue.

Secondary axes worth a mention but not a deep dive: ordering scope (global vs per-key vs none) and delivery guarantee (at-most-once / at-least-once / "exactly-once" — in practice almost always at-least-once plus an idempotent consumer).

The families

A. Job Queue — work distribution

Mental model: a to-do list shared by a pool of workers, where the list itself tracks each item's full lifecycle.

• Payload: command / task

• Delivery: point-to-point, competing consumers

• After read: destroyed (or moved to a completed/failed set)

• Position: broker-owned, with a rich per-job state machine

What separates a job queue from a plain message queue isn't the transport — it's that the broker owns the lifecycle of each job, not just its delivery. That buys you, out of the box:

• retries with backoff strategies

• delayed jobs and repeatable/cron jobs

• priorities and per-queue concurrency limits

• rate limiting

• progress reporting and a dashboard

• a failed-job store you can inspect and retry

Use it for: background work you triggered yourself — image processing, sending emails, generating reports, syncing with a slow third-party API.

Don't use it for: distributing facts to other teams' systems — nothing here replays history, and the producer is implicitly commanding a specific kind of worker.

Examples: BullMQ, Sidekiq, Celery, Resque, Hangfire.

B. Message Queue — durable point-to-point transport

Mental model: a pipe with a buffer between two services.

• Payload: usually a command, sometimes an event

• Delivery: point-to-point, competing consumers

• After read: destroyed on acknowledgement

• Position: broker-owned (in-flight / visibility tracking)

The value here is temporal decoupling — the receiver can be down, slow, or mid-restart, and the work just waits. It also gives you load leveling: a burst of 50k requests becomes a queue that drains at whatever rate the consumers can sustain, instead of taking the service down.

**It's still service-to-service communication — just async instead of sync.**That's easy to lose sight of, since there's no direct call between the two services. But that's exactly the point: instead of Service A calling Service B over HTTP and waiting for a response, Service A drops a message and moves on; Service B picks it up whenever it's ready.

Direct call (sync):
Service A ──── HTTP request ────► Service B
              (A blocks and waits for a response)

Through a queue (async):
Service A ──► [ Queue ] ◄── Service B
        (A moves on immediately)   (B pulls when ready)

Both are communication between two services. The difference is that a queue removes the requirement that both sides be available and fast at the same moment. The trade-off: there's no built-in way to get a response back. If Service A needs an answer, you either poll for a result, use a separate "reply" queue, or reconsider whether this should be a queue at all.

Key mechanics: acknowledgement, redelivery on a missing ack, visibility timeout, dead letter queue for poison messages.

Use it for: decoupling two services, absorbing bursts, protecting a fragile downstream.

Don't use it for: notifying many systems about the same fact — you'd need N queues, and the producer would have to know all N.

Examples: AWS SQS, RabbitMQ (queue), Azure Service Bus Queue, ActiveMQ.

🔍 Job Queue vs. Message Queue — the difference that actually confuses people

On the surface these look identical: one producer, one consumer, message deleted after. The real difference is who owns the job's lifecycle.

Message queue — the broker is a dumb pipe. It holds the message until someone picks it up:

produced → waiting → delivered → acked → gone

If the consumer crashes mid-work, the message becomes visible again after the visibility timeout and gets redelivered — but that's a side effect of "no ack arrived," not a designed feature. The broker has no concept of retry count, backoff, scheduling, or concurrency limits. All of that is your application's problem.

Job queue — the broker owns the lifecycle. A job moves through real states:

waiting → active → (failed → retrying → waiting again → active)
                 → (stalled → requeued)
                 → completed
                 → failed (retries exhausted → dead)

And it's configurable: attempts: 5, backoff: { type: 'exponential', delay: 1000 }, repeat: { cron: '0 * * * *' }, delay: 30 * 60 * 1000, concurrency: 3, limiter: { max: 100, duration: 60000 } — the broker enforces all of it, and a failed job's failedReason is stored and queryable.

The one-liner: a message queue delivers a message once and forgets it; a job queue manages the entire life of the work item.

If you want retry counting in a plain queue like SQS, you build it yourself — read ApproximateReceiveCount and decide. In a job queue, that's the default behavior.

C. Publish / Subscribe — fan-out of notifications

Mental model: a radio broadcast. The producer talks to a topic, not to anyone in particular.

• Payload: event

• Delivery: fan-out, one copy per subscriber

• After read: destroyed per-subscriber; usually no history

• Position: broker-owned

• Routing: consumer subscribes to a topic or pattern

The producer becomes totally ignorant of its consumers — add a fifth subscriber without touching the producer. The classic failure mode is that it's fire-and-forget: a subscriber offline at publish time typically never sees the message, unless the system fans out into durable per-subscriber queues underneath (SNS → several SQS queues is the standard shape for this).

Use it for: notifying multiple independent systems about the same fact.

Don't use it for: anything that needs replay, or onboarding a consumer that needs the past.

Examples: AWS SNS, Google Cloud Pub/Sub, RabbitMQ exchanges, MQTT, Redis Pub/Sub.

D. Event Streaming — an append-only log of facts

Mental model: not a pipe — a ledger. Nothing gets consumed away; consumers read through it at their own pace.

• Payload: event / fact

• Delivery: fan-out across consumer groups, point-to-point within a group

• After read: retained for a configured period or size

• Position: consumer-owned offsets

• Routing: consumer subscribes to a topic; a key determines the partition

The log is partitioned, each partition strictly ordered, and a key (say, subscriberId) guarantees all events for that key land in the same partition — ordering per entity, without needing global ordering. Within a consumer group, partitions are distributed among instances, which is how you get competing-consumer behavior inside a fundamentally broadcast system.

Retention plus offsets unlock what nothing else here can do: replay after a bug, late consumers reading a year of history on day one, independent progress per consumer, and stream processing (joins, windowed aggregations, materialized views).

The cost: you now own offset management, rebalance behavior, and schema evolution over long-lived data.

Use it for: event pipelines, audit history as a first-class asset, analytics feeds, event sourcing, CDC.

Don't use it for: simple background jobs — there's no built-in retry, delay, or priority; you build retry topics and a DLQ topic yourself.

Examples: Apache Kafka, AWS Kinesis, Azure Event Hubs, Apache Pulsar, Redpanda, Redis Streams.

E. Event Bus / Event Router — routing as configuration

Mental model: a smart switchboard between many producers and many consumers.

• Payload: event

• Delivery: fan-out, decided per rule

• After read: typically not retained

• Position: broker-owned

• Routing: broker rules over message content

The distinguishing feature is content-based routing declared outside the application — routing lives as configuration on the bus, not as if statements inside a producer or a fan-out subscriber list it has to maintain.

Here's what that looks like end to end. Say an e-commerce system emits one event type, OrderCreated, and four different teams each care about it for different reasons:

                         ┌──► Inventory       (always, decrement stock)
OrderCreated event ──► Event Bus ├──► Notifications    (always, send confirmation)
                         ├──► Analytics       (always, log for dashboards)
                         └──► Fraud Detection (only if amount > $10,000)

With pub/sub, the producer publishes to a topic and every subscriber gets every event — Fraud Detection would receive all orders and have to filter out the small ones itself. With an event bus, you instead write a rule directly on the bus:

IF event.type = "OrderCreated" AND event.amount > 10000
THEN route to fraud-detection-queue

The bus evaluates that rule per event and only delivers to Fraud Detection when it matches. Inventory, Notifications, and Analytics get their own rules (usually "match everything") pointing at their own targets. If a fifth team joins next quarter and only cares about orders from one country, that's a new rule on the bus — no change to the order service, and no change to any existing consumer.

That's the core value: routing logic that used to live scattered across producers and subscribers becomes one place you can read, audit, and change without deploying code. Buses typically also offer light transformation (reshape the event before delivery), a schema registry, and pre-built connectors to many kinds of targets — queues, functions, HTTP endpoints, SaaS platforms — so "route this to Slack" or "route this to a Lambda" is a target you pick, not an integration you write from scratch.

The line between pub/sub and an event bus is genuinely blurry in practice — think of a bus as pub/sub plus a rules engine plus integration glue. Many real systems use both: a Kafka or SNS layer for the raw firehose of events, and an event bus on top for the handful of cross-team routing rules that change often.

Use it for: integrating many heterogeneous systems where routing changes more often than code — cross-team event distribution, SaaS-to-SaaS integration glue, "notify these different things under these different conditions" scenarios.

Don't use it for: high-throughput pipelines (per-event rule evaluation adds overhead pub/sub doesn't have), or anywhere you need replay and strict ordering — most buses don't retain history by default.

Examples: AWS EventBridge, Azure Event Grid, Google Eventarc.

One product, several models

Keep this in mind reading any vendor's docs, because it's the main source of confusion:

• RabbitMQ — queues are family B; exchanges (fanout/topic) give you family C; header/topic exchanges edge toward family E; the streams plugin approaches family D.

• Redis — Pub/Sub is a non-durable family C; Streams is family D; BullMQ on top is family A.

• SQS + SNS — deliberately split: SNS fans out, SQS gives durability. Combined, you get reliable pub/sub.

• Kafka — family D, but used as B (single consumer group) or C (many groups) constantly in practice.

• Pulsar — explicitly supports queue and stream subscription modes on the same topic.

Don't ask "what is it called?" Ask the five axis questions instead.

How to choose, in order

1. Command or fact? Command → A or B. Fact → C, D, or E.

2. One thing needs to happen, or many independent things? One → A/B. Many → C/D/E.

3. Will anyone ever need the history — to replay, backfill, audit, or onboard a new consumer? Yes → D. Usually the deciding question.

4. Do you need per-item retries, delays, scheduling, priorities? → A.

5. Does routing change more often than code, across heterogeneous systems? → E.

6. Do you need ordering? Only D gives you dependable per-key ordering at scale.

Misconceptions worth unlearning

1. "Kafka is a faster RabbitMQ." Different semantics entirely — Kafka retains, RabbitMQ deletes.

2. "Pub/sub is just a queue with multiple consumers." A queue with multiple consumers splits the load; pub/sub duplicates it.

3. "I'll use Kafka so I can retry jobs." Kafka has no per-message retry or delay — you build retry topics yourself.

4. "Queues guarantee ordering." Most don't by default, and ordering conflicts with parallel consumers everywhere.

5. "Exactly-once delivery is a feature I can just turn on." Treat it as at-least-once plus idempotency; anything stronger is scoped to a specific broker's internal boundaries.

6. "An event bus and a message broker are the same layer." A bus routes; a broker transports. Buses are usually built on top of brokers.

7. "The message queue is where I put things I don't want to lose." Queues drain and delete — if retention matters, you want a log or a database.