The problem
The order service of our online shop writes its
events to an outbox table, in the order's transaction. A
relay reads that table, publishes on the
app.events exchange, waits for the broker's
confirmation and marks the row published: this is
the Outbox pattern.
What remains is the path between "the row is in the outbox" and "the message is in the right queue". Failures on it are silent: no exception, no failed message.
A release adds order.return.requested.v1, a
return request: only shipping consumes it, to
have the parcel picked up. The
order.order.*.v1 wildcard of
shipping.events does not cover it, and
order is deployed before its binding. Each
request is published, confirmed by RabbitMQ, then dropped:
no queue holds it.
The relay marks those rows published.
shipping never receives these requests, and the
returns are never picked up. Other failures lurk on the same
path:
- two relays active during a deployment;
- a relay stopped in the middle of a batch;
- a binding removed from the intended topology, still active on the broker;
- a message that, under an allowed key, claims to be something else;
- a RabbitMQ user left on a default password.
The options
All of them assume publisher confirms. They differ on one point: what does the relay know about a message with no route?
Publisher confirms alone
- What they solve: an ack means the broker has taken charge of the message. A persistent message is then written to disk (durable queue) or accepted by a majority of replicas (quorum queue).
- What they cost: RabbitMQ also confirms a message that no queue receives, as soon as it finds there is no route. For the relay, "taken charge of" and "dropped" give the same ack.
An alternate exchange
- What it solves: messages with no route go to a fallback exchange, whose queue keeps them. Nothing is lost on the broker side.
- What it costs: the relay still marks the row published. Someone has to watch the fallback queue, then send its messages again once the binding exists.
The mandatory flag
-
What it solves: RabbitMQ returns a
mandatorymessage with no route to its publisher, with abasic.return(code 312,NO_ROUTE) sent before the ack. The relay knows, row by row, what arrived nowhere. -
What it costs: the client must handle
that return, which Messenger's AMQP transport does not do.
And
mandatorymeans "at least one queue": one missing subscriber among several goes unnoticed.
The two cannot be combined: a message dropped into the
alternate exchange counts as routed, and
mandatory no longer returns it.
Our recommendation
The relay publishes with mandatory and
publisher confirms. It marks a row published only after an
ack with no basic.return, and the topology is
deployed before the producers.
Three answers, three moves
-
An ack alone: at least one queue has
accepted the message. The relay sets
published_at. -
A
basic.return, then the ack: no queue. The row stays unpublished, with a new attempt later. The relay moves on to the next row: the fault lies with one routing key, not with the broker. - A nack, a timeout or a lost connection: the broker is at fault. The relay stops the batch and opens a new channel on the next one.
A nack does not always prevent delivery. A full queue set to
reject-publish refuses the message, but the
other queues keep their copy. The new attempt creates
duplicates, which the
idempotent consumer
discards.
What the PHP clients do with the return
Messenger's AMQP transport accepts the flag (new AmqpStamp($routingKey, \AMQP_MANDATORY)), but sets no return callback, in Symfony 7.4 as in 8.1.
Without a callback, ext-amqp raises an
E_NOTICE and stops waiting.
In production, Symfony logs that notice without throwing
(php_errors.throw defaults to
%kernel.debug%). send() returns,
and the relay marks the row published. php-amqplib, for its
part, ignores the return without
set_return_listener().
So our relay talks to ext-amqp directly, with
setReturnCallback(). The consumers stay on
Messenger.
The topology before the producers
The deployment order becomes a rule: the topology
(exchanges, queues, bindings, users, permissions), then the
consumers, then the producers. Whether it comes from
messenger:setup-transports or from definitions
imported by the broker, it precedes the first message.
With mandatory, a wrong order for a new event
costs a delay, not a loss: the rows wait for the binding.
For a new subscriber to an event already consumed,
mandatory sees nothing. The order remains the
only protection, and a replay from the outbox repairs an
omission.
An event nobody listens to also stays unpublished. That is intended: in a monorepo, the contract, the subscription and the handler arrive together, and an event with no subscriber points to an omission.
The trade-offs we accept
One more AMQP client. The relay leaves Messenger: connection, channel and callbacks are ours to handle, and to test against a real RabbitMQ.
Ordering across keys. A returned row is
skipped, the following ones go out. While
order.return.requested.v1 waits for its
binding, the events published after it overtake it: a state
copy checks a version before writing.
Attempts with no effect. A row with no
subscriber is published again after 30 seconds, then at an
interval that doubles, up to 15 minutes. An alert on
last_error = 'NO_ROUTE' names each key
concerned.
No alternate exchange on app.events.
Our topology
suggested one, but it would silence mandatory.
A message published outside the relay therefore has no
safety net.
The signal that should make us revisit this choice: frequent returns, or the need to know that every expected subscriber received the event. The bindings then have to be compared with a catalogue of subscriptions.
Implementation
The examples target PHP 8.4, Symfony 7.4 or 8.1, ext-amqp
2.2, RabbitMQ 4.3 and PostgreSQL 18. The
outbox table and the Scheduler task that runs
the relay are those of
the article on the Outbox pattern.
1. Track failures in the outbox
// src/Entity/OutboxMessage.php (order service, excerpt)
#[ORM\Column(length: 255, nullable: true)]
private ?string $lastError = null;
// No new attempt before this date; NULL for a fresh row.
#[ORM\Column(type: Types::DATETIMETZ_IMMUTABLE, nullable: true)]
private ?\DateTimeImmutable $nextAttemptAt = null;
public function recordFailure(string $error, ?\DateTimeImmutable $nextAttemptAt = null): void
{
++$this->attempts;
$this->lastError = mb_substr($error, 0, 255);
$this->nextAttemptAt = $nextAttemptAt;
}
2. Publish as mandatory with ext-amqp
// src/Outbox/ConfirmedPublisher.php (order service)
namespace App\Outbox;
use App\Entity\OutboxMessage;
use AppContracts\Routing;
final class ConfirmedPublisher
{
private ?\AMQPExchange $exchange = null;
private int $sequence = 0;
/** @var array<string, string> message_id => reason for the return */
private array $returned = [];
public function __construct(
// host, vhost, login, password: read from the environment (RABBITMQ_*).
private readonly array $credentials,
private readonly float $confirmTimeout = 5.0,
) {
}
/** true: routed to at least one queue. Throws \AMQPException on a nack or a timeout. */
public function publish(OutboxMessage $message): bool
{
$this->exchange ??= $this->open();
$channel = $this->exchange->getChannel();
$id = $message->getEnvelope()['event_id'];
$sequence = ++$this->sequence;
$channel->setConfirmCallback(
// false ends the wait: our ack has arrived, alone or grouped.
static fn (int $tag, bool $multiple): bool => $tag < $sequence,
static fn (int $tag, bool $multiple, bool $requeue): bool => $tag < $sequence
?: throw new \AMQPException('Nack from the broker.'),
);
$this->exchange->publish(
json_encode($message->getEnvelope(), \JSON_THROW_ON_ERROR),
$message->getRoutingKey(),
\AMQP_MANDATORY,
[
'message_id' => $id,
'content_type' => 'application/json',
'delivery_mode' => \AMQP_DELIVERY_MODE_PERSISTENT,
],
);
$channel->waitForConfirm($this->confirmTimeout);
if (isset($this->returned[$id])) {
unset($this->returned[$id]);
return false;
}
return true;
}
public function close(): void
{
$this->exchange?->getConnection()->disconnect();
$this->exchange = null;
}
private function open(): \AMQPExchange
{
$connection = new \AMQPConnection($this->credentials);
$connection->connect();
$channel = new \AMQPChannel($connection);
$channel->confirmSelect();
$this->sequence = 0;
$channel->setReturnCallback(
function (int $code, string $text, string $exchange, string $routingKey, \AMQPBasicProperties $properties, string $body): bool {
// 312 NO_ROUTE: no binding matches the routing key.
$this->returned[(string) $properties->getMessageId()] = $code.' '.$text;
return true; // carry on: this message's ack follows the return.
},
);
$exchange = new \AMQPExchange($channel);
// No declareExchange(): the topology belongs to the broker.
$exchange->setName(Routing::EXCHANGE);
return $exchange;
}
}
3. The relay: publish, then mark or postpone
// src/Outbox/OutboxRelay.php (order service), excerpt of relay()
// The batch query gains a condition, before ORDER BY ... FOR UPDATE SKIP LOCKED:
// AND (o.next_attempt_at IS NULL OR o.next_attempt_at <= now())
$count = 0;
foreach ($messages as $message) {
try {
$routed = $this->publisher->publish($message);
} catch (\AMQPException $e) {
// The broker is at fault: stop the batch, new channel on the next one.
$this->publisher->close();
$message->recordFailure($e->getMessage());
break;
}
++$count;
if (!$routed) {
// Confirmed but dropped: the row stays unpublished, with a growing delay.
$delay = min(30 * 2 ** $message->getAttempts(), 900);
$message->recordFailure('NO_ROUTE', new \DateTimeImmutable("+{$delay} seconds"));
continue;
}
$message->setPublishedAt(new \DateTimeImmutable());
}
// $count counts published or returned rows; wrapInTransaction() commits the batch.
return $count;
During a deployment, SKIP LOCKED lets two
relays overlap: each one skips the rows the other has
locked.
A stop between the ack and the commit rolls the marking back: the next batch publishes the row again, a duplicate and never a loss.
4. Stop the relay cleanly
The relay runs inside
messenger:consume scheduler_default. With the
pcntl extension, the worker handles SIGTERM after the
current task; without it, SIGTERM kills PHP in the middle of
the batch.
As PID 1 in a container, a process ignores the signals it
does not handle itself. Docker then sends SIGKILL after 10
seconds by default. init: true puts an init
process at PID 1, which forwards the signal.
# services/order/compose.yml (relay, excerpt)
services:
outbox-relay:
# Image of the order service, built with pcntl.
command: ['php', 'bin/console', 'messenger:consume', 'scheduler_default', '--time-limit=3600']
init: true
# Beyond that, SIGKILL: the current batch is rolled back, then published again.
stop_grace_period: 30s
environment:
RABBITMQ_HOST: rabbitmq
RABBITMQ_VHOST: app
RABBITMQ_USER: order
# Without a value, Compose refuses to start: no default password.
RABBITMQ_PASSWORD: ${RABBITMQ_ORDER_PASSWORD:?missing}
5. One user per service, with no default password
Each service has its own RabbitMQ user, with rights limited to its own queues. A topic permission also confines its publications to its prefix:
rabbitmqctl set_topic_permissions -p app order app.events '^order\.' '.*'
The trap is provisioning that falls back to a default
password when a variable is missing. And a compose file that
lists its variables under environment: passes
no other one to the container. We prefer a failure, with
${VAR:?error}.
6. Check the actual topology
Neither messenger:setup-transports nor a
definitions import removes a binding. A binding removed from
the reference therefore keeps delivering, for instance a
v1 nobody reads any more. We compare on every
broker deployment:
# tools/check-bindings.sh
rabbitmqctl export_definitions /tmp/live.json
query='.bindings[] | select(.source == "app.events") | "\(.destination) \(.routing_key)"'
diff <(jq -r "$query" deploy/rabbitmq/definitions.json | sort) <(jq -r "$query" /tmp/live.json | sort)
A line found only on the broker is deleted by hand:
rabbitmqadmin --vhost app bindings delete --source app.events \
--destination-type queue --destination shipping.events --routing-key 'order.order.*.v1'
7. A guard on the consumer side
The topic permission checks the routing key, not the
message. order can publish under
order.order.placed.v1 a body whose
event_name names another message, which
stock would handle as its own. The guard
therefore refuses, without retry, a message from another
service whose event_name differs from the key.
// src/Messenger/RoutingKeyGuardMiddleware.php (stock service)
namespace App\Messenger;
use Symfony\Component\Messenger\Bridge\Amqp\Transport\AmqpReceivedStamp;
use Symfony\Component\Messenger\Envelope;
use Symfony\Component\Messenger\Exception\UnrecoverableMessageHandlingException;
use Symfony\Component\Messenger\Middleware\MiddlewareInterface;
use Symfony\Component\Messenger\Middleware\StackInterface;
final class RoutingKeyGuardMiddleware implements MiddlewareInterface
{
// The service's own traffic, retries included (key "stock.events").
private const OWN_PREFIX = 'stock.';
public function handle(Envelope $envelope, StackInterface $stack): Envelope
{
$routingKey = $envelope->last(AmqpReceivedStamp::class)?->getAmqpEnvelope()->getRoutingKey();
if (null !== $routingKey && !str_starts_with($routingKey, self::OWN_PREFIX)) {
$eventName = $envelope->last(EventMetadataStamp::class)?->eventName;
if ($eventName !== $routingKey) {
throw new UnrecoverableMessageHandlingException(sprintf(
'Message refused: routing key "%s", event "%s".',
$routingKey,
$eventName ?? 'missing',
));
}
}
return $stack->next()->handle($envelope, $stack);
}
}
It reads EventMetadataStamp, added by
the serializer of our contracts, and goes before doctrine_transaction. The
refused message goes to the failure transport. Write rights
matter just as much: whoever writes on the default exchange
(amq.default) reaches any queue by its name,
with no topic check.
Messenger's delay queues send retries back through it, which
requires that right. The
delay[arguments] option (x-dead-letter-exchange: app.events) sends them back through app.events instead,
with a binding of each queue on its own name.
8. Count failures in the right place
The DLQ of a quorum queue receives one copy per Messenger
retry, not only the final failure (our note on dead letter queues). Definitive failures are counted with
messenger:failed:show.
Checklist
-
Publisher confirms and the
mandatoryflag on every publication of the relay. -
A row marked published only after an ack with no
basic.return. - A return callback on the channel, tested against a real RabbitMQ.
-
Returned rows postponed without blocking the others, and
an alert on
NO_ROUTE. - The topology deployed before the consumers, and those before the producers.
-
FOR UPDATE SKIP LOCKEDin the relay's query. -
pcntl in the image,
init: trueand a suitable stop grace period. - One user per service, a topic permission on its prefix, no default password.
- Actual bindings compared with the intended ones on every broker deployment.
-
A guard that refuses without retry an
event_namethat differs from the routing key.
Sources
-
RabbitMQ,
Publisher Confirms: messages with no route,
basic.returnbeforebasic.ack, persistence and quorum queues. -
RabbitMQ,
unroutable messages,
Alternate Exchanges
and
reject-publish. -
AMQP 0-9-1,
protocol reference:
mandatory,basic.return, code 312no-route. -
RabbitMQ,
Access Control
(topic permissions,
amq.default), rabbitmqctl and Dead Letter Exchanges. - RabbitMQ, Definitions, rabbitmqadmin and the rabbitmq-users list, additive import (RabbitMQ team, 27 October 2023).
- ext-amqp 2.2.0 (2 January 2026), AMQPChannel and basic.return without a callback; php-amqplib, AMQPChannel.
-
Symfony,
Messenger
(AMQP transport, pcntl),
AMQP transport code
and
php_errors.throw. -
PostgreSQL,
locking clause:
SKIP LOCKED. -
Docker,
docker run
(PID 1), Compose
services
(
init,stop_grace_period) and interpolation. - Lychen, pull requests 253 and 236: an open-source implementation of the relay with confirms, the guard and the per-service passwords, in a moon monorepo.