The problem

The order service of our online shop writes its events to an outbox table, in the order's transaction. A relay reads that table, publishes on the app.events exchange, waits for the broker's confirmation and marks the row published: this is the Outbox pattern.

What remains is the path between "the row is in the outbox" and "the message is in the right queue". Failures on it are silent: no exception, no failed message.

A release adds order.return.requested.v1, a return request: only shipping consumes it, to have the parcel picked up. The order.order.*.v1 wildcard of shipping.events does not cover it, and order is deployed before its binding. Each request is published, confirmed by RabbitMQ, then dropped: no queue holds it.

The relay marks those rows published. shipping never receives these requests, and the returns are never picked up. Other failures lurk on the same path:

  • two relays active during a deployment;
  • a relay stopped in the middle of a batch;
  • a binding removed from the intended topology, still active on the broker;
  • a message that, under an allowed key, claims to be something else;
  • a RabbitMQ user left on a default password.

The options

All of them assume publisher confirms. They differ on one point: what does the relay know about a message with no route?

Publisher confirms alone

  • What they solve: an ack means the broker has taken charge of the message. A persistent message is then written to disk (durable queue) or accepted by a majority of replicas (quorum queue).
  • What they cost: RabbitMQ also confirms a message that no queue receives, as soon as it finds there is no route. For the relay, "taken charge of" and "dropped" give the same ack.

An alternate exchange

  • What it solves: messages with no route go to a fallback exchange, whose queue keeps them. Nothing is lost on the broker side.
  • What it costs: the relay still marks the row published. Someone has to watch the fallback queue, then send its messages again once the binding exists.

The mandatory flag

  • What it solves: RabbitMQ returns a mandatory message with no route to its publisher, with a basic.return (code 312, NO_ROUTE) sent before the ack. The relay knows, row by row, what arrived nowhere.
  • What it costs: the client must handle that return, which Messenger's AMQP transport does not do. And mandatory means "at least one queue": one missing subscriber among several goes unnoticed.

The two cannot be combined: a message dropped into the alternate exchange counts as routed, and mandatory no longer returns it.

Our recommendation

The relay publishes with mandatory and publisher confirms. It marks a row published only after an ack with no basic.return, and the topology is deployed before the producers.

Published means routed: an ack, and no basic.return before it Outbox table PostgreSQL Relay order service RabbitMQ app.events reads a batch FOR UPDATE SKIP LOCKED basic.publish , mandatory order.return.requested.v1 A binding matches: at least one queue accepts it basic.ack published_at = now() No binding: confirmed, then dropped by the broker basic.return 312 NO_ROUTE basic.ack attempts + 1, next_attempt_at the row stays unpublished batch COMMIT Without mandatory , no basic.return : the ack alone would mark a lost row as published.

Three answers, three moves

  • An ack alone: at least one queue has accepted the message. The relay sets published_at.
  • A basic.return, then the ack: no queue. The row stays unpublished, with a new attempt later. The relay moves on to the next row: the fault lies with one routing key, not with the broker.
  • A nack, a timeout or a lost connection: the broker is at fault. The relay stops the batch and opens a new channel on the next one.

A nack does not always prevent delivery. A full queue set to reject-publish refuses the message, but the other queues keep their copy. The new attempt creates duplicates, which the idempotent consumer discards.

What the PHP clients do with the return

Messenger's AMQP transport accepts the flag (new AmqpStamp($routingKey, \AMQP_MANDATORY)), but sets no return callback, in Symfony 7.4 as in 8.1. Without a callback, ext-amqp raises an E_NOTICE and stops waiting.

In production, Symfony logs that notice without throwing (php_errors.throw defaults to %kernel.debug%). send() returns, and the relay marks the row published. php-amqplib, for its part, ignores the return without set_return_listener().

So our relay talks to ext-amqp directly, with setReturnCallback(). The consumers stay on Messenger.

The topology before the producers

The deployment order becomes a rule: the topology (exchanges, queues, bindings, users, permissions), then the consumers, then the producers. Whether it comes from messenger:setup-transports or from definitions imported by the broker, it precedes the first message.

With mandatory, a wrong order for a new event costs a delay, not a loss: the rows wait for the binding. For a new subscriber to an event already consumed, mandatory sees nothing. The order remains the only protection, and a replay from the outbox repairs an omission.

An event nobody listens to also stays unpublished. That is intended: in a monorepo, the contract, the subscription and the handler arrive together, and an event with no subscriber points to an omission.

The trade-offs we accept

One more AMQP client. The relay leaves Messenger: connection, channel and callbacks are ours to handle, and to test against a real RabbitMQ.

Ordering across keys. A returned row is skipped, the following ones go out. While order.return.requested.v1 waits for its binding, the events published after it overtake it: a state copy checks a version before writing.

Attempts with no effect. A row with no subscriber is published again after 30 seconds, then at an interval that doubles, up to 15 minutes. An alert on last_error = 'NO_ROUTE' names each key concerned.

No alternate exchange on app.events. Our topology suggested one, but it would silence mandatory. A message published outside the relay therefore has no safety net.

The signal that should make us revisit this choice: frequent returns, or the need to know that every expected subscriber received the event. The bindings then have to be compared with a catalogue of subscriptions.

Implementation

The examples target PHP 8.4, Symfony 7.4 or 8.1, ext-amqp 2.2, RabbitMQ 4.3 and PostgreSQL 18. The outbox table and the Scheduler task that runs the relay are those of the article on the Outbox pattern.

1. Track failures in the outbox

// src/Entity/OutboxMessage.php (order service, excerpt)
#[ORM\Column(length: 255, nullable: true)]
private ?string $lastError = null;

// No new attempt before this date; NULL for a fresh row.
#[ORM\Column(type: Types::DATETIMETZ_IMMUTABLE, nullable: true)]
private ?\DateTimeImmutable $nextAttemptAt = null;

public function recordFailure(string $error, ?\DateTimeImmutable $nextAttemptAt = null): void
{
    ++$this->attempts;
    $this->lastError = mb_substr($error, 0, 255);
    $this->nextAttemptAt = $nextAttemptAt;
}

2. Publish as mandatory with ext-amqp

// src/Outbox/ConfirmedPublisher.php (order service)
namespace App\Outbox;

use App\Entity\OutboxMessage;
use AppContracts\Routing;

final class ConfirmedPublisher
{
    private ?\AMQPExchange $exchange = null;
    private int $sequence = 0;
    /** @var array<string, string> message_id => reason for the return */
    private array $returned = [];

    public function __construct(
        // host, vhost, login, password: read from the environment (RABBITMQ_*).
        private readonly array $credentials,
        private readonly float $confirmTimeout = 5.0,
    ) {
    }

    /** true: routed to at least one queue. Throws \AMQPException on a nack or a timeout. */
    public function publish(OutboxMessage $message): bool
    {
        $this->exchange ??= $this->open();
        $channel = $this->exchange->getChannel();
        $id = $message->getEnvelope()['event_id'];
        $sequence = ++$this->sequence;

        $channel->setConfirmCallback(
            // false ends the wait: our ack has arrived, alone or grouped.
            static fn (int $tag, bool $multiple): bool => $tag < $sequence,
            static fn (int $tag, bool $multiple, bool $requeue): bool => $tag < $sequence
                ?: throw new \AMQPException('Nack from the broker.'),
        );

        $this->exchange->publish(
            json_encode($message->getEnvelope(), \JSON_THROW_ON_ERROR),
            $message->getRoutingKey(),
            \AMQP_MANDATORY,
            [
                'message_id' => $id,
                'content_type' => 'application/json',
                'delivery_mode' => \AMQP_DELIVERY_MODE_PERSISTENT,
            ],
        );
        $channel->waitForConfirm($this->confirmTimeout);

        if (isset($this->returned[$id])) {
            unset($this->returned[$id]);

            return false;
        }

        return true;
    }

    public function close(): void
    {
        $this->exchange?->getConnection()->disconnect();
        $this->exchange = null;
    }

    private function open(): \AMQPExchange
    {
        $connection = new \AMQPConnection($this->credentials);
        $connection->connect();

        $channel = new \AMQPChannel($connection);
        $channel->confirmSelect();
        $this->sequence = 0;

        $channel->setReturnCallback(
            function (int $code, string $text, string $exchange, string $routingKey, \AMQPBasicProperties $properties, string $body): bool {
                // 312 NO_ROUTE: no binding matches the routing key.
                $this->returned[(string) $properties->getMessageId()] = $code.' '.$text;

                return true; // carry on: this message's ack follows the return.
            },
        );

        $exchange = new \AMQPExchange($channel);
        // No declareExchange(): the topology belongs to the broker.
        $exchange->setName(Routing::EXCHANGE);

        return $exchange;
    }
}

3. The relay: publish, then mark or postpone

// src/Outbox/OutboxRelay.php (order service), excerpt of relay()
// The batch query gains a condition, before ORDER BY ... FOR UPDATE SKIP LOCKED:
//   AND (o.next_attempt_at IS NULL OR o.next_attempt_at <= now())
$count = 0;
foreach ($messages as $message) {
    try {
        $routed = $this->publisher->publish($message);
    } catch (\AMQPException $e) {
        // The broker is at fault: stop the batch, new channel on the next one.
        $this->publisher->close();
        $message->recordFailure($e->getMessage());
        break;
    }

    ++$count;
    if (!$routed) {
        // Confirmed but dropped: the row stays unpublished, with a growing delay.
        $delay = min(30 * 2 ** $message->getAttempts(), 900);
        $message->recordFailure('NO_ROUTE', new \DateTimeImmutable("+{$delay} seconds"));
        continue;
    }

    $message->setPublishedAt(new \DateTimeImmutable());
}

// $count counts published or returned rows; wrapInTransaction() commits the batch.
return $count;

During a deployment, SKIP LOCKED lets two relays overlap: each one skips the rows the other has locked.

A stop between the ack and the commit rolls the marking back: the next batch publishes the row again, a duplicate and never a loss.

4. Stop the relay cleanly

The relay runs inside messenger:consume scheduler_default. With the pcntl extension, the worker handles SIGTERM after the current task; without it, SIGTERM kills PHP in the middle of the batch.

As PID 1 in a container, a process ignores the signals it does not handle itself. Docker then sends SIGKILL after 10 seconds by default. init: true puts an init process at PID 1, which forwards the signal.

# services/order/compose.yml (relay, excerpt)
services:
  outbox-relay:
    # Image of the order service, built with pcntl.
    command: ['php', 'bin/console', 'messenger:consume', 'scheduler_default', '--time-limit=3600']
    init: true
    # Beyond that, SIGKILL: the current batch is rolled back, then published again.
    stop_grace_period: 30s
    environment:
      RABBITMQ_HOST: rabbitmq
      RABBITMQ_VHOST: app
      RABBITMQ_USER: order
      # Without a value, Compose refuses to start: no default password.
      RABBITMQ_PASSWORD: ${RABBITMQ_ORDER_PASSWORD:?missing}

5. One user per service, with no default password

Each service has its own RabbitMQ user, with rights limited to its own queues. A topic permission also confines its publications to its prefix:

rabbitmqctl set_topic_permissions -p app order app.events '^order\.' '.*'

The trap is provisioning that falls back to a default password when a variable is missing. And a compose file that lists its variables under environment: passes no other one to the container. We prefer a failure, with ${VAR:?error}.

6. Check the actual topology

Neither messenger:setup-transports nor a definitions import removes a binding. A binding removed from the reference therefore keeps delivering, for instance a v1 nobody reads any more. We compare on every broker deployment:

# tools/check-bindings.sh
rabbitmqctl export_definitions /tmp/live.json
query='.bindings[] | select(.source == "app.events") | "\(.destination) \(.routing_key)"'
diff <(jq -r "$query" deploy/rabbitmq/definitions.json | sort) <(jq -r "$query" /tmp/live.json | sort)

A line found only on the broker is deleted by hand:

rabbitmqadmin --vhost app bindings delete --source app.events \
  --destination-type queue --destination shipping.events --routing-key 'order.order.*.v1'

7. A guard on the consumer side

The topic permission checks the routing key, not the message. order can publish under order.order.placed.v1 a body whose event_name names another message, which stock would handle as its own. The guard therefore refuses, without retry, a message from another service whose event_name differs from the key.

// src/Messenger/RoutingKeyGuardMiddleware.php (stock service)
namespace App\Messenger;

use Symfony\Component\Messenger\Bridge\Amqp\Transport\AmqpReceivedStamp;
use Symfony\Component\Messenger\Envelope;
use Symfony\Component\Messenger\Exception\UnrecoverableMessageHandlingException;
use Symfony\Component\Messenger\Middleware\MiddlewareInterface;
use Symfony\Component\Messenger\Middleware\StackInterface;

final class RoutingKeyGuardMiddleware implements MiddlewareInterface
{
    // The service's own traffic, retries included (key "stock.events").
    private const OWN_PREFIX = 'stock.';

    public function handle(Envelope $envelope, StackInterface $stack): Envelope
    {
        $routingKey = $envelope->last(AmqpReceivedStamp::class)?->getAmqpEnvelope()->getRoutingKey();

        if (null !== $routingKey && !str_starts_with($routingKey, self::OWN_PREFIX)) {
            $eventName = $envelope->last(EventMetadataStamp::class)?->eventName;

            if ($eventName !== $routingKey) {
                throw new UnrecoverableMessageHandlingException(sprintf(
                    'Message refused: routing key "%s", event "%s".',
                    $routingKey,
                    $eventName ?? 'missing',
                ));
            }
        }

        return $stack->next()->handle($envelope, $stack);
    }
}

It reads EventMetadataStamp, added by the serializer of our contracts, and goes before doctrine_transaction. The refused message goes to the failure transport. Write rights matter just as much: whoever writes on the default exchange (amq.default) reaches any queue by its name, with no topic check.

Messenger's delay queues send retries back through it, which requires that right. The delay[arguments] option (x-dead-letter-exchange: app.events) sends them back through app.events instead, with a binding of each queue on its own name.

8. Count failures in the right place

The DLQ of a quorum queue receives one copy per Messenger retry, not only the final failure (our note on dead letter queues). Definitive failures are counted with messenger:failed:show.

Checklist

  • Publisher confirms and the mandatory flag on every publication of the relay.
  • A row marked published only after an ack with no basic.return.
  • A return callback on the channel, tested against a real RabbitMQ.
  • Returned rows postponed without blocking the others, and an alert on NO_ROUTE.
  • The topology deployed before the consumers, and those before the producers.
  • FOR UPDATE SKIP LOCKED in the relay's query.
  • pcntl in the image, init: true and a suitable stop grace period.
  • One user per service, a topic permission on its prefix, no default password.
  • Actual bindings compared with the intended ones on every broker deployment.
  • A guard that refuses without retry an event_name that differs from the routing key.

Sources