Handling failed messages

When a message handler throws an exception, the exception is logged and reported through the Laravel exception handler. What happens to the message next depends on how the consumer is configured.

Sponsorship

Support Laravel Kafka by sponsoring me!

Laravel Kafka is free and Open Source software, built to empower developers like you. Your support helps maintain and enhance the project. If you find it valuable, please consider sponsoring me on GitHub. Every contribution makes a difference and keeps the development going strong! Thank you!

Become a Sponsor Want to hide this message? Sponsor at any tier of $10/month or more!

Without a dead letter queue, the consumer stops when a message fails, and the offset of the failed message is not committed. The consumer is closed, leaving the consumer group, and consume() throws a Junges\Kafka\Exceptions\ConsumerException. The original exception is available through the getPrevious method. The next consumer of the partition, including the same consumer once it is restarted, starts from the failed message, so failed messages are not lost.

use Junges\Kafka\Exceptions\ConsumerException;

$consumer = \Junges\Kafka\Facades\Kafka::consumer(['orders'])
    ->withGroupId('orders-group')
    ->withHandler(new OrderHandler)
    ->build();

$consumer->consume(); // Throws a ConsumerException when a message fails

The consumer process is expected to exit, and to be restarted by a process monitor such as Supervisor. Keep in mind that:

  • Messages are delivered at least once. A message can be processed again after a restart, so handlers should be idempotent.
  • A message that always fails stops the consumer every time it is consumed, blocking its partition until the cause is fixed. Use a dead letter queue to move such messages out of the way.
  • Make sure your process monitor keeps restarting the consumer. Supervisor, for instance, considers a process that exits within startsecs seconds of starting as a failed start, and gives up after startretries failed starts.

This works both in auto commit mode and in manual commit mode. In auto commit mode, the consumer sets the enable.auto.offset.store option to false and stores the offset of each message only after it is processed. This keeps librdkafka from committing the offset of a failed message in the background, which it would otherwise do as soon as the message is fetched.

Failed messages can be retried before they are handled as failed. After that, they can also be sent to a dead letter queue, or skipped.

Failures caused by a temporary problem, such as a dependency that is briefly unavailable, can be retried with the retryFailedMessages method. It receives the number of retries and, optionally, the time to wait before each retry in milliseconds:

$consumer = \Junges\Kafka\Facades\Kafka::consumer(['orders'])
    ->withGroupId('orders-group')
    ->retryFailedMessages(3, backoffInMs: 1000)
    ->withDlq()
    ->withHandler(new OrderHandler)
    ->build();

When the handler throws an exception, it is called again with the same message, up to the given number of times. Middlewares run again on every attempt. The getAttempts method of the message returns how many times the handler was called with it, including the current call:

use Junges\Kafka\Contracts\ConsumerMessage;
use Junges\Kafka\Contracts\Consumer;

function (ConsumerMessage $message, Consumer $consumer) {
    if ($message->getAttempts() > 1) {
        logger()->info('Retrying message', ['offset' => $message->getOffset()]);
    }

    // ...
}

Once all retries are used, the message is handled as failed: the failure callback is called, and the message is sent to the dead letter queue, stops the consumer, or is skipped, depending on the configuration. Retries also end early when the consumer is asked to stop, for instance by a termination signal.

Some failures will never succeed no matter how many times the handler is called, such as a validation error or a database constraint violation. To retry only some of them, pass a closure to the when argument. It receives the exception thrown by the handler, and the message is retried only when the closure returns true:

use Illuminate\Http\Client\ConnectionException;

$consumer = \Junges\Kafka\Facades\Kafka::consumer(['orders'])
    ->withGroupId('orders-group')
    ->retryFailedMessages(3, backoffInMs: 1000, when: fn (Throwable $e) => $e instanceof ConnectionException)
    ->withHandler(new OrderHandler)
    ->build();

The closure is called after each failed attempt. When it returns false, the remaining retries are skipped and the message is handled as failed right away.

To wait longer before each retry, pass an array of backoffs instead. Each retry waits for the value at its position, and the last value is used for the remaining retries:

$consumer = \Junges\Kafka\Facades\Kafka::consumer(['orders'])
    ->retryFailedMessages(5, backoffInMs: [1000, 5000, 10000])
    ->withHandler(new OrderHandler)
    ->build();

The consumer waits during the backoff, so no other message is consumed while a message is being retried. Keep the total time spent retrying a message (the number of retries multiplied by the backoff, plus the time the handler takes) well below the max.poll.interval.ms consumer option, 5 minutes by default. A consumer that does not poll Kafka within that interval is removed from the consumer group. Longer outages are better handled by a dead letter queue or by letting the consumer stop.

When a dead letter queue is configured with withDlq, failed messages are published to the dead letter queue topic before their offsets are committed, and the consumer moves on to the next message instead of stopping. See configuring a dead letter queue for details.

If losing a failed message is acceptable, use the skipFailedMessages method. The consumer then commits the offset of the failed message and moves on to the next one, so the failed message is not consumed again:

$consumer = \Junges\Kafka\Facades\Kafka::consumer(['page-views'])
    ->withGroupId('analytics')
    ->skipFailedMessages()
    ->withHandler(new PageViewHandler)
    ->build();

A Junges\Kafka\Events\MessageSkipped event is dispatched for every skipped message, with the message and the exception that made it fail, so you can monitor them:

use Illuminate\Support\Facades\Event;
use Junges\Kafka\Events\MessageSkipped;

Event::listen(function (MessageSkipped $event) {
    logger()->warning('Kafka message skipped', [
        'topic' => $event->message->getTopicName(),
        'offset' => $event->message->getOffset(),
        'exception' => $event->throwable->getMessage(),
    ]);
});

When a dead letter queue is also configured, failed messages are sent to it instead of being skipped.

Skipping is useful when a failure is permanent, like a message that can not be deserialized, which would stop the consumer every time it is restarted. A failure that is temporary, like a database or a cache that is down, is better handled by stopping the consumer, because the message can succeed once the consumer is restarted. To skip only some failures, pass a closure to the skipFailedMessages method. It receives the exception that made the message fail, and the message is skipped only when the closure returns true:

use Illuminate\Validation\ValidationException;

$consumer = \Junges\Kafka\Facades\Kafka::consumer(['orders'])
    ->withGroupId('orders-group')
    ->skipFailedMessages(fn (Throwable $e) => $e instanceof ValidationException)
    ->withHandler(new OrderHandler)
    ->build();

When the closure returns false, the consumer stops, as it would without skipFailedMessages. The closure receives the exception thrown while processing the message, not an exception thrown by the failure callback, and it is not called when the message is sent to a dead letter queue.

To run some code when a message is handled as failed, once its retries are used, override the failed method of a consumer class :

use Illuminate\Support\Facades\Notification;
use Junges\Kafka\Contracts\ConsumerMessage;
use Throwable;

public function failed(ConsumerMessage $message, Throwable $exception): void
{
    Notification::route('slack', config('services.slack.alerts'))
        ->notify(new KafkaMessageFailed($message, $exception));
}

When building a consumer yourself, use the onMessageFailed method of the consumer builder:

$consumer = \Junges\Kafka\Facades\Kafka::consumer(['orders'])
    ->onMessageFailed(function (ConsumerMessage $message, Throwable $exception) {
        // ...
    })
    ->withHandler(new OrderHandler)
    ->build();

The callback runs before the message is sent to the dead letter queue, skipped, or stops the consumer, and it can't change what happens to it. An exception thrown by the callback is reported, and the message is handled as usual.

To be notified of the failed messages of every consumer, listen to the MessageFailed event , which is dispatched right before the failure callback is called. The RetryingMessage event is dispatched before each retry.