Configuring your kafka producer | Laravel Kafka [ K Laravel Kafka ](/) [Docs](/docs) [Blog](https://laravelkafka.com/blog) [GitHub](https://github.com/mateusjunges/laravel-kafka) Search ⌘K [ Login with GitHub Login ](https://laravelkafka.com/oauth/github/redirect) Docs for version selected v3.0 v2.13 v2.12 v2.11 v2.10 v2.9 v2.8 v1.13 - - [ Introduction ](/docs/v3.0/introduction) - [ Requirements ](/docs/v3.0/requirements) - [ Installation and Setup ](/docs/v3.0/installation-and-setup) - [ Questions and issues ](/docs/v3.0/questions-and-issues) - [ Changelog ](/docs/v3.0/changelog) - [ Upgrade guide ](/docs/v3.0/upgrade-guide) - [ Example docker-compose file ](/docs/v3.0/example-docker-compose) - Producing messages ------------------ - [ Producing messages ](/docs/v3.0/producing-messages/producing-messages) - [ Configuring your kafka producer ](/docs/v3.0/producing-messages/configuring-producers) - [ Configuring message payload ](/docs/v3.0/producing-messages/configuring-message-payload) - [ Custom serializers ](/docs/v3.0/producing-messages/custom-serializers) - [ Publishing to kafka ](/docs/v3.0/producing-messages/publishing-to-kafka) - Consuming messages ------------------ - [ Creating a kafka consumer ](/docs/v3.0/consuming-messages/creating-consumer) - [ Subscribing to kafka topics ](/docs/v3.0/consuming-messages/subscribing-to-kafka-topics) - [ Using regex to subscribe to kafka topics ](/docs/v3.0/consuming-messages/using-regex-to-subscribe-to-kafka-topics) - [ Assigning consumers to a topic partition ](/docs/v3.0/consuming-messages/assigning-partitions) - [ Consuming messages from specific offsets ](/docs/v3.0/consuming-messages/consuming-from-specific-offsets) - [ Consumer groups ](/docs/v3.0/consuming-messages/consumer-groups) - [ Partition Discovery and Dynamic Assignment ](/docs/v3.0/consuming-messages/partition-discovery) - [ Message handlers ](/docs/v3.0/consuming-messages/message-handlers) - [ Configuring consumer options ](/docs/v3.0/consuming-messages/configuring-consumer-options) - [ Handling failed messages ](/docs/v3.0/consuming-messages/handling-failed-messages) - [ Custom deserializers ](/docs/v3.0/consuming-messages/custom-deserializers) - [ Consuming messages ](/docs/v3.0/consuming-messages/consuming-messages) - [ Pausing partitions ](/docs/v3.0/consuming-messages/pausing-partitions) - [ Consumer classes ](/docs/v3.0/consuming-messages/class-structure) - [ Using consumers with Laravel Telescope ](/docs/v3.0/consuming-messages/laravel-telescope) - Advanced usage -------------- - [ Connections ](/docs/v3.0/advanced-usage/connections) - [ Replacing the default serializer/deserializer ](/docs/v3.0/advanced-usage/replacing-default-serializer) - [ Graceful shutdown ](/docs/v3.0/advanced-usage/graceful-shutdown) - [ Running consumers in production ](/docs/v3.0/advanced-usage/running-consumers-in-production) - [ SASL Authentication ](/docs/v3.0/advanced-usage/sasl-authentication) - [ Custom Committers ](/docs/v3.0/advanced-usage/custom-committers) - [ Manual Commit ](/docs/v3.0/advanced-usage/manual-commit) - [ Middlewares ](/docs/v3.0/advanced-usage/middlewares) - [ Stop consumer when there are no messages left ](/docs/v3.0/advanced-usage/stop-consumer-after-last-message) - [ Stop consumer on demand ](/docs/v3.0/advanced-usage/stopping-a-consumer) - [ Writing custom loggers ](/docs/v3.0/advanced-usage/custom-loggers) - [ Before and after callbacks ](/docs/v3.0/advanced-usage/before-callbacks) - [ Setting global configurations ](/docs/v3.0/advanced-usage/setting-global-configuration) - [ Sending multiple messages with the same producer ](/docs/v3.0/advanced-usage/sending-multiple-messages-with-the-same-producer) - [ Events ](/docs/v3.0/advanced-usage/events) - [ Consumer lag ](/docs/v3.0/advanced-usage/consumer-lag) - Testing ------- - [ Kafka fake ](/docs/v3.0/testing/fake) - [ Assert Published ](/docs/v3.0/testing/assert-published) - [ Assert published On ](/docs/v3.0/testing/assert-published-on) - [ Assert not published ](/docs/v3.0/testing/assert-not-published) - [ Assert nothing published ](/docs/v3.0/testing/assert-nothing-published) - [ Assert published times ](/docs/v3.0/testing/assert-published-times) - [ Assert published on times ](/docs/v3.0/testing/assert-published-on-times) - [ Mocking your kafka consumer ](/docs/v3.0/testing/mocking-your-kafka-consumer) Configuring your kafka producer ================================= Each connection has a single producer, shared by every message published through that connection. The producer is configured in the connection configuration, in your `config/kafka.php` file. Sponsorship Support Laravel Kafka by sponsoring me! Laravel Kafka is free and Open Source software, built to empower developers like you. Your support helps maintain and enhance the project. If you find it valuable, please consider sponsoring me on GitHub. Every contribution makes a difference and keeps the development going strong! Thank you! [ Become a Sponsor ](https://github.com/sponsors/mateusjunges) Want to hide this message? Sponsor at any tier of $10/month or more! ### [](#content-defining-configuration-options "Permalink")Defining configuration options The `producer.options` key of a connection accepts any librdkafka option. You can check all available options [ here ](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md). Options defined in the `options` key of the connection are applied to both producers and consumers. ``` 'connections' => [ 'default' => [ 'brokers' => env('KAFKA_BROKERS', 'localhost:9092'), 'options' => [ 'client.id' => 'my-application', ], 'producer' => [ 'options' => [ 'compression.codec' => 'snappy', 'enable.idempotence' => true, 'linger.ms' => 5, ], ], ], ], ``` To enable debug mode while developing your application, set the `debug` option: ``` 'options' => [ 'debug' => 'all', ], ``` ### [](#content-flushing "Permalink")Flushing The `flush_timeout_ms`, `flush_retries` and `flush_retry_sleep_ms` keys of the `producer` configuration define how long to wait for queued messages to be delivered when flushing the producer, and how many times to retry before giving up. To find out about messages that could not be delivered, listen to the `MessageDeliveryFailed` [ event ](../advanced-usage/events). To be notified of the delivery of every message, register a delivery report callback on the connection with `onDeliveryReport()`. ### [](#content-configuration-callbacks "Permalink")Configuration callbacks librdkafka configuration callbacks, such as the error, log and OAUTHBEARER token refresh callbacks, are registered on the connection. Because the producer is created the first time a message is published, register them before publishing any message, for example in the `boot` method of a service provider: ``` use Junges\Kafka\Facades\Kafka; Kafka::connection() ->onError(function ($kafka, int $err, string $reason) { logger()->error($reason); }) ->onLog(function ($kafka, int $level, string $facility, string $message) { logger()->debug($message); }); ``` Delivery failures are also dispatched as `Junges\Kafka\Events\MessageDeliveryFailed` events, so you usually don't need a delivery report callback, registered with `onDeliveryReport()`, to find out about them. Callbacks registered on a connection are applied to its producer and to the consumers created using it. Registering a callback after the connection's producer was created throws a `LogicException`, because librdkafka can't change the configuration of an existing producer. ### [](#content-transactions "Permalink")Transactions Transactions deliver a group of messages all together, or not at all. To use them, set a `transactional.id` in the producer options of a dedicated connection. Every message published through that connection must then be published inside a transaction: ``` 'connections' => [ 'payments' => [ 'brokers' => env('KAFKA_BROKERS'), 'producer' => [ 'options' => [ 'transactional.id' => 'payments-'.gethostname().'-'.getmypid(), ], ], ], ], ``` Then, publish the messages inside the `transaction` method of the connection: ``` use Junges\Kafka\Connection; use Junges\Kafka\Facades\Kafka; Kafka::connection('payments')->transaction(function (Connection $connection) { $connection->publish('ledger')->withBody(['order_id' => 1, 'amount' => 100])->send(); $connection->publish('invoices')->withBody(['order_id' => 1])->send(); }); ``` The transaction is committed once the callback returns, and the value returned by the callback is returned by `transaction`. If the callback throws, the transaction is aborted, none of its messages are delivered, and the exception is rethrown. Kafka reports some transaction errors as temporary. When committing fails with a retriable error, the commit is retried, and when Kafka requires the transaction to be aborted, it is aborted and the callback runs again. Both happen up to 3 times, which you can change with the `attempts` argument: ``` Kafka::connection('payments')->transaction($callback, attempts: 5); ``` As the callback may run more than once, avoid side effects in it other than publishing messages. Kafka allows a single producer at a time for each `transactional.id`: when a producer starts a transaction, older producers using the same id are fenced off and their transactions fail. Every process publishing transactions, such as each PHP-FPM worker or queue worker, must therefore use its own id, which is why the example includes the host name and the process id. Previous [ Producing messages ](https://laravelkafka.com/docs/v3.0/producing-messages/producing-messages) Next [ Configuring message payload ](https://laravelkafka.com/docs/v3.0/producing-messages/configuring-message-payload) Sponsors [ version="1.0" encoding="UTF-8"? EasyCal ](https://easycal.app/) [ Search ⌘ K ](https://typesense.org/)