AWS Lambda adds queue mode for self-managed Kafka event sources

AWS Lambda has added queue support to self-managed Apache Kafka event source mappings (ESMs), allowing processing to scale beyond a topic’s partition count. The November 2, 2026 announcement covers customer-operated clusters and managed offerings such as Confluent Cloud, provided the cluster has Kafka share groups enabled.

Parallelism without more partitions

AWS describes the previous Lambda Kafka consumption model as using consumer groups, which assign each partition to exactly one consumer within a group. That ties consumer parallelism to the topic’s partition count.

Queue mode instead uses Kafka share groups, allowing multiple Lambda event pollers to consume concurrently from the same partition. Each poller supports up to 10 concurrent invocations—not a limit of 10 for the entire mapping.

AWS says this approach requires no changes to producers, topic configuration or function code. It also says a slow or failed event no longer holds up other events in the same partition. These are vendor claims, rather than measured performance results.

The trade-off is explicit: queue mode does not preserve partition ordering. AWS positions it for independent work such as media encoding, notification delivery and telemetry ingestion, rather than workloads whose correctness depends on ordered processing.

Availability, configuration and pricing

According to the launch notice, the capability is available in all AWS commercial Regions where self-managed Kafka ESM is available. It requires a cluster with share groups enabled; AWS gives Apache Kafka 4.2 or later as an example.

To get started, create a self-managed Kafka ESM with ConsumptionMode set to Queue. AWS lists configuration support through the console, Lambda API, AWS CLI, CloudFormation, SAM and CDK.

Standard Lambda pricing applies. Customers configuring provisioned mode for their self-managed Kafka ESM also pay for Event Poller Unit (EPU) usage.

Check downstream capacity before scaling

For engineers, the useful distinction is between partitioning to organize data and partitioning solely to unlock more workers. Queue mode is worth evaluating for the latter case, provided ordering is not required.

Before rollout, confirm that out-of-order processing is acceptable and test downstream database and API capacity at the intended concurrency. Treat the removal of the partition-count constraint as an opportunity to scale—not a guarantee that the rest of the application can absorb the extra work.