It seems like the problem is that kafka's default unit of parallelism is the partition (N consumers are mapped to N partitions enabling N degree of parallelism), but what Uber wanted is parallelism even within a partition.
Wouldn't a much simpler solution be to increase the number of partitions to `numConsumers * numThreadsPerConsumer` and make every consumer thread a separate kafka consumer of its own? I believe this is how the Kafka creators envisioned it to be done.
Another option would be to use Amazon SQS instead of Kafka where there is no concept of partitions and we can simply scale up the number of consumers however we want.
Wouldn't it be simpler to just publish each message type to a separate topic?
This solution is not so simple to monitor - if you have 3 types of messages in one topic, your topic lag isn't very descriptive - you need to go deeper to understand the issue
Publishing of messages into a different topic type is not a problem. We can do that as much as we want. The problem is that ordering of messages within a partition is limiting the consumption throughput and that's why Uber designed this Kafka consumer proxy to parallellize the processing of messages within a partition.
How is the order of the messages maintained? Proxy layer could read multiple messages from the same user and send the messages to downstream services in parallel. Can this lead to race conditions?
It seems like the problem is that kafka's default unit of parallelism is the partition (N consumers are mapped to N partitions enabling N degree of parallelism), but what Uber wanted is parallelism even within a partition.
Wouldn't a much simpler solution be to increase the number of partitions to `numConsumers * numThreadsPerConsumer` and make every consumer thread a separate kafka consumer of its own? I believe this is how the Kafka creators envisioned it to be done.
Another option would be to use Amazon SQS instead of Kafka where there is no concept of partitions and we can simply scale up the number of consumers however we want.
Wouldn't it be simpler to just publish each message type to a separate topic?
This solution is not so simple to monitor - if you have 3 types of messages in one topic, your topic lag isn't very descriptive - you need to go deeper to understand the issue
Publishing of messages into a different topic type is not a problem. We can do that as much as we want. The problem is that ordering of messages within a partition is limiting the consumption throughput and that's why Uber designed this Kafka consumer proxy to parallellize the processing of messages within a partition.
How does the consumers register themselves with the proxy?
How is the order of the messages maintained? Proxy layer could read multiple messages from the same user and send the messages to downstream services in parallel. Can this lead to race conditions?