#StackBounty: #apache-kafka #publish-subscribe #telemetry #kafka-topic What are the practical limits of Kafka regex-topics / listening …

Bounty: 100

I am exploring different PubSub platforms and I was wondering what the limits are in Kafka for listening to multiple topics. Consider for instance this Use Case. We have trains, station entry gates, devices that all publish their telemetry. Currently this is done on a MQ but as data rates increase, smart trains etc. we need to move to a new PubSub/streaming platform and Kafka is on that list of course.

As I see it there are two strategies for aggregating this telemetry into a stream:

  1. aggregate on consumption, in which each train/device initially gets its own topic and topic aggregation is done using a regex-topic / virtual topic
  2. aggregate on production, in which all trains produces to an single topic and consumers use filters if neccessary to single out individual producers

As I understood Kafka is not particularly suited for high number of topics (>10.000), but it could be done. Would a regex-topic be able to aggregate 2000, 3000 topics?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.