Dynamic Kafka Topics in Quarkus

Previously I posted on using the Apache KafkaProducer to route events to Kafka topics dynamically, but there’s a better way. The KafkaProducer is pretty much obsolete since Quarkus 2, and all metadata is ignored. I ended up with serialization issues.

The solution is to use a standard Emitter, and add metadata to your message. For example, the service I’m developing consumes an ingested event from a kafka topic. It then forwards it to this class to route to selected topics:

The “reflector” channel is initialized in the application properties like this:

mp.messaging.outgoing.reflector.cloud-events-source=reflectorservice
mp.messaging.outgoing.reflector.connector=smallrye-kafka
mp.messaging.outgoing.reflector.value.serializer=org.apache.kafka.common.serialization.StringSerializer
mp.messaging.outgoing.reflector.cloud-events-mode=structured
mp.messaging.outgoing.reflector.cloud-events-type=reflector

The emit function simply creates metadata with the specific topic, then creates a message, adds the metadata, and sends it via the emitter.

This approach works much better.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s