insert mutli index on the same topic about kafka-connect-elasticsearch-source HOT 3 OPEN

Hi,
you can't have multiple index.prefix, but you have two solutions:
in your example you can use my_ as index prefix, or alternatively you might create two instances of the connetor (each one using a single index.prefix).

Then, the output topic is based on the name of the input index. But if you need to put the output in a single topic, you might use the confluent connect transformer https://docs.confluent.io/platform/current/connect/transforms/regexrouter.html#regexrouter in order to rename the topic as you prefer.

Hope this helps

from kafka-connect-elasticsearch-source.

Hi there,

Is there another means of doing this?

I need to send multiple indexes to one fixed topic name but the connector seems to be completely ignoring the regex configuration entries.

For example qradar-wcf-000011, qradar-wcf-000012, qradar-wcf-000013 (they increment on elasticsearch over time) all to a kafka topic called "qradar-wcf".

Thanks,
Liz

from kafka-connect-elasticsearch-source.

Add the following to your config part :

"transforms": "dropTopicPrefix",
"transforms.dropTopicPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropTopicPrefix.regex": "(.*)qradar-wcf-(.*)",
"transforms.dropTopicPrefix.replacement": "qradar-wcf",

this will publish all the messages to the topic qradar-wcf

from kafka-connect-elasticsearch-source.