Batch size question about kafka-connect-elasticsearch-source HOT 5 CLOSED

Hi,

I didn't realize the incrementing.field.name should be unique, thanks for mentioning that.

I am using it with a a timestamp but multiple records could have the same time stamp if they are processed at the same time.

I think the secondary sort field makes sense.

from kafka-connect-elasticsearch-source.

Hi @albertwgchu, I've published a pre-release (1.4) with the secondary sort feature.

You can use the incrementing.secondary.field.name settings to specify the secondary sort field. Note that text fields cannot be sorted, so if you need to sort string fields you will need to use keyword fields.

If everything is fine also for your side, I will mark version 1.4 as stable.

from kafka-connect-elasticsearch-source.

Hi @albertwgchu,
the incrementing incrementing.field.name should be a strictly increasing field (no duplicates), to avoid the described issue.
So, in the described situation you may loose some records.

Maybe in a new version I can add a secondary field for pagination, to avoid this issue. The _id field would be great, but it is not possible to sort it. However I can add a custom secondary sort field, so let say when you have duplicates n the primary increasing field, the secondary will help in pagination avoiding data losses.

Could it be useful to you?

from kafka-connect-elasticsearch-source.

Cool, I will add configurable secondary sort field in the next version.

from kafka-connect-elasticsearch-source.

I will take a look in the next few days.

from kafka-connect-elasticsearch-source.