ElasticSearch7 create unlimited amount of indices about conductor-community HOT 5 OPEN

Investigating code I see

    private void createIndexesTemplates() {
        try {
            initIndexesTemplates();
            updateIndexesNames();
            Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);
        } catch (Exception e) {
            logger.error("Error creating index templates!", e);
        }
    }

See this line
Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);

Where updateIndexesNames creates new index every week.
Can someone explain why does it change names for indices?

I'd like to change this behavior, because we reaching limits on shards.

from conductor-community.

I also checked below methods are not used anywhere:

IndexDAO.getEventExecutions
IndexDAO.getMessages

so it is probably safe to stop indexing them:

conductor.app.eventMessageIndexingEnabled=false
conductor.app.eventExecutionIndexingEnabled=false

However to stop creating new indices we need to change this config:
conductor.elasticsearch.autoIndexManagementEnabled=false
And provision indices for conductor manually.

Can anybody comment on this?

from conductor-community.

Hi @astelmashenko , Thanks for reporting this. We will add support for archival from ES also.
Let me know if this works, or we can chat here for more realtime collaboration.

from conductor-community.

@manan164 ,

We will add support for archival from ES

Yes, it will help.
Additionally, I'd add reaction to indexing flag, e.g. if conductor.app.eventMessageIndexingEnabled=false is set then do not create new indices once a week. Maybe think also about changing period of creating new indices, e.g. WEEK, MONTH, YEAR
Thanks

from conductor-community.

@manan164 , are there any work done on archival of indicies?

from conductor-community.