Investigate possibility of tools modifying persisted blocks about tsdb HOT 14 CLOSED

@fabxc @Bplotka do you think that now with thanos this issue is addressed?
Not sure if there is an actual cli in thanos for this, but it seems that it solves that particular problem described here.

from tsdb.

Not really.

If I understand this right, we are talking here about multiple readers and single (or multiple even!) writer problem on TSDB blocks in filesystem.

This is related to this: thanos-io/thanos#206

Because this issue is still open, we cannot use local compaction because we have no idea when Prometheus TSDB starts compaction and removes some blocks while we are potentially in process of uploading those to object store.

This can be mitigated by Snapshot API, but I think it requires some change for Prometheus flag (Admin API?). This is not the best, but at least we have an workaround.

Without this workaround we could add some endpoints to turn of local compaction for short time.. not sure. It might be a smart move to just stick to workaround unless it causes more problems.

Any thoughts @fabxc @gouthamve ?

from tsdb.

I am fine with closing this for now, we can reopen when we will have clear indication that this is serious problem (workaround being not sufficient)

from tsdb.

No let's keep it open as there are obviously some blockers here.

from tsdb.

@bwplotka tsdb already has an api to disable/enable compactions
https://github.com/prometheus/tsdb/blob/407e12d051f7907e1bde47ccd59258eedbd10715/db.go#L743

so how about

• disable compaction
• upload the blocks needed
• enable compaction

from tsdb.

🎉 Now is the question... when Prometheus will expose it as API endpoint? (or is it done already?)

from tsdb.

if you think will work for you I can open a PR

from tsdb.

That will definitely fix this issue!

from tsdb.

I’m not sure it’s that easy, we typically don’t have endpoints mutating the configuration of the running process. I’m just wary of new use cases that could be argued for if we added this. Don’t get me wrong I have use cases for this as well, just trying to make sure we’re not making a mistake here.

from tsdb.

adding it will be very simple so why not open the PR and build a docker image so you can test it and will take it from there.

from tsdb.

I didn’t mean code wise, I meant accepting such functionality maintenance wise :)

from tsdb.

Yup, why can you not do this via the Snapshot API?

from tsdb.

I didn’t mean code wise, I meant accepting such functionality maintenance wise :)

Yup, hard to justify it.

As I said in very beggining, yup - we can use Snapshot API for it, as well as compaction disable API (if there would be any), so it's not blocking. (:

from tsdb.

I think we can close this one as this overlaps quite a lot with what we are discussing in #346

from tsdb.