job-list: default 'since' to something != 0 about flux-core HOT 5 OPEN

Another alternative would be to limit the number of matches (which I think we already do? max_entries=1000 or something...)

from flux-core.

Another alternative would be to limit the number of matches (which I think we already do? max_entries=1000 or something...)

Yeah, we already do that. The issue is if you have some obscure search and don't match anything (or very little). So you'd be scanning the entire database for potential matches, but not actually matching anything.

from flux-core.

Ah, so your worry is users inadvertently creating searches that tax the database too heavily? I can see the sense in that, but it does seem like modern databases should be able to handle this kind of query better than we think.

My worry would be incorrect answers by default. For example, a user queries "give me all my jobs that ran on fluke123" and none have run in the past week on that host, so the answer is an empty list.

from flux-core.

My worry would be incorrect answers by default. For example, a user queries "give me all my jobs that ran on fluke123" and none have run in the past week on that host, so the answer is an empty list.

Ahhh I didn't think of it that way. That is a good point.

Ah, so your worry is users inadvertently creating searches that tax the database too heavily? I can see the sense in that, but it does seem like modern databases should be able to handle this kind of query better than we think.

Yeah, we'll have to see how the job constraints stuff ends up working out. Now that we're splitting out the job-archive into flux-acounting (flux-framework/flux-accounting#357) we have less "dependency" and can tweak the database per our own needs. Given the initial job constraints implementation (#5126) and the initial job db in #4336, I think we'd just iterate through every inactive job in the database ... which is bad of course. (side note, #4336 has to be re-worked given job constraints.)

from flux-core.

By cahnce, I noticed that in sacct there are "time windows" with their searches. Only jobs that started in the specified time range (-S, -E) will be returned. That's a alternate way to deal with the general "don't scan the whole database".

from flux-core.