Backend: Improve sorting of workflows by last updated about browsertrix HOT 6 CLOSED

@SuaYoo in the draft PR, this currently only sorts on the new order if you explicitly pass the ?sortBy=lastCrawlTime param (modified the existing sort by field instead of adding a new one, making it effectively lastUpdated).

I see your point though and will look into changing it to return a lastUpdated field and use that instead

from browsertrix.

Another option:

• Add currCrawlStartTime to db
• Sort desc by {currCrawlStartTime -1, lastCrawlTime -1}, which should return current crawls first, then finished crawls that have completed ordered descending by lastCrawlTime. A multi-field index would speed this up as well.

or a variant: sort on lastCrawlStartTime (as current crawl), lastCrawlTime, and status.

Will try a few options and see what is the simplest solution with the desired result.

from browsertrix.

Sorry for the late response--to clarify, is this work handling default sorting by the backend, or adding a new field lastUpdated? The latter would be preferred, so that the frontend both sort by the value and show the value without having to re-calculate it.

from browsertrix.

Can sortBy accept a secondary/additional sort fields? Then we can keep lastCrawlTime and the frontend can request to filter sortBy=lastCrawlTime,finished,started,modified,created.

from browsertrix.

Can sortBy accept a secondary/additional sort fields? Then we can keep lastCrawlTime and the frontend can request to filter sortBy=lastCrawlTime,finished,started,modified,created.

We can do that! The trade-off would be either we need to create a bunch of indices for the possible combinations, or lookups might be slower than they would with a predictable combination and accompanying index.

from browsertrix.

@SuaYoo if you're cool with it I think I'd prefer to just set a lastUpdated field in the db and build an index around that to keep the query and sorting fast. Otherwise we might get back into the slow lookups territory we moved away from recently.

Sorting options for workflows would then be: ("created", "modified", "firstSeed", "lastCrawlTime", "lastUpdated"), with lastUpdated being either the current crawl start time if a workflow is running or the finished time of the last crawl if not.

from browsertrix.