Review jobs workers about taxonworks HOT 3 OPEN

Initial thoughts:

DELAYED_JOB_PRIORITY = {
  import_dataset_import: 0,
  import_dataset_stage: 10,
  query_batch_update: 20,
  cache: 50,

  # TODO: refine to `async_export` and `sync_export` ?
  project_download: 100,
  basic_nomenclature_export: 200,
  dwca_export: 500,  
  coldp_export: 510,

  dwc_occurrence_index: 2000,  # TODO: merge with below?
  dwca_build_index: 2010, 

  cached_map: 3000,
}.freeze

from taxonworks.

import_dataset_import are jobs that once completed a new one is immediately queued to continue processing the staged import data. Unless priority ages all other jobs would not be processed for quite a long time if someone happens to fire a large dataset import.

from taxonworks.

Understood. As you found the problem is that import_dataset_import fires other delayed jobs related to caching/indexing. We don't want those to get in the way. Somehow we need to take that into account.

from taxonworks.