Create Crawl Configuration/Template/Definition UI about browsertrix HOT 7 CLOSED

Thoughts on adding a name field to easily label and differentiate configs?

from browsertrix.

Thoughts on adding a name field to easily label and differentiate configs?

Yes, definitely a good idea. Just free-form text that can be searched by, right?

from browsertrix.

@ikreymer initial pass at mockup: https://app.mockplus.com/run/rp/TgN5xi_FPJvV/2DkJibt0vw?cps=expand&rps=expand&nav=1&ha=1&la=1&fc=0&out=0&rt=1

q's:

1. what are sane defaults for all these fields? I'll update the mockup to reflect default values.
2. can you go into more detail as to how scope type and page limit should apply to the URLs? Should the term seed be user facing? or should the UI display seeds as something like "URL group"/"Page groups"?
3. should the JSON configuration represent the whole config/template, or just the nested .config object? IMO the former is more user-friendly if we expect users to download JSON configs and then upload or copy-paste them in the future

from browsertrix.

Simplified config for now, just send URL list for seeds, everything else moved to outer scope, eg.

{
  "schedule": null,
  "runNow": true,
  "config": {
    "seeds": [
      "https://webrecorder.net/"
    ],
    "scopeType": "prefix"
  },
  "name": "Example Name",
  "colls": [],
  "crawlTimeout": "0",
  "parallel": 1
}

from browsertrix.

per discussion on call:

1. change default to "run instantly" and schedule to "weekly", @ikreymer to document other defaults
2. seed URL is user-facing term (TODO link glossary), group seed configs in UI
3. render whole JSON object

from browsertrix.

@ikreymer minor issue with the POST crawlconfigs endpoint: leaving out the trailing / throws a Method Not Allowed error on the server

from browsertrix.

All done!
Crawl scaling and tags to be separate issues.

from browsertrix.