anticompositenumber / linkspam Goto Github PK

It makes most sense to run linkspam jobs on the Grid, but grid jobs can't be started from k8s (where the webservice runs). Options:

1. Split linkspam into it's own tool, with it's own frontend webservice running on the grid.
2. Split linkspam into it's own tool, with it's own non-user-facing webservice running on the grid that only takes requests from anticompositetools
   2.1. Do that, but on AntiCompositeBot
3. Use a grid webservice or other communication method on anticompositetools to run linkspam jobs
4. Run the main anticompositetools webservice on the grid.

Blocks AntiCompositeNumber/anticompositetools#1 and partially AntiCompositeNumber/anticompositetools#4

Toolforge provides a redis instance, which means that I could spin up Celery instead of using the grid.

Pro:

• The grid is becoming more and more outdated
• Newer python versions available sooner
• Webservices run from k8s, meaning I can use the toolforge-maintained configuration instead of uwsgi-plain

Con:

• I don't know redis and celery very well
• I'd have to set up and maintain more things instead of depending on the grid

Questions:

• What is inter-process communication like?
• What is database and file access like?
• How does monitoring and control work?

https://flask.palletsprojects.com/en/1.1.x/patterns/celery/
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_continuous_jobs
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge

• Finish writing comments in app.py
• Update comments in global_linkspam.py
• Write development documentation in readme.md
• Add a page with user documentation

If I'm going to learn to SQL in #5, I might as well use it for report generation as well.

Timestamps should be displayed nicely, instead of as a raw ISO 8601 string. Relative dates are probably best for the front page, but I think I want locale-formatted dates on the report pages. In either case, the ISO string should be shown as a tooltip.

Currently in use are "new", "default", and "automatic". Manual reports will have statuses for recently created reports and stale reports.

Old manual reports should expire and be removed. Somewhere between 2 weeks and 1 month is probably a good balance between stability and staleness.

Reports with status "new" should be changed to "automatic" or the equivalent for manual jobs. #17 for what that should be.

Right now, only automatic reports are displayed. When manual reports are introduced, recent ones should also appear on the front page. Automatic reports should be more prominent.

The non-maintainer shutoff is currently a runpage on enwiki. Now that this tool doesn't have a static frontend, it would make sense to have this shutoff in the tool itself, if a shutoff is necessary at all.

It would be a good idea to hide it behind OAUTH though.

• Determine if a global shutoff is necessary
  • If yes, move to web app
    • Require OAUTH auth to disable
    • Display obvious messages that the tool is disabled
    • Stop running jobs as well as preventing new ones
  • If no, remove unneded code

Even for automatic jobs, it would be helpful to not have to manually edit the JSON file. Saving should be limited to authenticated users (#9), and automatic jobs that aren't yet configured in cron should be marked as disabled.

Authentication to OAUTH should be required to submit a new job. Manual jobs can be submitted directly after authenticating, but automatic jobs require server-side action.