wwoods / lamegame_tasking Goto Github PK

lamegame_tasking

Packages provided:
  lgTask

Dependencies:
  pymongo

Summary:
lamegame_tasking provides a self-organizing task processing framework.  The central database is a mongoDb instance.

Note that all tasks run in threads in a single process.

Example usage:
Programming a task (must be in a file in tasks directory):
class MyAppTask(lgTask.Task):
    def run(taskDb, arg, **kwargs):
        # Pass mongodb:// conn string to get a pymongo collection, database,
        # or connection (based on the URL).  This is parameter magic that can
        # be overridden
        taskDb['collection'].insert({ 'arg': arg })
        self.taskConnection.createTask(
            "MyAppTask"
            , taskDb=taskDb
            , arg=arg + 1
            )



Programming a task that should abort if an instance is already running:
class MyAppSingleton(lgTask.SingletonTask):
    def run(**kwargs):
        # ...Normal run function

The task is singleton'd based on the "taskName" parameter, which for classes 
defaults to the class name.  If you have e.g. a per-database singleton, simply 
make scheduling or creating the task extract the database name and initialize 
"taskName" accordingly.



=== Event-Drive Tasks ===

Creating a sample task to run:
>> import lgTask
>> c = lgTask.Connection()
>> c.createTask("MyAppTask"[, taskName="MyAppTaskAltName"], arg="Hello, world!")



Creating a task to run at some point in the future:
>> import lgTask
>> c = lgTask.Connection()
>> c.delayedTask(datetime object OR "1 day/hour/minute/second"
    , "MyAppTask", arg="Delayed!")



=== Scheduled and batched tasks ===

Creating a batched task (if already scheduled, don't schedule again; otherwise
schedule the task):
>> import lgTask
>> c = lgTask.Connection()
>> c.batchTask(datetime object OR "1 day/hour/minute/second", "MyAppTask")

If the task was already scheduled (and not executed yet), these functions do 
nothing; that task is presumed to cover whatever this newly requested task
is supposed to accomplish.  Be careful with that.  But this is mostly for
batching things when batches are gradually built in the db.



Creating a recurring scheduled task (executed immediately and at given 
intervals):
>> import lgTask
>> c = lgTask.Connection()
>> c.intervalTask(datetime object OR "1 day/hour/minute/second", "MyAppTask")

If an intervalTask is called with a name that already exists but different 
kwargs or interval, this will raise an exception of type TaskKwargError rather 
than rewriting that task's schedule / kwargs.  If called with identical
arguments to the existing, does nothing.




Unscheduling a scheduled task (does not raise exception if the task does
not exist):
>> c.intervalTaskRemove("MyAppTask"[, taskName="name"])



=== Task Processors ===

Configuring and running processors for development:
At the beginning of your app's web server / task generator:

>> import lgTask
>> processor = lgTask.Processor("/path/to/config"[, taskName="myProcessor"])
>> processor.start()

This will spawn the processor in a new thread.  If you want to specify the 
config in code, you can pass a dict or lgTask.lib.reprconf.Config object 
instead of a file path.  

If you want to block on the processor until it can no longer consume any
tasks, use:

>> processor.stop(onNoTasksToConsume=True)



Configuring and running processors for production:
1. Symlink lamegame_tasking/bin/lgTaskProcessor to your PATH (or execute directly)

2. Setup a user to run the task processors; in their home directory, check out your task source code and dependencies.

2. Setup a configuration file for your deployment, like the following (myAppProcessor.cfg):

[processor] # This name is required; on the bright side, this allows you
                # to put other configuration items in the file
taskDatabase = 'mongodb://[user:password]@server:port[/mytaskdb]'
pythonPath = [ '../' ] # Folders relative to config file to add to python path
                       # when running the Processor.

3. Setup cron to run lgTaskProcessor every minute (to prevent it from erroring out; a processor will automatically not run two instances) with the following parameters:
  a. Config file - absolute path to task processor configuration
  b. [optional] Number of processors to spawn - if unspecified, spawn 1.
  z. Example: lgTaskProcessor /home/user/myAppProcessor.cfg 2

Each Processor runs one task at a time.  They are named according to the format "hostname-index", with index starting at 1.  If only one processor is spawned b y the lgTaskProcessor dscript, the "-index" portion of the name is dropped.