Job management for scikit learn classifiers. Inspired by / copied from Joseph Turian https://github.com/turian/batchtrain/blob/master/jobman.py

This package provides an interface to transparently save Scikit-Learn classifiers to a Mongo database. Features include:

• Transparent saving of trained classifiers to a (potentially non-local) mongo database
• Knowledge of when a classifier has been trained with the same hyperparameters and training data

• scikit-learn
• pymongo

pip install git+git://github.com/ChrisBeaumont/joblog.git

from joblog import JobFactory
from sklearn.svm import SVC
from sklearn.datasets import load_iris

jf = JobFactory('localhost') # connect to a mongo database running on localhost

iris = load_iris()
X, Y = iris.data, iris.target
params = {'C': 5.0, 'kernel': 'rbf'}

job = jf.job(SVC, X, Y, params)
clf = job.run()   # train and return a classifier.

A pickled version of clf is auto-saved to the database whenever job.run is called. Calling job.run again returns this cached result

clf2 = job.run()  # classifier is not re-trained
clf2 = job.rerun() # force it to refit

Similarly, if a new job is created with the same inputs, the cached result is retrieved

job = jf.job(SVC, X, Y, params)  # duplicate job
job.duplicate    # True
clf = job.run()  # cached result is returned

joblog uses checksums to detect changes in the input arrays.

X[0] += 1
job = jf.job(SVC, X, Y, params)  # X is different, this is a new job
job.duplicate    # False
clf = job.run()  # new result is computed

Similarly, any change in the classification class, training data, or hyperparameters results in a new classifier being trained and stored

By default, joblog saves a pickled version of the trained classifier. This is accessible via the result property:

clf = job.result  # possibly None, if job hasn't been run

However, you can assign any (pickleable) value to job.result, which is saved in the database

clf = job.run()
job.result = job.score(X, Y)
job.run() # now returns the score

 #connect to a remote database
jf = JobFactory('mongodb://<dbuser>:<dbpassword>@dbh29.mongolab.com:xyz/abc')

 # specify which database and collection to use
jf = JobFactory(db='db_name', collection='collection_name')

Jobs can be created with an optional label

j = jf.job(SVC, X, Y, {}, label='svc_run')

Two jobs with different labels but the same inputs are considered unique.

You can delete all jobs in a collection via JobFactory.clear_jobs()

Extra attributes can be associated with each job using key/value syntax. These values are stored in the database

job['recall'] = 0.95
assert job['recall'] == 0.95

The job_grid method creates and returns a grid of jobs for all combinations of parameters, similar to the scikit-learn IterGrid class

jf = JobFactory()
param_grid = {C=[.1, 1, 10], kernel=['linear', 'rbf']}
for job in jf.job_grid(SVC, X, Y, param_grid):
	job.run()

By default, job_grid skips duplicate jobs. You can turn this off

for job in jf.job_grid(SVC, X, Y, param_grid, filter_duplicates=False):
	job.rerun()	# force retraining

Classifiers can sometimes be very large, and too cumbersome to store in the database. You can customize what classifier information is stored via the store keyword in run():

job.run(store='classifier')  # default, stores full classifier
job.run(store='score')  # store classifier.score(X, Y)
job.run(store='none')   # store nothing

This also affects what is returned from the result property, as well as what gets returned during subsequent calls to run().