Scripts for evaluation of contextual bandit algorithms in Vowpal Wabbit. See the following paper for details:
A. Bietti, A. Agarwal, and J. Lanford. Practical Evaluation and Optimization of Contextual Bandit Algorithms. arXiv preprint, 2018.
You will need an OML account to get the datasets.
- create an account at openml.org and login
- click icon at the top right once logged in
- click api authentication button
- create file config.py in this directory with contents
OML_API_KEY="..."
VW_DS_DIR="..."
- set up to facilitate parallelism
- first argument is
task_id
: should span from 0 to number-of-parallel-tasks - second argument is
num_tasks
: number of parallel tasks ./run_vw_job.py 0 1
runs everything with no parallelism