Remove evaluation codes from tyra when we have the repo for evaluation

Tyra

Prediction model evaluation dashboard

Screenshot

Quick Start

Install requirements: pip3 install -r requirements.txt
Copy example_default_profile.yaml to default_profile.yaml and modify to point to your database
Run python3 run_webapp.py
Enter http://localhost:5001/ in your local browser

Database Schema

Tyra is designed to work on different data science projects, but to accomplish this it expects a standard model storage interface; a Postgres database with a results schema containing three tables, models, evaluations, and predictions.

models

model_id - integer
run_time - timestamp
model_type - string
model_group_id - integer
test - boolean
model_comment - string
train_end_time - timestamp

model_groups

model_group_id - integer

evaluations

model_id - integer
metric - string (each value should be one of the Available Metrics below)
parameter - string
value - numeric
evaluation_start_time - timestamp

predictions

model_id - integer
entity_id - bigint
unit_score - numeric
label_value - int
as_of_date - timestamp

feature_importances

model_id - integer
feature - string
feature_importance - numeric

ranked_table (cache_table.sql is an example for creating a ranked table)

model_group_id - integer
metric_parameter - string
avg - numeric
run_time - teimstamp
model_comment - string

Whether or not the model_id and as_of_date columns in predictions and evaluations are defined as a foreign key, they should act as one as the tables are joined using it.

No assertions are made on other tables in the results schema or other schemas, or on other columns in these tables.

Available Metrics

The list of metrics that will be displayed to the user currently lives in parameters.yaml. The currently shipped list looks like this:

precision
recall
auc
true positives
true negatives
false positives
false negatives

Both absolute value and percent thresholding are supported for metric parameters. The format for each one in the results.evaluations.parameter is as follows:

100_abs (top 100 entities) 10.0_pct (top 10%)

Tyra expects these metrics to be computed ahead of time, so if your evaluation code does not compute all of these thresholds for all of the threshold-able metrics ahead of time, we recommend removing entries from parameters.yaml to make the lists match up and avoid confusion in the UI.

Running multiple instances

Since Tyra is built to look at different project databases, being able to run multiple instances side-by-side is helpful. To accomplish this, you can pass environment variables to configure the database credential file path and the port. So these two commands will work:

PROFILE=/path/to/first/profile.yaml PORT=5001 python3 run_webapp.py

PROFILE=/path/to/second/profile.yaml PORT=5002 python3 run_webapp.py

Dev Front-end setup

Tyra uses NodeJS and Webpack to organize and bundle frontend dependencies.

Initial setup (If you don't see output.js in /webapp/static/)

Install NodeJS (https://nodejs.org/en/)
cd frontend
npm install to install dependencies (the initial install will take a few minutes, go have a snack!)
npm run build to initially create the bundled javascript file

Troubleshooting

Sometimes node and npm versions from package managers are ancient and need to be upgraded before installation will work.

To upgrade node to the latest stable version:

sudo npm cache clean -f
sudo npm install -g n
sudo n stable

To upgrade npm:

sudo npm install npm@latest -g

During development

npm run start will start a webpack '--watch' command that watches your javascript and compiles it to webapp/static/output.js. The initial startup will probably take 10-15 seconds, but every time you save a javascript file the recompilation will be much quicker.

New components can be added in the frontend/components directory. There is a directory for each component, because soon (not yet) we will start bundling styles in individual component directories. Other components will be able to import your new component right away, but if you would like the component to made available globally (in other words, a Flask template), you will have to add this to frontend/index.js

Installing new modules

In the frontend directory, install the package you want with npm install --save <pkg-name>. The --save option will persist this change to package.json.

	else:
	logging.warning('No config file found, using empty config')
	config = {}
	dburl = ''
	dbschema = {}

	lookup_query = """
	SELECT
	model_group_id
	FROM results.ranked_table
	ORDER BY avg DESC
	"""

	profile_file = os.environ.get('PROFILE', 'default_profile.yaml')
	schema_config = os.environ.get('SCHEMA', 'db_schema.yaml')

	numpy
	scipy
	scikit-learn

	codecov
	pytest-cov
	tox

dssg / tyra Goto Github PK

tyra's Introduction

Tyra

Prediction model evaluation dashboard

Screenshot

Quick Start

Database Schema

models

model_groups

evaluations

predictions

feature_importances

ranked_table (cache_table.sql is an example for creating a ranked table)

Available Metrics

Running multiple instances

Dev Front-end setup

Initial setup (If you don't see output.js in /webapp/static/)

Troubleshooting

During development

Installing new modules

tyra's People

Contributors

Stargazers

Watchers

Forkers

tyra's Issues

Recommend Projects

Recommend Topics

Recommend Org