Giter VIP home page Giter VIP logo

fairness-indicators's Introduction

Fairness Indicators BETA

Fairness_Indicators

Fairness Indicators is designed to support teams in evaluating and improving models for fairness concerns in partnership with the broader Tensorflow toolkit.

The tool is currently actively used internally by many of our products, and is now available in BETA for you to try for your own use cases. We would love to partner with you to understand where Fairness Indicators is most useful, and where added functionality would be valuable. Please reach out at [email protected]. You can provide any feedback on your experience, and feature requests, here.

Key links

What is Fairness Indicators?

Fairness Indicators enables easy computation of commonly-identified fairness metrics for binary and multiclass classifiers.

Many existing tools for evaluating fairness concerns don’t work well on large scale datasets and models. At Google, it is important for us to have tools that can work on billion-user systems. Fairness Indicators will allow you to evaluate across any size of use case.

In particular, Fairness Indicators includes the ability to:

  • Evaluate the distribution of datasets
  • Evaluate model performance, sliced across defined groups of users
    • Feel confident about your results with confidence intervals and evals at multiple thresholds
  • Dive deep into individual slices to explore root causes and opportunities for improvement

This case study, complete with videos and programming exercises, of how Fairness Indicators can be used on one of our own products to evaluate fairness concerns over time.

The pip package download includes:

  • Tensorflow Data Analysis (TFDV) [analyze distribution of your dataset]
  • Tensorflow Model Analysis (TFMA) [analyze model performance]
    • Fairness Indicators [an addition to TFMA that adds fairness metrics and the ability to easily compare performance across slices]
  • The What-If Tool (WIT) [an interactive visual interface designed to probe your models better]

How can I use Fairness Indicators?

Tensorflow Models

  • Access Fairness Indicators as part of the Evaluator component in Tensorflow Extended [docs]
  • Access Fairness Indicators in Tensorboard when evaluating other real-time metrics [docs]

Not using existing Tensorflow tools? No worries!

  • Download the Fairness Indicators pip package, and use Tensorflow Model Analysis as a standalone tool [docs]

Non-Tensorflow Models

  • Model Agnostic TFMA enables you to compute Fairness Indicators based on the output of any model [docs]

Examples

The examples directory contains several examples.

More questions?

For more information on how to think about fairness evaluation in the context of your use case, see this link.

If you have found a bug in Fairness Indicators, please file a GitHub issue with as much supporting information as you can provide.

fairness-indicators's People

Contributors

shuklak13 avatar catherinaxu avatar jindalshivam09 avatar christinagreer avatar fhuanming avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.