Comments (5)
The purpose of this feature is to create an identity card of a model that links it to datasets, algorithms, initial models.
This topic is related to the different works of Substra Foundation, both on the framework with composite training but also the topics trustworthy AI.
This notion of model genaology does not exist today, even if there are works in progress to be studied.
In particular DVC
from substra-documentation.
Outside of Substra Framework
The objective is to create a template that allows to list all the history of the predictive model.
DATA:
- How were they collected? Single dataset or assembly of several datasets?
- By whom?
- Is synthetic data used?
- Are personal or confidential data present?
- How the split between train, test and validation has been done?
- Use of PETs?
- How bias has been prevented?
ALGO:
- What is the learning algorithm used?
- By whom was the learning algorithm created?
MODELS:
- What is the Initial model?
- Performances of the model?
- Decision thresholds and ranges of indecision?
- Condition of use? Limits?
- How the model is used? How the model is reviewed over time?
- List of incidents
ACCOUNTABILITY:
- Who is responsible in case of incident? Human agency?
- What are the different levels of responsibilities between stakeholders?
- Use of subcontractors?
Substra Framework
How some of this elements could be collected in Substra framework? How Substra Framework can be used as element of proof?
from substra-documentation.
A very interesting paper on that matter: Model Cards for Model Reporting.
Also, Google's Model cards.
from substra-documentation.
Update from last MAP committee (10/09/2020):
- The objective here is to create an identity card of a model that links it to datasets, algorithms, initial models; that includes its “genealogy”!
- [Camille] an early early draft in the frontend already, sort of simple identity card. Not yet clean, readable, user-friendly. Could be great to work on improving that!
- The idea is to move forward in the different work groups on this ID card, and to see how to integrate it into the Substra Framework.
- [Amine] About monitoring data quality, and how it impacts model performance, continuous learning setups (or regularly re-trained models).
- [Camille] This is rather done outside the framework today but we could probably imagine integrations with external tools via queries.
- Next step: the topic will be studied during the new season of DataForGood!
from substra-documentation.
Closing stale issue.
from substra-documentation.
Related Issues (20)
- Add reference to instance demo
- Add walkthrough titanic example in demo page HOT 1
- Add chaincode examples
- [BUG] Dead link in Titanic Example HOT 1
- [FEATURE_REQUEST] Doc about the Compute Plan HOT 1
- Update Glossary
- Add fake_data
- Add a more step by step guide HOT 2
- [FEATURE_REQUEST] Extend to unsupervised ML use cases HOT 1
- Fix login commands on demo page
- Add permissions note for assets HOT 3
- Add tips for GPU check HOT 1
- [k8s] Map ports used for a deployment HOT 1
- Describe errors and exceptions HOT 1
- Debugging - Troubleshooting for Docker container crash HOT 4
- Add a warning about one-session-only HOT 1
- Broken links HOT 1
- [BUG] myst_parser : No module named 'attr' HOT 1
- [BUG] Broken link in TorchAlgo link in example
- [Typos] HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from substra-documentation.