Giter VIP home page Giter VIP logo

substrafl's Introduction



Substra


Substra is an open source federated learning (FL) software. It enables the training and validation of machine learning models on distributed datasets. It provides a flexible Python interface and a web application to run federated learning training at scale. This specific repository is the low-level Python library used to interact with a Substra network.

Substra's main usage is in production environments. It has already been deployed and used by hospitals and biotech companies (see the MELLODDY project for instance). Substra can also be used on a single machine to perform FL simulations and debug code.

Substra was originally developed by Owkin and is now hosted by the Linux Foundation for AI and Data. Today Owkin is the main contributor to Substra.

Join the discussion on Slack and subscribe here to our newsletter.

To start using Substra

Have a look at our documentation.

Try out our MNIST example.

Support

If you need support, please either raise an issue on Github or ask on Slack.

Contributing

Substra warmly welcomes any contribution. Feel free to fork the repo and create a pull request.

Setup

To setup the project in development mode, run:

pip install -e ".[dev]"

To run all tests, use the following command:

make test

Some of the tests require Docker running on your machine before running them.

Code formatting

You can opt into auto-formatting of code on pre-commit using Black.

This relies on hooks managed by pre-commit, which you can set up as follows.

Install pre-commit, then run:

pre-commit install

Documentation generation

To generate the command line interface documentation, sdk and schemas documentation, the python version must be 3.8. Run the following command:

make doc

Documentation will be available in the references/ directory.

Changelog generation

The changelog is managed with towncrier. To add a new entry in the changelog, add a file in the changes folder. The file name should have the following structure: <unique_id>.<change_type>. The unique_id is a unique identifier, we currently use the PR number. The change_type can be of the following types: added, changed, removed, fixed.

To generate the changelog (for example during a release), use the following command (you must have the dev dependencies installed):

towncrier build --version=<x.y.z>

You can use the --draft option to see what would be generated without actually writing to the changelog (and without removing the fragments).

substrafl's People

Contributors

alexandrepicosson avatar aureliengasser avatar cboillet-dev avatar esadruhn avatar fabien-gelus avatar grouane avatar guilhem-barthes avatar kelvin-m avatar louishulot avatar maikia avatar mandreux-owkin avatar mblottiere avatar milouu avatar oleobal avatar olivierdehaene avatar romaingoussault avatar sdgjlbl avatar tanguy-marchand avatar thbcmlowk avatar thibaultfy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

substrafl's Issues

BUG: circular recursive copy of files when using Dependency object with local_code from within local_code

What are you trying to do?

I run a script inside my package that I want to list as a dependency, while copying my package it creates a temp directory holding the copy, and then it tries to copy that folder as well, which already contains the package and so forth until disk space runs out.

Issue Description (what is happening?)

Pretty much self explanatory

Expected Behavior (what should happen?)

It should copy only relevant files

Reproducible Example

No response

Operating system

MacOS

Python version

3.9.16

Installed Substra versions

0.35.Orc1

Installed versions of dependencies

No response

Logs / Stacktrace

No response

BUG: can't change aggregation lr in Scaffold

What are you trying to do?

Change aggregation lr in Scaffold

Issue Description (what is happening?)

It is impossible to set a different aggregation_lr than 1.

Expected Behavior (what should happen?)

Result of experiment should change

Reproducible Example

No response

Operating system

MacOS (but irrelevant)

Python version

3.10

Installed Substra versions

substrafl 0.34.0

Installed versions of dependencies

No response

Logs / Stacktrace

No response

[BUG] Wrong instructions related to metrics_keys in TestDataNodes doc

Describe the bug

TestDataNode docs states:
metric_keys (List[[str](https://docs.python.org/3/library/stdtypes.html#str)]) – Substra metric keys to the metrics, use substra.Client().add_algo().

It should instead talk about add_metric not add_algo and provide a more detailed explanation such as:
the metric key is the hash of the metric created by the add_metric method of the Client (substra.Client().add_metric()). It can be accessed by inspecting the result of the method's call or retrieved from the frontend in deployed mode.

Screenshots

Capture d’écran 2023-02-22 à 15 52 24

Environment

NA

BUG: When giving a larger batch-size than the dataset size the logic handling it is not working

What are you trying to do?

I am passing a larger than the dataset's size batch size so that I am sure that the whole dataset is loaded at every update on all clients.
I get this log:

2023-04-13 08:56:18,519 - INFO - The batch size (9223372036854775807) is greater than the number of samples: n_samples.This is not allowed. Batch_size is now updated to equal number of samples (200)

Then afterwards I get a memory allocation error in local_predict because it's allocating:

batch = [0] * self.batch_size # with self.batch_size=9223372036854775807

And self.batch_size was not updated to 200 contrary to what the log says.

Issue Description (what is happening?)

OOM because the fix is not applied

Expected Behavior (what should happen?)

The batch_size should be set to 200

Reproducible Example

No response

Operating system

MacOS

Python version

3.9.16

Installed Substra versions

0.35.0

Installed versions of dependencies

No response

Logs / Stacktrace

No response

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.