One big barrier to adapting Brightway to other databases is its tight coupling between projects, the default database backend, and the IO library. To decouple these, we first need to define data formats, interfaces, and endpoints.

The interface is easy to define - we will use Python, with normal data structures, either built in or with attrs or similar.

The data format will be eILCD. eILCD is an XML format for life cycle inventory and impact category data. We are choosing eILCD instead of developing something on our own, or using a different existing format, because:

• eILCD is the only format really battle tested with multiple implementations and use in a variety of contexts
• eILCD is the transfer format for GLAD and PEF
• eILCD is extensively documented, both with XSD files and guidance

Don't get the wrong impression, eILCD is everything you hate about XML, but arguably even worse than usual as it was mostly designed by scientists instead of programmers. But it is still better than the alternatives.

The hardest part are the API endpoints. These are Python methods or functions which programs can use to interface with any database backend that follow the Brightway common database standard; database implementors can choose to only support part of this standard, but we should define this standard in a parsimonious fashion, such that open source implementations do choose to implement the whole standard.

Based on the current default database backend, the following CRUD endpoints should be considered:

• Node (Can be called activity or process)
• Edge (Can be called exchange)
• Impact category
• Characterization factor

It might also be necessary to have a common endpoint for bw_projects for when a project is activated.

After finishing this section, programmers will be able to choose whatever database and schema they want, and will know exactly which functions are expected by bw_io such that the IO classes can write or update data. There will also be a reference implementation, compatible with existing bw2data code and semantics, which will demonstrate how users can import data from bw_io, and query data from an ORM.

Specific tasks:

• B1 Choose, document, and justify the data structures used for passing data following the common API
• B2 Adapt eILCD to a standard Python data structures. Any deviations from eILCD nomenclature or structures must be documented and justified.
• B3 Design a set of Python signals to be emitted by database backends on CRUD operations. This is needed for search indices or other integrated platforms. Follow Activity Browser signals when possible to maximize compatibility.
• B4 Specify and provide a reference implementation of the Brightway Common Database standard
• B5 Port bw2data to the standard as bw_data

Nice-to-have:

• B6 Reference common database implementation against a different type of database (e.g. graph database)

This is needed for search indices or other integrated platforms to be notified that data has changed. Follow Activity Browser signals when possible to maximize compatibility.

Needs #6 before it can be finalized.

An important part of Brightway maturing is a formal decision making process for making changes to and prioritizing development in Brightway. To that end, we will:

• H1 Create a structured process for making breaking changes or adding major features to Brightway (see the proposal template), based on Python enhancement proposals.
• H2 Create a Steering Committee which will vote on BEPs
• H3 Create a Brightway Users Group as a formal forum to provide feedback from Brightway users and key stakeholders

Départ de Sentier (DdS) will manage the Steering Committee and Users Group. Managing the Brightway project, and organizing the Brightcon conference, are two core activities of DdS.

See the proposal template.

Motivation

Every time we need to make a change, e.g. add a tool, update Github Workflows system images, etc. we need to manually change every library. Manual changes can be inconsistent and are a pain, leading to procrastination. Larger changes are basically impossible to imagine.

Desired template attributes

We need a standard template and guide for libraries which has the following qualities:

• Uses Github workflows to test all three OSes with both the latest and next-to-last system images and Py 3.7-3.10.
• Test installation using both conda and pip
• Uses setup.cfg instead of setup.py. Include sections for dev (has e.g. pytest, pip-tools), stable (all requirements pinned), and normal installation (requirements have ranges of >=)
• Reports 1-click code coverage results
• Provides badges for CI results, code coverage (e.g. shields.io)
• Has a solution for hosted HTML documentation including napoleon
• Uses tooling to help improve code quality
• Single-source the library version (why these solutions so hacky!?)
• Automated builds and uploads to anaconda on tagged commits

Adapting the hypermodern template

A lot of these attributes are found in the hypermodern template, though @cmutel would argue against the following:

• Use of poetry: Better to stick with standard tools instead of more opinionated ones in components designed to be integrated into applications.
• Use of tox or nox: Not necessary, we do testing in CI, and setting up a venv for each test run is quite heavy for our components.
• Automated build notes. Could be considered in the future, we don't yet have the discipline to use this consistently, and it makes life harder for new contributors.

We should use the following (copied from their docs):

• Documentation with Sphinx, MyST, and Read the Docs using the furo theme
• Automated uploads to PyPI and TestPyPI

pre-commit hooks

Note that we assume the following are done as pre-commit hooks by the template (instead of by Github Workflows):

• Import sorting with isort
• Code formatting with Black and Prettier
• Black formatting in documentation snippets
• Static type-checking with mypy
• Runtime type-checking with Typeguard

cookiecutter instead of pyscaffold

pyscaffold ticks a lot of boxes, but has some controversial choices which are hard to remove, including the use of setuptools-scm and tox, has custom commands instead of pip and friends (not great for new users), and would need to adapted to include the Azure Workflows config in any case. @cmutel thinks that a custom cookiecutter template is probably the best choice. Ideally, someone would already have such a template, as the cookiecutter ecosystem is large, but no suitable template was found browsing cookiecutter-templates.

pip, not poetry or pipenv

Let's skip the drama, pip with pip-tools and pipreqs works just fine.

Useful links

• Python packaging authority
• PyPA: Using Github Workflows to publish packages
• pip-tools
• pipreqs
• cookiecutter conda template
• setuptools on setup.cfg
• Nice intro to pyproject and setup.cfg
• Workflow from @romainsacchi which does automatic package building and uploading

Tasks

• Make cookiecutter template that satisfies the required attributes
• Document use of this template, both for new library creation and for updating libraries
• Create example project from the example template
• Have regular checks to ensure that all Brightway libraries work correctly as dependent libraries evolve

e.g. a graph database

This is already quite advanced. Specific tasks:

• A1 Updating presamples or porting concept of campaigns to another library (or bw2data calculation setups) to allow easier management of datapackages
• A2 Provide equivalent of ComparativeMonteCarlo

Nice-to-have:

• A3 Standard report format for calculation results
• A4 Packaging/scripts (plus containerization) to easily get calculation running on web servers
• A5 Web API (e.g. Flask app) for running calculations, with rate-limited implementation on DdS server
• A6 Provide support and specification for multiple impact categories in one calculation (MultiLCA)

A formal forum to provide feedback from Brightway users and key stakeholders

Once #6 is closed, this could be a very simple library using an ORM and SQLite to show how a database backend could interact with BW2IO.

The intent here is to choose between a mix of lists, dicts, sets, and something more structured, like attrs, dataclasses, or pydantic.

The current interface between bw2data and bw2io is inadequate - it does not include a lot of important metadata, it grew organically without clear design or vision, it doesn't not distinguish between inventory and impact assessment, and it barely specified and therefore impossible to implement by outside collaborators.

Specific tasks:

• E1 bw_io needs to to support the Brightway Common Database standard
• E2 Python decorator to use old-style bw_io strategies without converting internal logic to ILCD data standard

Nice-to-have:

• E3 bw_io strategies rewritten to use the ported ILCD data standard internally
• E4 Code audit of bw_io for test coverage and strategy obsolescence

eILCD. Unclear why this is referred to as eILCD instead of ILCD version 2, but whatever.