os-climate / os-climate-community-hub Goto Github PK

With the new DBT/OpenMetadata pattern for managing metadata, I suspect that this repo, which was never fleshed out, is obsolete and can be deleted:

https://github.com/os-climate/osc-update-metadata

I plan to delete along with other obsolete/unused repos.

Requesting credentials for github user: eartvit

Requesting credentials for the bucket: redhat-osc-physical-landing-647521352890

Email address to send a credential download link: [email protected]

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Now that OS-Climate is an open source project, the contents of the repo https://github.com/os-climate/open-source-readiness should be merged into other existing documentation and archived. Better to host good material in more active spaces, such as OS-Climate-Community-Hub, where it can stay up-to-date.

Assigning to @HeatherAck as principal manager of OS-Cliamte-Community-Hub. Volunteers for OS-Climate community managers welcome!

Need to define processes for data joining. For example, we want to add LEI data to PUDL data. Determine if this process is unique from Data Enrichment

I was looking to see whether we have yet integrated the teachings about how to fork and commit to repositories. I came across this line:

Want more information on how to contribute code or data? Please see [Contribution Guidelines](https://github.com/os-climate/OS-Climate-Community-Hub/blob/main/Contributing.md)Want more information on how to contribute code or data? Please see [Contribution Guidelines](https://github.com/os-climate/OS-Climate-Community-Hub/blob/main/Contributing.md)

The Contributing.md file leads to a 404. Interestingly, GitHub itself issues a popup telling me that Contributing Guidelines changed 17 hours ago, to here: https://github.com/os-climate/OS-Climate-Community-Hub/blob/80ee6a9679eca5fef88008e79015f286909c946c/CONTRIBUTING.md

I expect that at some point this will merge back into main?

Process should scale up for a multitude of community requests and fulfill them in organized way (i.e. Issue Template).

Process should include prioritization mechanism for data source acquisition when resources are constrained.

Process should establish standardized data ingestion pipeline processes

• Bottoms up data:

• [] PUDL
• [] RMI (US-based utilities sector only)
• [] GeoAsset (steel and cement)

• Tops-down data:

• [] ESSD (global GHG emissions data; also includes sector and region definitions used by IPCC)

Tops-down data is useful as a coverage check for our bottoms-up data.

Leverage corp data pipeline, PhysiRisk and ITR repos, to file data requests and ensure data requests are visible to a data sourcing workstream.

Requesting onboarding for github user: dunstanm

Request access to github team: os-climate-itr-tool-project-team

View teams here: https://github.com/orgs/os-climate/teams

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• new user invited to os-climate github org
• user invited to requested teams

Requesting credentials for github user: SKZHA

Requesting credentials for the bucket: redhat-osc-physical-landing-647521352890

Email address to send a credential download link: [email protected]

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Thanks!

Requesting onboarding for github user: @romeokienzler

Request access to github team: odh-env-users
View teams here: https://github.com/orgs/os-climate/teams

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• new user invited to os-climate github org
• user invited to requested teams

Requesting credentials for github user: DaBeIDS

Requesting credentials for the bucket: redhat-osc-physical-landing-647521352890

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Requesting onboarding for github user: suppathak

Request access to github team: <team-name(s)>

View teams here: https://github.com/orgs/os-climate/teams

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• new user invited to os-climate github org
• user invited to requested teams

Due to a security incident, I heard that we had to ask for a new access key for OSC S3 bucket. Would it be possible to issue that for me? Thanks! @redmikhail

Requesting onboarding for github user: helena-intel

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• new user invited to os-climate github org
• user invited to requested teams

Requesting onboarding for github user: yoann-diep

Request access to github team: <team-name(s)>

View teams here: https://github.com/orgs/os-climate/teams

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• new user invited to os-climate github org
• user invited to requested teams

Requesting onboarding for github user: mandy-chessell

Request access to github team: <team-name(s)>
View teams here: https://github.com/orgs/os-climate/teams

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• new user invited to os-climate github org
• user invited to requested teams

Requesting credentials for github user: dunstanm

Requesting credentials for the bucket: itr

Email address to send a credential download link: [email protected]

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Requesting credentials for github user:

Requesting credentials for the bucket:

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Requesting credentials for github user: joemoorhouse

Requesting credentials for the bucket: redhat-osc-physical-landing-647521352890

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Create OS-C calendar with recurring meetings

Requesting credentials for github user: suppathak

Requesting credentials for the bucket:

Email address to send a credential download link: [email protected]

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

I don't know what I was thinking when I created a whole repo to demonstrate the use of plotly in a notebook. I should have created a notebook in the data_platform_demo repository. This issue can be closed by migrating the notebook so that it works in the data_platform_demo repo and this repo deleted.

@erikerlandson any concerns about that?

Requesting credentials for github user: helena-intel

Requesting credentials for the bucket: redhat-osc-physical-landing-647521352890

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Requesting credentials for github user: MichaelTiemannOSC

Requesting credentials for the bucket:

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Define data federation process(es) where users can consume Data Commons data sources. Make several data sources available with consistent API/access method, mapped to a legal entity (without normalization across sources, and without data quality and curation) and get users' adoption and feedback.

• Determine which data sources
• Determine how users know what sources are available and where to find them (catalog, one storage location or multiple locations)
• Define requirements for API and/or method of data access

These two repos are placeholders over a year old. I think they should be culled.

https://github.com/os-climate/esg_model_pipeline_notebook
https://github.com/os-climate/esg_data_pipeline_notebook

Requesting credentials for github user: mriefer
Requesting credentials for the bucket: redhat-osc-physical-landing-647521352890

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Requesting credentials for github user: Jonathan-Belhassen

Requesting credentials for the bucket: redhat-osc-physical-landing-647521352890

Email address to send a credential download link: [email protected]

Request checklist:

• [x ] User can establish connection with the bucket
• [ x] User can read files from the bucket
• User can write files to the bucket

Requesting credentials for github user: dunstanm

Requesting credentials for the bucket: itr

Email address to send a credential download link: [email protected]

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

The onboarding materials state:

All project contributors are expected to adhere to the Linux Foundation DCO Policy
https://wiki.linuxfoundation.org/dco

We need a reference document about how to make this work for our expected developer environments: Jupyter Notebooks running either on OS-Climate nodes or on developers' desktops. But also developers who use other IDEs and use GitHub Desktop and/or the GitHub CLI.

We also need instructions on how to fix DCO errors, and in particular, resources we can go to when we are not comfortable doing things to GitHub repos that are not part of the normal push/pull process.

Requesting onboarding for github user: MichaelTiemann

I already have MichaelTiemannOSC credentials tied to an os-climate.org email address, but that email pathway blocks SuperSet's concept of how to share dashboards via email. So I'm requesting alternate credentials tied to a gmail.com address in hopes that I can then share a SuperSet dashboard via THAT email pathway.

Request access to github team: I should only need odh-env-team membership (so I can access SuperSet).

View teams here: https://github.com/orgs/os-climate/teams

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• new user invited to os-climate github org
• user invited to requested teams

This Elyra tutorial exposes a lot of details about building the underlying infra for supporting data pipelines. Most of the demo notebooks take this infra for granted, leading to challenges when pipelines are ready to be implemented "for real".

For this EPIC we need to:

• Validate/update the basic Elyra tutorial to match current Op1st realities (we don't want to build on out-of-date information)
• Update Data Commons developer guidance
• update notebooks in Data Platform Demo that can exemplify good behavior
• update ingestion pipelines, tagging and/or releasing those that have been properly harmonized

Ideally, this will lead to regularization of how dependencies are managed and how data exploration with Jupyter Notebooks can lead to reproducible data transformation and management with Elyra pipeline nodes that are efficient in the libraries they load and the code they execute.

Hi team
Sapna and I met internally with our ESG team. A KPI-related topic was the need to distinguish and indicate if a KPI is an observation or an estimation.

•      Observations are harder to find, and less data is available.
  

•      Estimations might not be accurate and might not be accepted by regulators.
  

Any thoughts?

From Vincent:
Hi Ofer,

This discussion would certainly deserve opening an issue in our GitHub (under Data Commons).

My quick thoughts more from the platform point of view:
• Defining which data is estimation (as well as associated information such as date, basis for estimation, etc...) is metadata we will want to manage.
• Ability to have metadata tagging for estimations vs. observations is something that we can definitely plan to use as a basis for data access / security but also could be used in quality gates / checks in data pipelines.
• It would be wonderful to have an example of a data domains where different sources can provide estimations / observations of the same metrics, so we could conceptually discuss the required handling in one of our storming sessions.
Thanks for raising this item to attention.

Requesting onboarding for github user: Jonathan-Belhassen

Request access to github team: [os-climate-physical-risk-tool-project]

View teams here: https://github.com/orgs/os-climate/teams

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• [ x] new user invited to os-climate github org
• [x ] user invited to requested teams

Requesting credentials for github user:

Requesting credentials for the bucket:

Email address to send a credential download link:

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Process should include validation steps and automated alarming/notification in the event of upload failures, missing data, etc.

Requesting onboarding for github user: thierryc01

Request access to GitHub teams: Don't know.... the link below does not work.
I'm today interested in the projects related to OSC Data Commons, Trino database, and NLP stuff.

View teams here: https://github.com/orgs/os-climate/teams

Onboarding resources:

• Data Commons environment setup
• credentials.env file

Onboarding checklist:

• new user invited to os-climate github org
• user invited to requested teams

Requesting credentials for github user: mamurak

Requesting credentials for the bucket: a bucket that contains samples for running the os climate demo

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

Create a standard readiness template with all the steps as open tasks and then fill in the tasks as they are completed member by member. We can also create cards for workstreams with open task lists for member signoff. When member reaches appropriate level of OSR, they can sign off. When project receives all signoffs, we can schedule making the repo public.

Requesting credentials for github user: ChristianMeyndt

Requesting credentials for the bucket: redhat-osc-physical-landing-647521352890

Request checklist:

• User can establish connection with the bucket
• User can read files from the bucket
• User can write files to the bucket

A provider can allow and/or restrict access and use of specific data fields. The provider can permit and/or restrict the publication of derived data, such as a temperature score or a vulnerability/exposure metric.