cbonoz / cortx22 Goto Github PK

Effortlessly move and onboard csv data between Cortx and IPFS into autogenerated starter Jupyter lab notebooks.

Built for the CortX 2022 hackathon on devpost: https://seagate-cortx-hackathon2022.devpost.com/

Demo (youtube): https://youtu.be/tK_bkdXqsmw

There are not a lot of great web tools that support moving between a hosted s3 instance and IPFS. I wanted to build a prototype app that made it easy to transfer csv data between IPFS and S3, and immediately import that data into a functioning jupyter notebook directly from a web interface.

FreeSheet is an app to help you manage the world of IPFS-hosted data sets and authorized AWS Cortx instances effectively from a web UI.

FreeSheet also 'onboards' any of your files into dynamically generated jupyter notebooks so you can start working with that data immediately.

FreeSheet supports three main actions:

• Upload - Upload a csv of data either directly to your CortX instance or IPFS.
• Exchange - push data sets effortlessly from your Cortx or AWS-hosted node to IPFS.
• Onboard - Onboard your IFPS data sets immediately into runnable starter Jupyter notebooks.

FreeSheet requires both the front and backend service to be running with the access keys below available.

Define the following environment variables:

    REACT_APP_STORAGE_KEY={YOUR WEB3 STORAGE KEY} # For connecting/uploading to IPFS.

Generate a web3.storage key here!

From the root folder: yarn; yarn start

The client (website) should now be running on port 3000.

Define the following environment variables:

    FS_S3_URL={YOUR AWS S3 SERVER URL} #  (ex: on seagate/cortx)
    FS_ACCESS_KEY={YOUR AWS ACCESS KEY} # cortx access key (for use with s3 sdk)
    FS_SECRET_KEY={YOUR AWS SECRET KEY} # cortx secret key (for use with s3 sdk)

From the /server folder: yarn; yarn start

The server should now be running on port 3001.

You can also create a bucket in your conncted CortX instance using the command: yarn create

If the front end is able to successfully connect to your running CortX or S3 instance, you should see a 'Connected' indicator here with the ability to select a bucket from the dropdown here. If you are not connected, a 'Not connected' indicator will show.

After this, you're ready to use FreeSheet as an interface to S3 and IPFS!

• ReactJS
• AWS sdk with Cortx
• Node.js / Fastify

• Integrating csv upload with the different data sources.
• Creating a simple interface
• Plugging in variable data source urls into pre-generated notebooks

It works.

• How to connect to a running CortX instance
• Dynamically generate a Python notebook from an arbitrary hosted data URL.
• Manage file uploads with Fastify (Node.js)

• Extend the autogenerated python utilities and code based on uploaded data sets.
• Push back data from IPFS into your S3 service.
• Perform pre-built visualizations of uploaded data sets within the FreeSheet app.
• Production hosting as a service (requires secure connections to your cortx/seagate system), or clone this repo and use it as a tool to connect to your S3 or CortX instance locally or within your private corporate network.