closechoice / dataframetrainer Goto Github PK

We need a table that saves all metadata of a user and a table that connects the user to the challenges he/she tried or completed.

add pre-commit to check formatting + tests

The "Sign Out" button is larger than "Sign In" and therefore everything gets skewed

The Code Editor page has various flaws:

• the output is not well formatted (we would look for something like jupyter notebook)
• just the returns are displayed and prints are not
• there is no resetting of the element that shows the output

Nice to have:

• Better shortcuts for the CodeEditor (e.g. ctrl + enter for code execution)

There should be one script which we use to challenges into the database.

• I would prefer if the variable names inside the docker containers were equal to them outside the container
• Variables in a group should be prefixed. e.g. DB_PORT or POSTGRES_PORT instead of just PORT since we also have frontend and backend port variables

Since the get_next_challenge endpoint only provides the POST method we should find a better name for it

The graphic on the User-Flow wiki page is linked as

![next_challenge](https://github.com/CloseChoice/dataframetrainer/assets/31857876/986b2f27-a02c-4138-8ad8-5e231f59781f)

In the markdown file. Can you upload the svg file in the wiki repo and add it with a local path?

![next_challenge](/next_challenge.drawio.svg)

steps to reproduce:
start the current main branch (docker compose up + npm run dev) visit http://localhost:5173/new_challenge/TransformWithConditions.

After adding a df at the end of the code terminal and clicking Run or Test nothing happens in the green output terminal

As of today we can debug github actions with tmate. But it would be great to use https://github.com/nektos/act so that we can use the github workflows instead of a makefile to test these locally

Also update the documentation after this is done

Currently the hypothesis is logged to the console. We need to show the result to the user. This includes the following steps:

1. Save the output of the result in the pyodide file system (maybe as xml or something).
2. Retrieve the relevant information from the file once it is written
3. Display a ✔️ or ❌ to the user and show the falsifying example

1. Conceptualize a basic ELO based Recommender System
2. Allow A/B testing for the Recommender System

Create a route which returns all available challenges

• check that get_next_challenge works for not registered users (also add test)
• fix routes in frontend-svelte/vite.config.ts, to use backend_server and backend (from the frontend). Especially the hardcoded localhost should be removed (but I haven't found a way around this, maybe there's an issue with axios and vite's proxy config)
• remove unused sessionStore

Currently the passwords for the dev database is checked in this repo. This is ok for now, since we are in the development phase. But we should change this sooner than later

We need a cookiebanner where we'll be transparent of the user data.

1. Having a banner with some dummy text that stores the information once it is agreed/declined
2. Fill banner with all necessary information

If the user writes a function, e.g.

def transform(df):
    return _helper(df)

def _helper(df):
    ...

The helper function is currently not extracted.

After the merge of the dev branch (#78 ) a couple of problems remain

Still a couple of things are not working correctly:

• Test output is not written into the test tab
• Logging statements are written in the stdout pane
• No resetting of stdout pane
• the pyodide-worker.module.ts is not used, though it is the better code
• example generation does not work

Since we have a server, we would need to deploy the code via CI. At the beginning it would be great to deploy the test branch (needs to be created) to the server and hosted at https://www.test.dataframetrainer.com

1. create test branch from main
2. create a new ssh key and store public key on server
3. add ssh private key to github secrets
4. set up a new CI pipeline that uses the the ssh key and deploys to the server on merge to test branch
5. set up nginx config to host website on the desired target

Later on we want to have the main branch on https://www.dataframetrainer.com, though it is unclear if we should get another server for this or use nginx-proxy further material:

• https://sitegeist.de/blog/typo3-blog/docker-compose-setup-mit-nginx-reverse-proxy.html
• https://linuxhandbook.com/nginx-reverse-proxy-docker/#:~:text=Setting%20up%20Nginx%20as%20reverse%20proxy%20to%20deploy,Run%20other%20service%20containers%20with%20reverse%20proxy%20
• https://github.com/nginx-proxy/nginx-proxy

create challenges where one needs to read (and maybe even write) to csv/excel, etc.

We need write these files in the pyodide filesystem and run the tests against this. Note that the transform function might not be the best choice for this. Maybe we need to generalize this

• New challenges should be addable in a challenges folder, two functions (challenge + solution, hopefully with a descriptive name) in one file for each new challenge. Docstring can be supplied as additional information for the challenge
• make sure that the functions are tested automatically
• add tooling to add functions to the database, which are not there

use .env file for local development
use github secrets for productive deployment

The frontend code is an absolute mess at the moment, would be great to have this refactored nicely

We need some frontend tests to check that the core functionality is not broken.

docker compose up should be capable to create a db with all challenge names, at best directly from the folder/file structure
(something like COPY challenges FROM PROGRAM 'ls challenges | grep challenge'; might be a good starting point)

As of now the docker container needs to be setup manually. We need a script to do this automatically.

Needed to improve developer experience

We recently introduced the get_next_challenge endpoint. The frontend add a button to get the next challenge and make the post request with the userid in the body.

To improve rendering of the dataframes, jupyterlite looks like it's worth a try: https://jupyterlite.readthedocs.io/en/latest/. This issue is a followup and one possible solution to #60

The static_example() function should also return a dictionary with keys coresponding to the parameter names like the create_df_func() does

depends on #41

Once the frontend tests are written, we would need to integrate them into the build-container workflow

The documentation is out-of-date. Please update.

To use Github and Google as Credential providers we need to register our app with them
Github
Google

In the svelte-docker branch both prod and dev docker compose scripts make the root .env file variables accessible to the frontend container via env_file: .env in the respective compose yml files.
The variables are imported inside src/hooks/server.ts from $env/static/private. When running npm run dev in frontend-svelte/Dockerfile.dev all variables are avilable but when running npm run build in frontend-svelte/Dockerfile.prod the variables are not available.
Error Message:
RollupError: "PGHOST" is not exported by "$env/static/private", imported by "src/hooks.server.ts"

The cause of this is likely that vite uses Rollup only when building for production and not when building for development and it seems that the importing of environment variables differs for Rollup

We need a view where all challenges are displayed and where we are linked to the specific challenge

Currently the tables are created with scripts that are run at the creation of the container/volume. We would need a seperate python container that actually updates all the tables if the changed. The container can stop afterwards.

Just recently we introduced the get_next_challenge endpoint. Once a user successfully requests a new challenge, we should write it to the user_challenges table.

Also add a CI test for this (though it is not clear how this can be implemented)

If someone prints or outputs a dataframe it should look like in a jupyter notebook

If somebody creates a new account, we write the same information twice to the DB: active_expires and idle_expires in the session table. This is redundant but worse, it is the current time and not the time of session expiry

1. Create new backend endpoint post_challenge_results
2. Once the users test their code we send a response to the backend including the code and the hypothesis status.
3. Update the results in users_challenges (currently no code is supported here, and that is fine for now). Note that there should already be an entry for this challenge and the current session of the user, since this was created when the challenge was proposed (or clicked on in the browsing option)

We need an impressum since the creators of the page are living in Germany.
1 . We should add a link to a dummy impressum page somewhere (maybe in the navigation bar or at the bottom of the page?)
2. Fill the page with all mandatory information