resonatecoop / api Goto Github PK

We need to copy over the code from the uploading repository (https://github.com/resonatecoop/upload)

• upload repository integrated
• audio uploading working
• backblazer uploading tested
• image uploading working
• email sending tested

Right now we have a ton of imports that look like ../../../../file-name.

This is a pain when moving files around. Is there no way to do this in Node JS in the year of our lord 2022?

https://stackoverflow.com/questions/10860244/how-to-make-node-js-require-absolute-instead-of-relative

https://gist.github.com/branneman/8048520

Currently the legacy migration scripts need to be run outside of the docker container because we don't have access to the SSH scripts needed in them to tunnel into the server and access the databases on them.

Is there a way to have them reference the .ssh files in the user's host machine? This will be required for running the scripts on the server (where node is not installed).

Edit: is it easier to just install node?

https://community.resonate.coop/t/trackgroups-playlists-and-usergroups/3259/2

The seeding data is very disjointed right now and needs to be made so that it's all related to each other and traversable. That's not the case right now.

Editing user basics like:

• display name
• country
• email
• password

Investigate what the user/admin stuff copied over from dashboard is for, and why it needs to be separate from the other endpoints.

You can see here on the id server

That the head contains references to static.resonate.is

  <link rel="preload" as="font" crossorigin="" href="https://static.resonate.is/fonts/Graphik-Semibold.woff2">
  <link rel="preload" as="font" crossorigin="" href="https://static.resonate.is/fonts/Graphik-Semibold.woff">
  <link rel="preload" as="font" crossorigin="" href="https://static.resonate.is/fonts/Graphik-Regular.woff2">
  <link rel="preload" as="font" crossorigin="" href="https://static.resonate.is/fonts/Graphik-Regular.woff">

It's not clear where this is stored.

Currently our workers have to be run manually but we should create a container that automatically spins them up.

all the api-doc.js files redefine errors and bad requests. That probably can just appear in one place.

Similarly a lot of the apiDocs.js are just the same.

While working on the tests for the Api.ts/Artists endpoint, I needed to make some small changes to the database in order to get the tests to pass.

Here is the SQL that I used:

insert into plays(track_id, user_id)
values('b6d160d1-be16-48a4-8c4f-0c0574c4c6aa', '251c01f6-7293-45f6-b8cd-242bdd76cd0d');

insert into plays(track_id, user_id)
values('b6d160d1-be16-48a4-8c4f-0c0574c4c6aa', '251c01f6-7293-45f6-b8cd-242bdd76cd0d');

update track_groups
set enabled = true
where enabled is null;

yes, there are two 'insert into plays...' statements.

This data should be added to the database as part of the seeding process which is run using

yarn docker:seed:all

It's not important how / where this is done in the seeder script/s, only that it happens as part of the seeding process.

Depends on #13

Some ideas:

https://github.com/schemathesis/schemathesis
https://github.com/apiaryio/dredd

But it might be fairly straightforward to roll something ourselves that uses the definitions in swagger to build requests, orchestrated alongside seeding tables in sequelize.

This is an issue for tracking things that look a little odd or off with the current database. Testing against https://dev.resonate.coop/docs

• user_group.description and user_group.bio columns look like they should have a lot more data in them. But since these tables are currently just copied over from the user-api table verbatim and the current production app has the same missing data, I think this is because the original migration into the user-api database didn't capture all the descriptions from the rsntr_user_meta table.
• Old membership information see #66
• user.display_name is blank for most users, but it should probably feed in the user's user_group if there is a one-to-one link.
• Don't migrate links that are empty strings
• Is there anything in place to actually track user group membership of itself? For example, artists being part of labels? As part of this, it kind of looks like a lot of the migrations into the DB that are labeled as "Label" are actually artists. 🤔 #101
• Why is there still a rsntr_usermeta table? Let's delete it! #15

Particularly when a request (GET w/ params or POST w/ body) is invalid. It's not clear why.

Gonna post a reproducible post.

One date parsing library pls.

Should really be based on the clients provided for OAuth #1

currently a sleuthful user could discover a way to infinitely add credits to their account by just access a pre-redirect URL

They broke. We fix.

• For getting the tests running inside a GitHub Action, this looks like a likely next step cc @mattburnett-repo
• deploy-stage looks like it doesn't exit and fail when a command fails.

When running docker compose up 10 or so of this message appears in the console:

resonate-api | [WARN] This Redis server's default user does not require a password, but a password was supplied

I suspect it has something to do with Redis' "protected-mode" feature, but haven't yet found a good path to a solution.

Here are some links collected while chasing this issue down:
redis/ioredis#1249
https://redis.io/docs/manual/security/
https://hub.docker.com/_/redis/
https://www.howtogeek.com/devops/how-to-quickly-deploy-redis-as-a-docker-container/

id.resonate uses a standard library for this and makes it available to the API. Are there other ways?

Currently search happens using elasticsearch. Is that overkill? Should we simplify it? Or integrate it?

Thoughts @jmborden @peterklingelhofer

Just a small organizational code detail

tracks, trackgroups, tag, stream, search, resolve, labels, artists. are all controllers and should be grouped as such

now:

src
-- artists
-- config
-- db
-- jobs
-- labels

proposed

src
-- controllers
----- artists
----- labels
----- etc
-- config
-- db

I want to propose that we build a list of all endpoints, go through them, analyze which ones can be pruned (maybe it's none!) because they don't get used, which ones have code in them we think we'll need to keep, or which ones should be combined.

Luckily we've got an initial list of endpoints at swagger:

Edit: for editability we'll use this doc https://cryptpad.fr/code/#/2/code/edit/sKozHnJxRc89quzgLDQfszil/

swagger allows us to return proper errors. Let's leverage that.

The work that needs to be done here is to convert the following endpoint to use the format of the rest of the API. This probably needs to happen in a couple of stages:

1. Set up an index.js file that sets up koa appropriately.
2. For each endpoint create the relevant GET, PUT, etc functions
3. Create the Swagger documentation for those functions

• user/earnigs
• user/files
• user/admin/earnings
• user/admin/files
• user/admin/users
• user/admin/plays
• user/admin/profile
• user/admin/queues
• user/admin/trackgroups
• user/admin/tracks
• /user/admin/user

• Artist external links (youtube, vimeo, twitter, etc)
• Member status (role) (band, member) (now stored in ex user-api db)
• Nickname, country, description (now stored in ex user-api db)
• mylabels
• mybands
• track -> user relationships

What is needed / Scope of effort

We need the ability to conduct testing in its own environment, separately from the dev environment.

This environment should enable basic testing of api endpoints. Use Case / End-to-end / Integration testing is a nice-to-have, but should be postponed until after we establish basic, fundamental functional testing.

Initial discussion of this effort indicates that we need:

• A test database, seeded with reliable, consistent, repeatable test data. This database should be completely separated from the dev database.
• A set of environment variables to be used when starting up the test version of the api Docker container.
• A way to start the Docker container that is properly configured to run tests.
• The tests should run against the most current codebase in the main branch.
• The ability to run the tests inside of the Docker container, as well as outside of it.
• A process to update the tests and the test environment when the codebase changes (eg. new functionality is added, etc.)

How

The following list is basically a brain-dump of steps needed to move forward on this effort. This list can be revised as we progress through this effort:

• Create a way to spin up a test version of the Docker container.
• Create a consistent, repeatable process for seeding the test database with test data. This will likely be labor-intensive, but once completed it will resolve the most immediate obstacle to test development that we face.
• Create a docker:migrate:test script
• Create a docker-compose-test.yml config file. Use this in the Docker container startup.
• Check / update environment settings. These are probably in api/src/config/databases.js

Additions / Changes / Corrections to this short document are expected, as we progress through this effort.

Once these issues are resolved, and we can reliably run tests against the api code, we should work on a process to keeping the tests current / up to date / reflective of updates to the api code.

There's a lot there we should capture.

Is there anything in place to actually track user group membership of itself? For example, artists being part of labels? As part of this, it kind of looks like a lot of the migrations into the DB that are labeled as "Label" are actually artists.

Edit: made the decision to do the following:

1. Create a user_group_members table.
2. on import from user-api db, import all user_groups as artists.
3. fetch all labels from rsntr_usermeta meta_key = 'mylabel'. This gives values user_id (member) of meta_value (label).
4. Find the label by user_group.legacy_id. Change that label to be of type label (3).
5. Find the artist by user_group.legacy_id. Insert them into the user_group_members table.

• Set up a media volume that receives images and audio (or be able to reference the current media volume?)
  Currently media is stored against the prod2-upload resonate server, in the media volume. It's not obvious if we can move a volume, but that might be something we can ask support for? We'd likely have to ask a couple of times though as we test and migrate data, which is a drag.
• Move images and audio from the old media volume
  Possibly with scp or rsync?

Probably just a column isMember on the User model

• Add new audio files to seed process, moving them to /media. Remove audio files on down migration
• [ ]

While the docs load properly, when you open an endpoint, it pops up an error saying it can't find Error definitions, and requests don't go through properly.

1. go to http://localhost:4000/docs
2. Click on the {id} route for users. Error pops up at top of screen.
3. Enter a user uuid. Request fails and sends to http://localhost:4000/users/:id instead of http://localhost:4000/api/v3/users/:id.

Stripe, sounds like. First step is authentication.

Ideally this will all be doable through the API. We need to sketch out more what needs to happen here.

This basically creates a second layer so that users can create separate artists, and don't have to "role" switch.

Look at dashboard roles and see how they work. Example in api/src/user/plays.js in the dashboard repo.

https://github.com/resonatecoop/dashboard/blob/develop/api/src/user/plays.js#L67

Plan

We'll want to switch off using resonate-id for authentication and switch to something easy, like panva/node-oidc-provider.

Reasons for doing this is that we'll have only one API, one place to manage clients, etc.

Here's an example app:

https://github.com/panva/node-oidc-provider/blob/main/example/koa.js

As part of this we'll also need to provide, on this server, a way to log in and register for users.

Right now, requests to images (for example when loading /explore/artists on beam) will cause the API to throttle further requests from the client.

We should make an exception so that these endpoints, in the development environment, don't cause throttling.

Currently user profiles are a way to say whether a user is in a band or a label. It's not clear if this is used anywhere in the UI or anywhere else in the platform.

Those numbers should be ported to the new user database, or the list of e-mails should be saved to encourage people to switch over to the new system once a year has passed.

We should also move over any old shares.

• Add a legacyId and legacyTable column to share_transactions.
• Create a column called legacy_transaction_id and
• member_orders -> share_transactions. Use uid to map to a user.legacyId
  • shares > 0
  • uid = share_transactions.user.legacyId
  • shares = quantity
  • Use details to match to a user. Create a user if no user exist.
  • legacy_transaction_id = member_orders.txid
  • date = share_transactions.created_at
• rsntr_gf_entry -> user_memberships
  • form_id = 35
  • Only do this for rsntr_gf_entry.payment_status = Paid
  • rsntr_gf_entry.created_by -> user_memberships.user.legacyId, create user if they don't exist?
  • copy rsntr_gf_entry.transaction_id into legacy_transaction_id
  • user_memberships.membership_class_id = 4 (Listener)
  • date_created = user_memberships.created_at
• rsntr_gf_entry -> share_transactions
  • form_id = 45
  • Only do payment_status = Paid
  • payment_amount = quantity
  • legacy_transaction_id = rsntr_gf_entry.transaction_id.
  • created_by = share_transaction.user.legacyId
  • rsntr_gf_entry.created_by -> user_memberships.user.legacyId, create user if they don't exist?

Then we need to figure out what to do with the orders table, which currently doesn't get used.

With the migrations, the API makes references to an Artist table that no longer exists. The majority of that now exists in the UserGroup table.

Make sure the controllers for artists reflect this.

Whatever our final data format ends up being

• Keep in mind that passwords should just transfer! Ideally people won't have to reset.

Current system has an SSL script that does this:

#!/bin/bash

COMPOSE="/usr/local/bin/docker-compose -f ./docker-compose-prod.yml --no-ansi"
DOCKER="/usr/bin/docker"

cd /home/apiserver/api

$COMPOSE run certbot renew && $COMPOSE kill -s SIGHUP webserver
$DOCKER system prune -af

 id                  |          created_at           | updated_at | deleted_at |               user_id                |         membership_class_id          |               
                                                                                                              membership_class                                                                
                                                              |       subscription_id        |             start             |              end              

Which keeps track of user membership.

subscription_id tracks the object they bought in Stripe.

Current roles in Resonate:

Note that we've also got these membership_classes

              id                  |          created_at           | updated_at | deleted_at |   name   |     product_id      |            price_id            

--------------------------------------|-------------------------------|------------|------------|----------|---------------------|--------------------------------
47523bb9-ce28-48a6-8327-e59bc28587cc | 2022-08-18 16:36:05.491964+00 | | | Listener | prod_LKF4kqYbTsNL6B | price_1KdaZyEPjyhP24QbJeJFqiGC
5ba32398-c7ab-425f-acba-7878a542b9a8 | 2022-08-18 16:36:05.491964+00 | | | Artist | prod_LKF3Uia6W8svBt | price_1KdaZaEPjyhP24QbKMV8Dpgg
3439daf3-6e5d-41cf-b466-8ab3872d1dae | 2022-08-18 16:36:05.491964+00 | | | Label | prod_LKF3LeCcpBOp2a | price_1KdaZSEPjyhP24QbMcjOjgAh

which seems more related to stripe

https://github.com/marketplace/actions/hetzner-deployer