uselotus / lotus Goto Github PK

Sentry Issue: PYTHON-DJANGO-CG

UniqueViolation: duplicate key value violates unique constraint "metering_billing_billing_organization_id_billing__09993474_uniq"
DETAIL:  Key (organization_id, billing_plan_id)=(1, ) already exists.

  File "django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)

IntegrityError: duplicate key value violates unique constraint "metering_billing_billing_organization_id_billing__09993474_uniq"
DETAIL:  Key (organization_id, billing_plan_id)=(1, ) already exists.

(21 additional frame(s) were not displayed)
...
  File "django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)

Is your feature request related to a problem? Please describe.
A company needs to know when a certain limit has been reached of usage for a given plan. For instance when the seat limit is reached and the user needs to upgrade to a new plan to add more seats.

Describe the solution you'd like
Adding a max-amount field to the plan component.

Describe the bug
When running docker-compose up, the env variables for Postgres are all "lotus", however they are being hardcoded in the docker compose file and if you change these env variables the backend fails with this error:

To Reproduce
Change the env variables to anything besides lotus.

Expected behavior
Changing env variables for postgres before running docker-compose up should create a postgres image with those config variables instead of the hardcoded "lotus" variables.
Screenshots

Dear,
after a look to the code, to my point of view providing a mix of Pipfile and pyproject.toml do not aims a gold standard.
Ideally, the build process have to generate a distributable archive such as a wheel.
The generated wheel is mount to the container and installed with help pip

Thus I would recommend to use poetry, which help a lot and ease this process

Best regards

Is your feature request related to a problem? Please describe.
Customers want to pay through paypal/braintree not just through Stripe

Describe the solution you'd like
Add braintree to our payment processor integrations. Similar to Stripe's integration.

This needs to be able to

• issue payments
• issue invoices
• receive webhooks on payment received
• import customers from Braintree and merge into lotus
• create customer in braintree

This should allow the next PSP integration to be much easier since these steps should be fairly reproducible and are already defined for Stripe in an extensible way in the backend.

The last bit of this will be creating a card in the frontend and connecting either via oauth or more easily (MVP) through a secret api key.

Describe the bug
Creating a non-superuser, doesn't then have the ability to login/authenticate.

Describe the bug
Need to create an active subscription if the subscription start_date is today.

Describe the bug
CSRF token verification fails when trying to login for the first time.

To Reproduce

1. Be logged out of Lotus.
2. Register a new account
3. Login in doesn't work
4. Refresh page and login works

Utilize an ETL framework (Singer, Airbyte) to enable export of our tables including

• invoices, customers, subscriptionrecords, plans, lineItems,

something along these lines:

https://medium.com/@ryangordon210/build-a-python-ci-cd-system-code-quality-with-pylint-f6dea78461c6

Describe the bug
Running dev and self-host on docker-compose doesn't work with our script. This is most likely an issue with our script or our docker not running on windows 64 correctly

To Reproduce
Steps to reproduce the behavior:

1. Follow contributing dev instructions
2. Run the ./scripts/dev.sh on the windows machine

Expected behavior
Docker compose should be running, including the backend container instead of erroring like below.
Screenshots

Desktop (please complete the following information):

• OS: Windows, potentially only windows 64
• Browser: N/A
• Version 0.9.1

Is your feature request related to a problem? Please describe.
Not everyone wants to put in a username and password.

Describe the solution you'd like
Google + github Oauth2 support using a django library.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• Replace dependency react-query with @tanstack/react-query
• Update github.com/posthog/posthog-go digest to 87b23fe
• Update github.com/uselotus/lotus/go/pkg digest to 0ea71f7
• Update dependency @types/react-syntax-highlighter to v15.5.11
• Update dependency depcheck to v1.4.7
• Update dependency django-extensions to v3.2.3
• Update dependency django-grappelli to v3.0.9
• Update dependency jinja2 to v3.1.3
• Update dependency prettier to v2.8.8
• Update dependency psycopg2 to v2.9.9
• Update dependency psycopg2-binary to v2.9.9
• Update dependency pylint to v2.17.7
• Update dependency react-content-loader to v6.2.1
• Update dependency react-query to v3.39.3
• Update dependency sql-formatter to v12.2.4
• Update dependency sqlalchemy to v2.0.29
• Update dependency ts-node to v10.9.2
• Update module github.com/lib/pq to v1.10.9
• Update redpanda-data/github-action action to v0.1.4
• Update Node.js to v18.20.2 (node, @types/node)
• Update Python
• Update dependency @ant-design/icons to v5.3.6
• Update dependency @nangohq/frontend to ^0.39.0
• Update dependency @types/lodash to v4.17.0
• Update dependency @types/node to v18.19.31
• Update dependency @vitejs/plugin-react to v2.2.0
• Update dependency ace-builds to v1.33.1
• Update dependency axios to v0.28.1
• Update dependency black to v23.12.1
• Update dependency boto3 to v1.34.93
• Update dependency braintree to v4.28.0
• Update dependency celery to v5.4.0
• Update dependency cronitor to v4.7.1
• Update dependency cypress to v12.17.4
• Update dependency dj-database-url to v2.1.0
• Update dependency django to v4.2.11
• Update dependency django-anymail to v10.3
• Update dependency django-celery-beat to v2.6.0
• Update dependency django-cors-headers to >=3.14,<3.15
• Update dependency django-model-utils to >=4.5,<4.6
• Update dependency django-money to >=3.4.1,<3.5
• Update dependency django-redis to v5.4.0
• Update dependency django-silk to v5.1.0
• Update dependency django-simple-history to <3.5.1
• Update dependency django-timezone-field to v5.1
• Update dependency django-vite to v2.1.3
• Update dependency djangorestframework to >=3.15.1,<3.16
• Update dependency djangorestframework-api-key to >=2.3,<2.4
• Update dependency drf-spectacular to v0.27.2
• Update dependency drf-standardized-errors to v0.13.0
• Update dependency eslint to v8.57.0
• Update dependency eslint-config-prettier to v8.10.0
• Update dependency eslint-plugin-import to v2.29.1
• Update dependency eslint-plugin-jsx-a11y to v6.8.0
• Update dependency eslint-plugin-react to v7.34.1
• Update dependency faker to v18.13.0
• Update dependency less to v4.2.0
• Update dependency less-loader to v11.1.4
• Update dependency locust to v2.26.0
• Update dependency mintlify to v1.3.0
• Update dependency model-bakery to v1.18.0
• Update dependency moment to v2.30.1
• Update dependency numpy to v1.26.4
• Update dependency openapi-typescript to v6.7.5
• Update dependency platformdirs to v3.11.0
• Update dependency posthog to v3.5.0
• Update dependency posthog-js to v1.130.0
• Update dependency pre-commit to >=2.21,<2.22
• Update dependency pyjwt to v2.8.0
• Update dependency pytest-cov to v4.1.0
• Update dependency pytest-django to v4.8.0
• Update dependency python-dateutil to >=2.9,<2.10
• Update dependency python-dotenv to >=0.21.1,<0.22
• Update dependency pytz to v2023.4
• Update dependency react-router-dom to v6.23.0
• Update dependency react-toastify to v9.1.3
• Update dependency reportlab to v4.2.0
• Update dependency ruamel.yaml to v0.18.6
• Update dependency ruff to v0.4.2
• Update dependency sentry-sdk to v1.45.0
• Update dependency sqlparse to v0.5.0
• Update dependency stripe to >=4.2,<4.3
• Update dependency tailwindcss to v3.4.3
• Update dependency usaddress-scourgify to v0.6.0
• Update dependency vite to v3.2.10
• Update dependency vite-plugin-imp to v2.4.0
• Update dependency whitenoise to v6.6.0
• Update dependency zustand to v4.5.2
• Update dnd-kit monorepo (@dnd-kit/core, @dnd-kit/utilities)
• Update docker.redpanda.com/vectorized/redpanda Docker tag to v22.3.25
• Update golang Docker tag to v1.22
• Update module github.com/labstack/echo/v4 to v4.12.0
• Update module github.com/redis/go-redis/v9 to v9.5.1
• Update module github.com/spf13/viper to v1.18.2
• Update module github.com/twmb/franz-go to v1.16.1
• Update nginx Docker tag to v1.26
• Update svix/svix-server Docker tag to v0.85
• Update typescript-eslint monorepo to v5.62.0 (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
• Update Node.js to v20 (node, @types/node)
• Update actions/cache action to v4
• Update actions/checkout action to v4
• Update actions/setup-node action to v4
• Update actions/setup-python action to v5
• Update dependency @ant-design/plots to v2
• Update dependency @ant-design/pro-components to v2
• Update dependency @dnd-kit/sortable to v8
• Update dependency @tanstack/react-query to v5
• Update dependency @tremor/react to v3
• Update dependency @vitejs/plugin-react to v4
• Update dependency antd to v5
• Update dependency axios to v1
• Update dependency black to v24
• Update dependency clsx to v2
• Update dependency cypress to v13
• Update dependency django to v5
• Update dependency django-activity-stream to v2
• Update dependency django-cors-headers to v4
• Update dependency django-grappelli to v4
• Update dependency django-timezone-field to v6
• Update dependency django-vite to v3
• Update dependency djangorestframework-api-key to v3
• Update dependency eslint to v9
• Update dependency eslint-config-prettier to v9
• Update dependency faker to v24
• Update dependency framer-motion to v11
• Update dependency gunicorn to v22
• Update dependency less-loader to v12
• Update dependency mintlify to v4
• Update dependency platformdirs to v4
• Update dependency pre-commit to v3
• Update dependency prettier to v3
• Update dependency pycountry to v23
• Update dependency pylint to v3
• Update dependency pytest-cov to v5
• Update dependency python-dotenv to v1
• Update dependency pytz to v2024
• Update dependency react-ace to v11
• Update dependency react-content-loader to v7
• Update dependency react-toastify to v10
• Update dependency redis to v5
• Update dependency sentry-sdk to v2
• Update dependency sql-formatter to v15
• Update dependency stripe to v9
• Update dependency svix to v1
• Update dependency typescript to v5
• Update dependency universal-cookie to v7
• Update dependency vite to v5
• Update dependency vite-tsconfig-paths to v4
• Update docker.redpanda.com/vectorized/redpanda Docker tag to v23
• Update svix/svix-server Docker tag to v1
• Update typescript-eslint monorepo to v7 (major) (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency @ant-design/plots to v1.2.6
• Update dependency @ant-design/pro-components to v1.1.25
• Update dependency autoprefixer to v10.4.19
• Update dependency babel-plugin-import to v1.13.8
• Update dependency dayjs to v1.11.10
• Update dependency postcss to v8.4.38
• Update dependency react-spinners to v0.13.8
• Update dependency @tanstack/react-query to v4.36.1
• Update dependency antd to v4.24.16
• Update react monorepo (@types/react, @types/react-dom, eslint-plugin-react-hooks, react, react-dom)
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Is your feature request related to a problem? Please describe.
Users want webhooks for these events:

• when a subscription ends/is cancelled
• when a subscription is created
• when a subscription is renewed
• customer created

Describe the solution you'd like
In the backend add this webhook functionality using our alerts framework and svix.

In the frontend simply add the checkbox in the create webhook url form.

Is your feature request related to a problem? Please describe.
Some users want the ability to use paddle instead of stripe for payment processing and invoicing.

Describe the solution you'd like
Create an integration for Paddle the same way the Stripe integration is currently created. This needs to be able to

• issue payments
• issue invoices
• receive webhooks on payment received
• import customers from paddle and merge into lotus
• create customer in paddle

This should allow the next PSP integration to be much easier since these steps should be fairly reproducible and are already defined for Stripe in an extensible way in the backend.

The last bit of this will be creating a card in the frontend and connecting either via oauth or more easily (MVP) through a secret api key.

This:

https://flower.readthedocs.io/en/latest/

Get Plans query (backend/api/views .. PlanViewSet, get_queryset method) was super slow originally despite some prefetching. Dug into the details of the nested relatiosn and managed to get it down from ~250 db hits and a response time of 1-1.5 second to ~180 db hits with a response time of 0.4-0.7 seconds. Would be good to dig into why it's making this many calls and try to bring it down further. Plans combine a bunch of different obejcts so we could reasonably end up needing ~20 queries but thats still an order of magnitude.

Sentry reference:
~250: https://sentry.io/organizations/lotus-ap/discover/prod:51f57f34f82947c6bbcc7b8d575749ce/?field=title&field=event.type&field=project&field=user.display&field=timestamp&name=All+Events&project=6617271&query=%2Fplans%2F&sort=-timestamp&statsPeriod=14d&yAxis=count%28%29

~180: https://sentry.io/organizations/lotus-ap/discover/lotus-dev:8b8c2bad184f408b9048ecc13d2c36cf/?field=title&field=event.type&field=project&field=user.display&field=timestamp&name=All+Events&project=6778627&query=&sort=-timestamp&statsPeriod=14d&yAxis=count%28%29

Currently, we have all the queries that power our metric aggregation stored in backend/metering_billing/aggregation. Though they are functionally working, there are some things worth considering and possibly fixing:

1. We are using jinja to template the queries and insert external variables. Is this the best tool for the job? Can we tidy up how the jinja renders the queries and enforce consistent styles (when inspecting queries there's a ton of funky looking whitespace)?
2. Is storing the queries as a string the best idea? Have no way of linting/formatting/checking the queries easily.
3. Can we identify ways to extract "common" snippets and turn them into stored procedures or find a way to combine the snippets as needed into a full fledged query? For example, we have the exact same statements used to identify the top_n customers in the total_daily_usage queries across counter, gauge, ands rate metrics.
4. We should decide on naming conventions: e.g. sometimes we call the time time_bucket, or time, or time_created. For similar concepts in queries across the files, we should enforce consistent naming.

If anyone wants to tackle one of these 4 / look into them and come up with a plan of action, might be a good idea to spin them out into their own issue.

Is your feature request related to a problem? Please describe.
Need better linting and auto formatting for any push.

Describe the solution you'd like
Do this for all our backend folder
Install ruff as our linter and run it everytime we push to a branch that will merge to main similar to our tests.
+
Add a black autoformatter whenever anything is pushed to main as a pre-commit action.
Describe alternatives you've considered
Potentially pylint but I think we would rather go with the trendy option of ruff.

We need to be able to have metrics that don't "reset" over a subscription period.

Is your feature request related to a problem? Please describe.
If an organization doesn't want to use Stripe to Invoice, they should be able to use an invoice generated from Lotus. Right now we have all of the information in a JSON but no physical display.

Describe the solution you'd like
Generate a PDF invoice. I gave attached example styles below, the goal is minimal. This should be executed in the generate invoice function and the invoice should then be stored in S3 if in the cloud or locally otherwise. The url for the invoice pdf should be stored in the invoice table under invoice_pdf

The invoice needs to include the issue date, due date, organization name and address, currency, and invoice number

line items should be able to be grouped and should have dates, descriptions, quantities, and prices

• Grouping of multiple subscription records should be similar to the draft invoice view.

Additional context
We will need to configure the S3 connection in new env variables so that self-hosted solutions can hook up their own S3 buckets with their own AWS accounts.

Is your feature request related to a problem? Please describe.
Loading pages and navigating between them isn't as fast as it could be.

Describe the solution you'd like

1. For most pages we should prefill the structure of the page and only have a loading symbol for data that needs to be loading. For example a table can have a loading spinner while the whole page shouldn't. Especially relevant in the settings tabs and when clicking on a customer.
2. In general can we speed up page loads to super fast.

Describe the bug
End date doesn't make sense to pass in to "api/subscriptions" post request.

Here is what needs to be fixed:

body for the post request should be:

customer -> customer_id
billing_plan -> plan_id
start_date -> start_date(optional) : on the backend if there is no start date passed in, then automatically make the start date now
end_date -> remove (backend should figure out what the end date is based on the plan)

We are having trouble waiting for the frontend to start in our GitHub Actions workflow in order to do Cypress testing. Here are the details:

We have a Django backend and a Vite/React frontend.

The backend is started successfully with pipenv run python manage.py runserver 0.0.0.0:8000 &.

We use curl to check if the backend is available with curl -IsS http://localhost:8000/api/healthcheck/ >/dev/null. This works fine and we can confirm that the backend is available.

We use yarn run dev --port 3000 & to start the frontend, and it runs successfully.

We tried using curl to check if the frontend is available with curl -s -o /dev/null -w "%{http_code}" http://localhost:3000. However, this always fails with curl: (7) Failed to connect to localhost port 3000

We also tried using the wait-on property of the Cypress GitHub Action to wait for http://localhost:3000/login to become available. However, this did not work and we got the following error:

The response we received from your web server was:

  > 404: Not Found

We also tried setting --host 0.0.0.0 for yarn run dev --port 3000 &, but this didn't solve the issue.

We suspect that there might be a problem with the way we are starting the frontend or the way we are checking if it is available. We would appreciate any help or guidance on how to resolve this issue.

Relevant changes are in #493

We need to refactor our current prod backend Dockerfile, currently it reuses the build/dev stage for the prod stage, but this means if we include dev dependencies in the dev stage then the final image will be bloated

Eventually, event ingestion is going to be the bottleneck of our system. We are going to need some sort of event streaming database, a way of asynchronously processing requests, maybe an endpoint written in a different language. We want to architect the system intentionally and thoughtfully before starting to build.

Currently, we're inconsistent with our use of backslaches, which has caused some pain when we go to the wrong address and get unexpected behavior. Some goals:

1. going to ...endpoint and ...endpoint/ should do the same thing
2. Ensure consistency across nginx, Django backend settings, and react redirects

Is your feature request related to a problem? Please describe.
Users need to be able to signup without already being given an account password.

Describe the solution you'd like
Registration and signup page for both cloud and self-hosted versions

As described in this summary of different types of authorization headers, API keys should either be transmitted in the query string or as a request header. as such: API-Key: abcdefgh123456789. Currently, the API key is nested inside an authorization header with a different name: Authorization: Api-Key t55vwqtd.jrb6QAGHswOBrZX70Jfz2qItjbtzavbv. This is preventing us from properly documenting the authoerization process when building the docs.

Hi everyone,

We are looking to use Lago for our membership system but we have an advanced setup and are trying to see if there is a workaround in Lotus. We are an E-Learning company and sell subjects with video teaching and other features. Right now our products have multiple layers and we would like to know if our layering is possible in this system.

We have a Subject(Main Product), for example, Math. In each Subject, we want to have different tier levels with certain features in each tier.

Now, we have curriculums (Sub Product) in that subject, and we want to be able to attach the tiers available for that curriculum. We can have multiple curriculums (books) that have all the tiers or some of the tiers.

Is this something possible or is there a workaround to it?

Is your feature request related to a problem? Please describe.
When creating a plan with many components, the components fill up the page rather than existing in a set heigh container that can be scrolled up and down to view.

Describe the solution you'd like
A scrollable container around the components.

Is your feature request related to a problem? Please describe.
Some users use Go for their backend but dont' want to use raw requests to access Lotus.

Describe the solution you'd like
A full SDK written in Go, similar to our python and typescript SDKs.

Is your feature request related to a problem? Please describe.
I would like to support Lotus by implementing Uffizzi preview environments.
Disclaimer: I work on Uffizzi.

Uffizzi is a Open Source full stack previews engine and our platform is available completely free for Lotus (and all open source projects). This will provide maintainers with preview environments of every PR in the cloud, which enables faster iterations and reduces time to merge. You can see the open source repos which are currently using Uffizzi over here

Uffizzi is purpose-built for the task of previewing PRs and it integrates with your workflow to deploy preview environments in the background without any manual steps for maintainers or contributors.

We can go ahead and create an Initial PoC for you right away if you think there is value in this proposal.

Figure out how to give users the option to generate a webhook on invoice creation, both on the frontend and the backend.

Is your feature request related to a problem? Please describe.
The only response we get back if we input invalid tiers is the Tiers are not valid response. This can be confusing to debug or figure out what you did wrong.

Describe the solution you'd like
An message relevant to the type of validation issue, either ranges are not correct or units are not correct etc.

In addition, if the border around the relevant cell could become red, signaling the location of the error that would be an additional improvement.

Is your feature request related to a problem? Please describe.
Our frontend tests don't run automatically in github actions.

Describe the solution you'd like
Update github actions to run our already defined cypress tests on push to a PR that merges into main.

In addition add an auto linter on push to origin.

Is your feature request related to a problem? Please describe.
We need to be able to enable extensibility and allow the community to contribute template metrics that are general enough for any project.

Describe the solution you'd like
A folder of jsons that define a billable metric that we can then call with an endpoint and see all templates in the frontend.

I do not suggest we actually store these "template" metrics in our database, instead let's just keep a folder of jsons that define them.

Proposed Integration with Suger.Io. This would only work with Lotus Cloud.