ahmedmoalla / redditube Goto Github PK

Remove com.amoalla.redditube.client.impl.MediaPostParser class and use default Spring json parsing

Each service should have its own liquibase migrations instead of a project grouping all migrations
Actions:

• Remove liquibase-migrations project
• Do necessary refactoring for all migrations to work as it was

We need to be able to switch authentication providers without modifying the whole code.
We need to make the code as generic as possible and just switch implementation if possible.

Code to modify:
DexUserIdPrincipalJwtAuthenticationConverter#getUserIdPrincipal
DexService should implement an interface which we'll use instead of using this service directly

The gateway needs to behave like a resource server. This will enable us to authenticate using jwt

Examples:

• LocalhostSecurityConfiguration and OAuthResourceServerConfiguration classes
• Exposing beans for monitoring via actuator
• etc...

• Dev url: https://dev-7464697-admin.okta.com/
• Use Okta by default on local development environment

Use a discovery server (Eureka) When in development to let services register themselves so we can use http://service-name instead of http://localhost:8080 .
This will help us when deploying to kubernetes later. We won't have to change the yaml files or create separate ones.

We will use OpenAPI Specification.
use this https://springdoc.org/#demos
Video presentation of the library: https://www.youtube.com/watch?v=utRxyPfFlDw&feature=youtu.be&ab_channel=SpringI%2FO
Result example: http://158.101.164.60:8081/swagger-ui/index.html?configUrl=%2Fv3%2Fapi-docs%2Fswagger-config&urls.primaryName=users

The result we are looking for is like the example. We need a list of definitions, one for each service.
The use of OpenAPI Specification will let us generate clients for our services.
Example of usage in intellij: https://www.jetbrains.com/help/idea/openapi.html?gclid=CjwKCAiAqJn9BRB0EiwAJ1Sztf4Ujj7rrbPPQJ0bbYbrD7nOaMKbCJ_PCP0ErJ_1_Qhd5QGGOk7FOhoC4YkQAvD_BwE

Access to swagger-ui should be exposed on the gateway

Handle Rate Limiting headers in reddit client. Maybe wait until rates a replenished before sending more requests.

• Subscription

  • Subscribe/Unsubscribe from a user or subreddit
  • Get a feed of posts of the users/subreddits a user is subscribed to

• Organize

  • Create folders/collections of images by theme or whatever
  • Can download the folder structure zipped
  • Star a post

Need to dockerize Dex to avoid cloning and building the project.
Dex need a database to start. Use docker compose to start both of them and make sure that Dex has access to Postgres

When connection is lost with reddit.com the library completely fails. We need some retry mechanism or to fail gracefully

Listen to NewMediaPostsAvailableEvents and store the received mediaposts as follows:

• Images, GIFs, videos stored in MinIO
• Other data is stored in Postgres

Add checks using SonarLint via Github actions on pull requests and prevent them of being merged if there is issues

• Have a centralized system to regroup error events on a dashboard
• This will help us in detecting errors and implement fixes to recover

Tasks:

• Implement error handling in MediaPostsScraperTask to recover from failed FetchMediaPostsSubTask
• Use Spring ApplicationEventPublisher to publish error events