redditube's People
redditube's Issues
Media post service should be able to start without explorer service up
Add rudimentary documentation for the structure of the project and plans for the future
Remove MediaPostParser and replace with a Jackson mapped class
Remove com.amoalla.redditube.client.impl.MediaPostParser
class and use default Spring json parsing
Replace Dex with Keycloak
Liquibase migrations per service
Each service should have its own liquibase migrations instead of a project grouping all migrations
Actions:
- Remove
liquibase-migrations
project - Do necessary refactoring for all migrations to work as it was
Make the project independent from Dex
We need to be able to switch authentication providers without modifying the whole code.
We need to make the code as generic as possible and just switch implementation if possible.
Code to modify:
DexUserIdPrincipalJwtAuthenticationConverter#getUserIdPrincipal
DexService
should implement an interface which we'll use instead of using this service directly
Use Vault for credentials storage
Emit new media posts available to RabbitMQ
Expose endpoint to get completion percentage of current scraping job
Add ability to authenticate to the gateway via jwt
The gateway needs to behave like a resource server. This will enable us to authenticate using jwt
Implement SubscriptionService.getFeed
Implement media posts organization into folders
Create a library that has common code used by all services
Examples:
LocalhostSecurityConfiguration
andOAuthResourceServerConfiguration
classes- Exposing beans for monitoring via actuator
- etc...
Try Okta instead of Dex as openid provider and use on local dev env
- Dev url: https://dev-7464697-admin.okta.com/
- Use Okta by default on local development environment
Replace occurences of service url by application name by using a discovery server
Use a discovery server (Eureka) When in development to let services register themselves so we can use http://service-name
instead of http://localhost:8080
.
This will help us when deploying to kubernetes later. We won't have to change the yaml files or create separate ones.
Add Swagger to all controllers via commons library
We will use OpenAPI Specification.
use this https://springdoc.org/#demos
Video presentation of the library: https://www.youtube.com/watch?v=utRxyPfFlDw&feature=youtu.be&ab_channel=SpringI%2FO
Result example: http://158.101.164.60:8081/swagger-ui/index.html?configUrl=%2Fv3%2Fapi-docs%2Fswagger-config&urls.primaryName=users
The result we are looking for is like the example. We need a list of definitions, one for each service.
The use of OpenAPI Specification will let us generate clients for our services.
Example of usage in intellij: https://www.jetbrains.com/help/idea/openapi.html?gclid=CjwKCAiAqJn9BRB0EiwAJ1Sztf4Ujj7rrbPPQJ0bbYbrD7nOaMKbCJ_PCP0ErJ_1_Qhd5QGGOk7FOhoC4YkQAvD_BwE
Access to swagger-ui should be exposed on the gateway
Add security with JWT to Grpc server
Try to use Spring Native for native image
Move constants to a Constant classes
Implement fallbacks for all services in gateway
Write integration tests
Handle reddit api Rate Limiting in reddit client library
Handle Rate Limiting headers in reddit client. Maybe wait until rates a replenished before sending more requests.
Remove duplicate scraped media posts
Implement mediaposts service
-
Subscription
- Subscribe/Unsubscribe from a user or subreddit
- Get a feed of posts of the users/subreddits a user is subscribed to
-
Organize
- Create folders/collections of images by theme or whatever
- Can download the folder structure zipped
- Star a post
Add MinIO to dev-env
Dockerize Dex
Need to dockerize Dex to avoid cloning and building the project.
Dex need a database to start. Use docker compose to start both of them and make sure that Dex has access to Postgres
Add a retry mechanism to BearerTokenProvider in reddit-client library
When connection is lost with reddit.com the library completely fails. We need some retry mechanism or to fail gracefully
Check PR built successfully before merge
Store MediaPosts in MinIO and Postgres
Listen to NewMediaPostsAvailableEvents
and store the received mediaposts as follows:
- Images, GIFs, videos stored in MinIO
- Other data is stored in Postgres
Add test coverage checks in PRs
Add Sonar checks in pull requests
Add checks using SonarLint via Github actions on pull requests and prevent them of being merged if there is issues
Add error handling mechanism for all services
- Have a centralized system to regroup error events on a dashboard
- This will help us in detecting errors and implement fixes to recover
Tasks:
- Implement error handling in
MediaPostsScraperTask
to recover from failedFetchMediaPostsSubTask
- Use Spring ApplicationEventPublisher to publish error events
Implement the scraper
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.