- Interface & Processing: OpenAI API,
- Dataset: Refined an existing dataset picked up from Kaggle, can be found here: https://www.kaggle.com/datasets/paramaggarwal/fashion-product-images-dataset
- Vector Database: Pinecone to store vector embedding of the products data of the kaggle dataset
- Backend: Python FastAPI
- Frontend: ReactJS
- Shift from traditional search to conversational fashion discovery.
- Need for personalized and trend-aligned outfit recommendations.
- Enhancing user experience and confidence in fashion choices.
- LLM-powered fashion outfit generator.
- Analyses user preferences, purchase history, social media trends.
- Generates complete, well-coordinated outfits.
navigate to the backend repository, create a virtual python enviroment.
python -m venv env
Activate the enviroment
env/Scripts/activate
Navigate to the file of requirements.txt and run Install Backend dependancies using
pip install -r requirements
Navigate to the root backend directory and run the command of
python server.py
to start the FastAPI backend server.
Create a .env file at the root director which should have the following variables-
PINECONE_API_KEY = "your api key"
PINECONE_ENVIRONMENT = "your env"
OPEN_AI_API_KEY = "YOUR_API_KEY_HERE"
USER_ID = os.getenv("YOUR_INSTAGRAM_ID_HERE")
ACCESS_TOKEN = os.getenv("ACCESS_TOKEN_FOR_INSTAGRAM")
Navigate to the frontend folder and install the dependancies using
npm install
command,
run the server locally using npm start
This will start your frontend server on the default localhost:3000
User insights : Gathering insights about the user, before suggesting outfit.
Step 1: Insights from User Prompt : Problem : Understanding what user wants from the prompt. Solution : User's prompt is processed to generate K-V pairs using OpenAI API to extract out what the user actually wants.
Step 2: Insights from Purchase History: Problem : Understanding User's preference Solution : By analysing the user purchase and view history, We identify user's preference in colours, brands, etc.
Step 3: Insights from Latest Fashion Trends: Problem : Finding & identifying latest trends from social media. Solution : Top trending fashion posts are scrapped from Instagram, Pinterest, etc. Which are processed to generate a dataset, later stored in a CSV file.
Searching the VectorDB: The three type of insights generated in the previous step are combined to generate a query to be used for searching in pinecone
Step 4: Processing Insights: Problem : We need to process K-V pairs from insights to form a usable search query for pinecone Solution : We combine the K-V pairs from the insights to generate search query & filters for pinecone, later converted to embeddings for searching.
Step 5: Searching in Database: Problem : Searching our database to find matching outfits for user query. Solution :. We upsert vector embeddings for both text & images of our dataset to Pinecone. Embedding generated from query is searched on Pinecone. Now a multi modal search is performed ( CLIP – for image embeddings, BM25 – to generate text embeddings)
Step 6: Serving user: Problem : Serving the results through Backend Solution : Using FastAPI to serve Pinecone search results to the user via a POST request.
Step 7: Incorporating the user requested outfit changes: Problem: Making changes to the outfit according to the user's changing prompt. Solution: GPT API identifies what needs to be changed and based on that VectorDB is searched again.
-
DB : Database
-
K-V Pairs : Key – Value pairs
-
CLIP : Contrastive Language-Image Pretraining : is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3.
-
BM25 : Best Match 25 : A term based ranking model that aims to provide accurate and relevant search results by scoring documents based on their term frequencies and document lengths.
-
GPT : Generative Pre-Trained. It is a Large Language Model developed by openAI.
-
- Users receive tailored outfit suggestions based on what they have described in the prompt and their individual preferences, including style, color choices, and past purchase history.
- The generator's analysis of user data ensures that every outfit recommendation aligns with the user's unique fashion taste.
-
- The outfit generator taps into social media trends and influencer styles to provide users with up-to-date fashion recommendations.
- Users can stay current with the latest trends, ensuring that their outfits are both personalized and in line with the fashion zeitgeist.
-
- Users can engage in a conversational manner to provide feedback on recommended outfits.
- The generator adapts based on user preferences, allowing them to tweak and modify outfits until they're satisfied.
Generating a personalized outfit based on user prompt, social media trends and user preferences.
Gathering insights about the user, before suggesting outfit.
-
- Insights from User Prompt :
- Problem : Understanding what user wants from the prompt.
- Solution : User's prompt is processed to generate K-V pairs using OpenAI API to extract out what the user actually wants.
- Insights from User Prompt :
-
- Insights from Purchase History:
- Problem : Understanding User's preference
- Solution : By analysing the user purchase and view history, We identify user's preference in colours, brands, etc.
- Insights from Purchase History:
-
- Insights from Latest Fashion Trends:
- Problem : Finding & identifying latest trends from social media.
- Solution : Top trending fashion posts are scrapped from Instagram, Pinterest, etc. Which are processed to generate a dataset, later stored in a CSV file.
- Insights from Latest Fashion Trends:
The three type of insights generated in the previous step are combined to generate a query to be used for searching in pinecone.
-
- Processing Insights
- Problem : We need to process K-V pairs from insights to form a usable search query for pinecone
- Solution : We combine the K-V pairs from the insights to generate search query & filters for pinecone, later converted to embeddings for searching.
- Processing Insights
-
- Searching in Database:
- Problem : Searching our database to find matching outfits for user query.
- Solution : We upsert vector embeddings for both text & images of our dataset to Pinecone. Embedding generated from query is searched on Pinecone. Now a multi modal search is performed ( CLIP – for image embeddings, BM25 – to generate text embeddings).
- Searching in Database:
-
- Serving user
- Problem : Serving the results through Backend
- Solution : Using FastAPI to serve Pinecone search results to the user via a POST request.
- Serving user
-
- Incorporating the user requested outfit changes
- Problem: Making changes to the outfit according to the user's changing prompt.
- Solution: GPT API identifies what needs to be changed and based on that VectorDB is searched again.
- Incorporating the user requested outfit changes
-
Limited dataset : Currently our searchable dataset is limited(42K products only).
-
Static purchase history : Currently the purchase history is static and is not updated automatically.
-
Scrapper & processing of trends data : Our scrapper works only for Instagram and the processing of scrapped data to csv is done manually.
-
Virtual Try-On : Virtual try-on allows you to see how clothes looks on the user before they buy it. It would use diffusion generative AI models like stable diffusion with control net to create photorealistic images of clothes on people with different body shapes, sizes, skin tones, and hair types.
-
Multi-Channel Integration: Extending the outfit generator's functionality to other communication channels, such as messaging apps and voice assistants, can enhance user convenience and accessibility
-
Global Expansion: Collaborating with fashion influencers, designers, and brands from around the world can help tailor the outfit generator's recommendations to diverse cultural and regional tastes.
-
Continuous Learning: Building models that learn and adapt from user feedback and preferences over time can create a more personalized and accurate outfit recommendation experience.