talktome's Introduction

talktome

Just another PoC with local LLM and FastAPI

Built on Macbook M1 Air 2020 MacOS 14.4 primarily

Model Used

LaMini-Flan-T5-77M for primary execution testing.
Llama-2-7b-chat-hf for actual execution. Planned.

Tests

Primary backend test with - FastAPI.
Python formatter used - Black.
weight file is ignored for git. can be downladed from this link, 308 MB in size.
chathistory.txt has all the previous outcomes during the soft tests.

Project Structure

.
└── talktome/
    ├── README.md
    ├── main.py
    ├── stapp.py
    ├── index.html
    ├── model/
    │   ├── weight_file.bin
    │   └── utilities.extensions
    ├── assets/
    │   └── media_attached.extensions
    └── static/
        ├── index.html
        ├── script.js
        └── style.css

Usage

Docker

to build docker image, run: docker build -t lamini-app .

run docker container: docker run -p 8000:8000 lamini-app

locally

load server: uvicorn --host 0.0.0.0 main:app

Open: index.html

run streamlit: python3.10 -m streamlit run stapp.py

Flow of completition

0.1 - FastAPI local server test

load server: uvicorn main:app --reload

run primary test: python test.py

0.2 - simple streamlit test

load server: uvicorn --host 0.0.0.0 main:app

run streamlit: python3.10 -m streamlit run stapp.py

0.3.1 - trial with custom UI

load server: uvicorn --host 0.0.0.0 main:app

Open: index.html

0.3.2 - [FAILED] trial with custom UI

Requires fixing the UI, so that it can show the chain of conversation.

0.4 - Dockerfile included

talktome's People

Contributors

Watchers

talktome's Issues

Code generating LLM not implemented.

Code generating LLM are under LFS that takes quite some time to download, so tried to utilize small LM to show the PoC. Goal is to deploy LLMs like, LLAMA, StarCoder, WizardLLM and so on.

Prompt Security Features Implementation

Preventing non-code o/p is implemented only.

Implementation like,

Input Validation: Validate input parameters to prevent injection attacks or malformed input using FastAPI's built-in features.
Output Sanitization: Remove potentially harmful information from the output before returning it to the client.
Access Control: Implement mechanisms like authentication and authorization to restrict access to endpoints as needed.
Secure Coding Practices: Follow best practices like avoiding hardcoded credentials and sanitizing inputs to prevent injection attacks.
Review for Vulnerabilities: Regularly review code for vulnerabilities like SQL injection or XSS and address them.
Regular Updates and Patch Management: Keep dependencies up-to-date to patch security vulnerabilities promptly.
Monitoring and Logging: Implement logging for monitoring and audit purposes, and set up systems to detect suspicious activities.
Encryption: Encrypt communication with HTTPS/TLS and sensitive data stored within the application.
Threat Modeling and Risk Assessment: Identify security threats and vulnerabilities specific to the application, and prioritize security measures accordingly.
Security Testing: Conduct penetration testing and vulnerability scanning, including automated testing in CI/CD pipelines.
Secure Deployment and Configuration: Deploy securely, following best practices for server and network security, and apply the principle of least privilege.