Python scripts to import database schema information from different data sources into a custom database.
- Python 3.8+
- pip
- Source db (postgres, snowflake, etc.)
- Destination db (your custom db)
-
Clone the repository to your local machine.
git clone https://github.com/betocmn/LLM-import-and-document-postgres-schema.git
-
Install all requirements
pip3 install -r requirements.txt
-
Use the env.example file to create a .env and fill it out.
-
Start by generating a new organization_id and importing all basic schema from the source. This command will output an "organization_id" that you will need next.
python3 scripts/import_from_postgres.py
-
Generate the AI documentation for all tables and columns. The first argument is the "organization_id" from before and the second is the number of tables to generate (because openAI requests take a while, you might want to do in chunks).
python3 scripts/document_from_postgres.py 8 30
-
Generate the LLM embeddings for this new org. The argument is the "organization_id" from before.
python3 scripts/generate_embeddings.py 8