https://github.com/databrickslabs/dolly

For example:

Hardcoded providers.

• OpenAI
• CohereAI
• ...

db.create_model(
    provider='openai',
    route='/new_model',
    params=[''],
    output='JSON',
)

In superduperdb.vector_search.pinecone.hashes create a class implementing search using pinecone. In addition, add support/ a flag to models to toggle on/ off adding vectors to pinecone during ingestion.

Use OpenAI python API: https://platform.openai.com/docs/api-reference/authentication?lang=python.

superduperdb.models.openai.wrapper.OpenAIBase.predict
superduperdb.models.openai.wrapper.OpenAIBase.predict_one

One configurable class for each type of OpenAI task - embeddings/ generation/ chat...

Depends on #68.

Deepspeed is a library for performing training and inference using model parallelism, with low code overhead. Because the code modifications are minimal, this may provide a nice modular way of doing large scale training for SuperDuperDB. This is useful if we want to enable training using Dolly 2.0.

• Create a concept, inference and training option to enable support for deepspeed.
• Benchmark using Dolly 2.0 example https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm

Dolly 2.0 is a fully permissive model developed by Databricks, shipping with weights and training data. We should be able to support self hosting of Dolly 2.0 at least intially for inference.

https://www.databricks.com/resources/webinar/build-your-own-large-language-model-dolly/thank-you?itm_data=dollyblog-intextcta-dollywebinar

https://huggingface.co/databricks/dolly-v2-12b

It's not clear if there's extra work here, or whether this can be supported with the right size hardware etc.. We may need to support saving a model in another way.

In order to make this type of thing work:

pip install superduperdb[all]
pip install superduperdb[torch]
pip install superduperdb[sklearn,pillow]

We would probably need some kind of lazy import scheme.

Overlaps with #68.

Llamaindex allows users to connect a vector store: https://gpt-index.readthedocs.io/en/latest/how_to/integrations/vector_stores.html. Add the right class/ connector to enable this functionality with Llamaindex.

https://python.langchain.com/en/latest/modules/chains/index_examples/vector_db_qa_with_sources.html

Create a class/ interface, which enables a SuperDuperDB collection to be passed to LangChain.

Depends on SuperDuperDB/roadmap#13. Create the minimal additional logic to enable DuckDB integation.

URI is a more accurate description of what superduperdb.utils.Downloader provides. Refactor naming,
and provide corresponding support for s3://.

Create an abstract base class.

Some libraries just work "out of the box". For example, once we have integrated sklearn then xgboost should work with no additional modifications. Make sure this is documented.

Use CohereAI python API: https://cohere-sdk.readthedocs.io/en/latest/.

superduperdb.models.cohereai.wrapper.CohereAIBase.predict
superduperdb.models.cohereai.wrapper.CohereBase.predict_one

One configurable class for each type of CohereAI task - embeddings/ generation/ chat...
Support is available for asyncio. Use this for batch processing.

Depends on #68.

Log-levels
Separation of concerns
Etc..

Msgpack is faster than the serializers pickle and dill. Investigate and incorporate features if this makes sense.

This will mean updating Database._create_serialized_file.

Add an integration in superduperdb.models.transformers enabling support for arbitrary Hugging Face specification.

Just need to work out how to pass a retriever to the langchain.

from langchain import OpenAI

chain = RetrievalQAWithSourcesChain.from_chain_type(OpenAI(temperature=0), chain_type="stuff", retriever=docsearch.as_retriever())

https://python.langchain.com/en/latest/modules/chains/index_examples/vector_db_qa_with_sources.html

If imputations are simple enough, it should be possible to not to need to supply the target function.

E.g.

collection.create_imputation('<identifier>', '<model-name>', '<key-name>', '<target-key-name>')

Support for high performance ray server to serve searchers/ questions/ model passes etc..

Should be possible to save code from Jupyter notebooks since this may often be the preferred way of working. It should be possible to define the critical objects - types, models etc. from this environment, which requires dill support:

https://pypi.org/project/dill/#:~:text=About%20Dill&text=Serialization%20is%20the%20process%20of,also%20includes%20some%20additional%20features. t

https://blythed.github.io/Packaging-Deep-Learning-Models-with-Dill/

Create support for users to supply their own class with a .train() method and compatible with the generic __init__ method. This should allow a greater range of learning tasks, such as GAN learning etc..

In order to hold the hands of our users, somewhat, create specific instances of create_learning_task:

• create_vector_search
• create_classifier
• create_clustering
• create_autoregression

E.g. in the case of create_classifier, specify input/ target fields, and encoding model.

`

• More example notebooks
  • Custom trainer
  • More types of tasks
  • Something with time-series data
  • ...
• More explanation
  • Everything is a bit thin on the ground
  • ...

https://github.com/facebookresearch/faiss

It should be possible to toggle between vanilla table scan look up and efficient lookup with FAISS.
Open questions:

• How manage recomputing every time/ vs. saving. index and updating
• How to synchronize arbitrary measure supplied with limited options in FAISS
• User/ warnings/ caveats
• Documentation

Add feature to create model based on external API call (e.g. from openAI API).
Should allow user to add credentials etc.. flexibly for multiple services.

Example notebook, showing how to add own vectors with OpenAI, and search with them.

Depends on #33 and #68.

• extract voice memos as mp3
• create model transcribing these:
  https://pytorch.org/hub/snakers4_silero-models_stt/
• create model indexing the transcribed texts
• ask questions/ search through these texts

Create the minimal logic and classes to enable Apache Iceberg on s3 integration. Depends on SuperDuperDB/roadmap#13.

Is it sufficient to use the duckdb connector https://www.dremio.com/blog/3-ways-to-use-python-with-apache-iceberg/?

There are currently many options to use with SuperDuperDB. Not all users will necessarily want to install all of these. Make it possible to install a base, and then power ups.

Here's an example:

https://github.com/lf1-io/padl-extensions/blob/main/setup.py

pip install superduperdb
pip install superduperdb[lightning]
pip install superduperdb[transformers]
pip install superduperdb[all]
etc...

There are points in the code where we try and infer the type of an object. This should be done in a lazy way with error handling, since otherwise we'll get errors if packages haven't been installed.

• numpy arrays
• torch tensors
• pillow images
• mp3/ wav etc.