Build a question answering system based on a given collection of documents with open-source LLMs

License: Other

Python 100.00%

hls-llm-doc-qa's Introduction

Biomedical Question Answering over Custom Datasets with 🦜️🔗 LangChain and Llama 2 from Hugging Face 🤗

Large Language Models produce some amazing results, chatting and answering questions with seeming intelligence. But how can you get LLMs to answer questions about your specific datasets? Imagine answering questions based on your company's knowledge base, docs or Slack chats. The good news is that this is easy with open-source tooling and LLMs. This example shows how to apply LangChain, Hugging Face transformers, and open source LLMs such as Llama-2 from Meta. This example can make use of any text-generation LLM or even OpenAI with minor changes. In this case, the data set is a set of freely available published papers in PDF format about cystic fibrosis from PubMed, but could be any corpus of text.

[email protected]

© 2022 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the Databricks License [https://databricks.com/db-license-source]. All included or referenced third party libraries are subject to the licenses set forth below.

library	description	license	source
transformers	Transformers ML	Apache 2.0	https://github.com/huggingface/transformers
sentence-transformers	Embeddings with BERT	Apache 2.0	https://github.com/UKPLab/sentence-transformers
langchain	LLM Applications	MIT	https://github.com/hwchase17/langchain
chromadb	Embedding Database	Apache 2.0	https://github.com/chroma-core/chroma
pypdf	Reading PDF files	Custom Open Source	https://github.com/py-pdf/pypdf
pycryptodome	Cryptographic library for Python	BSD 2-Clause	https://github.com/Legrandin/pycryptodome
accelerate	Train and use PyTorch models with multi-GPU, TPU, mixed-precision	Apache 2.0	https://github.com/huggingface/accelerate
unstructured	Build custom preprocessing pipelines for ML	Apache 2.0	https://github.com/yaml/pyyaml
sacremoses	Tokenizer, truecaser and normalizer	MIT	https://github.com/hplt-project/sacremoses
ninja	Small build system with a focus on speed	Apache 2.0	https://github.com/ninja-build/ninja
pytorch-lightning	Lightweight PyTorch wrapper	Apache 2.0	https://github.com/Lightning-AI/lightning
xformers	Transformers building blocks	BSD 3-Clause	https://github.com/facebookresearch/xformers
triton	Triton language and compiler	MIT	https://github.com/openai/triton/

Getting started

Although specific solutions can be downloaded as .dbc archives from our websites, we recommend cloning these repositories onto your databricks environment. Not only will you get access to latest code, but you will be part of a community of experts driving industry best practices and re-usable solutions, influencing our respective industries.

To start using a solution accelerator in Databricks simply follow these steps:

Clone solution accelerator repository in Databricks using Databricks Repos
Attach the RUNME notebook to any cluster and execute the notebook via Run-All. A multi-step-job describing the accelerator pipeline will be created, and the link will be provided. The job configuration is written in the RUNME notebook in json format.
Execute the multi-step-job to see how the pipeline runs.
You might want to modify the samples in the solution accelerator to your need, collaborate with other users and run the code samples against your own data. To do so start by changing the Git remote of your repository to your organization’s repository vs using our samples repository (learn more). You can now commit and push code, collaborate with other user’s via Git and follow your organization’s processes for code development.

The cost associated with running the accelerator is the user's responsibility.

Project support

Please note the code in this project is provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects. The source in this project is provided subject to the Databricks License. All included or referenced third party libraries are subject to the licenses set forth below.

Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support.

hls-llm-doc-qa's People

Contributors

Stargazers

Watchers

hls-llm-doc-qa's Issues

Traceback Error in 003 QA in CMD 17

I seem to get these errors when using the questions in the cells

: in _fwd_kernel:21

KeyError:
('2-.-0-.-0--394352f6a8351feaac334fbb8cc63fa4-46c7c5d46afed8316facd72e7e581bec-4da92a6240b2b826fdd8860ba75a31b4-39e
3c68a052760cc345a9147b0d68f7d-5c5e32ff210f3b7f56c98ca29917c25e-06f0df2d61979d629033f4a22eff5198-4ac47e74762ba6a774c
ceea0e1e75ae6-13b7ffc189bd9fba7696034bbcfee151', (torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32,
torch.bfloat16, torch.float32, torch.float32, 'fp32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32',
'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32'), ('vector', True,
128, False, False, True, 128, 128), (True, True, True, True, True, True, True, (False,), (True, False), (True,
False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False),
(True, False), (False, False), (True, False), (True, False), (True, False), (True, False), (True, False), (False,
False), (False, False), (True, False), (True, False), (False, False), (False, False)))

During handling of the above exception, another exception occurred:

in :1 │
│ in answer_question:3 │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/base.py:140 in call │
│ │
│ 137 │ │ │ ) │
│ 138 │ │ except (KeyboardInterrupt, Exception) as e: │
│ 139 │ │ │ run_manager.on_chain_error(e) │
│ ❱ 140 │ │ │ raise e │
│ 141 │ │ run_manager.on_chain_end(outputs) │
│ 142 │ │ return self.prep_outputs(inputs, outputs, return_only_outputs) │
│ 143 │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/base.py:134 in call │
│ │
│ 131 │ │ ) │
│ 132 │ │ try: │
│ 133 │ │ │ outputs = ( │
│ ❱ 134 │ │ │ │ self._call(inputs, run_manager=run_manager) │
│ 135 │ │ │ │ if new_arg_supported │
│ 136 │ │ │ │ else self._call(inputs) │
│ 137 │ │ │ ) │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/combine_documents/base.py:84 in _call │
│ │
│ 81 │ │ docs = inputs[self.input_key] │
│ 82 │ │ # Other keys are assumed to be needed for LLM prediction │
│ 83 │ │ other_keys = {k: v for k, v in inputs.items() if k != self.input_key} │
│ ❱ 84 │ │ output, extra_return_dict = self.combine_docs( │
│ 85 │ │ │ docs, callbacks=_run_manager.get_child(), **other_keys │
│ 86 │ │ ) │
│ 87 │ │ extra_return_dict[self.output_key] = output │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/combine_documents/stuff.py:87 in combine_docs │
│ │
│ 84 │ │ """Stuff all documents into one prompt and pass to LLM.""" │
│ 85 │ │ inputs = self._get_inputs(docs, **kwargs) │
│ 86 │ │ # Call predict on the LLM. │
│ ❱ 87 │ │ return self.llm_chain.predict(callbacks=callbacks, **inputs), {} │
│ 88 │ │
│ 89 │ async def acombine_docs( │
│ 90 │ │ self, docs: List[Document], callbacks: Callbacks = None, **kwargs: Any │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/llm.py:213 in predict │
│ │
│ 210 │ │ │ │ │
│ 211 │ │ │ │ completion = llm.predict(adjective="funny") │
│ 212 │ │ """ │
│ ❱ 213 │ │ return self(kwargs, callbacks=callbacks)[self.output_key] │
│ 214 │ │
│ 215 │ async def apredict(self, callbacks: Callbacks = None, **kwargs: Any) -> str: │
│ 216 │ │ """Format prompt with kwargs and pass to LLM. │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/base.py:140 in call │
│ │
│ 137 │ │ │ ) │
│ 138 │ │ except (KeyboardInterrupt, Exception) as e: │
│ 139 │ │ │ run_manager.on_chain_error(e) │
│ ❱ 140 │ │ │ raise e │
│ 141 │ │ run_manager.on_chain_end(outputs) │
│ 142 │ │ return self.prep_outputs(inputs, outputs, return_only_outputs) │
│ 143 │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/base.py:134 in call │
│ │
│ 131 │ │ ) │
│ 132 │ │ try: │
│ 133 │ │ │ outputs = ( │
│ ❱ 134 │ │ │ │ self._call(inputs, run_manager=run_manager) │
│ 135 │ │ │ │ if new_arg_supported │
│ 136 │ │ │ │ else self._call(inputs) │
│ 137 │ │ │ ) │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/llm.py:69 in _call │
│ │
│ 66 │ │ inputs: Dict[str, Any], │
│ 67 │ │ run_manager: Optional[CallbackManagerForChainRun] = None, │
│ 68 │ ) -> Dict[str, str]: │
│ ❱ 69 │ │ response = self.generate([inputs], run_manager=run_manager) │
│ 70 │ │ return self.create_outputs(response)[0] │
│ 71 │ │
│ 72 │ def generate( │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/chains/llm.py:79 in generate │
│ │
│ 76 │ ) -> LLMResult: │
│ 77 │ │ """Generate LLM result from inputs.""" │
│ 78 │ │ prompts, stop = self.prep_prompts(input_list, run_manager=run_manager) │
│ ❱ 79 │ │ return self.llm.generate_prompt( │
│ 80 │ │ │ prompts, stop, callbacks=run_manager.get_child() if run_manager else None │
│ 81 │ │ ) │
│ 82 │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/llms/base.py:134 in generate_prompt │
│ │
│ 131 │ │ callbacks: Callbacks = None, │
│ 132 │ ) -> LLMResult: │
│ 133 │ │ prompt_strings = [p.to_string() for p in prompts] │
│ ❱ 134 │ │ return self.generate(prompt_strings, stop=stop, callbacks=callbacks) │
│ 135 │ │
│ 136 │ async def agenerate_prompt( │
│ 137 │ │ self, │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/llms/base.py:191 in generate │
│ │
│ 188 │ │ │ │ ) │
│ 189 │ │ │ except (KeyboardInterrupt, Exception) as e: │
│ 190 │ │ │ │ run_manager.on_llm_error(e) │
│ ❱ 191 │ │ │ │ raise e │
│ 192 │ │ │ run_manager.on_llm_end(output) │
│ 193 │ │ │ return output │
│ 194 │ │ if len(missing_prompts) > 0: │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/llms/base.py:185 in generate │
│ │
│ 182 │ │ │ ) │
│ 183 │ │ │ try: │
│ 184 │ │ │ │ output = ( │
│ ❱ 185 │ │ │ │ │ self._generate(prompts, stop=stop, run_manager=run_manager) │
│ 186 │ │ │ │ │ if new_arg_supported │
│ 187 │ │ │ │ │ else self._generate(prompts, stop=stop) │
│ 188 │ │ │ │ ) │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/llms/base.py:436 in _generate │
│ │
│ 433 │ │ new_arg_supported = inspect.signature(self._call).parameters.get("run_manager") │
│ 434 │ │ for prompt in prompts: │
│ 435 │ │ │ text = ( │
│ ❱ 436 │ │ │ │ self._call(prompt, stop=stop, run_manager=run_manager) │
│ 437 │ │ │ │ if new_arg_supported │
│ 438 │ │ │ │ else self._call(prompt, stop=stop) │
│ 439 │ │ │ ) │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/langchain/llms/huggingface_pipeline.py:168 in _call │
│ │
│ 165 │ │ stop: Optional[List[str]] = None, │
│ 166 │ │ run_manager: Optional[CallbackManagerForLLMRun] = None, │
│ 167 │ ) -> str: │
│ ❱ 168 │ │ response = self.pipeline(prompt) │
│ 169 │ │ if self.pipeline.task == "text-generation": │
│ 170 │ │ │ # Text generation return includes the starter text. │
│ 171 │ │ │ text = response[0]["generated_text"][len(prompt) :] │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/transformers/pipelines/text_generation.py:201 in call │
│ │
│ 198 │ │ │ - generated_token_ids (torch.Tensor or tf.Tensor, present when `retu │
│ 199 │ │ │ ids of the generated text. │
│ 200 │ │ """ │
│ ❱ 201 │ │ return super().call(text_inputs, **kwargs) │
│ 202 │ │
│ 203 │ def preprocess(self, prompt_text, prefix="", handle_long_generation=None, **generate │
│ 204 │ │ inputs = self.tokenizer( │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/transformers/pipelines/base.py:1119 in call │
│ │
│ 1116 │ │ │ │ ) │
│ 1117 │ │ │ ) │
│ 1118 │ │ else: │
│ ❱ 1119 │ │ │ return self.run_single(inputs, preprocess_params, forward_params, postproces │
│ 1120 │ │
│ 1121 │ def run_multi(self, inputs, preprocess_params, forward_params, postprocess_params): │
│ 1122 │ │ return [self.run_single(item, preprocess_params, forward_params, postprocess_par │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/transformers/pipelines/base.py:1126 in run_single │
│ │
│ 1123 │ │
│ 1124 │ def run_single(self, inputs, preprocess_params, forward_params, postprocess_params): │
│ 1125 │ │ model_inputs = self.preprocess(inputs, **preprocess_params) │
│ ❱ 1126 │ │ model_outputs = self.forward(model_inputs, **forward_params) │
│ 1127 │ │ outputs = self.postprocess(model_outputs, **postprocess_params) │
│ 1128 │ │ return outputs │
│ 1129 │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/transformers/pipelines/base.py:1025 in forward │
│ │
│ 1022 │ │ │ │ inference_context = self.get_inference_context() │
│ 1023 │ │ │ │ with inference_context(): │
│ 1024 │ │ │ │ │ model_inputs = self._ensure_tensor_on_device(model_inputs, device=se │
│ ❱ 1025 │ │ │ │ │ model_outputs = self._forward(model_inputs, **forward_params) │
│ 1026 │ │ │ │ │ model_outputs = self._ensure_tensor_on_device(model_outputs, device= │
│ 1027 │ │ │ else: │
│ 1028 │ │ │ │ raise ValueError(f"Framework {self.framework} is not supported") │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/transformers/pipelines/text_generation.py:263 in _forward │
│ │
│ 260 │ │ │ │ generate_kwargs["min_length"] += prefix_length │
│ 261 │ │ │
│ 262 │ │ # BS x SL │
│ ❱ 263 │ │ generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=att │
│ 264 │ │ out_b = generated_sequence.shape[0] │
│ 265 │ │ if self.framework == "pt": │
│ 266 │ │ │ generated_sequence = generated_sequence.reshape(in_b, out_b // in_b, *genera │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/torch/utils/_contextlib.py:115 in decorate_context │
│ │
│ 112 │ @functools.wraps(func) │
│ 113 │ def decorate_context(*args, **kwargs): │
│ 114 │ │ with ctx_factory(): │
│ ❱ 115 │ │ │ return func(*args, **kwargs) │
│ 116 │ │
│ 117 │ return decorate_context │
│ 118 │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/transformers/generation/utils.py:1565 in generate │
│ │
│ 1562 │ │ │ ) │
│ 1563 │ │ │ │
│ 1564 │ │ │ # 13. run sample │
│ ❱ 1565 │ │ │ return self.sample( │
│ 1566 │ │ │ │ input_ids, │
│ 1567 │ │ │ │ logits_processor=logits_processor, │
│ 1568 │ │ │ │ logits_warper=logits_warper, │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/transformers/generation/utils.py:2612 in sample │
│ │
│ 2609 │ │ │ model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs) │
│ 2610 │ │ │ │
│ 2611 │ │ │ # forward pass to get next token │
│ ❱ 2612 │ │ │ outputs = self( │
│ 2613 │ │ │ │ **model_inputs, │
│ 2614 │ │ │ │ return_dict=True, │
│ 2615 │ │ │ │ output_attentions=output_attentions, │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/torch/nn/modules/module.py:1501 in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /root/.cache/huggingface/modules/transformers_modules/mosaicml/mpt-7b-instruct/925e0d80e50e77aad │
│ daf9c3ced41ca4ea23a1025/modeling_mpt.py:270 in forward │
│ │
│ 267 │ │ use_cache = use_cache if use_cache is not None else self.config.use_cache │
│ 268 │ │ if inputs_embeds is not None: │
│ 269 │ │ │ raise NotImplementedError('inputs_embeds has to be None (for hf/peft support │
│ ❱ 270 │ │ outputs = self.transformer(input_ids=input_ids, past_key_values=past_key_values, │
│ 271 │ │ logits = self.transformer.wte(outputs.last_hidden_state.to(self.transformer.wte. │
│ 272 │ │ if self.logit_scale is not None: │
│ 273 │ │ │ if self.logit_scale == 0: │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/torch/nn/modules/module.py:1501 in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /root/.cache/huggingface/modules/transformers_modules/mosaicml/mpt-7b-instruct/925e0d80e50e77aad │
│ daf9c3ced41ca4ea23a1025/modeling_mpt.py:202 in forward │
│ │
│ 199 │ │ │ │ assert all_hidden_states is not None │
│ 200 │ │ │ │ all_hidden_states = all_hidden_states + (x,) │
│ 201 │ │ │ past_key_value = past_key_values[b_idx] if past_key_values is not None else │
│ ❱ 202 │ │ │ (x, attn_weights, past_key_value) = block(x, past_key_value=past_key_value, │
│ 203 │ │ │ if past_key_values is not None: │
│ 204 │ │ │ │ past_key_values[b_idx] = past_key_value │
│ 205 │ │ │ if output_attentions: │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/torch/nn/modules/module.py:1501 in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /root/.cache/huggingface/modules/transformers_modules/mosaicml/mpt-7b-instruct/925e0d80e50e77aad │
│ daf9c3ced41ca4ea23a1025/blocks.py:36 in forward │
│ │
│ 33 │ │
│ 34 │ def forward(self, x: torch.Tensor, past_key_value: Optional[Tuple[torch.Tensor]]=Non │
│ 35 │ │ a = self.norm_1(x) │
│ ❱ 36 │ │ (b, attn_weights, past_key_value) = self.attn(a, past_key_value=past_key_value, │
│ 37 │ │ x = x + self.resid_attn_dropout(b) │
│ 38 │ │ m = self.norm_2(x) │
│ 39 │ │ n = self.ffn(m) │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/torch/nn/modules/module.py:1501 in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /root/.cache/huggingface/modules/transformers_modules/mosaicml/mpt-7b-instruct/925e0d80e50e77aad │
│ daf9c3ced41ca4ea23a1025/attention.py:201 in forward │
│ │
│ 198 │ │ │ dtype = query.dtype │
│ 199 │ │ │ query = self.q_ln(query).to(dtype) │
│ 200 │ │ │ key = self.k_ln(key).to(dtype) │
│ ❱ 201 │ │ (context, attn_weights, past_key_value) = self.attn_fn(query, key, value, self.n │
│ 202 │ │ return (self.out_proj(context), attn_weights, past_key_value) │
│ 203 │
│ 204 class MultiQueryAttention(nn.Module): │
│ │
│ /root/.cache/huggingface/modules/transformers_modules/mosaicml/mpt-7b-instruct/925e0d80e50e77aad │
│ daf9c3ced41ca4ea23a1025/attention.py:147 in triton_flash_attn_fn │
│ │
│ 144 │ │ key = key.expand(*key.shape[:2], n_heads, key.size(-1)) │
│ 145 │ │ value = value.expand(*value.shape[:2], n_heads, value.size(-1)) │
│ 146 │ reset_is_causal = _reset_is_causal(query.size(1), key.size(1), is_causal) │
│ ❱ 147 │ attn_output = flash_attn_func(query, key, value, attn_bias, reset_is_causal, softmax │
│ 148 │ output = attn_output.view(*attn_output.shape[:2], -1) │
│ 149 │ return (output, None, past_key_value) │
│ 150 │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/torch/autograd/function.py:506 in apply │
│ │
│ 503 │ │ if not torch._C._are_functorch_transforms_active(): │
│ 504 │ │ │ # See NOTE: [functorch vjp and autograd interaction] │
│ 505 │ │ │ args = _functorch.utils.unwrap_dead_wrappers(args) │
│ ❱ 506 │ │ │ return super().apply(*args, **kwargs) # type: ignore[misc] │
│ 507 │ │ │
│ 508 │ │ if cls.setup_context == SingleLevelFunction.setup_context: │
│ 509 │ │ │ raise RuntimeError( │
│ │
│ /root/.cache/huggingface/modules/transformers_modules/mosaicml/mpt-7b-instruct/925e0d80e50e77aad │
│ daf9c3ced41ca4ea23a1025/flash_attn_triton.py:469 in forward │
│ │
│ 466 │ │ │ │ ALiBi mask for non-causal would have shape (1, nheads, seqlen_q, seqlen │
│ 467 │ │ """ │
│ 468 │ │ (q, k, v) = [x if x.stride(-1) == 1 else x.contiguous() for x in [q, k, v]] │
│ ❱ 469 │ │ (o, lse, ctx.softmax_scale) = _flash_attn_forward(q, k, v, bias=bias, causal=cau │
│ 470 │ │ ctx.save_for_backward(q, k, v, o, lse, bias) │
│ 471 │ │ ctx.causal = causal │
│ 472 │ │ return o │
│ │
│ /root/.cache/huggingface/modules/transformers_modules/mosaicml/mpt-7b-instruct/925e0d80e50e77aad │
│ daf9c3ced41ca4ea23a1025/flash_attn_triton.py:363 in _flash_attn_forward │
│ │
│ 360 │ BLOCK = 128 │
│ 361 │ num_warps = 4 if d <= 64 else 8 │
│ 362 │ grid = lambda META: (triton.cdiv(seqlen_q, META['BLOCK_M']), batch * nheads) │
│ ❱ 363 │ _fwd_kernel[grid](q, k, v, bias, o, lse, tmp, softmax_scale, q.stride(0), q.stride(2 │
│ 364 │ return (o, lse, softmax_scale) │
│ 365 │
│ 366 def _flash_attn_backward(do, q, k, v, o, lse, dq, dk, dv, bias=None, causal=False, softm │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/triton_pre_mlir/runtime/jit.py:106 in launcher │
│ │
│ 103 │ │ memorizes the grid. │
│ 104 │ │ """ │
│ 105 │ │ def launcher(*args, **kwargs): │
│ ❱ 106 │ │ │ return self.run(*args, grid=grid, **kwargs) │
│ 107 │ │ return launcher │
│ 108 │
│ 109 │
│ │
│ /local_disk0/.ephemeral_nfs/envs/pythonEnv-e4039dcb-6324-43cf-ad2f-10278d712478/lib/python3.10/s │
│ ite-packages/triton_pre_mlir/runtime/autotuner.py:200 in run │
│ │
│ 197 │ def run(self, *args, **kwargs): │
│ 198 │ │ for v, heur in self.values.items(): │
│ 199 │ │ │ kwargs[v] = heur({**dict(zip(self.arg_names, args)), **kwargs}) │
│ ❱ 200 │ │ return self.fn.run(*args, **kwargs) │
│ 201 │
│ 202 │
│ 203 def heuristics(values): │
│ in _fwd_kernel:43 │
╰───────────────────────────────

RuntimeError: Triton Error [CUDA]: invalid argument

400 Bad request error on create_endpoint()

I am getting this error:

400 Client Error: Bad Request for url: https://<my_endpoint>.azuredatabricks.net/api/2.0/serving-endpoints
When executing this line:

create_endpoint(endpoint_name, served_models)

I think is because GPU Medium is not an option when registering an endpoint.

Above that line we can see that we are declaring the workload_type:

served_models = [
    {
      "name": model_name,
      "model_name": model_name,
      "model_version": model_version,
      "workload_size": "Small",
      "workload_type": "GPU_MEDIUM",
      "scale_to_zero_enabled": False
    }
]

I tried the registry in Isolation and getting this error:

{
  "error_code": "INVALID_PARAMETER_VALUE",
    "message": "Workload type 'GPU_MEDIUM' with size 'Small' is not supported. Please choose a node type from "
}

Cannot pass integration test on Azure and GCP

The VM instance types with A10 GPU is not available in internal testing environment for Azure and GCP. As a result, Azure and GCP Github action tests are disabled.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

databricks-industry-solutions / hls-llm-doc-qa Goto Github PK

hls-llm-doc-qa's Introduction

Biomedical Question Answering over Custom Datasets with 🦜️🔗 LangChain and Llama 2 from Hugging Face 🤗

Getting started

Project support

hls-llm-doc-qa's People

Contributors

Stargazers

Watchers

Forkers

hls-llm-doc-qa's Issues

Traceback Error in 003 QA in CMD 17

400 Bad request error on create_endpoint()

Cannot pass integration test on Azure and GCP

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent