Comments (8)
ohh my apologies. I'll fix that bug (thanks a lot for bringing it to our attention ❤️ ). In the meantime could you set the env-var with some random text as a workaround till we release the fix?
from ragas.
Hey @seanbenhur! We're glad you like it 😊
how are you planning to use Llama2? I would recommend using via langchain because there is a lot of support for it in the community something like this could help Llama 2 in LangChain — FIRST Open Source Conversational Agent! - YouTube
but let me know if that is not what you have in mind. Also, why are you trying to move away from langchain?
from ragas.
Thanks for the prompt response, I was thinking of using Llama without langchain since I feel langchain is an extra dependency for the project
from ragas.
got it. Unfortunately, that is not possible right now and we don't plan to add it to the roadmap. There is also another issue which is that we have not tested Llama2's performance for evaluation so we wouldn't recommend using it for evaluation in critical applications.
hope you understand...
from ragas.
Sure No problems!
I tried to use Llama with Langchain without using OpenAI, but now I am getting error since I didn't pass the OAI key on the importing stage itself
`ValidationError Traceback (most recent call last)
Cell In[39], line 6
4 import torch
5 from langchain.llms import HuggingFacePipeline
----> 6 from ragas.metrics import Faithfulness
9 DATASET_PATH = "explodinggradients/ragas-wikiqa"
10 MODEL_PATH = "llama-2-7b-chat-hf"
File ~/.local/lib/python3.8/site-packages/ragas/__init__.py:1
----> 1 from ragas.evaluation import evaluate
3 try:
4 from ._version import version as __version__
File ~/.local/lib/python3.8/site-packages/ragas/evaluation.py:9
6 from datasets import Dataset, concatenate_datasets
8 from ragas._analytics import EvaluationEvent, track
----> 9 from ragas.metrics.base import Metric
10 from ragas.metrics.critique import AspectCritique
11 from ragas.validation import validate_column_dtypes, validate_evaluation_modes
File ~/.local/lib/python3.8/site-packages/ragas/metrics/__init__.py:2
1 from ragas.metrics.answer_relevance import AnswerRelevancy, answer_relevancy
----> 2 from ragas.metrics.context_relevance import ContextRelevancy, context_relevancy
3 from ragas.metrics.critique import AspectCritique
4 from ragas.metrics.faithfulnes import Faithfulness, faithfulness
File ~/.local/lib/python3.8/site-packages/ragas/metrics/context_relevance.py:173
168 scores.append(agr_score * np.mean(overlap_scores))
170 return dataset.add_column(f"{self.name}", scores) # type: ignore
--> 173 context_relevancy = ContextRelevancy()
File <string>:4, in __init__(self, batch_size, llm, name, evaluation_mode, strictness, agreement_metric, model_name)
File ~/.local/lib/python3.8/site-packages/ragas/metrics/base.py:69, in _llm_factory()
68 def _llm_factory():
---> 69 return ChatOpenAI(model_name="gpt-3.5-turbo-16k")
File ~/.local/lib/python3.8/site-packages/langchain/load/serializable.py:74, in Serializable.__init__(self, **kwargs)
73 def __init__(self, **kwargs: Any) -> None:
---> 74 super().__init__(**kwargs)
75 self._lc_kwargs = kwargs
File ~/.local/lib/python3.8/site-packages/pydantic/main.py:341, in pydantic.main.BaseModel.__init__()
ValidationError: 1 validation error for ChatOpenAI
__root__
Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass `openai_api_key` as a named parameter. (type=value_error)`
from ragas.
this is because the metrics are still using OAI endpoints. Can you refer custom llm guide already? it will show you how to change the LLM. In case it does not work can you share the code and ragas,python versions?
from ragas.
Yes I referred the above nb, the issue is I get the error, when I import the metric
from ragas.metrics import Faithfulness
from ragas.
@seanbenhur made a PR for this: #118
from ragas.
Related Issues (20)
- time out
- Azure Open AI Error when creating synthetic test data - Chat Completion Models Not Supported HOT 2
- Python library must be updated to the latest version HOT 3
- When I use my own LLM model to evaluate answer_relevancy, the following error occurs.
- Context Precision Error using own LLMs
- I'm not able to reproduce documentation example HOT 6
- How can we use Azure OpenAI key with Ragas? HOT 1
- ImportError: cannot import name 'LangchainLLM' from 'ragas.llms' HOT 3
- TypeError: create() got an unexpected keyword argument 'deployment_name' HOT 1
- IndexError: Invalid key: 0 is out of bounds for size 0 HOT 2
- ValueError: Project root not found!
- ValueError: diag requires an array of at least two dimensions
- answer_correctness : too many values to unpack (expected 3) HOT 8
- Azure OpenAI NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} HOT 6
- Context Precision Prompt Example wrong? HOT 1
- Adaptation of answer_correction leads to wrong example HOT 2
- Answer Semantic Similarity gives output not in range 0-1 HOT 6
- Metric to account for entities HOT 5
- Adapt prompts in test generation
- Add community tag in docs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ragas.