Giter VIP home page Giter VIP logo

Comments (8)

jjmachan avatar jjmachan commented on June 15, 2024 1

ohh my apologies. I'll fix that bug (thanks a lot for bringing it to our attention ❤️ ). In the meantime could you set the env-var with some random text as a workaround till we release the fix?

from ragas.

jjmachan avatar jjmachan commented on June 15, 2024

Hey @seanbenhur! We're glad you like it 😊

how are you planning to use Llama2? I would recommend using via langchain because there is a lot of support for it in the community something like this could help Llama 2 in LangChain — FIRST Open Source Conversational Agent! - YouTube

but let me know if that is not what you have in mind. Also, why are you trying to move away from langchain?

from ragas.

seanbenhur avatar seanbenhur commented on June 15, 2024

Thanks for the prompt response, I was thinking of using Llama without langchain since I feel langchain is an extra dependency for the project

from ragas.

jjmachan avatar jjmachan commented on June 15, 2024

got it. Unfortunately, that is not possible right now and we don't plan to add it to the roadmap. There is also another issue which is that we have not tested Llama2's performance for evaluation so we wouldn't recommend using it for evaluation in critical applications.

hope you understand...

from ragas.

seanbenhur avatar seanbenhur commented on June 15, 2024

Sure No problems!

I tried to use Llama with Langchain without using OpenAI, but now I am getting error since I didn't pass the OAI key on the importing stage itself

`ValidationError                           Traceback (most recent call last)
Cell In[39], line 6
      4 import torch
      5 from langchain.llms import HuggingFacePipeline
----> 6 from ragas.metrics import Faithfulness
      9 DATASET_PATH = "explodinggradients/ragas-wikiqa"
     10 MODEL_PATH = "llama-2-7b-chat-hf"

File ~/.local/lib/python3.8/site-packages/ragas/__init__.py:1
----> 1 from ragas.evaluation import evaluate
      3 try:
      4     from ._version import version as __version__

File ~/.local/lib/python3.8/site-packages/ragas/evaluation.py:9
      6 from datasets import Dataset, concatenate_datasets
      8 from ragas._analytics import EvaluationEvent, track
----> 9 from ragas.metrics.base import Metric
     10 from ragas.metrics.critique import AspectCritique
     11 from ragas.validation import validate_column_dtypes, validate_evaluation_modes

File ~/.local/lib/python3.8/site-packages/ragas/metrics/__init__.py:2
      1 from ragas.metrics.answer_relevance import AnswerRelevancy, answer_relevancy
----> 2 from ragas.metrics.context_relevance import ContextRelevancy, context_relevancy
      3 from ragas.metrics.critique import AspectCritique
      4 from ragas.metrics.faithfulnes import Faithfulness, faithfulness

File ~/.local/lib/python3.8/site-packages/ragas/metrics/context_relevance.py:173
    168             scores.append(agr_score * np.mean(overlap_scores))
    170         return dataset.add_column(f"{self.name}", scores)  # type: ignore
--> 173 context_relevancy = ContextRelevancy()

File <string>:4, in __init__(self, batch_size, llm, name, evaluation_mode, strictness, agreement_metric, model_name)

File ~/.local/lib/python3.8/site-packages/ragas/metrics/base.py:69, in _llm_factory()
     68 def _llm_factory():
---> 69     return ChatOpenAI(model_name="gpt-3.5-turbo-16k")

File ~/.local/lib/python3.8/site-packages/langchain/load/serializable.py:74, in Serializable.__init__(self, **kwargs)
     73 def __init__(self, **kwargs: Any) -> None:
---> 74     super().__init__(**kwargs)
     75     self._lc_kwargs = kwargs

File ~/.local/lib/python3.8/site-packages/pydantic/main.py:341, in pydantic.main.BaseModel.__init__()

ValidationError: 1 validation error for ChatOpenAI
__root__
  Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass  `openai_api_key` as a named parameter. (type=value_error)`

from ragas.

jjmachan avatar jjmachan commented on June 15, 2024

this is because the metrics are still using OAI endpoints. Can you refer custom llm guide already? it will show you how to change the LLM. In case it does not work can you share the code and ragas,python versions?

from ragas.

seanbenhur avatar seanbenhur commented on June 15, 2024

Yes I referred the above nb, the issue is I get the error, when I import the metric

from ragas.metrics import Faithfulness

from ragas.

ishaan-jaff avatar ishaan-jaff commented on June 15, 2024

@seanbenhur made a PR for this: #118

from ragas.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.