HalluPAQ is a systematic toolkit for efficient confidence scoring and hallucination detection in domain-specific Large Language Models (LLMs). This toolkit is designed to assess the credibility of LLM outputs by utilizing a vector database to index and query a corpus of Probable Asked Questions (PAQ), efficiently determining the veracity of LLM responses by measuring their similarity to known knowledge. Our method not only provides rapid evaluations but also significantly outperforms existing methods in ROC-AUC scores, establishing a new standard for safe LLM deployment.

• Introduction
  • HalluPAQ System Overview
  • Data
• Installation
• Usage
• Methodology
• Results

Training Set: Ground truth texts from StatPearls and Anatomy Gray, processed by OpenAI models to generate questions and answers.

Validation Set & Test Set:

• Paraphrased Questions: From the training set for adaptability testing.
• Out-of-Coverage Questions: Challenge the model with diverse biomedical topics from PubMed not present in the training data.

Ground-truth tagging rules

Clone the repository and install dependencies:

git clone https://github.com/orangejustin/HalluPAQ.git
cd HalluPAQ
pip install -r requirements.txt

1. To generate group Q&A pairs, follow these steps:
   1. Modify the input corpus and output directory settings as needed.
   2. Run the Q&A generation script:

python3 qa_generation/qa_generation.py

We obtain the generated Q&A pairs in the knowledge_souce directory.

2. Running HalluPAQ Analysis for Hallucination Detection
   1. Run simulation with a RAG system processes to evaluate and tag the generated Q&A pairs.
   2. Determine if any of the responses are hallucinations based on the confidence scores.

cd HalluPAQ
python3 analysis_scripts/simulated_rag.py \
        --input_jsonl knowledge_source/knowledge_source.jsonl \
        --output_jsonl path/to/output.jsonl --arn_role your_sagemaker_role_arn \
        --shut_down

• --input_jsonl: Specifies the path to the JSONL file containing the generated Q&A pairs.
• --output_jsonl: Defines the path where the output, including the tags for hallucination detection, will be stored.
• --arn_role: The AWS SageMaker role ARN that is required for accessing SageMaker resources during the analysis.
• --shut_down: An optional flag to shut down the SageMaker instance after processing to manage resources efficiently.

This process will analyze each Q&A pair, comparing the generated answer to the content derived from a knowledge base to check for discrepancies that may indicate hallucinations. The results will be saved in the specified output JSONL file, tagged with confidence scores indicating the likelihood of each response being a hallucination.

HalluPAQ employs a novel approach to enhance the reliability of LLM outputs through efficient indexing and querying of a PAQ corpus. This methodology ensures rapid evaluations and high accuracy in determining the credibility of LLM responses.

HalluPAQ achieves an average ROC-AUC score of 0.76952 and an average overhead time of 0.00455 seconds, outperforming existing methods such as SelfCheckGPT-NLI and FactScore.