[Question]: How to see the intermediate step in the response synthesis about llama_index HOT 2 CLOSED

To inspect the intermediate steps in the response synthesis, including the context used for each Large Language Model (LLM) call and the associated answers in the refined response mode, you can utilize the debugging and tracing features of LlamaIndex. Here's how you can achieve this:

1. Enable Debug Logging: Start by enabling debug logging in your application. This will allow you to log operations at a basic level. You can do this by configuring the logging module in your Python script as follows:

   import logging
   import sys
   
   logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

2. Use TreeSelectLeafRetriever for Detailed Tracing: The TreeSelectLeafRetriever class in LlamaIndex is designed for traversing the index graph to search for the best leaf node that answers your query. It supports detailed debugging and tracing, including logging debug information such as the selected node and the current answer response at each level of the tree traversal.

3. Customize Query Processing: For advanced observability, customize the behavior of the TreeSelectLeafRetriever class by providing your own implementations of the BasePromptTemplate and modifying the query templates used for LLM calls. This customization allows for detailed control over how queries are processed, how responses are synthesized and refined, and enables tracing the intermediate steps and the context used for each LLM call.

4. Structured Answer Filtering and Node Selection: The class also supports structured answer filtering and node selection based on LLM responses, which can be fine-tuned by setting the child_branch_factor and using appropriate query templates. This feature allows for granular control over the answers provided by the LLM, including filtering out unsatisfactory responses and selecting the most relevant nodes for further querying.

For more detailed guidance, refer to the LlamaIndex documentation and specifically look into the TreeSelectLeafRetriever class and the refine.py module within the LlamaIndex repository. These resources will provide you with the necessary information to understand and implement the debugging and tracing features for your project.

To continue the conversation, mention @dosu.

from llama_index.

You can enable logging for every llm input and output

import llama_index.core

llama_index.core.set_global_handler("simple")

from llama_index.