1. Illustrated Technical Book Hindi Webpage with PDFs Link

2. Rabi Crop Details English Webpages with Text Link

Content sheet - https://docs.google.com/spreadsheets/d/12NPm3r4odguaI_3dqEiyJKSzhIsCZXp30BFFrMss1UY/edit?pli=1#gid=1447499902

• Text Context;
• Image Content - Context tagging to the heading is done manually
• @Amruth-Vamshi to add chunks
• @aashutosh-samagra to verify

@Amruth-Vamshi kindlt detail out the ticket for Karun

Services needed:

1. BFF - copy Bhashini branch from AKAI
2. FusionAuth
3. Userservice
4. Transport socket
5. Minio

• Share plan

Sample pdf here

We need to be able to extract text from it and be able to chunk it in the form of headings and related chunks.

We have tired 2 different approaches :

1. Using Deepdoc detection to extract the text headings and structure of each page and converting it into a json format : here
2. Using Pymupdf to get the boundaries of the text from the pdf and then using that to figure out the headings and the content pieces : here

Overall ambition :

We should be able able to parse a pdf such that we are able to get this following structure out of it.
It includes the following key capabilities :

1. Ability to process pdfs with multiple languages - English/Odia/Hindi
2. Ability to create chunks with headings on the basis of the way the pdf is structured. We should be able to recognize that some texts are headings, some are content and then be able to convert that into the structure above.
3. Be able to process images and tables and convert them into chunks that can be passed to an LLM to answer questions based on them.

Where are we on this now :

Chunking

Free text chunking :

We are able to chunk free text (unstructured text) here

Structured pdf chunking

We have looked at 2 approaches for chunking text :

1. Using Deepdoc detection to extract the text headings and structure of each page and converting it into a json format : here
2. Using Pymupdf to get the boundaries of the text from the pdf and then using that to figure out the headings and the content pieces : here

What is a good chunk

• Should be around 100 to 200 words.
• The text/topic in a chunk should be on a similar topic which makes semantic sense.
• The text/topic in a chunk should be different from other chunks
• Ideally it should cover a small topic in its entirety. It could cover multiple topics but these small topics should not be a part of some other chunk.

For example :
Bad Chunk :

Here is a list of links : 
Cab booking  :  http:/sdjnsdkgj/  
Hotel form :  http:/sfjgkjnfsgn/  

This is a bad chunk because :

1. The chunk is small is size
2. The links cover multiple topics at once. Cab booking form should be a part of the chunk that should be a part that describes how to book a cab. Similarly, for the hotel booking lunch