PDF upload failed. How to upload size of PDF about openchat HOT 6 CLOSED

I managed to upload the PDF by deleting some pages and reducing their size. However, it seems that the text was not imported properly. The error from llm_server is as follows:

event - compiled successfully in 118 ms (55 modules)
[WARN] Importing from 'langchain/document_loaders' is deprecated. Import from eg. 'langchain/document_loaders/fs/text' or 'langchain/document_loaders/web/cheerio' instead. See https://js.langchain.com/docs/getting-started/install#updating-from-0052 for upgrade instructions.
Error
    at UnknownErrorExceptionClosure (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:443:37)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:446:2)
    at __w_pdfjs_require__ (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:7939:23)
    at __w_pdfjs_require__ (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at /usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:88:18
    at /usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:91:10
    at webpackUniversalModuleDefinition (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:18:20)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:25:3)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at PDF (/usr/src/app/node_modules/pdf-parse/lib/pdf-parse.js:63:29)
    at CustomPDFLoader.parse (webpack-internal:///(api)/./utils/customPDFLoader.ts:42:30)
    at async DirectoryLoader.load (file:///usr/src/app/node_modules/langchain/dist/document_loaders/fs/directory.js:66:40)
    at async pdfHandler (webpack-internal:///(api)/./data-sources/pdfHandler.ts:29:25)
    at async handler (webpack-internal:///(api)/./pages/api/ingest.ts:38:20)
    at async Object.apiResolver (/usr/src/app/node_modules/next/dist/server/api-utils/node.js:372:9)
    at async DevServer.runApi (/usr/src/app/node_modules/next/dist/server/next-server.js:513:9)
    at async Object.fn (/usr/src/app/node_modules/next/dist/server/next-server.js:815:35)
    at async Router.execute (/usr/src/app/node_modules/next/dist/server/router.js:243:32)
    at async DevServer.runImpl (/usr/src/app/node_modules/next/dist/server/base-server.js:432:29)
    at async DevServer.run (/usr/src/app/node_modules/next/dist/server/dev/next-dev-server.js:814:20)
    at async DevServer.handleRequestImpl (/usr/src/app/node_modules/next/dist/server/base-server.js:375:20)
    at async /usr/src/app/node_modules/next/dist/server/base-server.js:157:99 {
  message: 'PDFDocument: stream must have data',
  details: 'Error: PDFDocument: stream must have data'
}

from openchat.

This happens when you upload non-scannable PDF files (contain images not text).

from openchat.

Hi, I would like to increase the upload size of my PDF files, can you tell me where to set the file size? Thank you.

The size limitation for parsing PDF files using pdf-parse would depend on the capabilities and limitations of the library itself. The code dynamically imports the pdf-parse library from the pdf-parse/lib/pdf-parse.js file. The specific limitations of pdf-parse may be documented in the library's documentation or source code.

llm-server/utils/customPDFLoader.ts

It's worth noting that processing very large PDF files can consume significant memory and processing resources. Therefore, even if there is no explicit limitation in the code, the performance and resource usage may be affected by the size and complexity of the PDF files being processed.

you can use this repo if this helps https://github.com/mayooear/gpt4-pdf-chatbot-langchain

from openchat.

The size of the PDF file is 25,277KB. However, when I attempted to upload the PDF, I encountered an error and was unable to complete the upload. Here is the link to the error details: https://flareapp.io/share/q5YNGnom#context

Should I make changes to the nginx settings to resolve this issue?

from openchat.

I managed to upload the PDF by deleting some pages and reducing their size. However, it seems that the text was not imported properly. The error from llm_server is as follows:

event - compiled successfully in 118 ms (55 modules)
[WARN] Importing from 'langchain/document_loaders' is deprecated. Import from eg. 'langchain/document_loaders/fs/text' or 'langchain/document_loaders/web/cheerio' instead. See https://js.langchain.com/docs/getting-started/install#updating-from-0052 for upgrade instructions.
Error
    at UnknownErrorExceptionClosure (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:443:37)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:446:2)
    at __w_pdfjs_require__ (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:7939:23)
    at __w_pdfjs_require__ (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at /usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:88:18
    at /usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:91:10
    at webpackUniversalModuleDefinition (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:18:20)
    at Object.<anonymous> (/usr/src/app/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:25:3)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at PDF (/usr/src/app/node_modules/pdf-parse/lib/pdf-parse.js:63:29)
    at CustomPDFLoader.parse (webpack-internal:///(api)/./utils/customPDFLoader.ts:42:30)
    at async DirectoryLoader.load (file:///usr/src/app/node_modules/langchain/dist/document_loaders/fs/directory.js:66:40)
    at async pdfHandler (webpack-internal:///(api)/./data-sources/pdfHandler.ts:29:25)
    at async handler (webpack-internal:///(api)/./pages/api/ingest.ts:38:20)
    at async Object.apiResolver (/usr/src/app/node_modules/next/dist/server/api-utils/node.js:372:9)
    at async DevServer.runApi (/usr/src/app/node_modules/next/dist/server/next-server.js:513:9)
    at async Object.fn (/usr/src/app/node_modules/next/dist/server/next-server.js:815:35)
    at async Router.execute (/usr/src/app/node_modules/next/dist/server/router.js:243:32)
    at async DevServer.runImpl (/usr/src/app/node_modules/next/dist/server/base-server.js:432:29)
    at async DevServer.run (/usr/src/app/node_modules/next/dist/server/dev/next-dev-server.js:814:20)
    at async DevServer.handleRequestImpl (/usr/src/app/node_modules/next/dist/server/base-server.js:375:20)
    at async /usr/src/app/node_modules/next/dist/server/base-server.js:157:99 {
  message: 'PDFDocument: stream must have data',
  details: 'Error: PDFDocument: stream must have data'
}

The same as me

from openchat.

@gharbat this also happens on pdfs that have both, text and images. There are some ocr solutions that overlap the text on a new layer so you still see the image but the pdf can be searched and the text selected.
Example: https://github.com/ocrmypdf/OCRmyPDF

from openchat.