Comments (7)
Looks like we need to add handling for those element types in elements_to_isd
. As of now, I think the element metadata gets lost when you read it in from JSON too. I'll update the issue description to reflect that. We'll be able to get a fix in for that shortly. Thanks for flagging!
https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py#L26-L33
from unstructured.
@Matthieu-Tinycoaching - There's a PR up to address the issue you flagged. That also adds helper functions for saving to/loading from JSON
from unstructured.
Right now! Just released the updated in 0.4.15
from unstructured.
@Matthieu-Tinycoaching - Checkout out convert_to_isd
and isd_to_elements
in our docs. You can serialize and deserialize from JSON like this:
with open("elements.json", "w") as f:
json.dump(convert_to_isd(elements), f)
with open("elements.json", "r") as f:
elements = isd_to_elements(json.load(f))
Would that meet your needs? We'd also be happy to include an elements_to_json
and elements_from_json
to wrap that.
from unstructured.
Hi @MthwRobinson thanks for the tip! It seems to do the job.
However, when trying this on the example data layout-parser-paper.pdf
, FigureCaption
and Text
data seem to be lost when reimporting data from the JSON file.
When counting for the types of elements present in the document just after parsing:
Counter({<class 'unstructured.documents.elements.ListItem'>: 223, <class 'unstructured.documents.elements.NarrativeText'>: 69, <class 'unstructured.documents.elements.Title'>: 15, <class 'unstructured.documents.elements.FigureCaption'>: 6, <class 'unstructured.documents.elements.Text'>: 2})
When doing this after reimporting from the JSON file:
Counter({<class 'unstructured.documents.elements.ListItem'>: 223, <class 'unstructured.documents.elements.NarrativeText'>: 69, <class 'unstructured.documents.elements.Title'>: 15})
Any idea?
from unstructured.
Added #270 to capture the missing elements. Will add some serialization/deserialization helper functions while we're in there.
from unstructured.
@MthwRobinson nice!
Do you know when the new release with this fix will be out?
from unstructured.
Related Issues (20)
- `infer_table_structure` in `partition_pdf` function causes CUDA RuntimeError
- `infer_table_structure` lead `Failed to initialize the model`
- chore: Update unstructured-client
- Clarify `orig_elements` documentation HOT 4
- ValueError: Detected a JSON file that does not conform to the Unstructured schema. partition_json currently only processes serialized Unstructured output.
- bug/KeyError with PDF partition fast strategy element ID — in old_to_new_mapping[parent_id] HOT 3
- feat/partition_metadata HOT 1
- bug/partition_pdf doesn't recognize given input parameter HOT 4
- 启动时能禁止nltk连网检查更新package吗?
- Text Extraction Issue: Greek Language PDFs Rendered with Incorrect Alphabet HOT 3
- feat/docx-field-codes
- feat: enable users to define retry logic when using `partition_via_api`
- bug/PDF parsing failed, `parent_id` not in `old_to_new_mapping` HOT 3
- feat/add the possibility to choose between tesseract and paddle in partition() function HOT 2
- chore/Allow python 3.12
- docs: Add docs showing users can set which OCR agent to use
- CCT `measure-table-structure-accuracy-command` doesn't drop index HOT 1
- partition_pdf: no text orientation detection?
- feat/chunking_by_title_tokens
- Partition DOC, DOCX, PPT, PPTX: No images detected HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unstructured.