One can use the notebook provided here to build a pipeline to parse and extract data from OCRed PDF files. Warning: When using LLMs for entity extraction, be sure to perform extensive quality control. They are very susceptible to distracting language (latching on to text that sounds "kind of like" what you're looking for) and missing language (making up content to fill any holes), and importantly, they do NOT provide any hints to when they may be erroring. You need to make sure random audits are part of your workflow! In the notebook we've worked out a workflow using regular expressions and LLMs to parse data from zoning board orders, but the process is generalizable.
- Collect a set of PDFs
- Place OCRed PDFs into the data folder
- Write regexes to pull out data
- Write LLM prompts to pull out data
- Test on random sample
- Iterate & improve