slaveofcode / boilerpipe3 Goto Github PK
View Code? Open in Web Editor NEWA fork of boilerpipe with python 3 and small fixes, ported from source `https://pypi.python.org/pypi/boilerpipe-py3.
A fork of boilerpipe with python 3 and small fixes, ported from source `https://pypi.python.org/pypi/boilerpipe-py3.
Using the Extractor leads to the following error, triggered by the lack of keyword argument convertStrings when boilerpipe launches the JVM.
This can be fixed by passing the argument convertStrings=False on line 10 of init() when the JVM is started. This will result in Java strings being returned which will need to be converted to Python strings on output.
Hi,
When I try to extract an article from varzesh3.com (for example https://www.varzesh3.com/news/1554055/) I get this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'your_url' is not defined
>>> your_url = 'https://www.varzesh3.com/news/1554055/'
>>> extractor = Extractor(extractor='ArticleExtractor', url=your_url)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/dist-packages/boilerpipe/extract/__init__.py", line 46, in __init__
self.data = str(self.data, encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
I solved this by replacing line 46 with:
self.data = self.data.decode(encoding, "ignore")
Hi!
I've been using Boilerpipe with Bitextor, and everything has worked out fine. The problem is that when I processed a PDF file, specifically this one, I run out of memory and the execution failed. The error message I got is:
Traceback (most recent call last):
File "BoilerpipeSAXInput.java", line 51, in de.l3s.boilerpipe.sax.BoilerpipeSAXInput.getTextDocument
File "BoilerpipeSAXInput.java", line 63, in de.l3s.boilerpipe.sax.BoilerpipeSAXInput.getTextDocument
File "org.apache.xerces.parsers.AbstractSAXParser.java", line -1, in org.apache.xerces.parsers.AbstractSAXParser.parse
File "org.apache.xerces.parsers.XMLParser.java", line -1, in org.apache.xerces.parsers.XMLParser.parse
File "HTMLConfiguration.java", line 452, in org.cyberneko.html.HTMLConfiguration.parse
File "HTMLConfiguration.java", line 499, in org.cyberneko.html.HTMLConfiguration.parse
File "HTMLScanner.java", line 907, in org.cyberneko.html.HTMLScanner.scanDocument
File "HTMLScanner.java", line 1967, in org.cyberneko.html.HTMLScanner$ContentScanner.scan
File "HTMLScanner.java", line 2291, in org.cyberneko.html.HTMLScanner$ContentScanner.scanCharacters
File "DefaultFilter.java", line 152, in org.cyberneko.html.filters.DefaultFilter.characters
File "HTMLTagBalancer.java", line 954, in org.cyberneko.html.HTMLTagBalancer.characters
File "org.apache.xerces.parsers.AbstractSAXParser.java", line -1, in org.apache.xerces.parsers.AbstractSAXParser.characters
File "BoilerpipeHTMLContentHandler.java", line 293, in de.l3s.boilerpipe.sax.BoilerpipeHTMLContentHandler.characters
File "BitSet.java", line 447, in java.util.BitSet.set
File "BitSet.java", line 352, in java.util.BitSet.expandTo
File "BitSet.java", line 337, in java.util.BitSet.ensureCapacity
File "Arrays.java", line 3308, in java.util.Arrays.copyOf
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module> File "/home/cgarcia/miniconda3/envs/bitextor/lib/python3.8/site-packages/boilerpipe/extract/__init__.py", line 67, in __init__ self.source = BoilerpipeSAXInput(InputSource(reader)).getTextDocument()
java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space
In order to get rid of Bitextor for the explanation of this issue, I attach to this issue the file which Bitextor generated from the PDF, which is an HTML, and the attached HTML is the one that causes this problem. The file size is 9.4 MB, which I don't know if is a size too big to make Boilerpipe fail. The problem is not related to the PDF itself, since I processed other PDFs and the process finished without errors.
In the end, I figured out that the problem was actually due to the memory (initially I though about a memory leak), what was really weird to me since it is a 9.4 MB file. I fixed the problem increasing the quantity of memory of jpype
. The total quantity of memory which a 9.4 MB HTML file required was of ~52 GB!!!!!!! My system has 126 GB, so the default max. heap size of the JVM is 30 GB. Since the process was requiring 52 GB and the max. heap size was 30 GB, I was running out of memory.
The reason of this issue is to alert other people which might have the same problem and to ask the following question: do these numbers make sense? I mean, 52 GB of memory for an HTML file of 9.4 MB?
The code which triggers the error:
from boilerpipe.extract import Extractor
text = ""
with open("boilerpipe_error.html") as f:
for l in f:
text += l
text = text.strip()
Extractor(extractor='ArticleExtractor', html=text)
The fix (run before the above code; it should work, but I haven't tested it out of the actual file, so I might have miss something):
import os
import jpype
import importlib
# Take 80 GB of memory for boilerpipe
boilerpipe_max_heap_size = 80 * 1024 # TODO change this value
if not jpype.isJVMStarted():
max_heap_size = f"-Xmx{str(options.boilerpipe_max_heap_size)}M" if options.boilerpipe_max_heap_size >= 0 else ''
jars = []
for top, dirs, files in os.walk(os.path.dirname(importlib.machinery.PathFinder().find_module("boilerpipe").get_filename()) + '/data'):
for nm in files:
if nm[-4:] == ".jar":
jars.append(os.path.join(top, nm))
jpype.addClassPath(os.pathsep.join(jars))
jargs = [jpype.getDefaultJVMPath()]
if max_heap_size != '':
jargs.append(max_heap_size)
jpype.startJVM(*jargs, convertStrings=False)
# ... run boilerpipe
First, thanks for the port.
When trying to use KeepEverythingWithMinKWordsExtractor
, I get the error:
Traceback (most recent call last):
File "test.py", line 4, in <module>
extractor = Extractor(extractor='KeepEverythingWithMinKWordsExtractor', url=url, kMin=20)
File "/private/tmp/html_extract/venv/lib/python3.6/site-packages/boilerpipe/extract/__init__.py", line 62, in __init__
"de.l3s.boilerpipe.extractors."+extractor).INSTANCE
AttributeError: type object 'de.l3s.boilerpipe.extractors.KeepEverythingWithMin' has no attribute 'INSTANCE'
The problem is that the KeepEverythingWithMinKWordsExtractor
constructor takes an argument (see the java code).
To fix this, line 60 in extract/__init__.py
should be replaced with:
if extractor == "KeepEverythingWithMinKWordsExtractor":
# handle argument
kMin = kwargs.get("kMin", 1) # set default to 1
self.extractor = jpype.JClass(
"de.l3s.boilerpipe.extractors."+extractor)(kMin)
else:
self.extractor = jpype.JClass(
"de.l3s.boilerpipe.extractors."+extractor).INSTANCE
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.