Question: check if page is readable? about trafilatura HOT 9 CLOSED

Yes, of course!

from trafilatura.

Hi @zirkelc, it makes perfect sense to give back a confidence score to the user. I used a binary criterion at some point and then removed it. The feature would need to be implemented before adding such a function.

from trafilatura.

Hi @adbar thanks for the quick response! Are you planning to add this feature? I could help you out if you have some more details implementation-wise.

from trafilatura.

Trafilatura actually uses a combination of several extractors so the different scores wouldn't be commensurable.

The best we could do would be to mimick a score and/or try to give back useful information about extraction quality to the user. Feel free to suggest something if you have an idea.

from trafilatura.

I thought about some options, but as you said, the combination of different extractors makes it difficult to calculate a real score.

One idea that might be feasible: Let's say you run 3 different extractors (baseline, justext, readability) and html2text for one page. Then remove all special characters such as #( ) [ ] | - _ *, which are normally used for formatting, images, links and tables, so that only the plain text remains. The entire text is then lowercased, split into individual words and all line breaks and spaces are removed. This gives you a kind of corpus for each extractor. Then you compare the word count for each extractor with the word count of the raw text in html2text and use the difference or ratio to calculate some sort of confidence score.

You might even convert the word list of each extractor and html2text into a set to get a unique word list. This makes the comparison between extractor and html2text even more meaningful. My assumption here is that the main content of a page consists of words with a high variance. The other elements such as header, footer and navigation only have a few unique words. This means that if the number of unique words in an extractor is within a certain range of the number of unique words in the html2text extraction (e.g. 50%, 75%, etc.), the result will be more meaningful. For pages without main content, the number of words in the header, footer, navigation etc. would probably be much higher than the result of the extractor and would therefore lead to a low score.

This is of course just a naive assumption without knowing the internals of trafilatura, so please feel free to let me know if this idea is complete nonsense :-D

from trafilatura.

Interesting idea, your "words among html2text" metric probably works well if you have several webpages from the same source. Then the variance will indeed be among the main text. This is what Trafilatura's deduplication option does to some extent, it filters out recurring text segments.

I'm not sure how it would work for a single webpage (which is what the extractor is evaluated on). Postal addresses in footers add a lot of variance for example, so does a "latest articles" column.

from trafilatura.

Yes, you might be right. Even though, some pre-filtering on the raw html, like removing semantic elements (<header>, <nav>, <footer>) should maybe decrease the variance a bit.

I will try to implement a small PoC the next few days/weeks, just to try out how good or bad it is. I could then use your evaluation dataset to compare the eval scores with the calculated confidence score. I assume a high eval score should correlate with a high confidence score. What do you think?

from trafilatura.

By all means, please go ahead. If everything works you'd have another extraction method and the confidence score would be a useful by-product, I'm curious to see how that goes.

from trafilatura.

@zirkelc I guess I can close this thread now that you've added the is_probably_readarable function?

from trafilatura.