remnestal / fakenews Goto Github PK

It seems like the data I have been using during development may contain an excessive amount of similar, albeit not identical, headlines. This loss of variance when large portions of the input headlines are in fact identical does not affect the generator directly, since all these occurances are correctly represented in the transition matrix, however, it also increases the amount of generated headlines identical or very similar to the source data.

This might be desirable in some scenarios, but not if you want to have a varied output. Thus, some tool for calculating thet statistical variance of the source data would be appreciated. 🔬📊

I want the fake news generator to generate real news.

• Implement real news?
• Also implement clickbait headlines?
• Can this fakenews generator also be a CLI text editor?

When the source data is either too small or has too little variation and the order-value is too large; overfitting will occur and generated headlines will start looking very similar if not even identical to counterparts in the source data.

Some way or another, I'd like to at the very least make the user aware of the likeness. Some options I've though of so far:

• Add option to limit generated data with a likeness-threshold value, for example: "only generate headlines that are at most 75% identical to one of the source headlines."
• Add the calculated likeness-value to the output so that the user can decide their relevance for themselves.

The ability to censor the output should perhaps not be part of the fakenews.py module itself to enforce modularity and to avoid forcing costly operations for every user. Checking a headline's likeness to the source data will likely be a fairly complex problem and may thus increase overhead significantly.