shaido987 / novel-dataset Goto Github PK

Always use the same value to show that no value exist.

• Create a class containing all web sites scraping methods (private / public as needed). Only keeping very simple code inside the notebook.
• Add this to a .py file and create a "code" folder to put both files inside.

The list with recommended series ids is missing a lot of entries.

For example, '32837' has 5 recommended series but the 1.0.3 dataset only contains a single entry ('36912') which is an id that is not in the dataset.

Run the complete 0.1 version of the program and upload the scraped csv file.

Currently "related_series_ids" and "recommended_series_ids" are lists of strings while the "recommendation_list_ids" column contains a list of ints.

Update "related_series_ids" and "recommended_series_ids" to lists of ints for consistency.

Can you add the option to retrieve metadata from www.wlnupdates.com ? It is much more frequently updated and there are some novels not included in the www.novelupdates.com dataset.

You can check wlnupdates.com API here:

https://github.com/fake-name/wlnupdates/blob/master/app/templates/api-docs.md

Add proper docstring comments to all functions.

Any suggestions on dataset or code enhancements are welcome.

In particular, if any issues with the dataset, how it is structured or missing information is identified it would be appreciated if pointed out.

Add an image to the readme with an graph.

• Nodes are novels
• Nodes colors is the main genres
• Edges are ???

Some suggestions for edges:

• same author (too few edges)
• Related / recommended series (directed graph)
• On same reading list

Add a progress bar when running to give an estimate on how long the run will take.

Create a more representative graph image for the readme (i.e., more nodes and better looking).

"Status in COO" is currently scraped incorrectly. Mostly None or a single character.

Remove the soup input from the chapter_info function. It should be unnecessary.

Novelupdates have added a new field called "Recommendation Lists". Add the ids of these lists to the collected data. It could be useful to connect series together that appears on the same lists.

When there are no related series the return right now is NaN while it should be None.

The list of recommended series is not correctly scraped and will always return an empty list.

Move all dataset creation code to scraper.py (maybe remove to web_scraper.py or scrape_dataset.py?). Put the code from the notebook into a if __name__ == "__main__": section.

Parameters that could potentially be used as input parameters (i.e., that can be used as args input):

• Debug mode
• Save file or version number

As of now, any requests to novelupdates returns an 403 error even when using cfparser. The cfparser package looks to be dead with no updates in recent years so it would most likely have to be changed to another package. An alternative would be cfscrape however it looks also to be quite dead...

Some further investigation is needed here before the dataset can be updated again.

The novelupdates website have changed slightly which results in a lot of novel's names to be missing.