scottymeyers / linkipedia Goto Github PK

Speed up the search functionality by initializing multiple child processes simultaneously.

Currently POSTing to /scrape forks a child process, then runs through an array of URLs. Each URL is run through one at a time, which can take forever to make a connection.

Each URL has a parent ID, perhaps we can search all the URLS simultaneously that share a parent ID without disrupting the order.

Currently, once a search is initialized the front-end polls the newly created record until a completed parameter it true. This should be completely removed to allow for setup w/o Mongo integration.

Currently, incomplete searches such as long running searches returning no results, or searches reset/cancelled save as null.

Create a timeout and also store boolean for completed search in db so front-end can display completed searches during a search.